class OffsetSeqLog extends HDFSMetadataLog[OffsetSeq]
This class is used to log offsets to persistent files in HDFS. Each file corresponds to a specific batch of offsets. The file format contains a version string in the first line, followed by a the JSON string representation of the offsets separated by a newline character. If a source offset is missing, then that line will contain a string value defined in the SERIALIZED_VOID_OFFSET variable in OffsetSeqLog companion object. For instance, when dealing with LongOffset types: v1 // version 1 metadata {0} // LongOffset 0 {3} // LongOffset 3
- // No offset for this source i.e., an invalid JSON string {2} // LongOffset 2 ...
- Alphabetic
- By Inheritance
- OffsetSeqLog
- HDFSMetadataLog
- Logging
- MetadataLog
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Instance Constructors
- new OffsetSeqLog(sparkSession: SparkSession, path: String)
Value Members
-
final
def
!=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
##(): Int
- Definition Classes
- AnyRef → Any
-
final
def
==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
def
add(batchId: Long, metadata: OffsetSeq): Boolean
Store the metadata for the specified batchId and return
true
if successful.Store the metadata for the specified batchId and return
true
if successful. If the batchId's metadata has already been stored, this method will returnfalse
.- Definition Classes
- HDFSMetadataLog → MetadataLog
-
def
addNewBatchByStream(batchId: Long)(fn: (OutputStream) ⇒ Unit): Boolean
Store the metadata for the specified batchId and return
true
if successful.Store the metadata for the specified batchId and return
true
if successful. This method fills the content of metadata via executing function. If the function throws an exception, writing will be automatically cancelled and this method will propagate the exception.If the batchId's metadata has already been stored, this method will return
false
.Writing the metadata is done by writing a batch to a temp file then rename it to the batch file.
There may be multiple HDFSMetadataLog using the same metadata path. Although it is not a valid behavior, we still need to prevent it from destroying the files.
- Definition Classes
- HDFSMetadataLog
-
def
applyFnToBatchByStream[RET](batchId: Long, skipExistingCheck: Boolean = false)(fn: (InputStream) ⇒ RET): RET
Apply provided function to each entry in the specific batch metadata log.
Apply provided function to each entry in the specific batch metadata log.
Unlike get which will materialize all entries into memory, this method streamlines the process via READ-AND-PROCESS. This helps to avoid the memory issue on huge metadata log file.
NOTE: This no longer fails early on corruption. The caller should handle the exception properly and make sure the logic is not affected by failing in the middle.
- Definition Classes
- HDFSMetadataLog
-
final
def
asInstanceOf[T0]: T0
- Definition Classes
- Any
-
val
batchCache: Map[Long, OffsetSeq]
Cache the latest two batches.
Cache the latest two batches. StreamExecution usually just accesses the latest two batches when committing offsets, this cache will save some file system operations.
- Attributes
- protected[sql]
- Definition Classes
- HDFSMetadataLog
-
val
batchFilesFilter: PathFilter
A
PathFilter
to filter only batch filesA
PathFilter
to filter only batch files- Attributes
- protected
- Definition Classes
- HDFSMetadataLog
-
def
batchIdToPath(batchId: Long): Path
- Attributes
- protected
- Definition Classes
- HDFSMetadataLog
-
def
clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
def
deserialize(in: InputStream): OffsetSeq
Read and deserialize the metadata from input stream.
Read and deserialize the metadata from input stream. If this method is overridden in a subclass, the overriding method should not close the given input stream, as it will be closed in the caller.
- Attributes
- protected
- Definition Classes
- OffsetSeqLog → HDFSMetadataLog
-
final
def
eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
equals(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
val
fileManager: CheckpointFileManager
- Attributes
- protected
- Definition Classes
- HDFSMetadataLog
-
def
finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( classOf[java.lang.Throwable] )
-
def
get(startId: Option[Long], endId: Option[Long]): Array[(Long, OffsetSeq)]
Return metadata for batches between startId (inclusive) and endId (inclusive).
Return metadata for batches between startId (inclusive) and endId (inclusive). If
startId
isNone
, just return all batches before endId (inclusive).- Definition Classes
- HDFSMetadataLog → MetadataLog
-
def
get(batchId: Long): Option[OffsetSeq]
Return the metadata for the specified batchId if it's stored.
Return the metadata for the specified batchId if it's stored. Otherwise, return None.
- Definition Classes
- HDFSMetadataLog → MetadataLog
-
final
def
getClass(): Class[_]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
def
getLatest(): Option[(Long, OffsetSeq)]
Return the latest batch Id and its metadata if exist.
Return the latest batch Id and its metadata if exist.
- Definition Classes
- HDFSMetadataLog → MetadataLog
-
def
getLatestBatchId(): Option[Long]
Return the latest batch id without reading the file.
Return the latest batch id without reading the file.
- Definition Classes
- HDFSMetadataLog
-
def
getOrderedBatchFiles(): Array[FileStatus]
Get an array of [FileStatus] referencing batch files.
Get an array of [FileStatus] referencing batch files. The array is sorted by most recent batch file first to oldest batch file.
- Definition Classes
- HDFSMetadataLog
-
def
getPrevBatchFromStorage(batchId: Long): Option[Long]
Get the id of the previous batch from storage
Get the id of the previous batch from storage
- batchId
get the previous batch id of this batch with batchId
- Definition Classes
- HDFSMetadataLog
-
def
hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
def
initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
- Attributes
- protected
- Definition Classes
- Logging
-
def
initializeLogIfNecessary(isInterpreter: Boolean): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
isBatchFile(path: Path): Boolean
- Attributes
- protected
- Definition Classes
- HDFSMetadataLog
-
final
def
isInstanceOf[T0]: Boolean
- Definition Classes
- Any
-
def
isTraceEnabled(): Boolean
- Attributes
- protected
- Definition Classes
- Logging
-
def
listBatches: Array[Long]
List the available batches on file system.
List the available batches on file system.
- Attributes
- protected
- Definition Classes
- HDFSMetadataLog
-
def
listBatchesOnDisk: Array[Long]
List the batches persisted to storage
-
def
log: Logger
- Attributes
- protected
- Definition Classes
- Logging
-
def
logDebug(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logDebug(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logError(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logError(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logInfo(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logInfo(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logName: String
- Attributes
- protected
- Definition Classes
- Logging
-
def
logTrace(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logTrace(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logWarning(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logWarning(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
val
metadataCacheEnabled: Boolean
- Attributes
- protected
- Definition Classes
- HDFSMetadataLog
-
val
metadataPath: Path
- Definition Classes
- HDFSMetadataLog
-
final
def
ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
final
def
notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
final
def
notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- def offsetSeqMetadataForBatchId(batchId: Long): Option[OffsetSeqMetadata]
-
def
pathToBatchId(path: Path): Long
- Attributes
- protected
- Definition Classes
- HDFSMetadataLog
-
def
purge(thresholdBatchId: Long): Unit
Removes all the log entry earlier than thresholdBatchId (exclusive).
Removes all the log entry earlier than thresholdBatchId (exclusive).
- Definition Classes
- HDFSMetadataLog → MetadataLog
-
def
purgeAfter(thresholdBatchId: Long): Unit
Removes all log entries later than thresholdBatchId (exclusive).
Removes all log entries later than thresholdBatchId (exclusive).
- Definition Classes
- HDFSMetadataLog
-
def
serialize(offsetSeq: OffsetSeq, out: OutputStream): Unit
Serialize the metadata and write to the output stream.
Serialize the metadata and write to the output stream. If this method is overridden in a subclass, the overriding method should not close the given output stream, as it will be closed in the caller.
- Attributes
- protected
- Definition Classes
- OffsetSeqLog → HDFSMetadataLog
-
final
def
synchronized[T0](arg0: ⇒ T0): T0
- Definition Classes
- AnyRef
-
def
toString(): String
- Definition Classes
- AnyRef → Any
-
final
def
wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
def
write(batchMetadataFile: Path, fn: (OutputStream) ⇒ Unit): Unit
- Attributes
- protected
- Definition Classes
- HDFSMetadataLog