org.apache.spark.sql.execution.streaming
Default implementation of FileManager using newer FileContext API.
A simple trait to abstract out the file management operations needed by HDFSMetadataLog.
Implementation of FileManager using older FileSystem API.
Implementation of FileManager using older FileSystem API. Note that this implementation cannot provide atomic renaming of paths, hence can lead to consistency issues. This should be used only as a backup option, when FileContextManager cannot be used.
Verify if batchIds are continuous and between startId
and endId
.
Verify if batchIds are continuous and between startId
and endId
.
the sorted ids to verify.
the start id. If it's set, batchIds should start with this id.
the start id. If it's set, batchIds should end with this id.