class AsyncStreamingQueryCheckpointMetadata extends StreamingQueryCheckpointMetadata
A version of StreamingQueryCheckpointMetadata that supports async state checkpointing.
- Alphabetic
- By Inheritance
- AsyncStreamingQueryCheckpointMetadata
- StreamingQueryCheckpointMetadata
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Instance Constructors
- new AsyncStreamingQueryCheckpointMetadata(sparkSession: SparkSession, resolvedCheckpointRoot: String, asyncWritesExecutorService: ThreadPoolExecutor, asyncProgressTrackingCheckpointingIntervalMs: Long, triggerClock: Clock)
- sparkSession
Spark session
- resolvedCheckpointRoot
The resolved checkpoint root path
- asyncWritesExecutorService
The executor service for async writes
- asyncProgressTrackingCheckpointingIntervalMs
The interval for async progress
- triggerClock
The clock to use for trigger time
Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- final def checkpointFile(name: String): String
Returns the path of a file with
namein the checkpoint directory.Returns the path of a file with
namein the checkpoint directory.- Attributes
- protected
- Definition Classes
- StreamingQueryCheckpointMetadata
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
- lazy val commitLog: AsyncCommitLog
A log that records the batch ids that have completed.
A log that records the batch ids that have completed. This is used to check if a batch was fully processed, and its output was committed to the sink, hence no need to process it again. This is used (for instance) during restart, to help identify which batch to run next.
- Definition Classes
- AsyncStreamingQueryCheckpointMetadata → StreamingQueryCheckpointMetadata
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- lazy val offsetLog: AsyncOffsetSeqLog
A write-ahead-log that records the offsets that are present in each batch.
A write-ahead-log that records the offsets that are present in each batch. In order to ensure that a given batch will always consist of the same data, we write to this log *before* any processing is done. Thus, the Nth record in this log indicated data that is currently being processed and the N-1th entry indicates which offsets have been durably committed to the sink.
- Definition Classes
- AsyncStreamingQueryCheckpointMetadata → StreamingQueryCheckpointMetadata
- final lazy val streamMetadata: StreamMetadata
Metadata associated with the whole query
Metadata associated with the whole query
- Definition Classes
- StreamingQueryCheckpointMetadata
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)