Get the end or formerly know as "available" offsets of the latest batch that has been planned

Await until all fields of the query have been initialized.

A lock used to wait/notify when batches complete. Use a fair lock to avoid thread starvation.

Waits for the termination of this query, either by query.stop() or by an exception. If the query has terminated with an exception, then the exception will be thrown. Otherwise, it returns whether the query has terminated or not within the timeoutMs milliseconds.

If the query has terminated, then all subsequent calls to this method will either return true immediately (if the query was terminated by stop()), or throw the exception immediately (if the query has terminated with exception).

Waits for the termination of this query, either by query.stop() or by an exception. If the query has terminated with an exception, then the exception will be thrown.

If the query has terminated, then all subsequent calls to this method will either return immediately (if the query was terminated by stop()), or throw the exception immediately (if the query has terminated with exception).

Returns the path of a file with name in the checkpoint directory.

Any clean up that needs to happen when the query is stopped or exits

Tracks how much data we have processed and committed to the sink or state store from each input source. Only the scheduler thread should modify this field, and only in atomic steps. Other threads should make a shallow copy if they are going to access this field more than once, since the field's value may change at any time.

Returns the StreamingQueryException if the query was terminated by an exception.

Prints the physical plan to the console for debugging purposes.

Prints the physical plan to the console for debugging purposes.

Expose for tests

Get the latest execution context .

Get the start offsets of the latest batch that has been planned

Returns the unique id of this query that persists across restarts from checkpoint data. That is, this id is generated when a query is started for the first time, and will be the same every time it is restarted from checkpoint data. Also see runId.

Interrupts the query execution thread and awaits its termination until until it exceeds the timeout. The timeout can be set on "spark.sql.streaming.stopTimeout".

Whether the query is currently active or not

Returns the most recent StreamingQueryProgress update of this streaming query.

The base logical plan which will be used across batch runs. Once the value is set, it should not be modified.

Called after the microbatch has completed execution. It takes care of committing the offset to commit log and other bookkeeping.

Method called once after the planning is done and before the start of the microbatch execution. It can be used to perform any pre-execution tasks.

Should not call super method as we need to do something completely different in this method for async progress tracking

Returns the user-specified name of the query, or null if not specified. This name can be specified in the org.apache.spark.sql.streaming.DataStreamWriter as dataframe.writeStream.queryName("query").start(). This name, if set, must be unique across all active queries.

A flag to indicate that a batch has completed with no new data available.

Populate the start offsets to start the execution at the current offsets stored in the sink (i.e. avoid reprocessing data that we have already processed). This function must be called before any processing occurs and will populate the following fields in the execution context of this micro-batch

• batchId
• startOffset
• endOffsets The basic structure of this method is as follows:

Identify (from the offset log) the offsets used to run the last batch IF last batch exists THEN Set the next batch to be executed as the last recovered batch Check the commit log to see which batch was committed last IF the last batch was committed THEN Call getBatch using the last batch start and end offsets // ^^^^ above line is needed since some sources assume last batch always re-executes Setup for a new batch i.e., start = last batch end, and identify new end DONE ELSE Identify a brand new batch DONE

Pretty identified string of printing in logs. Format is If name is set "queryName [id = xyz, runId = abc]" else "[id = xyz, runId = abc]"

Blocks until all available data in the source has been processed and committed to the sink. This method is intended for testing. Note that in the case of continually arriving data, this method may block forever. Additionally, this method is only guaranteed to block until data that has been synchronously appended data to a org.apache.spark.sql.execution.streaming.Source prior to invocation. (i.e. getOffset must immediately reflect the addition).

The thread that runs the micro-batches of this stream. Note that this thread must be org.apache.spark.util.UninterruptibleThread to workaround KAFKA-1894: interrupting a running KafkaConsumer may cause endless loop.

Returns an array of the most recent StreamingQueryProgress updates for this query. The number of progress updates retained for each stream is configured by Spark session configuration spark.sql.streaming.numRecentProgressUpdates.

Repeatedly attempts to run batches as data arrives.

Returns the unique id of this run of the query. That is, every start/restart of a query will generate a unique runId. Therefore, every time a query is restarted from checkpoint, it will have the same id but different runIds.

We will only set the lastExecutionContext only if the batch id is larger than the batch id of the current latestExecutionContext. This is done to make sure we will always tracking the latest execution context i.e. we will never set latestExecutionContext to a earlier / older batch.

The list of stream instances which will be used across batch runs. Once the value is set, it should not be modified.

Returns the SparkSession associated with this.

Starts the execution. This returns only after the thread has started and QueryStartedEvent has been posted to all the listeners.

Defines the internal state of execution

Returns the current status of the query.

Signals to the thread executing micro-batches that it should stop running after the next batch. This method blocks until the thread stops running.

Stops all streaming sources safely.

Metadata associated with the whole query

Used to report metrics to coda-hale. This uses id for easier tracking across restarts.

A list of unique sources in the query plan. This will be set when generating logical plan.

Conduct sanity checks on the offset log to make sure it is correct and expected. Also return the previous offset written to the offset log

A map of current watermarks, keyed by the position of the watermark operator in the physical plan.

This state is 'soft state', which does not affect the correctness and semantics of watermarks and is not persisted across query restarts. The fault-tolerant watermark state is in offsetSeqMetadata.