class IncrementalExecution extends QueryExecution with Logging
A variant of QueryExecution that allows the execution of the given LogicalPlan plan incrementally. Possibly preserving state in between each execution.
- Alphabetic
- By Inheritance
- IncrementalExecution
- QueryExecution
- Logging
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Instance Constructors
- new IncrementalExecution(sparkSession: classic.SparkSession, logicalPlan: LogicalPlan, outputMode: OutputMode, checkpointLocation: String, queryId: UUID, runId: UUID, currentBatchId: Long, prevOffsetSeqMetadata: Option[OffsetSeqMetadata], offsetSeqMetadata: OffsetSeqMetadata, watermarkPropagator: WatermarkPropagator, isFirstBatch: Boolean, currentStateStoreCkptId: Map[Long, Array[Array[String]]] = MutableMap[Long, Array[Array[String]]](), stateSchemaMetadatas: Map[Long, StateSchemaBroadcast] = MutableMap[Long, StateSchemaBroadcast](), mode: CommandExecutionMode.Value = CommandExecutionMode.ALL)
- currentStateStoreCkptId
checkpoint ID for the latest committed version. It is operatorID -> array of checkpointIDs. Array index n represents checkpoint ID for the nth shuffle partition.
- stateSchemaMetadatas
map that keeps track of the StateSchemaBroadcasts per operator. It is passed in by reference from MicrobatchExecution, so it is populated during planning in the first batch, and passed in to the StatefulOperatorStateInfo on every subsequent batch
Type Members
- implicit class LogStringContext extends AnyRef
- Definition Classes
- Logging
- sealed trait SparkPlanPartialRule extends AnyRef
Value Members
- object debug
A special namespace for commands that can be used to debug query execution.
A special namespace for commands that can be used to debug query execution.
- Definition Classes
- QueryExecution
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- def analyzed: LogicalPlan
- Definition Classes
- QueryExecution
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def assertAnalyzed(): Unit
no need to try-catch again as this is already done once
no need to try-catch again as this is already done once
- Definition Classes
- IncrementalExecution → QueryExecution
- def assertCommandExecuted(): Unit
- Definition Classes
- QueryExecution
- def assertExecutedPlanPrepared(): Unit
- Definition Classes
- QueryExecution
- def assertOptimized(): Unit
- Definition Classes
- QueryExecution
- def assertSparkPlanPrepared(): Unit
- Definition Classes
- QueryExecution
- def assertSupported(): Unit
No need assert supported, as this check has already been done
No need assert supported, as this check has already been done
- Definition Classes
- IncrementalExecution → QueryExecution
- val checkpointLocation: String
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
- def commandExecuted: LogicalPlan
- Definition Classes
- QueryExecution
- val currentBatchId: Long
- val currentStateStoreCkptId: Map[Long, Array[Array[String]]]
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- def executePhase[T](phase: String)(block: => T): T
- Attributes
- protected
- Definition Classes
- QueryExecution
- def executedPlan: SparkPlan
- Definition Classes
- QueryExecution
- def explainString(mode: ExplainMode): String
- Definition Classes
- QueryExecution
- def extendedExplainInfo(append: (String) => Unit, plan: SparkPlan): Unit
- Definition Classes
- QueryExecution
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- val id: Long
- Definition Classes
- QueryExecution
- def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
- Attributes
- protected
- Definition Classes
- Logging
- def initializeLogIfNecessary(isInterpreter: Boolean): Unit
- Attributes
- protected
- Definition Classes
- Logging
- val isFirstBatch: Boolean
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- lazy val isLazyAnalysis: Boolean
- Definition Classes
- QueryExecution
- def isTraceEnabled(): Boolean
- Attributes
- protected
- Definition Classes
- Logging
- val lazyToRdd: LazyTry[SQLExecutionRDD]
- Definition Classes
- QueryExecution
- def log: Logger
- Attributes
- protected
- Definition Classes
- Logging
- def logDebug(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logDebug(entry: LogEntry, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logDebug(entry: LogEntry): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logDebug(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logError(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logError(entry: LogEntry, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logError(entry: LogEntry): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logError(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logInfo(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logInfo(entry: LogEntry, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logInfo(entry: LogEntry): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logInfo(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logName: String
- Attributes
- protected
- Definition Classes
- Logging
- def logTrace(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logTrace(entry: LogEntry, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logTrace(entry: LogEntry): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logTrace(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logWarning(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logWarning(entry: LogEntry, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logWarning(entry: LogEntry): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logWarning(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- val logical: LogicalPlan
- Definition Classes
- QueryExecution
- val mode: CommandExecutionMode.Value
- Definition Classes
- QueryExecution
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def normalized: LogicalPlan
- Definition Classes
- QueryExecution
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- def observedMetrics: Map[String, Row]
Get the metrics observed during the execution of the query plan.
Get the metrics observed during the execution of the query plan.
- Definition Classes
- QueryExecution
- val offsetSeqMetadata: OffsetSeqMetadata
- lazy val optimizedPlan: LogicalPlan
See [SPARK-18339] Walk the optimized logical plan and replace CurrentBatchTimestamp with the desired literal
See [SPARK-18339] Walk the optimized logical plan and replace CurrentBatchTimestamp with the desired literal
- Definition Classes
- IncrementalExecution → QueryExecution
- val outputMode: OutputMode
- val planner: SparkPlanner
- Definition Classes
- IncrementalExecution → QueryExecution
- def preparations: Seq[Rule[SparkPlan]]
- Definition Classes
- IncrementalExecution → QueryExecution
- val prevOffsetSeqMetadata: Option[OffsetSeqMetadata]
- val queryId: UUID
- val runId: UUID
- def shouldRunAnotherBatch(newMetadata: OffsetSeqMetadata): Boolean
Should the MicroBatchExecution run another batch based on this execution and the current updated metadata.
Should the MicroBatchExecution run another batch based on this execution and the current updated metadata.
This method performs simulation of watermark propagation against new batch (which is not planned yet), which is required for asking the needs of another batch to each stateful operator.
- val shuffleCleanupMode: ShuffleCleanupMode
- Definition Classes
- QueryExecution
- def simpleString: String
- Definition Classes
- QueryExecution
- def sparkPlan: SparkPlan
- Definition Classes
- QueryExecution
- val sparkSession: classic.SparkSession
- Definition Classes
- QueryExecution
- val state: Rule[SparkPlan]
- val stateSchemaMetadatas: Map[Long, StateSchemaBroadcast]
- def stringWithStats: String
- Definition Classes
- QueryExecution
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toRdd: RDD[InternalRow]
Internal version of the RDD.
Internal version of the RDD. Avoids copies and has no schema. Note for callers: Spark may apply various optimization including reusing object: this means the row is valid only for the iteration it is retrieved. You should avoid storing row and accessing after iteration. (Calling
collect()is one of known bad usage.) If you want to store these rows into collection, please apply some converter or copy row which produces new object per iteration. Given QueryExecution is not a public class, end users are discouraged to use this: please useDataset.rddinstead where conversion will be applied.- Definition Classes
- QueryExecution
- def toString(): String
- Definition Classes
- QueryExecution → AnyRef → Any
- val tracker: QueryPlanningTracker
- Definition Classes
- QueryExecution
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- val watermarkPropagator: WatermarkPropagator
- def withCachedData: LogicalPlan
- Definition Classes
- QueryExecution
- def withLogContext(context: Map[String, String])(body: => Unit): Unit
- Attributes
- protected
- Definition Classes
- Logging
- object ConvertLocalLimitRule extends SparkPlanPartialRule
- object ShufflePartitionsRule extends SparkPlanPartialRule
- object StateOpIdRule extends SparkPlanPartialRule
- object StateSchemaAndOperatorMetadataRule extends SparkPlanPartialRule
- object WatermarkPropagationRule extends SparkPlanPartialRule
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)