class IncrementalExecution extends QueryExecution with Logging
A variant of QueryExecution that allows the execution of the given LogicalPlan plan incrementally. Possibly preserving state in between each execution.
Linear Supertypes
Ordering
- Alphabetic
- By Inheritance
Inherited
- IncrementalExecution
- QueryExecution
- Logging
- AnyRef
- Any
- Hide All
- Show All
Visibility
- Public
- Protected
Instance Constructors
- new IncrementalExecution(sparkSession: SparkSession, logicalPlan: LogicalPlan, outputMode: OutputMode, checkpointLocation: String, queryId: UUID, runId: UUID, currentBatchId: Long, prevOffsetSeqMetadata: Option[OffsetSeqMetadata], offsetSeqMetadata: OffsetSeqMetadata, watermarkPropagator: WatermarkPropagator, isFirstBatch: Boolean)
Type Members
- implicit class LogStringContext extends AnyRef
- Definition Classes
- Logging
- sealed trait SparkPlanPartialRule extends AnyRef
Value Members
- object debug
A special namespace for commands that can be used to debug query execution.
A special namespace for commands that can be used to debug query execution.
- Definition Classes
- QueryExecution
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- lazy val analyzed: LogicalPlan
- Definition Classes
- QueryExecution
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def assertAnalyzed(): Unit
no need to try-catch again as this is already done once
no need to try-catch again as this is already done once
- Definition Classes
- IncrementalExecution → QueryExecution
- def assertCommandExecuted(): Unit
- Definition Classes
- QueryExecution
- def assertExecutedPlanPrepared(): Unit
- Definition Classes
- QueryExecution
- def assertOptimized(): Unit
- Definition Classes
- QueryExecution
- def assertSparkPlanPrepared(): Unit
- Definition Classes
- QueryExecution
- def assertSupported(): Unit
No need assert supported, as this check has already been done
No need assert supported, as this check has already been done
- Definition Classes
- IncrementalExecution → QueryExecution
- val checkpointLocation: String
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
- lazy val commandExecuted: LogicalPlan
- Definition Classes
- QueryExecution
- val currentBatchId: Long
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- def executePhase[T](phase: String)(block: => T): T
- Attributes
- protected
- Definition Classes
- QueryExecution
- lazy val executedPlan: SparkPlan
- Definition Classes
- QueryExecution
- def explainString(mode: ExplainMode): String
- Definition Classes
- QueryExecution
- def extendedExplainInfo(append: (String) => Unit, plan: SparkPlan): Unit
- Definition Classes
- QueryExecution
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- val id: Long
- Definition Classes
- QueryExecution
- def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
- Attributes
- protected
- Definition Classes
- Logging
- def initializeLogIfNecessary(isInterpreter: Boolean): Unit
- Attributes
- protected
- Definition Classes
- Logging
- val isFirstBatch: Boolean
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- def isTraceEnabled(): Boolean
- Attributes
- protected
- Definition Classes
- Logging
- def log: Logger
- Attributes
- protected
- Definition Classes
- Logging
- def logDebug(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logDebug(entry: LogEntry, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logDebug(entry: LogEntry): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logDebug(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logError(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logError(entry: LogEntry, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logError(entry: LogEntry): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logError(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logInfo(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logInfo(entry: LogEntry, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logInfo(entry: LogEntry): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logInfo(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logName: String
- Attributes
- protected
- Definition Classes
- Logging
- def logTrace(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logTrace(entry: LogEntry, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logTrace(entry: LogEntry): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logTrace(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logWarning(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logWarning(entry: LogEntry, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logWarning(entry: LogEntry): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logWarning(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- val logical: LogicalPlan
- Definition Classes
- QueryExecution
- val mode: CommandExecutionMode.Value
- Definition Classes
- QueryExecution
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- lazy val normalized: LogicalPlan
- Definition Classes
- QueryExecution
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- def observedMetrics: Map[String, Row]
Get the metrics observed during the execution of the query plan.
Get the metrics observed during the execution of the query plan.
- Definition Classes
- QueryExecution
- val offsetSeqMetadata: OffsetSeqMetadata
- lazy val optimizedPlan: LogicalPlan
See [SPARK-18339] Walk the optimized logical plan and replace CurrentBatchTimestamp with the desired literal
See [SPARK-18339] Walk the optimized logical plan and replace CurrentBatchTimestamp with the desired literal
- Definition Classes
- IncrementalExecution → QueryExecution
- val outputMode: OutputMode
- val planner: SparkPlanner
- Definition Classes
- IncrementalExecution → QueryExecution
- def preparations: Seq[Rule[SparkPlan]]
- Definition Classes
- IncrementalExecution → QueryExecution
- val prevOffsetSeqMetadata: Option[OffsetSeqMetadata]
- val queryId: UUID
- val runId: UUID
- def shouldRunAnotherBatch(newMetadata: OffsetSeqMetadata): Boolean
Should the MicroBatchExecution run another batch based on this execution and the current updated metadata.
Should the MicroBatchExecution run another batch based on this execution and the current updated metadata.
This method performs simulation of watermark propagation against new batch (which is not planned yet), which is required for asking the needs of another batch to each stateful operator.
- val shuffleCleanupMode: ShuffleCleanupMode
- Definition Classes
- QueryExecution
- def simpleString: String
- Definition Classes
- QueryExecution
- lazy val sparkPlan: SparkPlan
- Definition Classes
- QueryExecution
- val sparkSession: SparkSession
- Definition Classes
- QueryExecution
- val state: Rule[SparkPlan]
- def stringWithStats: String
- Definition Classes
- QueryExecution
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- lazy val toRdd: RDD[InternalRow]
Internal version of the RDD.
Internal version of the RDD. Avoids copies and has no schema. Note for callers: Spark may apply various optimization including reusing object: this means the row is valid only for the iteration it is retrieved. You should avoid storing row and accessing after iteration. (Calling
collect()is one of known bad usage.) If you want to store these rows into collection, please apply some converter or copy row which produces new object per iteration. Given QueryExecution is not a public class, end users are discouraged to use this: please useDataset.rddinstead where conversion will be applied.- Definition Classes
- QueryExecution
- def toString(): String
- Definition Classes
- QueryExecution → AnyRef → Any
- val tracker: QueryPlanningTracker
- Definition Classes
- QueryExecution
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- val watermarkPropagator: WatermarkPropagator
- lazy val withCachedData: LogicalPlan
- Definition Classes
- QueryExecution
- def withLogContext(context: HashMap[String, String])(body: => Unit): Unit
- Attributes
- protected
- Definition Classes
- Logging
- object ConvertLocalLimitRule extends SparkPlanPartialRule
- object ShufflePartitionsRule extends SparkPlanPartialRule
- object StateOpIdRule extends SparkPlanPartialRule
- object WatermarkPropagationRule extends SparkPlanPartialRule
- object WriteStatefulOperatorMetadataRule extends SparkPlanPartialRule
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)