Packages

c

org.apache.spark.sql.execution.streaming

IncrementalExecution

class IncrementalExecution extends QueryExecution with Logging

A variant of QueryExecution that allows the execution of the given LogicalPlan plan incrementally. Possibly preserving state in between each execution.

Linear Supertypes
QueryExecution, Logging, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. IncrementalExecution
  2. QueryExecution
  3. Logging
  4. AnyRef
  5. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new IncrementalExecution(sparkSession: SparkSession, logicalPlan: LogicalPlan, outputMode: OutputMode, checkpointLocation: String, queryId: UUID, runId: UUID, currentBatchId: Long, prevOffsetSeqMetadata: Option[OffsetSeqMetadata], offsetSeqMetadata: OffsetSeqMetadata)

Value Members

  1. object debug

    A special namespace for commands that can be used to debug query execution.

    A special namespace for commands that can be used to debug query execution.

    Definition Classes
    QueryExecution
  2. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  3. final def ##: Int
    Definition Classes
    AnyRef → Any
  4. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  5. lazy val analyzed: LogicalPlan
    Definition Classes
    QueryExecution
  6. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  7. def assertAnalyzed(): Unit
    Definition Classes
    QueryExecution
  8. def assertCommandExecuted(): Unit
    Definition Classes
    QueryExecution
  9. def assertSupported(): Unit

    No need assert supported, as this check has already been done

    No need assert supported, as this check has already been done

    Definition Classes
    IncrementalExecution → QueryExecution
  10. val checkpointLocation: String
  11. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @native()
  12. lazy val commandExecuted: LogicalPlan
    Definition Classes
    QueryExecution
  13. val currentBatchId: Long
  14. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  15. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  16. val eventTimeWatermarkForEviction: Long
  17. val eventTimeWatermarkForLateEvents: Long
  18. def executePhase[T](phase: String)(block: => T): T
    Attributes
    protected
    Definition Classes
    QueryExecution
  19. lazy val executedPlan: SparkPlan
    Definition Classes
    QueryExecution
  20. def explainString(mode: ExplainMode): String
    Definition Classes
    QueryExecution
  21. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable])
  22. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  23. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  24. val id: Long
    Definition Classes
    QueryExecution
  25. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  26. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  27. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  28. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  29. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  30. def logDebug(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  31. def logDebug(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  32. def logError(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  33. def logError(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  34. def logInfo(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  35. def logInfo(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  36. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  37. def logTrace(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  38. def logTrace(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  39. def logWarning(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  40. def logWarning(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  41. val logical: LogicalPlan
    Definition Classes
    QueryExecution
  42. val mode: CommandExecutionMode.Value
    Definition Classes
    QueryExecution
  43. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  44. lazy val normalized: LogicalPlan
    Definition Classes
    QueryExecution
  45. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  46. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  47. def observedMetrics: Map[String, Row]

    Get the metrics observed during the execution of the query plan.

    Get the metrics observed during the execution of the query plan.

    Definition Classes
    QueryExecution
  48. val offsetSeqMetadata: OffsetSeqMetadata
  49. lazy val optimizedPlan: LogicalPlan

    See [SPARK-18339] Walk the optimized logical plan and replace CurrentBatchTimestamp with the desired literal

    See [SPARK-18339] Walk the optimized logical plan and replace CurrentBatchTimestamp with the desired literal

    Definition Classes
    IncrementalExecution → QueryExecution
  50. val outputMode: OutputMode
  51. val planner: SparkPlanner
    Definition Classes
    IncrementalExecution → QueryExecution
  52. def preparations: Seq[Rule[SparkPlan]]
    Definition Classes
    IncrementalExecution → QueryExecution
  53. val prevOffsetSeqMetadata: Option[OffsetSeqMetadata]
  54. val queryId: UUID
  55. val runId: UUID
  56. def shouldRunAnotherBatch(newMetadata: OffsetSeqMetadata): Boolean

    Should the MicroBatchExecution run another batch based on this execution and the current updated metadata.

  57. def simpleString: String
    Definition Classes
    QueryExecution
  58. lazy val sparkPlan: SparkPlan
    Definition Classes
    QueryExecution
  59. val sparkSession: SparkSession
    Definition Classes
    QueryExecution
  60. val state: Rule[SparkPlan]

    Locates save/restore pairs surrounding aggregation.

  61. def stringWithStats: String
    Definition Classes
    QueryExecution
  62. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  63. lazy val toRdd: RDD[InternalRow]

    Internal version of the RDD.

    Internal version of the RDD. Avoids copies and has no schema. Note for callers: Spark may apply various optimization including reusing object: this means the row is valid only for the iteration it is retrieved. You should avoid storing row and accessing after iteration. (Calling collect() is one of known bad usage.) If you want to store these rows into collection, please apply some converter or copy row which produces new object per iteration. Given QueryExecution is not a public class, end users are discouraged to use this: please use Dataset.rdd instead where conversion will be applied.

    Definition Classes
    QueryExecution
  64. def toString(): String
    Definition Classes
    QueryExecution → AnyRef → Any
  65. val tracker: QueryPlanningTracker
    Definition Classes
    QueryExecution
  66. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  67. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  68. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  69. lazy val withCachedData: LogicalPlan
    Definition Classes
    QueryExecution

Inherited from QueryExecution

Inherited from Logging

Inherited from AnyRef

Inherited from Any

Ungrouped