org.apache.spark.sql.execution.streaming

FlatMapGroupsWithStateExec

Companion object FlatMapGroupsWithStateExec

case class FlatMapGroupsWithStateExec(func: (Any, Iterator[Any], LogicalGroupState[Any]) => Iterator[Any], keyDeserializer: Expression, valueDeserializer: Expression, initialStateDeserializer: Expression, groupingAttributes: Seq[Attribute], initialStateGroupAttrs: Seq[Attribute], dataAttributes: Seq[Attribute], initialStateDataAttrs: Seq[Attribute], outputObjAttr: Attribute, stateInfo: Option[StatefulOperatorStateInfo], stateEncoder: ExpressionEncoder[Any], stateFormatVersion: Int, outputMode: OutputMode, timeoutConf: GroupStateTimeout, batchTimestampMs: Option[Long], eventTimeWatermark: Option[Long], initialState: SparkPlan, hasInitialState: Boolean, child: SparkPlan) extends SparkPlan with BinaryExecNode with ObjectProducerExec with StateStoreWriter with WatermarkSupport with Product with Serializable

Physical operator for executing FlatMapGroupsWithState

func: function called on each group
keyDeserializer: used to extract the key object for each group.
valueDeserializer: used to extract the items in the iterator from an input row.
initialStateDeserializer: used to extract the state object from the initialState dataset
groupingAttributes: used to group the data
dataAttributes: used to read the data
outputObjAttr: Defines the output object
stateEncoder: used to serialize/deserialize state before calling func
outputMode: the output mode of func
timeoutConf: used to timeout groups that have not received data in a while
batchTimestampMs: processing timestamp of the current batch.
eventTimeWatermark: event time watermark for the current batch
initialState: the user specified initial state
hasInitialState: indicates whether the initial state is provided or not
child: the physical plan for the underlying data

Linear Supertypes

WatermarkSupport, StateStoreWriter, StatefulOperator, ObjectProducerExec, BinaryExecNode, BinaryLike[SparkPlan], SparkPlan, Serializable, Logging, QueryPlan[SparkPlan], SQLConfHelper, TreeNode[SparkPlan], TreePatternBits, Product, Equals, AnyRef, Any

Ordering

Alphabetic
By Inheritance

Inherited

FlatMapGroupsWithStateExec
WatermarkSupport
StateStoreWriter
StatefulOperator
ObjectProducerExec
BinaryExecNode
BinaryLike
SparkPlan
Serializable
Logging
QueryPlan
SQLConfHelper
TreeNode
TreePatternBits
Product
Equals
AnyRef
Any

Hide All
Show All

Visibility

Public
Protected

Instance Constructors

new FlatMapGroupsWithStateExec(func: (Any, Iterator[Any], LogicalGroupState[Any]) => Iterator[Any], keyDeserializer: Expression, valueDeserializer: Expression, initialStateDeserializer: Expression, groupingAttributes: Seq[Attribute], initialStateGroupAttrs: Seq[Attribute], dataAttributes: Seq[Attribute], initialStateDataAttrs: Seq[Attribute], outputObjAttr: Attribute, stateInfo: Option[StatefulOperatorStateInfo], stateEncoder: ExpressionEncoder[Any], stateFormatVersion: Int, outputMode: OutputMode, timeoutConf: GroupStateTimeout, batchTimestampMs: Option[Long], eventTimeWatermark: Option[Long], initialState: SparkPlan, hasInitialState: Boolean, child: SparkPlan)
func
function called on each group
keyDeserializer
used to extract the key object for each group.
valueDeserializer
used to extract the items in the iterator from an input row.
initialStateDeserializer
used to extract the state object from the initialState dataset
groupingAttributes
used to group the data
dataAttributes
used to read the data
outputObjAttr
Defines the output object
stateEncoder
used to serialize/deserialize state before calling func
outputMode
the output mode of func
timeoutConf
used to timeout groups that have not received data in a while
batchTimestampMs
processing timestamp of the current batch.
eventTimeWatermark
event time watermark for the current batch
initialState
the user specified initial state
hasInitialState
indicates whether the initial state is provided or not
child
the physical plan for the underlying data

Type Members

class InputProcessor extends AnyRef
Helper class to update the state store

Value Members

final def !=(arg0: Any): Boolean
Definition Classes
AnyRef → Any
final def ##: Int
Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean
Definition Classes
AnyRef → Any
lazy val allAttributes: AttributeSeq
Definition Classes
QueryPlan
def apply(number: Int): TreeNode[_]
Definition Classes
TreeNode
def applyRemovingRowsOlderThanWatermark(iter: Iterator[InternalRow], predicateDropRowByWatermark: BasePredicate): Iterator[InternalRow]
Attributes
protected
Definition Classes
StateStoreWriter
def argString(maxFields: Int): String
Definition Classes
TreeNode
def asCode: String
Definition Classes
TreeNode
final def asInstanceOf[T0]: T0
Definition Classes
Any
val batchTimestampMs: Option[Long]
final lazy val canonicalized: SparkPlan
Definition Classes
QueryPlan
Annotations
@transient()
val child: SparkPlan
Definition Classes
FlatMapGroupsWithStateExec → WatermarkSupport
final lazy val children: Seq[SparkPlan]
Definition Classes
BinaryLike
Annotations
@transient()
def cleanupResources(): Unit
Cleans up the resources used by the physical operator (if any).
Cleans up the resources used by the physical operator (if any). In general, all the resources should be cleaned up when the task finishes but operators like SortMergeJoinExec and LimitExec may want eager cleanup to free up tight resources (e.g., memory).
Attributes
protected[sql]
Definition Classes
SparkPlan
def clone(): SparkPlan
Definition Classes
TreeNode → AnyRef
def collect[B](pf: PartialFunction[SparkPlan, B]): Seq[B]
Definition Classes
TreeNode
def collectFirst[B](pf: PartialFunction[SparkPlan, B]): Option[B]
Definition Classes
TreeNode
def collectLeaves(): Seq[SparkPlan]
Definition Classes
TreeNode
def collectWithSubqueries[B](f: PartialFunction[SparkPlan, B]): Seq[B]
Definition Classes
QueryPlan
def conf: SQLConf
Definition Classes
SparkPlan → SQLConfHelper
final def containsAllPatterns(patterns: TreePattern*): Boolean
Definition Classes
TreePatternBits
final def containsAnyPattern(patterns: TreePattern*): Boolean
Definition Classes
TreePatternBits
lazy val containsChild: Set[TreeNode[_]]
Definition Classes
TreeNode
final def containsPattern(t: TreePattern): Boolean
Definition Classes
TreePatternBits
Annotations
@inline()
def copyTagsFrom(other: SparkPlan): Unit
Definition Classes
TreeNode
def customStatefulOperatorMetrics: Seq[StatefulOperatorCustomMetric]
Set of stateful operator custom metrics.
Set of stateful operator custom metrics. These are captured as part of the generic key-value map StateOperatorProgress.customMetrics. Stateful operators can extend this method to provide their own unique custom metrics.
Attributes
protected
Definition Classes
StateStoreWriter
val dataAttributes: Seq[Attribute]
lazy val deterministic: Boolean
Definition Classes
QueryPlan
def doCanonicalize(): SparkPlan
Attributes
protected
Definition Classes
QueryPlan
def doExecute(): RDD[InternalRow]
Produces the result of the query as an RDD[InternalRow]
Produces the result of the query as an RDD[InternalRow]
Overridden by concrete implementations of SparkPlan.
Attributes
protected
Definition Classes
FlatMapGroupsWithStateExec → SparkPlan
def doExecuteBroadcast[T](): Broadcast[T]
Produces the result of the query as a broadcast variable.
Produces the result of the query as a broadcast variable.
Overridden by concrete implementations of SparkPlan.
Attributes
protected[sql]
Definition Classes
SparkPlan
def doExecuteColumnar(): RDD[ColumnarBatch]
Produces the result of the query as an RDD[ColumnarBatch] if supportsColumnar returns true.
Produces the result of the query as an RDD[ColumnarBatch] if supportsColumnar returns true. By convention the executor that creates a ColumnarBatch is responsible for closing it when it is no longer needed. This allows input formats to be able to reuse batches if needed.
Attributes
protected
Definition Classes
SparkPlan
def doPrepare(): Unit
Overridden by concrete implementations of SparkPlan.
Overridden by concrete implementations of SparkPlan. It is guaranteed to run before any execute of SparkPlan. This is helpful if we want to set up some state before executing the query, e.g., BroadcastHashJoin uses it to broadcast asynchronously.
Attributes
protected
Definition Classes
SparkPlan
Note
prepare method has already walked down the tree, so the implementation doesn't have to call children's prepare methods. This will only be called once, protected by this.
final def eq(arg0: AnyRef): Boolean
Definition Classes
AnyRef
val eventTimeWatermark: Option[Long]
The watermark value.
The watermark value.
Definition Classes
FlatMapGroupsWithStateExec → WatermarkSupport
final def execute(): RDD[InternalRow]
Returns the result of this query as an RDD[InternalRow] by delegating to doExecute after preparations.
Returns the result of this query as an RDD[InternalRow] by delegating to doExecute after preparations.
Concrete implementations of SparkPlan should override doExecute.
Definition Classes
SparkPlan
final def executeBroadcast[T](): Broadcast[T]
Returns the result of this query as a broadcast variable by delegating to doExecuteBroadcast after preparations.
Returns the result of this query as a broadcast variable by delegating to doExecuteBroadcast after preparations.
Concrete implementations of SparkPlan should override doExecuteBroadcast.
Definition Classes
SparkPlan
def executeCollect(): Array[InternalRow]
Runs this query returning the result as an array.
Runs this query returning the result as an array.
Definition Classes
SparkPlan
def executeCollectPublic(): Array[Row]
Runs this query returning the result as an array, using external Row format.
Runs this query returning the result as an array, using external Row format.
Definition Classes
SparkPlan
final def executeColumnar(): RDD[ColumnarBatch]
Returns the result of this query as an RDD[ColumnarBatch] by delegating to doColumnarExecute after preparations.
Returns the result of this query as an RDD[ColumnarBatch] by delegating to doColumnarExecute after preparations.
Concrete implementations of SparkPlan should override doColumnarExecute if supportsColumnar returns true.
Definition Classes
SparkPlan
final def executeQuery[T](query: => T): T
Executes a query after preparing the query and adding query plan information to created RDDs for visualization.
Executes a query after preparing the query and adding query plan information to created RDDs for visualization.
Attributes
protected
Definition Classes
SparkPlan
def executeTail(n: Int): Array[InternalRow]
Runs this query returning the last n rows as an array.
Runs this query returning the last n rows as an array.
This is modeled after RDD.take but never runs any job locally on the driver.
Definition Classes
SparkPlan
def executeTake(n: Int): Array[InternalRow]
Runs this query returning the first n rows as an array.
Runs this query returning the first n rows as an array.
This is modeled after RDD.take but never runs any job locally on the driver.
Definition Classes
SparkPlan
def executeToIterator(): Iterator[InternalRow]
Runs this query returning the result as an iterator of InternalRow.
Runs this query returning the result as an iterator of InternalRow.
Definition Classes
SparkPlan
Note
Triggers multiple jobs (one for each partition).
def exists(f: (SparkPlan) => Boolean): Boolean
Definition Classes
TreeNode
final def expressions: Seq[Expression]
Definition Classes
QueryPlan
def fastEquals(other: TreeNode[_]): Boolean
Definition Classes
TreeNode
def finalize(): Unit
Attributes
protected[lang]
Definition Classes
AnyRef
Annotations
@throws(classOf[java.lang.Throwable])
def find(f: (SparkPlan) => Boolean): Option[SparkPlan]
Definition Classes
TreeNode
def flatMap[A](f: (SparkPlan) => TraversableOnce[A]): Seq[A]
Definition Classes
TreeNode
def foreach(f: (SparkPlan) => Unit): Unit
Definition Classes
TreeNode
def foreachUp(f: (SparkPlan) => Unit): Unit
Definition Classes
TreeNode
def formattedNodeName: String
Attributes
protected
Definition Classes
QueryPlan
val func: (Any, Iterator[Any], LogicalGroupState[Any]) => Iterator[Any]
def generateTreeString(depth: Int, lastChildren: Seq[Boolean], append: (String) => Unit, verbose: Boolean, prefix: String, addSuffix: Boolean, maxFields: Int, printNodeId: Boolean, indent: Int): Unit
Definition Classes
TreeNode
final def getClass(): Class[_ <: AnyRef]
Definition Classes
AnyRef → Any
Annotations
@native()
def getDefaultTreePatternBits: BitSet
Attributes
protected
Definition Classes
TreeNode
def getProgress(): StateOperatorProgress
Get the progress made by this stateful operator after execution.
Get the progress made by this stateful operator after execution. This should be called in the driver after this SparkPlan has been executed and metrics have been updated.
Definition Classes
StateStoreWriter
def getStateInfo: StatefulOperatorStateInfo
Attributes
protected
Definition Classes
StatefulOperator
def getTagValue[T](tag: TreeNodeTag[T]): Option[T]
Definition Classes
TreeNode
val groupingAttributes: Seq[Attribute]
val hasInitialState: Boolean
def hashCode(): Int
Definition Classes
TreeNode → AnyRef → Any
val id: Int
Definition Classes
SparkPlan
val initialState: SparkPlan
val initialStateDataAttrs: Seq[Attribute]
val initialStateDeserializer: Expression
val initialStateGroupAttrs: Seq[Attribute]
def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
Attributes
protected
Definition Classes
Logging
def initializeLogIfNecessary(isInterpreter: Boolean): Unit
Attributes
protected
Definition Classes
Logging
def innerChildren: Seq[QueryPlan[_]]
Definition Classes
QueryPlan → TreeNode
def inputSet: AttributeSet
Definition Classes
QueryPlan
def isCanonicalizedPlan: Boolean
Attributes
protected
Definition Classes
QueryPlan
final def isInstanceOf[T0]: Boolean
Definition Classes
Any
def isRuleIneffective(ruleId: RuleId): Boolean
Attributes
protected
Definition Classes
TreeNode
def isTraceEnabled(): Boolean
Attributes
protected
Definition Classes
Logging
def jsonFields: List[JField]
Attributes
protected
Definition Classes
TreeNode
val keyDeserializer: Expression
def keyExpressions: Seq[Attribute]
The keys that may have a watermark attribute.
The keys that may have a watermark attribute.
Definition Classes
FlatMapGroupsWithStateExec → WatermarkSupport
def left: SparkPlan
Definition Classes
FlatMapGroupsWithStateExec → BinaryLike
final def legacyWithNewChildren(newChildren: Seq[SparkPlan]): SparkPlan
Attributes
protected
Definition Classes
TreeNode
def log: Logger
Attributes
protected
Definition Classes
Logging
def logDebug(msg: => String, throwable: Throwable): Unit
Attributes
protected
Definition Classes
Logging
def logDebug(msg: => String): Unit
Attributes
protected
Definition Classes
Logging
def logError(msg: => String, throwable: Throwable): Unit
Attributes
protected
Definition Classes
Logging
def logError(msg: => String): Unit
Attributes
protected
Definition Classes
Logging
def logInfo(msg: => String, throwable: Throwable): Unit
Attributes
protected
Definition Classes
Logging
def logInfo(msg: => String): Unit
Attributes
protected
Definition Classes
Logging
def logName: String
Attributes
protected
Definition Classes
Logging
def logTrace(msg: => String, throwable: Throwable): Unit
Attributes
protected
Definition Classes
Logging
def logTrace(msg: => String): Unit
Attributes
protected
Definition Classes
Logging
def logWarning(msg: => String, throwable: Throwable): Unit
Attributes
protected
Definition Classes
Logging
def logWarning(msg: => String): Unit
Attributes
protected
Definition Classes
Logging
def logicalLink: Option[LogicalPlan]
returns
The logical plan this plan is linked to.
Definition Classes
SparkPlan
def longMetric(name: String): SQLMetric
returns
SQLMetric for the name.
Definition Classes
SparkPlan
def makeCopy(newArgs: Array[AnyRef]): SparkPlan
Overridden make copy also propagates sqlContext to copied plan.
Overridden make copy also propagates sqlContext to copied plan.
Definition Classes
SparkPlan → TreeNode
def map[A](f: (SparkPlan) => A): Seq[A]
Definition Classes
TreeNode
final def mapChildren(f: (SparkPlan) => SparkPlan): SparkPlan
Definition Classes
BinaryLike
def mapExpressions(f: (Expression) => Expression): FlatMapGroupsWithStateExec.this.type
Definition Classes
QueryPlan
def mapProductIterator[B](f: (Any) => B)(implicit arg0: ClassTag[B]): Array[B]
Attributes
protected
Definition Classes
TreeNode
def markRuleAsIneffective(ruleId: RuleId): Unit
Attributes
protected
Definition Classes
TreeNode
lazy val metrics: Map[String, SQLMetric]
returns
All metrics containing metrics of this SparkPlan.
Definition Classes
StateStoreWriter → SparkPlan
final def missingInput: AttributeSet
Definition Classes
QueryPlan
final def ne(arg0: AnyRef): Boolean
Definition Classes
AnyRef
def nodeName: String
Definition Classes
TreeNode
val nodePatterns: Seq[TreePattern]
Attributes
protected
Definition Classes
TreeNode
final def notify(): Unit
Definition Classes
AnyRef
Annotations
@native()
final def notifyAll(): Unit
Definition Classes
AnyRef
Annotations
@native()
def numberedTreeString: String
Definition Classes
TreeNode
val origin: Origin
Definition Classes
TreeNode
def otherCopyArgs: Seq[AnyRef]
Attributes
protected
Definition Classes
TreeNode
def output: Seq[Attribute]
Definition Classes
ObjectProducerExec → QueryPlan
val outputMode: OutputMode
val outputObjAttr: Attribute
Definition Classes
FlatMapGroupsWithStateExec → ObjectProducerExec
def outputObjectType: DataType
Definition Classes
ObjectProducerExec
def outputOrdering: Seq[SortOrder]
Specifies how data is ordered in each partition.
Specifies how data is ordered in each partition.
Definition Classes
SparkPlan
def outputPartitioning: Partitioning
Specifies how data is partitioned across different nodes in the cluster.
Specifies how data is partitioned across different nodes in the cluster. Note this method may fail if it is invoked before EnsureRequirements is applied since PartitioningCollection requires all its partitionings to have the same number of partitions.
Definition Classes
SparkPlan
lazy val outputSet: AttributeSet
Definition Classes
QueryPlan
Annotations
@transient()
def p(number: Int): SparkPlan
Definition Classes
TreeNode
final def prepare(): Unit
Prepares this SparkPlan for execution.
Prepares this SparkPlan for execution. It's idempotent.
Definition Classes
SparkPlan
def prepareSubqueries(): Unit
Finds scalar subquery expressions in this plan node and starts evaluating them.
Finds scalar subquery expressions in this plan node and starts evaluating them.
Attributes
protected
Definition Classes
SparkPlan
def prettyJson: String
Definition Classes
TreeNode
def printSchema(): Unit
Definition Classes
QueryPlan
def processDataWithPartition(iter: Iterator[InternalRow], store: StateStore, processor: InputProcessor, initialStateIterOption: Option[Iterator[InternalRow]] = None): CompletionIterator[InternalRow, Iterator[InternalRow]]
Process data by applying the user defined function on a per partition basis.
Process data by applying the user defined function on a per partition basis.
iter
- Iterator of the data rows
store
- associated state store for this partition
processor
- handle to the input processor object.
initialStateIterOption
- optional initial state iterator
def producedAttributes: AttributeSet
Definition Classes
ObjectProducerExec → QueryPlan
def productElementNames: Iterator[String]
Definition Classes
Product
lazy val references: AttributeSet
Definition Classes
QueryPlan
Annotations
@transient()
def removeKeysOlderThanWatermark(storeManager: StreamingAggregationStateManager, store: StateStore): Unit
Attributes
protected
Definition Classes
WatermarkSupport
def removeKeysOlderThanWatermark(store: StateStore): Unit
Attributes
protected
Definition Classes
WatermarkSupport
def requiredChildDistribution: Seq[Distribution]
Distribute by grouping attributes - We need the underlying data and the initial state data to have the same grouping so that the data are co-lacated on the same task.
Distribute by grouping attributes - We need the underlying data and the initial state data to have the same grouping so that the data are co-lacated on the same task.
Definition Classes
FlatMapGroupsWithStateExec → SparkPlan
def requiredChildOrdering: Seq[Seq[SortOrder]]
Ordering needed for using GroupingIterator.
Ordering needed for using GroupingIterator. We need the initial state to also use the ordering as the data so that we can co-locate the keys from the underlying data and the initial state.
Definition Classes
FlatMapGroupsWithStateExec → SparkPlan
def resetMetrics(): Unit
Resets all the metrics.
Resets all the metrics.
Definition Classes
SparkPlan
def rewriteAttrs(attrMap: AttributeMap[Attribute]): SparkPlan
Definition Classes
QueryPlan
def right: SparkPlan
Definition Classes
FlatMapGroupsWithStateExec → BinaryLike
final def sameResult(other: SparkPlan): Boolean
Definition Classes
QueryPlan
lazy val schema: StructType
Definition Classes
QueryPlan
def schemaString: String
Definition Classes
QueryPlan
final def semanticHash(): Int
Definition Classes
QueryPlan
final val session: SparkSession
Definition Classes
SparkPlan
def setLogicalLink(logicalPlan: LogicalPlan): Unit
Set logical plan link recursively if unset.
Set logical plan link recursively if unset.
Definition Classes
SparkPlan
def setOperatorMetrics(numStateStoreInstances: Int = 1): Unit
Set the operator level metrics
Set the operator level metrics
Attributes
protected
Definition Classes
StateStoreWriter
def setStoreMetrics(store: StateStore): Unit
Set the SQL metrics related to the state store.
Set the SQL metrics related to the state store. This should be called in that task after the store has been updated.
Attributes
protected
Definition Classes
StateStoreWriter
def setTagValue[T](tag: TreeNodeTag[T], value: T): Unit
Definition Classes
TreeNode
def shortName: String
Name to output in StreamingOperatorProgress to identify operator type
Name to output in StreamingOperatorProgress to identify operator type
Definition Classes
FlatMapGroupsWithStateExec → StateStoreWriter
def shouldRunAnotherBatch(newMetadata: OffsetSeqMetadata): Boolean
Should the MicroBatchExecution run another batch based on this stateful operator and the current updated metadata.
Should the MicroBatchExecution run another batch based on this stateful operator and the current updated metadata.
Definition Classes
FlatMapGroupsWithStateExec → StateStoreWriter
def simpleString(maxFields: Int): String
Definition Classes
QueryPlan → TreeNode
def simpleStringWithNodeId(): String
Definition Classes
QueryPlan → TreeNode
def sparkContext: SparkContext
Attributes
protected
Definition Classes
SparkPlan
val stateEncoder: ExpressionEncoder[Any]
val stateFormatVersion: Int
val stateInfo: Option[StatefulOperatorStateInfo]
Definition Classes
FlatMapGroupsWithStateExec → StatefulOperator
def statePrefix: String
Attributes
protected
Definition Classes
QueryPlan
def stringArgs: Iterator[Any]
Attributes
protected
Definition Classes
TreeNode
lazy val subqueries: Seq[SparkPlan]
Definition Classes
QueryPlan
Annotations
@transient()
def subqueriesAll: Seq[SparkPlan]
Definition Classes
QueryPlan
def supportsColumnar: Boolean
Return true if this stage of the plan supports columnar execution.
Return true if this stage of the plan supports columnar execution. A plan can also support row-based execution (see supportsRowBased). Spark will decide which execution to be called during query planning.
Definition Classes
SparkPlan
def supportsRowBased: Boolean
Return true if this stage of the plan supports row-based execution.
Return true if this stage of the plan supports row-based execution. A plan can also support columnar execution (see supportsColumnar). Spark will decide which execution to be called during query planning.
Definition Classes
SparkPlan
final def synchronized[T0](arg0: => T0): T0
Definition Classes
AnyRef
def timeTakenMs(body: => Unit): Long
Records the duration of running body for the next query progress update.
Records the duration of running body for the next query progress update.
Attributes
protected
Definition Classes
StateStoreWriter
val timeoutConf: GroupStateTimeout
def toJSON: String
Definition Classes
TreeNode
def toRowBased: SparkPlan
Converts the output of this plan to row-based if it is columnar plan.
Converts the output of this plan to row-based if it is columnar plan.
Definition Classes
SparkPlan
def toString(): String
Definition Classes
TreeNode → AnyRef → Any
def transform(rule: PartialFunction[SparkPlan, SparkPlan]): SparkPlan
Definition Classes
TreeNode
def transformAllExpressions(rule: PartialFunction[Expression, Expression]): FlatMapGroupsWithStateExec.this.type
Definition Classes
QueryPlan
def transformAllExpressionsWithPruning(cond: (TreePatternBits) => Boolean, ruleId: RuleId)(rule: PartialFunction[Expression, Expression]): FlatMapGroupsWithStateExec.this.type
Definition Classes
QueryPlan
def transformDown(rule: PartialFunction[SparkPlan, SparkPlan]): SparkPlan
Definition Classes
TreeNode
def transformDownWithPruning(cond: (TreePatternBits) => Boolean, ruleId: RuleId)(rule: PartialFunction[SparkPlan, SparkPlan]): SparkPlan
Definition Classes
TreeNode
def transformDownWithSubqueries(f: PartialFunction[SparkPlan, SparkPlan]): SparkPlan
Definition Classes
QueryPlan
def transformDownWithSubqueriesAndPruning(cond: (TreePatternBits) => Boolean, ruleId: RuleId)(f: PartialFunction[SparkPlan, SparkPlan]): SparkPlan
Definition Classes
QueryPlan
def transformExpressions(rule: PartialFunction[Expression, Expression]): FlatMapGroupsWithStateExec.this.type
Definition Classes
QueryPlan
def transformExpressionsDown(rule: PartialFunction[Expression, Expression]): FlatMapGroupsWithStateExec.this.type
Definition Classes
QueryPlan
def transformExpressionsDownWithPruning(cond: (TreePatternBits) => Boolean, ruleId: RuleId)(rule: PartialFunction[Expression, Expression]): FlatMapGroupsWithStateExec.this.type
Definition Classes
QueryPlan
def transformExpressionsUp(rule: PartialFunction[Expression, Expression]): FlatMapGroupsWithStateExec.this.type
Definition Classes
QueryPlan
def transformExpressionsUpWithPruning(cond: (TreePatternBits) => Boolean, ruleId: RuleId)(rule: PartialFunction[Expression, Expression]): FlatMapGroupsWithStateExec.this.type
Definition Classes
QueryPlan
def transformExpressionsWithPruning(cond: (TreePatternBits) => Boolean, ruleId: RuleId)(rule: PartialFunction[Expression, Expression]): FlatMapGroupsWithStateExec.this.type
Definition Classes
QueryPlan
def transformUp(rule: PartialFunction[SparkPlan, SparkPlan]): SparkPlan
Definition Classes
TreeNode
def transformUpWithBeforeAndAfterRuleOnChildren(cond: (SparkPlan) => Boolean, ruleId: RuleId)(rule: PartialFunction[(SparkPlan, SparkPlan), SparkPlan]): SparkPlan
Definition Classes
TreeNode
def transformUpWithNewOutput(rule: PartialFunction[SparkPlan, (SparkPlan, Seq[(Attribute, Attribute)])], skipCond: (SparkPlan) => Boolean, canGetOutput: (SparkPlan) => Boolean): SparkPlan
Definition Classes
QueryPlan
def transformUpWithPruning(cond: (TreePatternBits) => Boolean, ruleId: RuleId)(rule: PartialFunction[SparkPlan, SparkPlan]): SparkPlan
Definition Classes
TreeNode
def transformUpWithSubqueries(f: PartialFunction[SparkPlan, SparkPlan]): SparkPlan
Definition Classes
QueryPlan
def transformWithPruning(cond: (TreePatternBits) => Boolean, ruleId: RuleId)(rule: PartialFunction[SparkPlan, SparkPlan]): SparkPlan
Definition Classes
TreeNode
def transformWithSubqueries(f: PartialFunction[SparkPlan, SparkPlan]): SparkPlan
Definition Classes
QueryPlan
lazy val treePatternBits: BitSet
Definition Classes
QueryPlan → TreeNode → TreePatternBits
def treeString(append: (String) => Unit, verbose: Boolean, addSuffix: Boolean, maxFields: Int, printOperatorId: Boolean): Unit
Definition Classes
TreeNode
final def treeString(verbose: Boolean, addSuffix: Boolean, maxFields: Int, printOperatorId: Boolean): String
Definition Classes
TreeNode
final def treeString: String
Definition Classes
TreeNode
def unsetTagValue[T](tag: TreeNodeTag[T]): Unit
Definition Classes
TreeNode
def updateOuterReferencesInSubquery(plan: SparkPlan, attrMap: AttributeMap[Attribute]): SparkPlan
Attributes
protected
Definition Classes
QueryPlan
val valueDeserializer: Expression
def vectorTypes: Option[Seq[String]]
The exact java types of the columns that are output in columnar processing mode.
The exact java types of the columns that are output in columnar processing mode. This is a performance optimization for code generation and is optional.
Definition Classes
SparkPlan
def verboseString(maxFields: Int): String
Definition Classes
QueryPlan → TreeNode
def verboseStringWithOperatorId(): String
Definition Classes
BinaryExecNode → QueryPlan
def verboseStringWithSuffix(maxFields: Int): String
Definition Classes
TreeNode
final def wait(): Unit
Definition Classes
AnyRef
Annotations
@throws(classOf[java.lang.InterruptedException])
final def wait(arg0: Long, arg1: Int): Unit
Definition Classes
AnyRef
Annotations
@throws(classOf[java.lang.InterruptedException])
final def wait(arg0: Long): Unit
Definition Classes
AnyRef
Annotations
@throws(classOf[java.lang.InterruptedException]) @native()
def waitForSubqueries(): Unit
Blocks the thread until all subqueries finish evaluation and update the results.
Blocks the thread until all subqueries finish evaluation and update the results.
Attributes
protected
Definition Classes
SparkPlan
lazy val watermarkExpression: Option[Expression]
Generate an expression that matches data older than the watermark
Generate an expression that matches data older than the watermark
Definition Classes
WatermarkSupport
lazy val watermarkPredicateForData: Option[BasePredicate]
Predicate based on the child output that matches data older than the watermark.
Predicate based on the child output that matches data older than the watermark.
Definition Classes
WatermarkSupport
lazy val watermarkPredicateForKeys: Option[BasePredicate]
Predicate based on keys that matches data older than the watermark
Predicate based on keys that matches data older than the watermark
Definition Classes
WatermarkSupport
final def withNewChildren(newChildren: Seq[SparkPlan]): SparkPlan
Definition Classes
TreeNode
def withNewChildrenInternal(newLeft: SparkPlan, newRight: SparkPlan): FlatMapGroupsWithStateExec
Attributes
protected
Definition Classes
FlatMapGroupsWithStateExec → BinaryLike
final def withNewChildrenInternal(newChildren: IndexedSeq[SparkPlan]): SparkPlan
Definition Classes
BinaryLike

Packages

FlatMapGroupsWithStateExec

Companion object FlatMapGroupsWithStateExec

Instance Constructors

Type Members

Value Members

Inherited from WatermarkSupport

Inherited from StateStoreWriter

Inherited from StatefulOperator

Inherited from ObjectProducerExec

Inherited from BinaryExecNode

Inherited from BinaryLike[SparkPlan]

Inherited from SparkPlan

Inherited from Serializable

Inherited from Logging

Inherited from QueryPlan[SparkPlan]

Inherited from SQLConfHelper

Inherited from TreeNode[SparkPlan]

Inherited from TreePatternBits

Inherited from Product

Inherited from Equals

Inherited from AnyRef

Inherited from Any

Ungrouped

Packages

FlatMapGroupsWithStateExec

Companion object FlatMapGroupsWithStateExec

Instance Constructors

Type Members

Value Members

Inherited from WatermarkSupport

Inherited from StateStoreWriter

Inherited from StatefulOperator

Inherited from ObjectProducerExec

Inherited from BinaryExecNode

Inherited from BinaryLike[SparkPlan]

Inherited from SparkPlan

Inherited from Serializable

Inherited from Logging

Inherited from QueryPlan[SparkPlan]

Inherited from SQLConfHelper

Inherited from TreeNode[SparkPlan]

Inherited from TreePatternBits

Inherited from Product

Inherited from Equals

Inherited from AnyRef

Inherited from Any

Ungrouped

FlatMapGroupsWithStateExec