org.apache.spark.sql.execution.streaming

FlatMapGroupsWithStateExec

Companion object FlatMapGroupsWithStateExec

case class FlatMapGroupsWithStateExec(func: (Any, Iterator[Any], LogicalGroupState[Any]) ⇒ Iterator[Any], keyDeserializer: Expression, valueDeserializer: Expression, initialStateDeserializer: Expression, groupingAttributes: Seq[Attribute], initialStateGroupAttrs: Seq[Attribute], dataAttributes: Seq[Attribute], initialStateDataAttrs: Seq[Attribute], outputObjAttr: Attribute, stateInfo: Option[StatefulOperatorStateInfo], stateEncoder: ExpressionEncoder[Any], stateFormatVersion: Int, outputMode: OutputMode, timeoutConf: GroupStateTimeout, batchTimestampMs: Option[Long], eventTimeWatermark: Option[Long], initialState: SparkPlan, hasInitialState: Boolean, child: SparkPlan) extends SparkPlan with BinaryExecNode with ObjectProducerExec with StateStoreWriter with WatermarkSupport with Product with Serializable

Physical operator for executing FlatMapGroupsWithState

func: function called on each group
keyDeserializer: used to extract the key object for each group.
valueDeserializer: used to extract the items in the iterator from an input row.
initialStateDeserializer: used to extract the state object from the initialState dataset
groupingAttributes: used to group the data
dataAttributes: used to read the data
outputObjAttr: Defines the output object
stateEncoder: used to serialize/deserialize state before calling func
outputMode: the output mode of func
timeoutConf: used to timeout groups that have not received data in a while
batchTimestampMs: processing timestamp of the current batch.
eventTimeWatermark: event time watermark for the current batch
initialState: the user specified initial state
hasInitialState: indicates whether the initial state is provided or not
child: the physical plan for the underlying data

Linear Supertypes

WatermarkSupport, StateStoreWriter, StatefulOperator, ObjectProducerExec, BinaryExecNode, BinaryLike[SparkPlan], SparkPlan, Serializable, Serializable, Logging, QueryPlan[SparkPlan], SQLConfHelper, TreeNode[SparkPlan], TreePatternBits, Product, Equals, AnyRef, Any

Ordering

Alphabetic
By Inheritance

Inherited

FlatMapGroupsWithStateExec
WatermarkSupport
StateStoreWriter
StatefulOperator
ObjectProducerExec
BinaryExecNode
BinaryLike
SparkPlan
Serializable
Serializable
Logging
QueryPlan
SQLConfHelper
TreeNode
TreePatternBits
Product
Equals
AnyRef
Any

Hide All
Show All

Visibility

Public
All

Instance Constructors

new FlatMapGroupsWithStateExec(func: (Any, Iterator[Any], LogicalGroupState[Any]) ⇒ Iterator[Any], keyDeserializer: Expression, valueDeserializer: Expression, initialStateDeserializer: Expression, groupingAttributes: Seq[Attribute], initialStateGroupAttrs: Seq[Attribute], dataAttributes: Seq[Attribute], initialStateDataAttrs: Seq[Attribute], outputObjAttr: Attribute, stateInfo: Option[StatefulOperatorStateInfo], stateEncoder: ExpressionEncoder[Any], stateFormatVersion: Int, outputMode: OutputMode, timeoutConf: GroupStateTimeout, batchTimestampMs: Option[Long], eventTimeWatermark: Option[Long], initialState: SparkPlan, hasInitialState: Boolean, child: SparkPlan)
func
function called on each group
keyDeserializer
used to extract the key object for each group.
valueDeserializer
used to extract the items in the iterator from an input row.
initialStateDeserializer
used to extract the state object from the initialState dataset
groupingAttributes
used to group the data
dataAttributes
used to read the data
outputObjAttr
Defines the output object
stateEncoder
used to serialize/deserialize state before calling func
outputMode
the output mode of func
timeoutConf
used to timeout groups that have not received data in a while
batchTimestampMs
processing timestamp of the current batch.
eventTimeWatermark
event time watermark for the current batch
initialState
the user specified initial state
hasInitialState
indicates whether the initial state is provided or not
child
the physical plan for the underlying data

Type Members

class InputProcessor extends AnyRef
Helper class to update the state store

Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
lazy val allAttributes: AttributeSeq

Definition Classes
QueryPlan
def apply(number: Int): TreeNode[_]

Definition Classes
TreeNode
def applyRemovingRowsOlderThanWatermark(iter: Iterator[InternalRow], predicateDropRowByWatermark: BasePredicate): Iterator[InternalRow]

Attributes
protected
Definition Classes
StateStoreWriter
def argString(maxFields: Int): String

Definition Classes
TreeNode
def asCode: String

Definition Classes
TreeNode
final def asInstanceOf[T0]: T0

Definition Classes
Any
val batchTimestampMs: Option[Long]
final lazy val canonicalized: SparkPlan

Definition Classes
QueryPlan
Annotations
@transient()
val child: SparkPlan

Definition Classes
FlatMapGroupsWithStateExec → WatermarkSupport
final lazy val children: Seq[SparkPlan]

Definition Classes
BinaryLike
Annotations
@transient()
def cleanupResources(): Unit
Cleans up the resources used by the physical operator (if any).
Cleans up the resources used by the physical operator (if any). In general, all the resources should be cleaned up when the task finishes but operators like SortMergeJoinExec and LimitExec may want eager cleanup to free up tight resources (e.g., memory).

Attributes
protected[sql]
Definition Classes
SparkPlan
def clone(): SparkPlan

Definition Classes
TreeNode → AnyRef
def collect[B](pf: PartialFunction[SparkPlan, B]): Seq[B]

Definition Classes
TreeNode
def collectFirst[B](pf: PartialFunction[SparkPlan, B]): Option[B]

Definition Classes
TreeNode
def collectLeaves(): Seq[SparkPlan]

Definition Classes
TreeNode
def collectWithSubqueries[B](f: PartialFunction[SparkPlan, B]): Seq[B]

Definition Classes
QueryPlan
def conf: SQLConf

Definition Classes
SparkPlan → SQLConfHelper
final def containsAllPatterns(patterns: TreePattern*): Boolean

Definition Classes
TreePatternBits
final def containsAnyPattern(patterns: TreePattern*): Boolean

Definition Classes
TreePatternBits
lazy val containsChild: Set[TreeNode[_]]

Definition Classes
TreeNode
final def containsPattern(t: TreePattern): Boolean

Definition Classes
TreePatternBits
Annotations
@inline()
def copyTagsFrom(other: SparkPlan): Unit

Definition Classes
TreeNode
def customStatefulOperatorMetrics: Seq[StatefulOperatorCustomMetric]
Set of stateful operator custom metrics.
Set of stateful operator custom metrics. These are captured as part of the generic key-value map StateOperatorProgress.customMetrics. Stateful operators can extend this method to provide their own unique custom metrics.

Attributes
protected
Definition Classes
StateStoreWriter
val dataAttributes: Seq[Attribute]
def doCanonicalize(): SparkPlan

Attributes
protected
Definition Classes
QueryPlan
def doExecute(): RDD[InternalRow]
Produces the result of the query as an RDD[InternalRow]
Produces the result of the query as an RDD[InternalRow]
Overridden by concrete implementations of SparkPlan.

Attributes
protected
Definition Classes
FlatMapGroupsWithStateExec → SparkPlan
def doExecuteBroadcast[T](): Broadcast[T]
Produces the result of the query as a broadcast variable.
Produces the result of the query as a broadcast variable.
Overridden by concrete implementations of SparkPlan.

Attributes
protected[sql]
Definition Classes
SparkPlan
def doExecuteColumnar(): RDD[ColumnarBatch]
Produces the result of the query as an RDD[ColumnarBatch] if supportsColumnar returns true.
Produces the result of the query as an RDD[ColumnarBatch] if supportsColumnar returns true. By convention the executor that creates a ColumnarBatch is responsible for closing it when it is no longer needed. This allows input formats to be able to reuse batches if needed.

Attributes
protected
Definition Classes
SparkPlan
def doPrepare(): Unit
Overridden by concrete implementations of SparkPlan.
Overridden by concrete implementations of SparkPlan. It is guaranteed to run before any execute of SparkPlan. This is helpful if we want to set up some state before executing the query, e.g., BroadcastHashJoin uses it to broadcast asynchronously.

Attributes
protected
Definition Classes
SparkPlan
Note
prepare method has already walked down the tree, so the implementation doesn't have to call children's prepare methods. This will only be called once, protected by this.
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
val eventTimeWatermark: Option[Long]
The watermark value.
The watermark value.

Definition Classes
FlatMapGroupsWithStateExec → WatermarkSupport
final def execute(): RDD[InternalRow]
Returns the result of this query as an RDD[InternalRow] by delegating to doExecute after preparations.
Returns the result of this query as an RDD[InternalRow] by delegating to doExecute after preparations.
Concrete implementations of SparkPlan should override doExecute.

Definition Classes
SparkPlan
final def executeBroadcast[T](): Broadcast[T]
Returns the result of this query as a broadcast variable by delegating to doExecuteBroadcast after preparations.
Returns the result of this query as a broadcast variable by delegating to doExecuteBroadcast after preparations.
Concrete implementations of SparkPlan should override doExecuteBroadcast.

Definition Classes
SparkPlan
def executeCollect(): Array[InternalRow]
Runs this query returning the result as an array.
Runs this query returning the result as an array.

Definition Classes
SparkPlan
def executeCollectPublic(): Array[Row]
Runs this query returning the result as an array, using external Row format.
Runs this query returning the result as an array, using external Row format.

Definition Classes
SparkPlan
final def executeColumnar(): RDD[ColumnarBatch]
Returns the result of this query as an RDD[ColumnarBatch] by delegating to doColumnarExecute after preparations.
Returns the result of this query as an RDD[ColumnarBatch] by delegating to doColumnarExecute after preparations.
Concrete implementations of SparkPlan should override doColumnarExecute if supportsColumnar returns true.

Definition Classes
SparkPlan
final def executeQuery[T](query: ⇒ T): T
Executes a query after preparing the query and adding query plan information to created RDDs for visualization.
Executes a query after preparing the query and adding query plan information to created RDDs for visualization.

Attributes
protected
Definition Classes
SparkPlan
def executeTail(n: Int): Array[InternalRow]
Runs this query returning the last n rows as an array.
Runs this query returning the last n rows as an array.
This is modeled after RDD.take but never runs any job locally on the driver.

Definition Classes
SparkPlan
def executeTake(n: Int): Array[InternalRow]
Runs this query returning the first n rows as an array.
Runs this query returning the first n rows as an array.
This is modeled after RDD.take but never runs any job locally on the driver.

Definition Classes
SparkPlan
def executeToIterator(): Iterator[InternalRow]
Runs this query returning the result as an iterator of InternalRow.
Runs this query returning the result as an iterator of InternalRow.

Definition Classes
SparkPlan
Note
Triggers multiple jobs (one for each partition).
final def expressions: Seq[Expression]

Definition Classes
QueryPlan
def fastEquals(other: TreeNode[_]): Boolean

Definition Classes
TreeNode
def finalize(): Unit

Attributes
protected[lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
def find(f: (SparkPlan) ⇒ Boolean): Option[SparkPlan]

Definition Classes
TreeNode
def flatMap[A](f: (SparkPlan) ⇒ TraversableOnce[A]): Seq[A]

Definition Classes
TreeNode
def foreach(f: (SparkPlan) ⇒ Unit): Unit

Definition Classes
TreeNode
def foreachUp(f: (SparkPlan) ⇒ Unit): Unit

Definition Classes
TreeNode
def formattedNodeName: String

Attributes
protected
Definition Classes
QueryPlan
val func: (Any, Iterator[Any], LogicalGroupState[Any]) ⇒ Iterator[Any]
def generateTreeString(depth: Int, lastChildren: Seq[Boolean], append: (String) ⇒ Unit, verbose: Boolean, prefix: String, addSuffix: Boolean, maxFields: Int, printNodeId: Boolean, indent: Int): Unit

Definition Classes
TreeNode
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
Annotations
@native()
def getDefaultTreePatternBits: BitSet

Attributes
protected
Definition Classes
TreeNode
def getProgress(): StateOperatorProgress
Get the progress made by this stateful operator after execution.
Get the progress made by this stateful operator after execution. This should be called in the driver after this SparkPlan has been executed and metrics have been updated.

Definition Classes
StateStoreWriter
def getStateInfo: StatefulOperatorStateInfo

Attributes
protected
Definition Classes
StatefulOperator
def getTagValue[T](tag: TreeNodeTag[T]): Option[T]

Definition Classes
TreeNode
val groupingAttributes: Seq[Attribute]
val hasInitialState: Boolean
def hashCode(): Int

Definition Classes
TreeNode → AnyRef → Any
val id: Int

Definition Classes
SparkPlan
val initialState: SparkPlan
val initialStateDataAttrs: Seq[Attribute]
val initialStateDeserializer: Expression
val initialStateGroupAttrs: Seq[Attribute]
def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean

Attributes
protected
Definition Classes
Logging
def initializeLogIfNecessary(isInterpreter: Boolean): Unit

Attributes
protected
Definition Classes
Logging
def innerChildren: Seq[QueryPlan[_]]

Definition Classes
QueryPlan → TreeNode
def inputSet: AttributeSet

Definition Classes
QueryPlan
def isCanonicalizedPlan: Boolean

Attributes
protected
Definition Classes
QueryPlan
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
def isRuleIneffective(ruleId: RuleId): Boolean

Attributes
protected
Definition Classes
TreeNode
def isTraceEnabled(): Boolean

Attributes
protected
Definition Classes
Logging
def jsonFields: List[JField]

Attributes
protected
Definition Classes
TreeNode
val keyDeserializer: Expression
def keyExpressions: Seq[Attribute]
The keys that may have a watermark attribute.
The keys that may have a watermark attribute.

Definition Classes
FlatMapGroupsWithStateExec → WatermarkSupport
def left: SparkPlan

Definition Classes
FlatMapGroupsWithStateExec → BinaryLike
final def legacyWithNewChildren(newChildren: Seq[SparkPlan]): SparkPlan

Attributes
protected
Definition Classes
TreeNode
def log: Logger

Attributes
protected
Definition Classes
Logging
def logDebug(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logDebug(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logError(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logError(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logInfo(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logInfo(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logName: String

Attributes
protected
Definition Classes
Logging
def logTrace(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logTrace(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logWarning(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logWarning(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logicalLink: Option[LogicalPlan]
returns
The logical plan this plan is linked to.

Definition Classes
SparkPlan
def longMetric(name: String): SQLMetric
returns
SQLMetric for the name.

Definition Classes
SparkPlan
def makeCopy(newArgs: Array[AnyRef]): SparkPlan
Overridden make copy also propagates sqlContext to copied plan.
Overridden make copy also propagates sqlContext to copied plan.

Definition Classes
SparkPlan → TreeNode
def map[A](f: (SparkPlan) ⇒ A): Seq[A]

Definition Classes
TreeNode
final def mapChildren(f: (SparkPlan) ⇒ SparkPlan): SparkPlan

Definition Classes
BinaryLike
def mapExpressions(f: (Expression) ⇒ Expression): FlatMapGroupsWithStateExec.this.type

Definition Classes
QueryPlan
def mapProductIterator[B](f: (Any) ⇒ B)(implicit arg0: ClassTag[B]): Array[B]

Attributes
protected
Definition Classes
TreeNode
def markRuleAsIneffective(ruleId: RuleId): Unit

Attributes
protected
Definition Classes
TreeNode
lazy val metrics: Map[String, SQLMetric]
returns
All metrics containing metrics of this SparkPlan.

Definition Classes
StateStoreWriter → SparkPlan
final def missingInput: AttributeSet

Definition Classes
QueryPlan
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def nodeName: String

Definition Classes
TreeNode
val nodePatterns: Seq[TreePattern]

Attributes
protected
Definition Classes
TreeNode
final def notify(): Unit

Definition Classes
AnyRef
Annotations
@native()
final def notifyAll(): Unit

Definition Classes
AnyRef
Annotations
@native()
def numberedTreeString: String

Definition Classes
TreeNode
val origin: Origin

Definition Classes
TreeNode
def otherCopyArgs: Seq[AnyRef]

Attributes
protected
Definition Classes
TreeNode
def output: Seq[Attribute]

Definition Classes
ObjectProducerExec → QueryPlan
val outputMode: OutputMode
val outputObjAttr: Attribute

Definition Classes
FlatMapGroupsWithStateExec → ObjectProducerExec
def outputObjectType: DataType

Definition Classes
ObjectProducerExec
def outputOrdering: Seq[SortOrder]
Specifies how data is ordered in each partition.
Specifies how data is ordered in each partition.

Definition Classes
SparkPlan
def outputPartitioning: Partitioning
Specifies how data is partitioned across different nodes in the cluster.
Specifies how data is partitioned across different nodes in the cluster. Note this method may fail if it is invoked before EnsureRequirements is applied since PartitioningCollection requires all its partitionings to have the same number of partitions.

Definition Classes
SparkPlan
lazy val outputSet: AttributeSet

Definition Classes
QueryPlan
Annotations
@transient()
def p(number: Int): SparkPlan

Definition Classes
TreeNode
final def prepare(): Unit
Prepares this SparkPlan for execution.
Prepares this SparkPlan for execution. It's idempotent.

Definition Classes
SparkPlan
def prepareSubqueries(): Unit
Finds scalar subquery expressions in this plan node and starts evaluating them.
Finds scalar subquery expressions in this plan node and starts evaluating them.

Attributes
protected
Definition Classes
SparkPlan
def prettyJson: String

Definition Classes
TreeNode
def printSchema(): Unit

Definition Classes
QueryPlan
def processDataWithPartition(iter: Iterator[InternalRow], store: StateStore, processor: InputProcessor, initialStateIterOption: Option[Iterator[InternalRow]] = None): CompletionIterator[InternalRow, Iterator[InternalRow]]
Process data by applying the user defined function on a per partition basis.
Process data by applying the user defined function on a per partition basis.
iter
- Iterator of the data rows
store
- associated state store for this partition
processor
- handle to the input processor object.
initialStateIterOption
- optional initial state iterator
def producedAttributes: AttributeSet

Definition Classes
ObjectProducerExec → QueryPlan
lazy val references: AttributeSet

Definition Classes
QueryPlan
Annotations
@transient()
def removeKeysOlderThanWatermark(storeManager: StreamingAggregationStateManager, store: StateStore): Unit

Attributes
protected
Definition Classes
WatermarkSupport
def removeKeysOlderThanWatermark(store: StateStore): Unit

Attributes
protected
Definition Classes
WatermarkSupport
def requiredChildDistribution: Seq[Distribution]
Distribute by grouping attributes - We need the underlying data and the initial state data to have the same grouping so that the data are co-lacated on the same task.
Distribute by grouping attributes - We need the underlying data and the initial state data to have the same grouping so that the data are co-lacated on the same task.

Definition Classes
FlatMapGroupsWithStateExec → SparkPlan
def requiredChildOrdering: Seq[Seq[SortOrder]]
Ordering needed for using GroupingIterator.
Ordering needed for using GroupingIterator. We need the initial state to also use the ordering as the data so that we can co-locate the keys from the underlying data and the initial state.

Definition Classes
FlatMapGroupsWithStateExec → SparkPlan
def resetMetrics(): Unit
Resets all the metrics.
Resets all the metrics.

Definition Classes
SparkPlan
def rewriteAttrs(attrMap: AttributeMap[Attribute]): SparkPlan

Definition Classes
QueryPlan
def right: SparkPlan

Definition Classes
FlatMapGroupsWithStateExec → BinaryLike
final def sameResult(other: SparkPlan): Boolean

Definition Classes
QueryPlan
lazy val schema: StructType

Definition Classes
QueryPlan
def schemaString: String

Definition Classes
QueryPlan
final def semanticHash(): Int

Definition Classes
QueryPlan
final val session: SparkSession

Definition Classes
SparkPlan
def setLogicalLink(logicalPlan: LogicalPlan): Unit
Set logical plan link recursively if unset.
Set logical plan link recursively if unset.

Definition Classes
SparkPlan
def setOperatorMetrics(numStateStoreInstances: Int = 1): Unit
Set the operator level metrics
Set the operator level metrics

Attributes
protected
Definition Classes
StateStoreWriter
def setStoreMetrics(store: StateStore): Unit
Set the SQL metrics related to the state store.
Set the SQL metrics related to the state store. This should be called in that task after the store has been updated.

Attributes
protected
Definition Classes
StateStoreWriter
def setTagValue[T](tag: TreeNodeTag[T], value: T): Unit

Definition Classes
TreeNode
def shortName: String
Name to output in StreamingOperatorProgress to identify operator type
Name to output in StreamingOperatorProgress to identify operator type

Definition Classes
FlatMapGroupsWithStateExec → StateStoreWriter
def shouldRunAnotherBatch(newMetadata: OffsetSeqMetadata): Boolean
Should the MicroBatchExecution run another batch based on this stateful operator and the current updated metadata.
Should the MicroBatchExecution run another batch based on this stateful operator and the current updated metadata.

Definition Classes
FlatMapGroupsWithStateExec → StateStoreWriter
def simpleString(maxFields: Int): String

Definition Classes
QueryPlan → TreeNode
def simpleStringWithNodeId(): String

Definition Classes
QueryPlan → TreeNode
def sparkContext: SparkContext

Attributes
protected
Definition Classes
SparkPlan
val stateEncoder: ExpressionEncoder[Any]
val stateFormatVersion: Int
val stateInfo: Option[StatefulOperatorStateInfo]

Definition Classes
FlatMapGroupsWithStateExec → StatefulOperator
def statePrefix: String

Attributes
protected
Definition Classes
QueryPlan
def stringArgs: Iterator[Any]

Attributes
protected
Definition Classes
TreeNode
def subqueries: Seq[SparkPlan]

Definition Classes
QueryPlan
def subqueriesAll: Seq[SparkPlan]

Definition Classes
QueryPlan
def supportsColumnar: Boolean
Return true if this stage of the plan supports columnar execution.
Return true if this stage of the plan supports columnar execution.

Definition Classes
SparkPlan
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
def timeTakenMs(body: ⇒ Unit): Long
Records the duration of running body for the next query progress update.
Records the duration of running body for the next query progress update.

Attributes
protected
Definition Classes
StateStoreWriter
val timeoutConf: GroupStateTimeout
def toJSON: String

Definition Classes
TreeNode
def toString(): String

Definition Classes
TreeNode → AnyRef → Any
def transform(rule: PartialFunction[SparkPlan, SparkPlan]): SparkPlan

Definition Classes
TreeNode
def transformAllExpressions(rule: PartialFunction[Expression, Expression]): FlatMapGroupsWithStateExec.this.type

Definition Classes
QueryPlan
def transformAllExpressionsWithPruning(cond: (TreePatternBits) ⇒ Boolean, ruleId: RuleId)(rule: PartialFunction[Expression, Expression]): FlatMapGroupsWithStateExec.this.type

Definition Classes
QueryPlan
def transformDown(rule: PartialFunction[SparkPlan, SparkPlan]): SparkPlan

Definition Classes
TreeNode
def transformDownWithPruning(cond: (TreePatternBits) ⇒ Boolean, ruleId: RuleId)(rule: PartialFunction[SparkPlan, SparkPlan]): SparkPlan

Definition Classes
TreeNode
def transformExpressions(rule: PartialFunction[Expression, Expression]): FlatMapGroupsWithStateExec.this.type

Definition Classes
QueryPlan
def transformExpressionsDown(rule: PartialFunction[Expression, Expression]): FlatMapGroupsWithStateExec.this.type

Definition Classes
QueryPlan
def transformExpressionsDownWithPruning(cond: (TreePatternBits) ⇒ Boolean, ruleId: RuleId)(rule: PartialFunction[Expression, Expression]): FlatMapGroupsWithStateExec.this.type

Definition Classes
QueryPlan
def transformExpressionsUp(rule: PartialFunction[Expression, Expression]): FlatMapGroupsWithStateExec.this.type

Definition Classes
QueryPlan
def transformExpressionsUpWithPruning(cond: (TreePatternBits) ⇒ Boolean, ruleId: RuleId)(rule: PartialFunction[Expression, Expression]): FlatMapGroupsWithStateExec.this.type

Definition Classes
QueryPlan
def transformExpressionsWithPruning(cond: (TreePatternBits) ⇒ Boolean, ruleId: RuleId)(rule: PartialFunction[Expression, Expression]): FlatMapGroupsWithStateExec.this.type

Definition Classes
QueryPlan
def transformUp(rule: PartialFunction[SparkPlan, SparkPlan]): SparkPlan

Definition Classes
TreeNode
def transformUpWithBeforeAndAfterRuleOnChildren(cond: (SparkPlan) ⇒ Boolean, ruleId: RuleId)(rule: PartialFunction[(SparkPlan, SparkPlan), SparkPlan]): SparkPlan

Definition Classes
TreeNode
def transformUpWithNewOutput(rule: PartialFunction[SparkPlan, (SparkPlan, Seq[(Attribute, Attribute)])], skipCond: (SparkPlan) ⇒ Boolean, canGetOutput: (SparkPlan) ⇒ Boolean): SparkPlan

Definition Classes
QueryPlan
def transformUpWithPruning(cond: (TreePatternBits) ⇒ Boolean, ruleId: RuleId)(rule: PartialFunction[SparkPlan, SparkPlan]): SparkPlan

Definition Classes
TreeNode
def transformUpWithSubqueries(f: PartialFunction[SparkPlan, SparkPlan]): SparkPlan

Definition Classes
QueryPlan
def transformWithPruning(cond: (TreePatternBits) ⇒ Boolean, ruleId: RuleId)(rule: PartialFunction[SparkPlan, SparkPlan]): SparkPlan

Definition Classes
TreeNode
lazy val treePatternBits: BitSet

Definition Classes
QueryPlan → TreeNode → TreePatternBits
def treeString(append: (String) ⇒ Unit, verbose: Boolean, addSuffix: Boolean, maxFields: Int, printOperatorId: Boolean): Unit

Definition Classes
TreeNode
final def treeString(verbose: Boolean, addSuffix: Boolean, maxFields: Int, printOperatorId: Boolean): String

Definition Classes
TreeNode
final def treeString: String

Definition Classes
TreeNode
def unsetTagValue[T](tag: TreeNodeTag[T]): Unit

Definition Classes
TreeNode
def updateOuterReferencesInSubquery(plan: SparkPlan, attrMap: AttributeMap[Attribute]): SparkPlan

Attributes
protected
Definition Classes
QueryPlan
val valueDeserializer: Expression
def vectorTypes: Option[Seq[String]]
The exact java types of the columns that are output in columnar processing mode.
The exact java types of the columns that are output in columnar processing mode. This is a performance optimization for code generation and is optional.

Definition Classes
SparkPlan
def verboseString(maxFields: Int): String

Definition Classes
QueryPlan → TreeNode
def verboseStringWithOperatorId(): String

Definition Classes
BinaryExecNode → QueryPlan
def verboseStringWithSuffix(maxFields: Int): String

Definition Classes
TreeNode
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... ) @native()
def waitForSubqueries(): Unit
Blocks the thread until all subqueries finish evaluation and update the results.
Blocks the thread until all subqueries finish evaluation and update the results.

Attributes
protected
Definition Classes
SparkPlan
lazy val watermarkExpression: Option[Expression]
Generate an expression that matches data older than the watermark
Generate an expression that matches data older than the watermark

Definition Classes
WatermarkSupport
lazy val watermarkPredicateForData: Option[BasePredicate]
Predicate based on the child output that matches data older than the watermark.
Predicate based on the child output that matches data older than the watermark.

Definition Classes
WatermarkSupport
lazy val watermarkPredicateForKeys: Option[BasePredicate]
Predicate based on keys that matches data older than the watermark
Predicate based on keys that matches data older than the watermark

Definition Classes
WatermarkSupport
final def withNewChildren(newChildren: Seq[SparkPlan]): SparkPlan

Definition Classes
TreeNode
def withNewChildrenInternal(newLeft: SparkPlan, newRight: SparkPlan): FlatMapGroupsWithStateExec

Attributes
protected
Definition Classes
FlatMapGroupsWithStateExec → BinaryLike
final def withNewChildrenInternal(newChildren: IndexedSeq[SparkPlan]): SparkPlan

Definition Classes
BinaryLike

Packages

FlatMapGroupsWithStateExec

Companion object FlatMapGroupsWithStateExec

Instance Constructors

Type Members

Value Members

Inherited from WatermarkSupport

Inherited from StateStoreWriter

Inherited from StatefulOperator

Inherited from ObjectProducerExec

Inherited from BinaryExecNode

Inherited from BinaryLike[SparkPlan]

Inherited from SparkPlan

Inherited from Serializable

Inherited from Serializable

Inherited from Logging

Inherited from QueryPlan[SparkPlan]

Inherited from SQLConfHelper

Inherited from TreeNode[SparkPlan]

Inherited from TreePatternBits

Inherited from Product

Inherited from Equals

Inherited from AnyRef

Inherited from Any

Ungrouped

Packages

FlatMapGroupsWithStateExec 

Companion object FlatMapGroupsWithStateExec

Instance Constructors

Type Members

Value Members

Inherited from WatermarkSupport

Inherited from StateStoreWriter

Inherited from StatefulOperator

Inherited from ObjectProducerExec

Inherited from BinaryExecNode

Inherited from BinaryLike[SparkPlan]

Inherited from SparkPlan

Inherited from Serializable

Inherited from Serializable

Inherited from Logging

Inherited from QueryPlan[SparkPlan]

Inherited from SQLConfHelper

Inherited from TreeNode[SparkPlan]

Inherited from TreePatternBits

Inherited from Product

Inherited from Equals

Inherited from AnyRef

Inherited from Any

Ungrouped

FlatMapGroupsWithStateExec