Packages

object AggUtils

Utility functions used by the query planner to convert our plan to new aggregation code path.

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. AggUtils
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  8. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable])
  9. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  10. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  11. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  12. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  13. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  14. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  15. def planAggregateWithOneDistinct(groupingExpressions: Seq[NamedExpression], functionsWithDistinct: Seq[AggregateExpression], functionsWithoutDistinct: Seq[AggregateExpression], distinctExpressions: Seq[Expression], normalizedNamedDistinctExpressions: Seq[NamedExpression], resultExpressions: Seq[NamedExpression], child: SparkPlan): Seq[SparkPlan]
  16. def planAggregateWithoutDistinct(groupingExpressions: Seq[NamedExpression], aggregateExpressions: Seq[AggregateExpression], resultExpressions: Seq[NamedExpression], child: SparkPlan): Seq[SparkPlan]
  17. def planStreamingAggregation(groupingExpressions: Seq[NamedExpression], functionsWithoutDistinct: Seq[AggregateExpression], resultExpressions: Seq[NamedExpression], stateFormatVersion: Int, child: SparkPlan): Seq[SparkPlan]

    Plans a streaming aggregation using the following progression:

    Plans a streaming aggregation using the following progression:

    • Partial Aggregation
    • Shuffle
    • Partial Merge (now there is at most 1 tuple per group)
    • StateStoreRestore (now there is 1 tuple from this batch + optionally one from the previous)
    • PartialMerge (now there is at most 1 tuple per group)
    • StateStoreSave (saves the tuple for the next batch)
    • Complete (output the current result of the aggregation)
  18. def planStreamingAggregationForSession(groupingExpressions: Seq[NamedExpression], sessionExpression: NamedExpression, functionsWithoutDistinct: Seq[AggregateExpression], resultExpressions: Seq[NamedExpression], stateFormatVersion: Int, mergeSessionsInLocalPartition: Boolean, child: SparkPlan): Seq[SparkPlan]

    Plans a streaming session aggregation using the following progression:

    Plans a streaming session aggregation using the following progression:

    • Partial Aggregation
      • all tuples will have aggregated columns with initial value
    • (If "spark.sql.streaming.sessionWindow.merge.sessions.in.local.partition" is enabled)
      • Sort within partition (sort: all keys)
      • MergingSessionExec
        • calculate session among tuples, and aggregate tuples in session with partial merge
    • Shuffle & Sort (distribution: keys "without" session, sort: all keys)
    • SessionWindowStateStoreRestore (group: keys "without" session)
      • merge input tuples with stored tuples (sessions) respecting sort order
    • MergingSessionExec
      • calculate session among tuples, and aggregate tuples in session with partial merge
      • NOTE: it leverages the fact that the output of SessionWindowStateStoreRestore is sorted
      • now there is at most 1 tuple per group, key with session
    • SessionWindowStateStoreSave (group: keys "without" session)
      • saves tuple(s) for the next batch (multiple sessions could co-exist at the same time)
    • Complete (output the current result of the aggregation)
  19. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  20. def toString(): String
    Definition Classes
    AnyRef → Any
  21. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  22. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  23. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()

Inherited from AnyRef

Inherited from Any

Ungrouped