Type Members

1.  trait AggregateCodegenSupport extends SparkPlan with BaseAggregateExec with BlockingOperatorWithCodegen with GeneratePredicateHelper

   An interface for those aggregate physical operators that support codegen.

2.  class AggregationBufferEntry extends AnyRef
3.  abstract class AggregationIterator extends Iterator[UnsafeRow] with Logging

   The base class of SortBasedAggregationIterator, TungstenAggregationIterator and ObjectAggregationIterator.

   The base class of SortBasedAggregationIterator, TungstenAggregationIterator and ObjectAggregationIterator. It mainly contains two parts: 1. It initializes aggregate functions. 2. It creates two functions, processRow and generateOutput based on AggregateMode of its aggregate functions. processRow is the function to handle an input. generateOutput is used to generate result.

4.  trait BaseAggregateExec extends SparkPlan with UnaryExecNode with AliasAwareOutputPartitioning

   Holds common logic for aggregate operators

5.  sealed trait BufferSetterGetterUtils extends AnyRef

   A helper trait used to create specialized setter and getter for types supported by org.apache.spark.sql.execution.UnsafeFixedWidthAggregationMap's buffer.

   A helper trait used to create specialized setter and getter for types supported by org.apache.spark.sql.execution.UnsafeFixedWidthAggregationMap's buffer. (see UnsafeFixedWidthAggregationMap.supportsAggregationBufferSchema).

6.  case class ComplexTypedAggregateExpression(aggregator: expressions.Aggregator[Any, Any, Any], inputDeserializer: Option[Expression], inputClass: Option[Class[_]], inputSchema: Option[StructType], bufferSerializer: Seq[Expression], bufferDeserializer: Expression, outputSerializer: Expression, dataType: DataType, nullable: Boolean, mutableAggBufferOffset: Int = 0, inputAggBufferOffset: Int = 0) extends TypedImperativeAggregate[Any] with TypedAggregateExpression with NonSQLExpression with Product with Serializable
7.  case class HashAggregateExec(requiredChildDistributionExpressions: Option[Seq[Expression]], isStreaming: Boolean, numShufflePartitions: Option[Int], groupingExpressions: Seq[NamedExpression], aggregateExpressions: Seq[AggregateExpression], aggregateAttributes: Seq[Attribute], initialInputBufferOffset: Int, resultExpressions: Seq[NamedExpression], child: SparkPlan) extends SparkPlan with AggregateCodegenSupport with Product with Serializable

   Hash-based aggregate operator that can also fallback to sorting when data exceeds memory size.

8.  abstract class HashMapGenerator extends AnyRef

   This is a helper class to generate an append-only row-based hash map that can act as a 'cache' for extremely fast key-value lookups while evaluating aggregates (and fall back to the BytesToBytesMap if a given key isn't found).

   This is a helper class to generate an append-only row-based hash map that can act as a 'cache' for extremely fast key-value lookups while evaluating aggregates (and fall back to the BytesToBytesMap if a given key isn't found). This is 'codegened' in HashAggregate to speed up aggregates w/ key.

   NOTE: the generated hash map currently doesn't support nullable keys and falls back to the BytesToBytesMap to store them.

9.  case class MergingSessionsExec(requiredChildDistributionExpressions: Option[Seq[Expression]], isStreaming: Boolean, numShufflePartitions: Option[Int], groupingExpressions: Seq[NamedExpression], sessionExpression: NamedExpression, aggregateExpressions: Seq[AggregateExpression], aggregateAttributes: Seq[Attribute], initialInputBufferOffset: Int, resultExpressions: Seq[NamedExpression], child: SparkPlan) extends SparkPlan with BaseAggregateExec with Product with Serializable

   This node is a variant of SortAggregateExec which merges the session windows based on the fact child node will provide inputs as sorted by group keys + the start time of session window.

   This node is a variant of SortAggregateExec which merges the session windows based on the fact child node will provide inputs as sorted by group keys + the start time of session window.

   When merging windows, it also applies aggregations on merged window, which eliminates the necessity on buffering inputs (which requires copying rows) and update the session spec for each input.

   This class receives requiredChildDistribution from caller, to enable merging session in local partition before shuffling. Specifying both parameters to None won't trigger shuffle, but sort would still happen per local partition.

   Refer MergingSessionsIterator for more details.

10.  class MergingSessionsIterator extends AggregationIterator

    This is a variant of SortAggregateIterator which merges the session windows based on the fact input rows are sorted by "group keys + the start time of session window".

    This is a variant of SortAggregateIterator which merges the session windows based on the fact input rows are sorted by "group keys + the start time of session window".

    When merging windows, it also applies aggregations on merged window, which eliminates the necessity on buffering inputs (which requires copying rows) and update the session spec for each input.

11.  class ObjectAggregationIterator extends AggregationIterator with Logging
12.  class ObjectAggregationMap extends AnyRef

    An aggregation map that supports using safe SpecificInternalRows aggregation buffers, so that we can support storing arbitrary Java objects as aggregate function states in the aggregation buffers.

    An aggregation map that supports using safe SpecificInternalRows aggregation buffers, so that we can support storing arbitrary Java objects as aggregate function states in the aggregation buffers. This class is only used together with ObjectHashAggregateExec.

13.  case class ObjectHashAggregateExec(requiredChildDistributionExpressions: Option[Seq[Expression]], isStreaming: Boolean, numShufflePartitions: Option[Int], groupingExpressions: Seq[NamedExpression], aggregateExpressions: Seq[AggregateExpression], aggregateAttributes: Seq[Attribute], initialInputBufferOffset: Int, resultExpressions: Seq[NamedExpression], child: SparkPlan) extends SparkPlan with BaseAggregateExec with Product with Serializable

    A hash-based aggregate operator that supports TypedImperativeAggregate functions that may use arbitrary JVM objects as aggregation states.

    A hash-based aggregate operator that supports TypedImperativeAggregate functions that may use arbitrary JVM objects as aggregation states.

    Similar to HashAggregateExec, this operator also falls back to sort-based aggregation when the size of the internal hash map exceeds the threshold. The differences are:

    • It uses safe rows as aggregation buffer since it must support JVM objects as aggregation states.
    • It tracks entry count of the hash map instead of byte size to decide when we should fall back. This is because it's hard to estimate the accurate size of arbitrary JVM objects in a lightweight way.
    • Whenever fallen back to sort-based aggregation, this operator feeds all of the rest input rows into external sorters instead of building more hash map(s) as what HashAggregateExec does. This is because having too many JVM object aggregation states floating there can be dangerous for GC.
    • CodeGen is not supported yet.

    This operator may be turned off by setting the following SQL configuration to false:

    spark.sql.execution.useObjectHashAggregateExec

    The fallback threshold can be configured by tuning:

    spark.sql.objectHashAggregate.sortBased.fallbackThreshold

14.  class RowBasedHashMapGenerator extends HashMapGenerator

    This is a helper class to generate an append-only row-based hash map that can act as a 'cache' for extremely fast key-value lookups while evaluating aggregates (and fall back to the BytesToBytesMap if a given key isn't found).

    This is a helper class to generate an append-only row-based hash map that can act as a 'cache' for extremely fast key-value lookups while evaluating aggregates (and fall back to the BytesToBytesMap if a given key isn't found). This is 'codegened' in HashAggregate to speed up aggregates w/ key.

    We also have VectorizedHashMapGenerator, which generates a append-only vectorized hash map. We choose one of the two as the 1st level, fast hash map during aggregation.

    NOTE: This row-based hash map currently doesn't support nullable keys and falls back to the BytesToBytesMap to store them.

15.  case class ScalaAggregator[IN, BUF, OUT](children: Seq[Expression], agg: expressions.Aggregator[IN, BUF, OUT], inputEncoder: ExpressionEncoder[IN], bufferEncoder: ExpressionEncoder[BUF], nullable: Boolean = true, isDeterministic: Boolean = true, mutableAggBufferOffset: Int = 0, inputAggBufferOffset: Int = 0, aggregatorName: Option[String] = None) extends TypedImperativeAggregate[BUF] with NonSQLExpression with UserDefinedExpression with ImplicitCastInputTypes with Logging with Product with Serializable
16.  case class ScalaUDAF(children: Seq[Expression], udaf: UserDefinedAggregateFunction, mutableAggBufferOffset: Int = 0, inputAggBufferOffset: Int = 0, udafName: Option[String] = None) extends ImperativeAggregate with NonSQLExpression with Logging with ImplicitCastInputTypes with UserDefinedExpression with Product with Serializable

    The internal wrapper used to hook a UserDefinedAggregateFunction udaf in the internal aggregation code path.

17.  case class SimpleTypedAggregateExpression(aggregator: expressions.Aggregator[Any, Any, Any], inputDeserializer: Option[Expression], inputClass: Option[Class[_]], inputSchema: Option[StructType], bufferSerializer: Seq[Expression], aggBufferAttributes: Seq[AttributeReference], bufferDeserializer: Expression, outputSerializer: Seq[Expression], outputExternalType: DataType, dataType: DataType, nullable: Boolean) extends DeclarativeAggregate with TypedAggregateExpression with NonSQLExpression with Product with Serializable
18.  case class SortAggregateExec(requiredChildDistributionExpressions: Option[Seq[Expression]], isStreaming: Boolean, numShufflePartitions: Option[Int], groupingExpressions: Seq[NamedExpression], aggregateExpressions: Seq[AggregateExpression], aggregateAttributes: Seq[Attribute], initialInputBufferOffset: Int, resultExpressions: Seq[NamedExpression], child: SparkPlan) extends SparkPlan with AggregateCodegenSupport with AliasAwareOutputOrdering with Product with Serializable

    Sort-based aggregate operator.

19.  class SortBasedAggregationIterator extends AggregationIterator

    An iterator used to evaluate org.apache.spark.sql.catalyst.expressions.aggregate.AggregateFunction.

    An iterator used to evaluate org.apache.spark.sql.catalyst.expressions.aggregate.AggregateFunction. It assumes the input rows have been sorted by values of groupingExpressions.

20.  class SortBasedAggregator extends AnyRef

    A class used to handle sort-based aggregation, used together with ObjectHashAggregateExec.

    A class used to handle sort-based aggregation, used together with ObjectHashAggregateExec.

    To do

    Try to eliminate this class by refactor and reuse code paths in SortAggregateExec.

21.  class TungstenAggregationIterator extends AggregationIterator with Logging

    An iterator used to evaluate aggregate functions.

    An iterator used to evaluate aggregate functions. It operates on UnsafeRows.

    This iterator first uses hash-based aggregation to process input rows. It uses a hash map to store groups and their corresponding aggregation buffers. If this map cannot allocate memory from memory manager, it spills the map into disk and creates a new one. After processed all the input, then merge all the spills together using external sorter, and do sort-based aggregation.

    The process has the following step:

    • Step 0: Do hash-based aggregation.
    • Step 1: Sort all entries of the hash map based on values of grouping expressions and spill them to disk.
    • Step 2: Create an external sorter based on the spilled sorted map entries and reset the map.
    • Step 3: Get a sorted KVIterator from the external sorter.
    • Step 4: Repeat step 0 until no more input.
    • Step 5: Initialize sort-based aggregation on the sorted iterator. Then, this iterator works in the way of sort-based aggregation.

    The code of this class is organized as follows:

    • Part 1: Initializing aggregate functions.
    • Part 2: Methods and fields used by setting aggregation buffer values, processing input rows from inputIter, and generating output rows.
    • Part 3: Methods and fields used by hash-based aggregation.
    • Part 4: Methods and fields used when we switch to sort-based aggregation.
    • Part 5: Methods and fields used by sort-based aggregation.
    • Part 6: Loads input and process input rows.
    • Part 7: Public methods of this iterator.
    • Part 8: A utility function used to generate a result when there is no input and there is no grouping expression.
22.  trait TypedAggregateExpression extends AggregateFunction

    A helper class to hook Aggregator into the aggregation system.

23.  class TypedAverage[IN] extends expressions.Aggregator[IN, (Double, Long), Double]
24.  class TypedCount[IN] extends expressions.Aggregator[IN, Long, Long]
25.  class TypedSumDouble[IN] extends expressions.Aggregator[IN, Double, Double]
26.  class TypedSumLong[IN] extends expressions.Aggregator[IN, Long, Long]
27.  case class UpdatingSessionsExec(isStreaming: Boolean, numShufflePartitions: Option[Int], groupingExpression: Seq[Attribute], sessionExpression: Attribute, child: SparkPlan) extends SparkPlan with UnaryExecNode with Product with Serializable

    This node updates the session window spec of each input rows via analyzing neighbor rows and determining rows belong to the same session window.

    This node updates the session window spec of each input rows via analyzing neighbor rows and determining rows belong to the same session window. The number of input rows remains the same. This node requires sort on input rows by group keys + the start time of session window.

    There are lots of overhead compared to MergingSessionsExec. Use MergingSessionsExec instead whenever possible. Use this node only when we cannot apply both calculations determining session windows and aggregating rows in session window altogether.

    Refer UpdatingSessionsIterator for more details.

28.  class UpdatingSessionsIterator extends Iterator[InternalRow]

    This class calculates and updates the session window for each element in the given iterator, which makes elements in the same session window having same session spec.

    This class calculates and updates the session window for each element in the given iterator, which makes elements in the same session window having same session spec. Downstream can apply aggregation to finally merge these elements bound to the same session window.

    This class works on the precondition that given iterator is sorted by "group keys + start time of session window", and this iterator still retains the characteristic of the sort.

    This class copies the elements to safely update on each element, as well as buffers elements which are bound to the same session window. Due to such overheads, MergingSessionsIterator should be used whenever possible.

29.  class VectorizedHashMapGenerator extends HashMapGenerator

    This is a helper class to generate an append-only vectorized hash map that can act as a 'cache' for extremely fast key-value lookups while evaluating aggregates (and fall back to the BytesToBytesMap if a given key isn't found).

    This is a helper class to generate an append-only vectorized hash map that can act as a 'cache' for extremely fast key-value lookups while evaluating aggregates (and fall back to the BytesToBytesMap if a given key isn't found). This is 'codegened' in HashAggregate to speed up aggregates w/ key.

    It is backed by a power-of-2-sized array for index lookups and a columnar batch that stores the key-value pairs. The index lookups in the array rely on linear probing (with a small number of maximum tries) and use an inexpensive hash function which makes it really efficient for a majority of lookups. However, using linear probing and an inexpensive hash function also makes it less robust as compared to the BytesToBytesMap (especially for a large number of keys or even for certain distribution of keys) and requires us to fall back on the latter for correctness. We also use a secondary columnar batch that logically projects over the original columnar batch and is equivalent to the BytesToBytesMap aggregate buffer.

    NOTE: This vectorized hash map currently doesn't support nullable keys and falls back to the BytesToBytesMap to store them.