aggregate

Type Members

1. abstract class AggregationIterator extends Iterator[InternalRow] with Logging

   

   The base class of SortBasedAggregationIterator and UnsafeHybridAggregationIterator.

   The base class of SortBasedAggregationIterator and UnsafeHybridAggregationIterator. It mainly contains two parts: 1. It initializes aggregate functions. 2. It creates two functions, processRow and generateOutput based on AggregateMode of its aggregate functions. processRow is the function to handle an input. generateOutput is used to generate result.

2. sealed trait BufferSetterGetterUtils extends AnyRef

   

   A helper trait used to create specialized setter and getter for types supported by org.apache.spark.sql.execution.UnsafeFixedWidthAggregationMap's buffer.

   A helper trait used to create specialized setter and getter for types supported by org.apache.spark.sql.execution.UnsafeFixedWidthAggregationMap's buffer. (see UnsafeFixedWidthAggregationMap.supportsAggregationBufferSchema).

3. case class SortBasedAggregate(requiredChildDistributionExpressions: Option[Seq[Expression]], groupingExpressions: Seq[NamedExpression], nonCompleteAggregateExpressions: Seq[AggregateExpression2], nonCompleteAggregateAttributes: Seq[Attribute], completeAggregateExpressions: Seq[AggregateExpression2], completeAggregateAttributes: Seq[Attribute], initialInputBufferOffset: Int, resultExpressions: Seq[NamedExpression], child: SparkPlan) extends SparkPlan with UnaryNode with Product with Serializable

   

4. class SortBasedAggregationIterator extends AggregationIterator

   

   An iterator used to evaluate AggregateFunction2.

   An iterator used to evaluate AggregateFunction2. It assumes the input rows have been sorted by values of groupingKeyAttributes.

5. case class TungstenAggregate(requiredChildDistributionExpressions: Option[Seq[Expression]], groupingExpressions: Seq[NamedExpression], nonCompleteAggregateExpressions: Seq[AggregateExpression2], completeAggregateExpressions: Seq[AggregateExpression2], initialInputBufferOffset: Int, resultExpressions: Seq[NamedExpression], child: SparkPlan) extends SparkPlan with UnaryNode with Product with Serializable

   

6. class TungstenAggregationIterator extends Iterator[UnsafeRow] with Logging

   

   An iterator used to evaluate aggregate functions.

   An iterator used to evaluate aggregate functions. It operates on UnsafeRows.

   This iterator first uses hash-based aggregation to process input rows. It uses a hash map to store groups and their corresponding aggregation buffers. If we this map cannot allocate memory from org.apache.spark.shuffle.ShuffleMemoryManager, it switches to sort-based aggregation. The process of the switch has the following step:

   • Step 1: Sort all entries of the hash map based on values of grouping expressions and spill them to disk.
   • Step 2: Create a external sorter based on the spilled sorted map entries.
   • Step 3: Redirect all input rows to the external sorter.
   • Step 4: Get a sorted KVIterator from the external sorter.
   • Step 5: Initialize sort-based aggregation. Then, this iterator works in the way of sort-based aggregation.

   The code of this class is organized as follows:

   • Part 1: Initializing aggregate functions.
   • Part 2: Methods and fields used by setting aggregation buffer values, processing input rows from inputIter, and generating output rows.
   • Part 3: Methods and fields used by hash-based aggregation.
   • Part 4: Methods and fields used when we switch to sort-based aggregation.
   • Part 5: Methods and fields used by sort-based aggregation.
   • Part 6: Loads input and process input rows.
   • Part 7: Public methods of this iterator.
   • Part 8: A utility function used to generate a result when there is no input and there is no grouping expression.