Type Members

1.  class FrameLessOffsetWindowFunctionFrame extends OffsetWindowFunctionFrameBase

   The frameless offset window frame is an internal window frame just used to optimize the performance for the window function that returns the value of the input column offset by a number of rows according to the current row.

   The frameless offset window frame is an internal window frame just used to optimize the performance for the window function that returns the value of the input column offset by a number of rows according to the current row. The internal window frame is not a popular window frame cannot be specified and used directly by the users. This window frame calculates frames containing LEAD/LAG statements.

2.  abstract class OffsetWindowFunctionFrameBase extends WindowFunctionFrame

   The offset window frame calculates frames containing LEAD/LAG statements.

3.  final class SlidingWindowFunctionFrame extends WindowFunctionFrame

   The sliding window frame calculates frames with the following SQL form: ...

   The sliding window frame calculates frames with the following SQL form: ... BETWEEN 1 PRECEDING AND 1 FOLLOWING

4.  final class UnboundedFollowingWindowFunctionFrame extends WindowFunctionFrame

   The UnboundFollowing window frame calculates frames with the following SQL form: ...

   The UnboundFollowing window frame calculates frames with the following SQL form: ... BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING

   There is only an upper bound. This is a slightly modified version of the sliding window. The sliding window operator has to check if both upper and the lower bound change when a new row gets processed, where as the unbounded following only has to check the lower bound.

   This is a very expensive operator to use, O(n * (n - 1) /2), because we need to maintain a buffer and must do full recalculation after each row. Reverse iteration would be possible, if the commutativity of the used window functions can be guaranteed.

5.  class UnboundedOffsetWindowFunctionFrame extends OffsetWindowFunctionFrameBase

   The unbounded offset window frame is an internal window frame just used to optimize the performance for the window function that returns the value of the input column offset by a number of rows within the frame and has specified ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING.

   The unbounded offset window frame is an internal window frame just used to optimize the performance for the window function that returns the value of the input column offset by a number of rows within the frame and has specified ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING. The internal window frame is not a popular window frame cannot be specified and used directly by the users. The unbounded offset window frame calculates frames containing NTH_VALUE statements. The unbounded offset window frame return the same value for all rows in the window partition.

6.  class UnboundedPrecedingOffsetWindowFunctionFrame extends OffsetWindowFunctionFrameBase

   The unbounded preceding offset window frame is an internal window frame just used to optimize the performance for the window function that returns the value of the input column offset by a number of rows within the frame and has specified ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW.

   The unbounded preceding offset window frame is an internal window frame just used to optimize the performance for the window function that returns the value of the input column offset by a number of rows within the frame and has specified ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW. The internal window frame is not a popular window frame cannot be specified and used directly by the users. The unbounded preceding offset window frame calculates frames containing NTH_VALUE statements. The unbounded preceding offset window frame return the same value for rows which index (starting from 1) equal to or greater than offset in the window partition.

7.  final class UnboundedPrecedingWindowFunctionFrame extends WindowFunctionFrame

   The UnboundPreceding window frame calculates frames with the following SQL form: ...

   The UnboundPreceding window frame calculates frames with the following SQL form: ... BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW

   There is only an upper bound. Very common use cases are for instance running sums or counts (row_number). Technically this is a special case of a sliding window. However a sliding window has to maintain a buffer, and it must do a full evaluation everytime the buffer changes. This is not the case when there is no lower bound, given the additive nature of most aggregates streaming updates and partial evaluation suffice and no buffering is needed.

8.  final class UnboundedWindowFunctionFrame extends WindowFunctionFrame

   The unbounded window frame calculates frames with the following SQL forms: ...

   The unbounded window frame calculates frames with the following SQL forms: ... (No Frame Definition) ... BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING

   Its results are the same for each and every row in the partition. This class can be seen as a special case of a sliding window, but is optimized for the unbound case.

9.  case class WindowExec(windowExpression: Seq[NamedExpression], partitionSpec: Seq[Expression], orderSpec: Seq[SortOrder], child: SparkPlan) extends SparkPlan with WindowExecBase with Product with Serializable

   This class calculates and outputs (windowed) aggregates over the rows in a single (sorted) partition.

   This class calculates and outputs (windowed) aggregates over the rows in a single (sorted) partition. The aggregates are calculated for each row in the group. Special processing instructions, frames, are used to calculate these aggregates. Frames are processed in the order specified in the window specification (the ORDER BY ... clause). There are four different frame types: - Entire partition: The frame is the entire partition, i.e. UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING. For this case, window function will take all rows as inputs and be evaluated once. - Growing frame: We only add new rows into the frame, Examples are:

   1. UNBOUNDED PRECEDING AND 1 PRECEDING 2. UNBOUNDED PRECEDING AND CURRENT ROW 3. UNBOUNDED PRECEDING AND 1 FOLLOWING Every time we move to a new row to process, we add some rows to the frame. We do not remove rows from this frame. - Shrinking frame: We only remove rows from the frame, Examples are:
   2. 1 PRECEDING AND UNBOUNDED FOLLOWING 2. CURRENT ROW AND UNBOUNDED FOLLOWING 3. 1 FOLLOWING AND UNBOUNDED FOLLOWING Every time we move to a new row to process, we remove some rows from the frame. We do not add rows to this frame. - Moving frame: Every time we move to a new row to process, we remove some rows from the frame and we add some rows to the frame. Examples are:
   3. 2 PRECEDING AND 1 PRECEDING 2. 1 PRECEDING AND CURRENT ROW 3. CURRENT ROW AND 1 FOLLOWING 4. 1 PRECEDING AND 1 FOLLOWING 5. 1 FOLLOWING AND 2 FOLLOWING - Offset frame: The frame consist of one row, which is an offset number of rows away from the current row. Only OffsetWindowFunctions can be processed in an offset frame. There are three implements of offset frame: FrameLessOffsetWindowFunctionFrame, UnboundedOffsetWindowFunctionFrame and UnboundedPrecedingOffsetWindowFunctionFrame.

   Different frame boundaries can be used in Growing, Shrinking and Moving frames. A frame boundary can be either Row or Range based: - Row Based: A row based boundary is based on the position of the row within the partition. An offset indicates the number of rows above or below the current row, the frame for the current row starts or ends. For instance, given a row based sliding frame with a lower bound offset of -1 and a upper bound offset of +2. The frame for row with index 5 would range from index 4 to index 7. - Range based: A range based boundary is based on the actual value of the ORDER BY expression(s). An offset is used to alter the value of the ORDER BY expression, for instance if the current order by expression has a value of 10 and the lower bound offset is -3, the resulting lower bound for the current row will be 10 - 3 = 7. This however puts a number of constraints on the ORDER BY expressions: there can be only one expression and this expression must have a numerical data type. An exception can be made when the offset is 0, because no value modification is needed, in this case multiple and non-numeric ORDER BY expression are allowed.

   This is quite an expensive operator because every row for a single group must be in the same partition and partitions must be sorted according to the grouping and sort order. The operator requires the planner to take care of the partitioning and sorting.

   The operator is semi-blocking. The window functions and aggregates are calculated one group at a time, the result will only be made available after the processing for the entire group has finished. The operator is able to process different frame configurations at the same time. This is done by delegating the actual frame processing (i.e. calculation of the window functions) to specialized classes, see WindowFunctionFrame, which take care of their own frame type: Entire Partition, Sliding, Growing & Shrinking. Boundary evaluation is also delegated to a pair of specialized classes: RowBoundOrdering & RangeBoundOrdering.

10.  trait WindowExecBase extends SparkPlan with UnaryExecNode

    Holds common logic for window operators

11.  abstract class WindowFunctionFrame extends AnyRef

    A window function calculates the results of a number of window functions for a window frame.

    A window function calculates the results of a number of window functions for a window frame. Before use a frame must be prepared by passing it all the rows in the current partition. After preparation the update method can be called to fill the output rows.

    Note: WindowFunctionFrame instances are reused during window execution. The prepare method will be called before processing the next partition, and must reset the states.