Aggregator

A base class for user-defined aggregations, which can be used in DataFrame and Dataset operations to take all of the elements of a group and reduce them to a single value.

For example, the following aggregator extracts an int from a specific class and adds them up:

case class Data(i: Int)

val customSummer =  new Aggregator[Data, Int, Int] {
  def zero: Int = 0
  def reduce(b: Int, a: Data): Int = b + a.i
  def merge(b1: Int, b2: Int): Int = b1 + b2
  def finish(r: Int): Int = r
}.toColumn()

val ds: Dataset[Data] = ...
val aggregated = ds.select(customSummer)

Based loosely on Aggregator from Algebird: https://github.com/twitter/algebird

I: The input type for the aggregation.
B: The type of the intermediate value of the reduction.
O: The type of the final output result.

Since: 1.6.0

Linear Supertypes

Serializable, Serializable, AnyRef, Any

Instance Constructors

new Aggregator()

Abstract Value Members

abstract def finish(reduction: B): O

Transform the output of the reduction.
Transform the output of the reduction.

Since
1.6.0
abstract def merge(b1: B, b2: B): B

Merge two intermediate values.
Merge two intermediate values.

Since
1.6.0
abstract def reduce(b: B, a: I): B

Combine two values to produce a new value.
Combine two values to produce a new value. For performance, the function may modify b and return it instead of constructing new object for b.

Since
1.6.0
abstract def zero: B

A zero value for this aggregation.
A zero value for this aggregation. Should satisfy the property that any b + zero = b.

Since
1.6.0

Concrete Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def asInstanceOf[T0]: T0

Definition Classes
Any
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def equals(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
def hashCode(): Int

Definition Classes
AnyRef → Any
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
def toColumn(implicit bEncoder: Encoder[B], cEncoder: Encoder[O]): TypedColumn[I, O]

Returns this Aggregator as a TypedColumn that can be used in Dataset or DataFrame operations.
Returns this Aggregator as a TypedColumn that can be used in Dataset or DataFrame operations.

Since
1.6.0
def toString(): String

Definition Classes
AnyRef → Any
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )

Related Doc: package expressions

abstract class Aggregator[-I, B, O] extends Serializable

Instance Constructors

new Aggregator()

Abstract Value Members

abstract def finish(reduction: B): O

abstract def merge(b1: B, b2: B): B

abstract def reduce(b: B, a: I): B

abstract def zero: B

Concrete Value Members

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: Any): Boolean

final def asInstanceOf[T0]: T0

def clone(): AnyRef

final def eq(arg0: AnyRef): Boolean

def equals(arg0: Any): Boolean

def finalize(): Unit

final def getClass(): Class[_]

def hashCode(): Int

final def isInstanceOf[T0]: Boolean

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

final def synchronized[T0](arg0: ⇒ T0): T0

def toColumn(implicit bEncoder: Encoder[B], cEncoder: Encoder[O]): TypedColumn[I, O]

def toString(): String

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

Inherited from Serializable

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped