udaf

Type Members

case class TDigestArrayUDAF[N](deltaV: Double, maxDiscreteV: Int)(implicit num: Numeric[N], dataTpe: TDigestUDAFDataType[N]) extends TDigestMultiUDAF with Product with Serializable

A UDAF for sketching a column of numeric ArrayData with an array of TDigest objects.
A UDAF for sketching a column of numeric ArrayData with an array of TDigest objects. Expected to be created using tdigestArrayUDAF.
N
the expected numeric type of the data; Double, Int, etc
deltaV
The delta value to be used by the TDigest objects
maxDiscreteV
The maxDiscrete value to be used by the TDigest objects
class TDigestDoubleUDAF extends UserDefinedAggregateFunction
case class TDigestMLLibVecUDAF(deltaV: Double, maxDiscreteV: Int) extends TDigestMultiUDAF with Product with Serializable

A UDAF for sketching a column of MLLib Vectors with an array of TDigest objects.
A UDAF for sketching a column of MLLib Vectors with an array of TDigest objects. Expected to be created using tdigestMLLibVecUDAF.
deltaV
The delta value to be used by the TDigest object
maxDiscreteV
The maxDiscrete value to be used by the TDigest object
case class TDigestMLVecUDAF(deltaV: Double, maxDiscreteV: Int) extends TDigestMultiUDAF with Product with Serializable

A UDAF for sketching a column of ML Vectors with an array of TDigest objects.
A UDAF for sketching a column of ML Vectors with an array of TDigest objects. Expected to be created using tdigestMLVecUDAF.
deltaV
The delta value to be used by the TDigest object
maxDiscreteV
The maxDiscrete value to be used by the TDigest object
abstract class TDigestMultiUDAF extends UserDefinedAggregateFunction

A base class that defines the common functionality for array sketching UDAFs
class TDigestUDAF[N] extends UserDefinedAggregateFunction

A UDAF for sketching numeric data with a TDigest.
A UDAF for sketching numeric data with a TDigest. Expected to be created using tdigestUDAF.
N
the expected numeric type of the data; Double, Int, etc
case class TDigestUDAFDataType[N](tpe: DataType) extends Product with Serializable

For declaring implicit values that map numeric types to corresponding DataType values

Value Members

object Static2TDigestDoubleUDAF extends UserDefinedAggregateFunction
object StaticTDigestDoubleUDAF extends TDigestDoubleUDAF
implicit def implicitTDigestArraySQLToTDigestArray(tdasql: TDigestArraySQL): Array[TDigest]

implicitly unpack a TDigestArraySQL to extract its Array[TDigest] payload
implicit def implicitTDigestSQLToTDigest(tdsql: TDigestSQL): TDigest

implicitly unpack a TDigestSQL to extract its TDigest payload
object python
implicit val tDigestUDAFDataTypeByte: TDigestUDAFDataType[Byte]
implicit val tDigestUDAFDataTypeDouble: TDigestUDAFDataType[Double]
implicit val tDigestUDAFDataTypeFloat: TDigestUDAFDataType[Float]
implicit val tDigestUDAFDataTypeInt: TDigestUDAFDataType[Int]
implicit val tDigestUDAFDataTypeLong: TDigestUDAFDataType[Long]
implicit val tDigestUDAFDataTypeShort: TDigestUDAFDataType[Short]

def tdigestArrayUDAF[N](implicit num: Numeric[N], dataType: TDigestUDAFDataType[N]): TDigestArrayUDAF[N]

Obtain a UDAF for sketching a numeric array-data Dataset column, using a t-digest for each element.

N: The numeric type of the array-data column; Double, Int, etc
returns: A UDAF that can be applied to a Dataset array-data column

Example:

import org.isarnproject.sketches.udaf._, org.apache.spark.isarnproject.sketches.udt._
// create a UDAF for t-digest array, adding custom settings for delta and maxDiscrete
val udafTD = tdigestArrayUDAF[Double].delta(0.1).maxDiscrete(25)
// apply the UDAF to get an array of t-digests for each element in the array-data
val agg = data.agg(udafTD($"NumericArrayColumn"))
// extract the t-digest array
val tdArray = agg.getAs[TDigestArraySQL](0).tdigests

def tdigestMLLibVecUDAF: TDigestMLLibVecUDAF

Obtain a UDAF for sketching an MLLib Vector Dataset column, using a t-digest for each element in the vector

returns: A UDAF that can be applied to a MLLib Vector column

Example:

import org.isarnproject.sketches.udaf._, org.apache.spark.isarnproject.sketches.udt._
// create a UDAF for t-digest array, adding custom settings for delta and maxDiscrete
val udafTD = tdigestMLLibVecUDAF[Double].delta(0.1).maxDiscrete(25)
// apply the UDAF to get an array of t-digests for each element in the array-data
val agg = data.agg(udafTD($"MLLibVecColumn"))
// extract the t-digest array
val tdArray = agg.getAs[TDigestArraySQL](0).tdigests

def tdigestMLVecUDAF: TDigestMLVecUDAF

Obtain a UDAF for sketching an ML Vector Dataset column, using a t-digest for each element in the vector

returns: A UDAF that can be applied to a ML Vector column

Example:

import org.isarnproject.sketches.udaf._, org.apache.spark.isarnproject.sketches.udt._
// create a UDAF for t-digest array, adding custom settings for delta and maxDiscrete
val udafTD = tdigestMLVecUDAF[Double].delta(0.1).maxDiscrete(25)
// apply the UDAF to get an array of t-digests for each element in the array-data
val agg = data.agg(udafTD($"MLVecColumn"))
// extract the t-digest array
val tdArray = agg.getAs[TDigestArraySQL](0).tdigests

def tdigestUDAF[N](implicit num: Numeric[N], dataType: TDigestUDAFDataType[N]): TDigestUDAF[N]

Obtain a UDAF for sketching a single numeric Dataset column using a t-digest

N: The numeric type of the column; Double, Int, etc
returns: A UDAF that can be applied to a Dataset column

Example:

import org.isarnproject.sketches.udaf._, org.apache.spark.isarnproject.sketches.udt._
// create a UDAF for a t-digest, adding custom settings for delta and maxDiscrete
val udafTD = tdigestUDAF[Double].delta(0.1).maxDiscrete(25)
// apply the UDAF to get a t-digest for a data column
val agg = data.agg(udafTD($"NumericColumn"))
// extract the t-digest
val td = agg.getAs[TDigestSQL](0).tdigest

package udaf

Type Members

case class TDigestArrayUDAF[N](deltaV: Double, maxDiscreteV: Int)(implicit num: Numeric[N], dataTpe: TDigestUDAFDataType[N]) extends TDigestMultiUDAF with Product with Serializable

class TDigestDoubleUDAF extends UserDefinedAggregateFunction

case class TDigestMLLibVecUDAF(deltaV: Double, maxDiscreteV: Int) extends TDigestMultiUDAF with Product with Serializable

case class TDigestMLVecUDAF(deltaV: Double, maxDiscreteV: Int) extends TDigestMultiUDAF with Product with Serializable

abstract class TDigestMultiUDAF extends UserDefinedAggregateFunction

class TDigestUDAF[N] extends UserDefinedAggregateFunction

case class TDigestUDAFDataType[N](tpe: DataType) extends Product with Serializable

Value Members

object Static2TDigestDoubleUDAF extends UserDefinedAggregateFunction

object StaticTDigestDoubleUDAF extends TDigestDoubleUDAF

implicit def implicitTDigestArraySQLToTDigestArray(tdasql: TDigestArraySQL): Array[TDigest]

implicit def implicitTDigestSQLToTDigest(tdsql: TDigestSQL): TDigest

object python

implicit val tDigestUDAFDataTypeByte: TDigestUDAFDataType[Byte]

implicit val tDigestUDAFDataTypeDouble: TDigestUDAFDataType[Double]

implicit val tDigestUDAFDataTypeFloat: TDigestUDAFDataType[Float]

implicit val tDigestUDAFDataTypeInt: TDigestUDAFDataType[Int]

implicit val tDigestUDAFDataTypeLong: TDigestUDAFDataType[Long]

implicit val tDigestUDAFDataTypeShort: TDigestUDAFDataType[Short]

def tdigestArrayUDAF[N](implicit num: Numeric[N], dataType: TDigestUDAFDataType[N]): TDigestArrayUDAF[N]

def tdigestMLLibVecUDAF: TDigestMLLibVecUDAF

def tdigestMLVecUDAF: TDigestMLVecUDAF

def tdigestUDAF[N](implicit num: Numeric[N], dataType: TDigestUDAFDataType[N]): TDigestUDAF[N]

Inherited from AnyRef

Inherited from Any

Ungrouped