o

ai.chronon.aggregator.row

StatsGenerator

object StatsGenerator

Module managing FeatureStats Schema, Aggregations to be used by type and aggregator construction.

Stats Aggregation has an offline/ batch component and an online component. The metrics defined for stats depend on the schema of the join. The dataTypes and column names. For the online side, we obtain this information from the JoinCodec/valueSchema For the offline side, we obtain this information directly from the outputTable. To keep the schemas consistent we sort the metrics in the schema by name. (one column can have multiple metrics).

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. StatsGenerator
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Type Members

  1. case class MetricTransform(name: String, expression: InputTransform, operation: Operation, suffix: String = "", argMap: Map[String, String] = null) extends Product with Serializable

    MetricTransform represents a single statistic built on top of an input column.

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. def PSIKllSketch(reference: AnyRef, comparison: AnyRef, bins: Int = 128, eps: Double = 0.000001): AnyRef

    PSI is a measure of the difference between two probability distributions.

    PSI is a measure of the difference between two probability distributions. However, it's not defined for cases where a bin can have zero elements in either distribution (meant for continuous measures). In order to support PSI for discrete measures we add a small eps value to perturb the distribution in bins.

    Existing rules of thumb are: PSI < 0.10 means "little shift", .10<PSI<.25 means "moderate shift", and PSI>0.25 means "significant shift, action required" https://scholarworks.wmich.edu/dissertations/3208

  5. def SeriesFinalizer(key: String, value: AnyRef): AnyRef

    Post processing for finalized values or IRs when generating a time series of stats.

    Post processing for finalized values or IRs when generating a time series of stats. In the case of percentiles for examples we reduce to 5 values in order to generate candlesticks.

  6. def anyTransforms(column: String): Seq[MetricTransform]

    Stats applied to any column

  7. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  8. def buildAggPart(m: MetricTransform): AggregationPart
  9. def buildAggregator(metrics: Seq[MetricTransform], selectedSchema: StructType): RowAggregator

    Build RowAggregator to use for computing stats on a dataframe based on metrics

  10. def buildMetrics(fields: Seq[(String, DataType)]): Seq[MetricTransform]

    For the schema of the data define metrics to be aggregated

  11. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @native()
  12. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  13. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  14. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable])
  15. val finalizedPercentilesMerged: Array[Double]
  16. val finalizedPercentilesSeries: Array[Double]
  17. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  18. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  19. val ignoreColumns: Seq[String]
  20. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  21. def lInfKllSketch(sketch1: AnyRef, sketch2: AnyRef, bins: Int = 128): AnyRef
  22. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  23. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  24. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  25. val nullRateSuffix: String
  26. val nullSuffix: String
  27. def numericTransforms(column: String): Seq[MetricTransform]

    Stats applied to numeric columns

  28. def regularize(doubles: Array[Double], eps: Double): Array[Double]

    Given a PMF add and substract small values to keep a valid probability distribution without zeros

  29. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  30. def toString(): String
    Definition Classes
    AnyRef → Any
  31. val totalColumn: String
  32. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  33. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  34. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  35. object InputTransform extends Enumeration

    InputTransform acts as a signal of how to process the metric.

    InputTransform acts as a signal of how to process the metric.

    IsNull: Check if the input is null.

    Raw: Operate in the input column.

    One: lit(true) in spark. Used for row counts leveraged to obtain null rate values.

Inherited from AnyRef

Inherited from Any

Ungrouped