org.apache.flink.api.java.summarize

Class NumericColumnSummary<T>

java.lang.Object

org.apache.flink.api.java.summarize.ColumnSummary

org.apache.flink.api.java.summarize.NumericColumnSummary<T>

Type Parameters:

T - the numeric type e.g. Integer, Double

All Implemented Interfaces:

Serializable

@PublicEvolving
public class NumericColumnSummary<T>
extends ColumnSummary
implements Serializable

Generic Column Summary for Numeric Types. Some values are considered "missing" where "missing" is defined as null, NaN, or Infinity. These values are ignored in some calculations like mean, variance, and standardDeviation. Uses the Kahan summation algorithm to avoid numeric instability when computing variance. The algorithm is described in: "Scalable and Numerically Stable Descriptive Statistics in SystemML", Tian et al, International Conference on Data Engineering 2012.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
NumericColumnSummary(long nonMissingCount, long nullCount, long nanCount, long infinityCount, T min, T max, T sum, Double mean, Double variance, Double standardDeviation)

Method Summary

Modifier and Type	Method and Description
long	getInfinityCount() Number of values that are positive or negative infinity.
T	getMax()
Double	getMean() Null, NaN, and Infinite values are ignored in this calculation.
T	getMin()
long	getMissingCount() The number of "missing" values where "missing" is defined as null, NaN, or Infinity.
long	getNanCount() Number of values that are NaN.
long	getNonMissingCount() The number of values that are not null, NaN, or Infinity.
long	getNonNullCount() The number of non-null values in this column
long	getNullCount() The number of null values in this column
Double	getStandardDeviation() Standard Deviation is a measure of variation in a set of numbers.
T	getSum()
Double	getVariance() Variance is a measure of how far a set of numbers are spread out.
String	toString()

Methods inherited from class org.apache.flink.api.java.summarize.ColumnSummary

containsNonNull, containsNull, getTotalCount

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

NumericColumnSummary

public NumericColumnSummary(long nonMissingCount,
                            long nullCount,
                            long nanCount,
                            long infinityCount,
                            T min,
                            T max,
                            T sum,
                            Double mean,
                            Double variance,
                            Double standardDeviation)

Method Detail

getMissingCount

public long getMissingCount()

The number of "missing" values where "missing" is defined as null, NaN, or Infinity. These values are ignored in some calculations like mean, variance, and standardDeviation.

getNonMissingCount

public long getNonMissingCount()

The number of values that are not null, NaN, or Infinity.

getNonNullCount

public long getNonNullCount()

The number of non-null values in this column

Specified by:

getNonNullCount in class ColumnSummary

getNullCount

public long getNullCount()

Description copied from class: ColumnSummary

The number of null values in this column

Specified by:

getNullCount in class ColumnSummary

getNanCount

public long getNanCount()

Number of values that are NaN. (always zero for types like Short, Integer, Long)

getInfinityCount

public long getInfinityCount()

Number of values that are positive or negative infinity. (always zero for types like Short, Integer, Long)

getMin

public T getMin()

getMax

public T getMax()

getSum

public T getSum()

getMean

public Double getMean()

Null, NaN, and Infinite values are ignored in this calculation.

See Also:

Arithmetic Mean

getVariance

public Double getVariance()

Variance is a measure of how far a set of numbers are spread out. Null, NaN, and Infinite values are ignored in this calculation.

See Also:

Variance

getStandardDeviation

public Double getStandardDeviation()

Standard Deviation is a measure of variation in a set of numbers. It is the square root of the variance. Null, NaN, and Infinite values are ignored in this calculation.

See Also:

Standard Deviation

toString

public String toString()

Overrides:

toString in class Object