ColumnStat

Statistics collected for a column.

1. Supported data types are defined in ColumnStat.supportsType. 2. The JVM data type stored in min/max is the internal data type for the corresponding Catalyst data type. For example, the internal type of DateType is Int, and that the internal type of TimestampType is Long. 3. There is no guarantee that the statistics collected are accurate. Approximation algorithms (sketches) might have been used, and the data collected can also be stale.

distinctCount: number of distinct values
min: minimum value
max: maximum value
nullCount: number of nulls
avgLen: average length of the values. For fixed-length types, this should be a constant.
maxLen: maximum length of the values. For fixed-length types, this should be a constant.

Linear Supertypes

Serializable, Serializable, Product, Equals, AnyRef, Any

Instance Constructors

new ColumnStat(distinctCount: BigInt, min: Option[Any], max: Option[Any], nullCount: BigInt, avgLen: Long, maxLen: Long)

distinctCount
number of distinct values
min
minimum value
max
maximum value
nullCount
number of nulls
avgLen
average length of the values. For fixed-length types, this should be a constant.
maxLen
maximum length of the values. For fixed-length types, this should be a constant.

Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def asInstanceOf[T0]: T0

Definition Classes
Any
val avgLen: Long

average length of the values.
average length of the values. For fixed-length types, this should be a constant.
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
val distinctCount: BigInt

number of distinct values
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
val max: Option[Any]

maximum value
val maxLen: Long

maximum length of the values.
maximum length of the values. For fixed-length types, this should be a constant.
val min: Option[Any]

minimum value
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
val nullCount: BigInt

number of nulls
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
def toMap(colName: String, dataType: DataType): Map[String, String]

Returns a map from string to string that can be used to serialize the column stats.
Returns a map from string to string that can be used to serialize the column stats. The key is the name of the field (e.g. "distinctCount" or "min"), and the value is the string representation for the value. min/max values are converted to the external data type. For example, for DateType we store java.sql.Date, and for TimestampType we store java.sql.Timestamp. The deserialization side is defined in ColumnStat.fromMap.
As part of the protocol, the returned map always contains a key called "version". In the case min/max values are null (None), they won't appear in the map.
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )

Related Docs: object ColumnStat | package logical

case class ColumnStat(distinctCount: BigInt, min: Option[Any], max: Option[Any], nullCount: BigInt, avgLen: Long, maxLen: Long) extends Product with Serializable

Instance Constructors

new ColumnStat(distinctCount: BigInt, min: Option[Any], max: Option[Any], nullCount: BigInt, avgLen: Long, maxLen: Long)

Value Members

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: Any): Boolean

final def asInstanceOf[T0]: T0

val avgLen: Long

def clone(): AnyRef

val distinctCount: BigInt

final def eq(arg0: AnyRef): Boolean

def finalize(): Unit

final def getClass(): Class[_]

final def isInstanceOf[T0]: Boolean

val max: Option[Any]

val maxLen: Long

val min: Option[Any]

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

val nullCount: BigInt

final def synchronized[T0](arg0: ⇒ T0): T0

def toMap(colName: String, dataType: DataType): Map[String, String]

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from AnyRef

Inherited from Any

Ungrouped