OpVectorColumnMetadata

Represents the metadata a column in a vector.

Because we expect every vector column to have been produced by some vectorization process, we provide the name of the feature that led to this column.

Also note that each column's indicator value should be unique, meaning that they represent mutually exclusive values. The output of a hashing vectorizer, for instance, does not produce mutually exclusive values.

parentFeatureName: The name of the parent feature(s) for the column. Usually a column has one parent feature, but can have many (eg. in the case of multiple Text columns being vectorized using a shared hash space)
parentFeatureType: The type of the parent feature(s) for the column
grouping: The name of the group an column belongs to (usually the parent feature, but in the case of Maps, this is the keys). Every other column in the same vector that has this grouping should be mutually exclusive to this one. If there is no grouping then this field is None
indicatorValue: A name for an binary indicator value (null indicator or result of a pivot or whatever that value is), otherwise None eg this is none when the column is from a numeric group that is not pivoted
descriptorValue: A name for a value that is continuous (not a binary indicator) eg for geolocation (lat, lon, accuracy) or for dates that have been converted to a circular representation the time window and x or y coordinate, otherwise None
index: Index of the vector this info is associated with (this is updated when OpVectorColumnMetadata is passed into OpVectorMetadata

Linear Supertypes

Product, Equals, JsonLike, Serializable, Serializable, AnyRef, Any

Instance Constructors

new OpVectorColumnMetadata(parentFeatureName: Seq[String], parentFeatureType: Seq[String], grouping: Option[String], indicatorValue: Option[String] = None, descriptorValue: Option[String] = None, index: Int = 0)

parentFeatureName
The name of the parent feature(s) for the column. Usually a column has one parent feature, but can have many (eg. in the case of multiple Text columns being vectorized using a shared hash space)
parentFeatureType
The type of the parent feature(s) for the column
grouping
The name of the group an column belongs to (usually the parent feature, but in the case of Maps, this is the keys). Every other column in the same vector that has this grouping should be mutually exclusive to this one. If there is no grouping then this field is None
indicatorValue
A name for an binary indicator value (null indicator or result of a pivot or whatever that value is), otherwise None eg this is none when the column is from a numeric group that is not pivoted
descriptorValue
A name for a value that is continuous (not a binary indicator) eg for geolocation (lat, lon, accuracy) or for dates that have been converted to a circular representation the time window and x or y coordinate, otherwise None
index
Index of the vector this info is associated with (this is updated when OpVectorColumnMetadata is passed into OpVectorMetadata

Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def asInstanceOf[T0]: T0

Definition Classes
Any
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
val descriptorValue: Option[String]

A name for a value that is continuous (not a binary indicator) eg for geolocation (lat, lon, accuracy) or for dates that have been converted to a circular representation the time window and x or y coordinate, otherwise None
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def featureGroup(): Option[String]

Get the feature grouping qualified by the parent feature name
Get the feature grouping qualified by the parent feature name
returns
Optional string of feature grouping
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
val grouping: Option[String]

The name of the group an column belongs to (usually the parent feature, but in the case of Maps, this is the keys).
The name of the group an column belongs to (usually the parent feature, but in the case of Maps, this is the keys). Every other column in the same vector that has this grouping should be mutually exclusive to this one. If there is no grouping then this field is None
def hasParentOfSubType[O <: FeatureType](implicit tt: scala.reflect.api.JavaUniverse.TypeTag[O]): Boolean

Does column have parent features of which are subtypes of feature type O
def hasParentOfType[O <: FeatureType](implicit tt: scala.reflect.api.JavaUniverse.TypeTag[O]): Boolean

Does column have parent features of specified feature type O
val index: Int

Index of the vector this info is associated with (this is updated when OpVectorColumnMetadata is passed into OpVectorMetadata
val indicatorValue: Option[String]

A name for an binary indicator value (null indicator or result of a pivot or whatever that value is), otherwise None eg this is none when the column is from a numeric group that is not pivoted
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
def isNullIndicator: Boolean

Is this column corresponds to a null-encoded value
Is this column corresponds to a null-encoded value
returns
true if this column corresponds to a null-encoded value
def isOtherIndicator: Boolean

Is this column corresponds the other category of a one hot encoded categorical
Is this column corresponds the other category of a one hot encoded categorical
returns
true if this column corresponds to the other category of a one hot encoded categorical
def makeColName(): String

Make unique name for this column
Make unique name for this column
returns
String name for this column
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
val parentFeatureName: Seq[String]

The name of the parent feature(s) for the column.
The name of the parent feature(s) for the column. Usually a column has one parent feature, but can have many (eg. in the case of multiple Text columns being vectorized using a shared hash space)
val parentFeatureType: Seq[String]

The type of the parent feature(s) for the column
def parentNamesWithMapKeys(): Seq[String]

Return parent features names with the key (grouping) from any map parents included in name
Return parent features names with the key (grouping) from any map parents included in name
returns
Sequence of parent feature names, simple names when features are not maps, names plus keys for columns with map parent features
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
def toJson(pretty: Boolean): String

Definition Classes
JsonLike
def toMetadata(): Metadata

Convert this column into Spark metadata.
Convert this column into Spark metadata.
returns
column Spark metadata
def toMetadata(ind: Array[Int]): Metadata

Convert this column into Spark metadata.
Convert this column into Spark metadata.
ind
Indexes of vector columns that match this OpVectorColumnMetadata
returns
column Spark metadata
def toString(): String

Definition Classes
JsonLike → AnyRef → Any
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )

Related Docs: object OpVectorColumnMetadata | package spark

case class OpVectorColumnMetadata(parentFeatureName: Seq[String], parentFeatureType: Seq[String], grouping: Option[String], indicatorValue: Option[String] = None, descriptorValue: Option[String] = None, index: Int = 0) extends JsonLike with Product with Serializable

Instance Constructors

new OpVectorColumnMetadata(parentFeatureName: Seq[String], parentFeatureType: Seq[String], grouping: Option[String], indicatorValue: Option[String] = None, descriptorValue: Option[String] = None, index: Int = 0)

Value Members

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: Any): Boolean

final def asInstanceOf[T0]: T0

def clone(): AnyRef

val descriptorValue: Option[String]

final def eq(arg0: AnyRef): Boolean

def featureGroup(): Option[String]

def finalize(): Unit

final def getClass(): Class[_]

val grouping: Option[String]

def hasParentOfSubType[O <: FeatureType](implicit tt: scala.reflect.api.JavaUniverse.TypeTag[O]): Boolean

def hasParentOfType[O <: FeatureType](implicit tt: scala.reflect.api.JavaUniverse.TypeTag[O]): Boolean

val index: Int

val indicatorValue: Option[String]

final def isInstanceOf[T0]: Boolean

def isNullIndicator: Boolean

def isOtherIndicator: Boolean

def makeColName(): String

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

val parentFeatureName: Seq[String]

val parentFeatureType: Seq[String]

def parentNamesWithMapKeys(): Seq[String]

final def synchronized[T0](arg0: ⇒ T0): T0

def toJson(pretty: Boolean): String

def toMetadata(): Metadata

def toMetadata(ind: Array[Int]): Metadata

def toString(): String

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

Inherited from Product

Inherited from Equals

Inherited from JsonLike

Inherited from Serializable

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped