types

Type Members

1. case class ArrayType(elementType: DataType, containsNull: Boolean) extends DataType with Product with Serializable

   

   :: DeveloperApi :: The data type for collections of multiple values.

   :: DeveloperApi :: The data type for collections of multiple values. Internally these are represented as columns that contain a scala.collection.Seq.

   Please use DataTypes.createArrayType() to create a specific instance.

   An ArrayType object comprises two fields, elementType: DataType and containsNull: Boolean. The field of elementType is used to specify the type of array elements. The field of containsNull is used to specify if the array has null values.

   elementType

   The data type of values.

   containsNull

   Indicates if values have null values

   Annotations

   @DeveloperApi()

2. class BinaryType extends NativeType with PrimitiveType

   

   :: DeveloperApi :: The data type representing Array[Byte] values.

   :: DeveloperApi :: The data type representing Array[Byte] values. Please use the singleton DataTypes.BinaryType.

   Annotations

   @DeveloperApi()

3. class BooleanType extends NativeType with PrimitiveType

   

   :: DeveloperApi :: The data type representing Boolean values.

   :: DeveloperApi :: The data type representing Boolean values. Please use the singleton DataTypes.BooleanType.

   Annotations

   @DeveloperApi()

4. class ByteType extends IntegralType

   

   :: DeveloperApi :: The data type representing Byte values.

   :: DeveloperApi :: The data type representing Byte values. Please use the singleton DataTypes.ByteType.

   Annotations

   @DeveloperApi()

5. abstract class DataType extends AnyRef

   

   :: DeveloperApi :: The base type of all Spark SQL data types.

   :: DeveloperApi :: The base type of all Spark SQL data types.

   Annotations

   @DeveloperApi()

6. class DataTypes extends AnyRef

   

7. class DateType extends NativeType

   

   :: DeveloperApi :: The data type representing java.sql.Date values.

   :: DeveloperApi :: The data type representing java.sql.Date values. Please use the singleton DataTypes.DateType.

   Annotations

   @DeveloperApi()

8. final class Decimal extends Ordered[Decimal] with Serializable

   

   A mutable implementation of BigDecimal that can hold a Long if values are small enough.

   A mutable implementation of BigDecimal that can hold a Long if values are small enough.

   The semantics of the fields are as follows: - _precision and _scale represent the SQL precision and scale we are looking for - If decimalVal is set, it represents the whole decimal value - Otherwise, the decimal value is longVal / (10 ** _scale)

9. case class DecimalType(precisionInfo: Option[PrecisionInfo]) extends FractionalType with Product with Serializable

   

   :: DeveloperApi :: The data type representing java.math.BigDecimal values.

   :: DeveloperApi :: The data type representing java.math.BigDecimal values. A Decimal that might have fixed precision and scale, or unlimited values for these.

   Please use DataTypes.createDecimalType() to create a specific instance.

   Annotations

   @DeveloperApi()

10. class DoubleType extends FractionalType

    

    :: DeveloperApi :: The data type representing Double values.

    :: DeveloperApi :: The data type representing Double values. Please use the singleton DataTypes.DoubleType.

    Annotations

    @DeveloperApi()

11. class FloatType extends FractionalType

    

    :: DeveloperApi :: The data type representing Float values.

    :: DeveloperApi :: The data type representing Float values. Please use the singleton DataTypes.FloatType.

    Annotations

    @DeveloperApi()

12. sealed abstract class FractionalType extends NumericType

    

    Attributes

    protected[org.apache.spark.sql]

13. class IntegerType extends IntegralType

    

    :: DeveloperApi :: The data type representing Int values.

    :: DeveloperApi :: The data type representing Int values. Please use the singleton DataTypes.IntegerType.

    Annotations

    @DeveloperApi()

14. sealed abstract class IntegralType extends NumericType

    

    Attributes

    protected[org.apache.spark.sql]

15. class LongType extends IntegralType

    

    :: DeveloperApi :: The data type representing Long values.

    :: DeveloperApi :: The data type representing Long values. Please use the singleton DataTypes.LongType.

    Annotations

    @DeveloperApi()

16. case class MapType(keyType: DataType, valueType: DataType, valueContainsNull: Boolean) extends DataType with Product with Serializable

    

    :: DeveloperApi :: The data type for Maps.

    :: DeveloperApi :: The data type for Maps. Keys in a map are not allowed to have null values.

    Please use DataTypes.createMapType() to create a specific instance.

    keyType

    The data type of map keys.

    valueType

    The data type of map values.

    valueContainsNull

    Indicates if map values have null values.

17. sealed class Metadata extends Serializable

    

    :: DeveloperApi ::

    :: DeveloperApi ::

    Metadata is a wrapper over Map[String, Any] that limits the value type to simple ones: Boolean, Long, Double, String, Metadata, Array[Boolean], Array[Long], Array[Double], Array[String], and Array[Metadata]. JSON is used for serialization.

    The default constructor is private. User should use either MetadataBuilder or Metadata.fromJson() to create Metadata instances.

    Annotations

    @DeveloperApi()

18. class MetadataBuilder extends AnyRef

    

    :: DeveloperApi ::

    :: DeveloperApi ::

    Builder for Metadata. If there is a key collision, the latter will overwrite the former.

    Annotations

    @DeveloperApi()

19. abstract class NativeType extends DataType

    

    Attributes

    protected[org.apache.spark.sql]

20. class NullType extends DataType

    

    :: DeveloperApi :: The data type representing NULL values.

    :: DeveloperApi :: The data type representing NULL values. Please use the singleton DataTypes.NullType.

    Annotations

    @DeveloperApi()

21. abstract class NumericType extends NativeType with PrimitiveType

    

    :: DeveloperApi :: Numeric data types.

22. case class PrecisionInfo(precision: Int, scale: Int) extends Product with Serializable

    

    Precision parameters for a Decimal

23. trait PrimitiveType extends DataType

    

    Attributes

    protected[org.apache.spark.sql]

24. class SQLUserDefinedType extends Annotation with Annotation with ClassfileAnnotation

    

25. class ShortType extends IntegralType

    

    :: DeveloperApi :: The data type representing Short values.

    :: DeveloperApi :: The data type representing Short values. Please use the singleton DataTypes.ShortType.

    Annotations

    @DeveloperApi()

26. class StringType extends NativeType with PrimitiveType

    

    :: DeveloperApi :: The data type representing String values.

    :: DeveloperApi :: The data type representing String values. Please use the singleton DataTypes.StringType.

    Annotations

    @DeveloperApi()

27. case class StructField(name: String, dataType: DataType, nullable: Boolean = true, metadata: Metadata = Metadata.empty) extends Product with Serializable

    

    A field inside a StructType.

    A field inside a StructType.

    name

    The name of this field.

    dataType

    The data type of this field.

    nullable

    Indicates if values of this field can be null values.

    metadata

    The metadata of this field. The metadata should be preserved during transformation if the content of the column is not modified, e.g, in selection.

28. case class StructType(fields: Array[StructField]) extends DataType with Seq[StructField] with Product with Serializable

    

    :: DeveloperApi :: A StructType object can be constructed by

    :: DeveloperApi :: A StructType object can be constructed by

    StructType(fields: Seq[StructField])

    For a StructType object, one or multiple StructFields can be extracted by names. If multiple StructFields are extracted, a StructType object will be returned. If a provided name does not have a matching field, it will be ignored. For the case of extracting a single StructField, a null will be returned. Example:

    import org.apache.spark.sql._
    
    val struct =
      StructType(
        StructField("a", IntegerType, true) ::
        StructField("b", LongType, false) ::
        StructField("c", BooleanType, false) :: Nil)
    
    // Extract a single StructField.
    val singleField = struct("b")
    // singleField: StructField = StructField(b,LongType,false)
    
    // This struct does not have a field called "d". null will be returned.
    val nonExisting = struct("d")
    // nonExisting: StructField = null
    
    // Extract multiple StructFields. Field names are provided in a set.
    // A StructType object will be returned.
    val twoFields = struct(Set("b", "c"))
    // twoFields: StructType =
    //   StructType(List(StructField(b,LongType,false), StructField(c,BooleanType,false)))
    
    // Any names without matching fields will be ignored.
    // For the case shown below, "d" will be ignored and
    // it is treated as struct(Set("b", "c")).
    val ignoreNonExisting = struct(Set("b", "c", "d"))
    // ignoreNonExisting: StructType =
    //   StructType(List(StructField(b,LongType,false), StructField(c,BooleanType,false)))

    A org.apache.spark.sql.Row object is used as a value of the StructType. Example:

    import org.apache.spark.sql._
    
    val innerStruct =
      StructType(
        StructField("f1", IntegerType, true) ::
        StructField("f2", LongType, false) ::
        StructField("f3", BooleanType, false) :: Nil)
    
    val struct = StructType(
      StructField("a", innerStruct, true) :: Nil)
    
    // Create a Row with the schema defined by struct
    val row = Row(Row(1, 2, true))
    // row: Row = [[1,2,true]]

    Annotations

    @DeveloperApi()

29. class TimestampType extends NativeType

    

    :: DeveloperApi :: The data type representing java.sql.Timestamp values.

    :: DeveloperApi :: The data type representing java.sql.Timestamp values. Please use the singleton DataTypes.TimestampType.

    Annotations

    @DeveloperApi()

30. abstract class UserDefinedType[UserType] extends DataType with Serializable

    

    ::DeveloperApi:: The data type for User Defined Types (UDTs).

    ::DeveloperApi:: The data type for User Defined Types (UDTs).

    This interface allows a user to make their own classes more interoperable with SparkSQL; e.g., by creating a UserDefinedType for a class X, it becomes possible to create a DataFrame which has class X in the schema.

    For SparkSQL to recognize UDTs, the UDT must be annotated with SQLUserDefinedType.

    The conversion via serialize occurs when instantiating a DataFrame from another RDD. The conversion via deserialize occurs when reading from a DataFrame.

    Annotations

    @DeveloperApi()