Packages

package encoders

Ordering
  1. Alphabetic
Visibility
  1. Public
  2. Protected

Type Members

  1. trait AgnosticEncoder[T] extends Encoder[T]

    A non implementation specific encoder.

    A non implementation specific encoder. This encoder containers all the information needed to generate an implementation specific encoder (e.g. InternalRow <=> Custom Object).

    The input of the serialization does not need to match the external type of the encoder. This is called lenient serialization. An example of this is lenient date serialization, in this case both java.sql.Date and java.time.LocalDate are allowed. Deserialization is never lenient; it will always produce instance of the external type.

Value Members

  1. object AgnosticEncoders
  2. object OuterScopes
  3. object RowEncoder

    A factory for constructing encoders that convert external row to/from the Spark SQL internal binary representation.

    A factory for constructing encoders that convert external row to/from the Spark SQL internal binary representation.

    The following is a mapping between Spark SQL types and its allowed external types:

    BooleanType -> java.lang.Boolean
ByteType -> java.lang.Byte
ShortType -> java.lang.Short
IntegerType -> java.lang.Integer
FloatType -> java.lang.Float
DoubleType -> java.lang.Double
StringType -> String
DecimalType -> java.math.BigDecimal or scala.math.BigDecimal or Decimal

DateType -> java.sql.Date if spark.sql.datetime.java8API.enabled is false
DateType -> java.time.LocalDate if spark.sql.datetime.java8API.enabled is true

TimestampType -> java.sql.Timestamp if spark.sql.datetime.java8API.enabled is false
TimestampType -> java.time.Instant if spark.sql.datetime.java8API.enabled is true

TimestampNTZType -> java.time.LocalDateTime

DayTimeIntervalType -> java.time.Duration
YearMonthIntervalType -> java.time.Period

BinaryType -> byte array
ArrayType -> scala.collection.Seq or Array
MapType -> scala.collection.Map
StructType -> org.apache.spark.sql.Row

Ungrouped