Convert a SQL datum to the user type
Convert the user type to a SQL datum
Convert the user type to a SQL datum
TODO: Can we make this take obj: UserType? The issue is in ScalaReflection.convertToCatalyst, where we need to convert Any to UserType.
Underlying storage type for this UDT
Class object for the UserType
Paired Python UDT class, if exists.
Matches any expression that evaluates to this DataType
Matches any expression that evaluates to this DataType
::DeveloperApi:: The data type for User Defined Types (UDTs).
This interface allows a user to make their own classes more interoperable with SparkSQL; e.g., by creating a UserDefinedType for a class X, it becomes possible to create a SchemaRDD which has class X in the schema.
For SparkSQL to recognize UDTs, the UDT must be annotated with org.apache.spark.sql.catalyst.annotation.SQLUserDefinedType.
The conversion via
serialize
occurs when instantiating aSchemaRDD
from another RDD. The conversion viadeserialize
occurs when reading from aSchemaRDD
.