Package

io.smartdatalake.util

evolution

Permalink

package evolution

Visibility
  1. Public
  2. All

Type Members

  1. case class UnsafeUnaryUdfExpression(child: Expression, udf: (Any) ⇒ Any, tgtDataType: DataType) extends UnaryExpression with Product with Serializable

    Permalink

    With Spark 3.0 the API for Udf's was made more typesafe.

    With Spark 3.0 the API for Udf's was made more typesafe. It's no longer possible to create a Udf and give it's return type as StructType. This implements an explicitly unsafe unary udf, which takes a function with signature Any -> Any as transformation. UnsafeUnaryUdfExpression is the Udf expression that is embedded and executed in Spark logical plan. Use UnsafeUnaryUdf to create the udf function to be used in Spark DataFrame API.

Value Members

  1. object UnsafeUnaryUdf

    Permalink

    This defines a helper method to create the Udf.

    This defines a helper method to create the Udf. Note that a Udf is a function which takes Columns as input and returns an derived column. The same UDF can be applied to several columns, each applications generates an UnsafeUnaryUdfExpression as derived column.

Ungrouped