customlogic

Type Members

1. trait CustomDfCreator extends Serializable

   

   Interface to define custom logic for DataFrame transformations

2. case class CustomDfCreatorConfig(className: Option[String] = None, scalaFile: Option[String] = None, scalaCode: Option[String] = None, options: Option[Map[String, String]] = None) extends Product with Serializable

   

3. class CustomDfCreatorWrapper extends CustomDfCreator

   

4. trait CustomDfTransformer extends Serializable

   

   Interface to define a custom Spark-DataFrame transformation (1:1)

5. case class CustomDfTransformerConfig(className: Option[String] = None, scalaFile: Option[String] = None, scalaCode: Option[String] = None, sqlCode: Option[String] = None, pythonFile: Option[String] = None, pythonCode: Option[String] = None, options: Map[String, String] = Map(), runtimeOptions: Map[String, String] = Map()) extends Product with Serializable

   

   Configuration of a custom Spark-DataFrame transformation between one input and one output (1:1)

   Configuration of a custom Spark-DataFrame transformation between one input and one output (1:1)

   Note about Python transformation: Environment with Python and PySpark needed. PySpark session is initialize and available under variables sc, session, sqlContext. Input DataFrame is available as inputDf. Output DataFrame must be set with setOutputDf(df).

   className

   Optional class name to load transformer code from

   scalaFile

   Optional file where scala code for transformation is loaded from

   scalaCode

   Optional scala code for transformation

   sqlCode

   Optional SQL code for transformation. Use tokens %{<key>} to replace with runtimeOptions in SQL code. Example: "select * from test where run = %{runId}"

   pythonFile

   Optional pythonFile to use for python transformation

   pythonCode

   Optional pythonCode to user for python transformation

   options

   Options to pass to the transformation

   runtimeOptions

   optional tuples of [key, spark sql expression] to be added as additional options when executing transformation. The spark sql expressions are evaluated against an instance of DefaultExpressionData.

6. trait CustomDfsTransformer extends Serializable

   

   Interface to define a custom Spark-DataFrame transformation (n:m) Same trait as CustomDfTransformer, but multiple input and outputs supported.

7. case class CustomDfsTransformerConfig(className: Option[String] = None, scalaFile: Option[String] = None, scalaCode: Option[String] = None, sqlCode: Map[DataObjectId, String] = Map(), options: Map[String, String] = Map(), runtimeOptions: Map[String, String] = Map()) extends Product with Serializable

   

   Configuration of a custom Spark-DataFrame transformation between several inputs and outputs (n:m)

   Configuration of a custom Spark-DataFrame transformation between several inputs and outputs (n:m)

   className

   Optional class name to load transformer code from

   scalaFile

   Optional file where scala code for transformation is loaded from

   scalaCode

   Optional scala code for transformation

   sqlCode

   Optional map of DataObjectId and corresponding SQL Code. Use tokens %{<key>} to replace with runtimeOptions in SQL code. Example: "select * from test where run = %{runId}"

   options

   Options to pass to the transformation

   runtimeOptions

   optional tuples of [key, spark sql expression] to be added as additional options when executing transformation. The spark sql expressions are evaluated against an instance of DefaultExpressionData.

8. trait CustomFileTransformer extends Serializable

   

   Interface to define custom file transformation for CustomFileAction

9. case class CustomFileTransformerConfig(className: Option[String] = None, scalaFile: Option[String] = None, scalaCode: Option[String] = None, options: Map[String, String] = Map()) extends Product with Serializable

   

   Configuration of custom file transformation between one input and one output (1:1)

   Configuration of custom file transformation between one input and one output (1:1)

   className

   Optional class name to load transformer code from

   scalaFile

   Optional file where scala code for transformation is loaded from

   scalaCode

   Optional scala code for transformation

   options

   Options to pass to the transformation

10. class CustomFileTransformerWrapper extends CustomFileTransformer with SmartDataLakeLogger