customlogic

Type Members

1. trait CustomDfCreator extends Serializable

   

   Interface to define custom logic for DataFrame creation

2. case class CustomDfCreatorConfig(className: Option[String] = None, scalaFile: Option[String] = None, scalaCode: Option[String] = None, options: Option[Map[String, String]] = None) extends Product with Serializable

   

   Configuration of a custom Spark-DataFrame creator as part of CustomDfDataObject Define a exec function which receives a map of options and returns a DataFrame to be used as input.

   Configuration of a custom Spark-DataFrame creator as part of CustomDfDataObject Define a exec function which receives a map of options and returns a DataFrame to be used as input. Optionally define a schema function to return a StructType used as schema in init-phase. See also trait CustomDfCreator.

   Note that for now implementing CustomDfCreator.schema method is only possible with className configuration attribute.

   className

   Optional class name implementing trait CustomDfCreator

   scalaFile

   Optional file where scala code for creator is loaded from. The scala code in the file needs to be a function of type fnExecType.

   scalaCode

   Optional scala code for creator. The scala code needs to be a function of type fnExecType.

   options

   Options to pass to the creator

3. class CustomDfCreatorWrapper extends CustomDfCreator

   

4. trait CustomDfTransformer extends Serializable

   

   Interface to define a custom Spark-DataFrame transformation (1:1)

5. case class CustomDfTransformerConfig(className: Option[String] = None, scalaFile: Option[String] = None, scalaCode: Option[String] = None, sqlCode: Option[String] = None, pythonFile: Option[String] = None, pythonCode: Option[String] = None, options: Map[String, String] = Map(), runtimeOptions: Map[String, String] = Map()) extends Product with Serializable

   

   Configuration of a custom Spark-DataFrame transformation between one input and one output (1:1) Define a transform function which receives a DataObjectIds, a DataFrames and a map of options and has to return a DataFrame, see also CustomDfTransformer.

   Configuration of a custom Spark-DataFrame transformation between one input and one output (1:1) Define a transform function which receives a DataObjectIds, a DataFrames and a map of options and has to return a DataFrame, see also CustomDfTransformer.

   Note about Python transformation: Environment with Python and PySpark needed. PySpark session is initialize and available under variables sc, session, sqlContext. Other variables available are - inputDf: Input DataFrame - options: Transformation options as Map[String,String] - dataObjectId: Id of input dataObject as String Output DataFrame must be set with setOutputDf(df).

   className

   Optional class name implementing trait CustomDfTransformer

   scalaFile

   Optional file where scala code for transformation is loaded from. The scala code in the file needs to be a function of type fnTransformType.

   scalaCode

   Optional scala code for transformation. The scala code needs to be a function of type fnTransformType.

   sqlCode

   Optional SQL code for transformation. Use tokens %{<key>} to replace with runtimeOptions in SQL code. Example: "select * from test where run = %{runId}"

   pythonFile

   Optional pythonFile to use for python transformation. The python code can use variables inputDf, dataObjectId and options. The transformed DataFrame has to be set with setOutputDf.

   pythonCode

   Optional pythonCode to user for python transformation. The python code can use variables inputDf, dataObjectId and options. The transformed DataFrame has to be set with setOutputDf.

   options

   Options to pass to the transformation

   runtimeOptions

   optional tuples of [key, spark sql expression] to be added as additional options when executing transformation. The spark sql expressions are evaluated against an instance of DefaultExpressionData.

6. trait CustomDfsTransformer extends Serializable

   

   Interface to define a custom Spark-DataFrame transformation (n:m) Same trait as CustomDfTransformer, but multiple input and outputs supported.

7. case class CustomDfsTransformerConfig(className: Option[String] = None, scalaFile: Option[String] = None, scalaCode: Option[String] = None, sqlCode: Map[DataObjectId, String] = Map(), options: Map[String, String] = Map(), runtimeOptions: Map[String, String] = Map()) extends Product with Serializable

   

   Configuration of a custom Spark-DataFrame transformation between several inputs and outputs (n:m).

   Configuration of a custom Spark-DataFrame transformation between several inputs and outputs (n:m). Define a transform function which receives a map of input DataObjectIds with DataFrames and a map of options and as to return a map of output DataObjectIds with DataFrames, see also trait CustomDfsTransformer.

   className

   Optional class name implementing trait CustomDfsTransformer

   scalaFile

   Optional file where scala code for transformation is loaded from. The scala code in the file needs to be a function of type fnTransformType.

   scalaCode

   Optional scala code for transformation. The scala code needs to be a function of type fnTransformType.

   sqlCode

   Optional map of DataObjectId and corresponding SQL Code. Use tokens %{<key>} to replace with runtimeOptions in SQL code. Example: "select * from test where run = %{runId}"

   options

   Options to pass to the transformation

   runtimeOptions

   optional tuples of [key, spark sql expression] to be added as additional options when executing transformation. The spark sql expressions are evaluated against an instance of DefaultExpressionData.

8. trait CustomFileCreator extends Serializable

   

9. case class CustomFileCreatorConfig(className: Option[String] = None, scalaFile: Option[String] = None, scalaCode: Option[String] = None, options: Option[Map[String, String]] = None) extends Product with Serializable

   

10. class CustomFileCreatorWrapper extends CustomFileCreator

    

11. trait CustomFileTransformer extends Serializable

    

    Interface to define custom file transformation for CustomFileAction

12. case class CustomFileTransformerConfig(className: Option[String] = None, scalaFile: Option[String] = None, scalaCode: Option[String] = None, options: Map[String, String] = Map()) extends Product with Serializable

    

    Configuration of custom file transformation between one input and one output (1:1)

    Configuration of custom file transformation between one input and one output (1:1)

    className

    Optional class name to load transformer code from

    scalaFile

    Optional file where scala code for transformation is loaded from

    scalaCode

    Optional scala code for transformation

    options

    Options to pass to the transformation

13. class CustomFileTransformerWrapper extends CustomFileTransformer with SmartDataLakeLogger

    

14. trait SparkUDFCreator extends AnyRef

    

    Interface to create a UserDefinedFunction object to be registered as udf.

15. case class SparkUDFCreatorConfig(className: String, options: Map[String, String] = Map()) extends Product with Serializable

    

    Configuration to register a UserDefinedFunction in the spark session of SmartDataLake.

    Configuration to register a UserDefinedFunction in the spark session of SmartDataLake.

    className

    fully qualified class name of class implementing SparkUDFCreator interface. The class needs a constructor without parameters.

    options

    Options are passed to SparkUDFCreator apply method.