classes to register for spark kryo serialization
spark options
enable hive for spark session
enable periodic memory usage logging, see detailed configuration MemoryLogTimerConfig
enable shutdown hook logger to trace shutdown cause
Define state listeners to be registered for receiving events of the execution of SmartDataLake job
Define UDFs to be registered in spark session. The registered UDFs are available in Spark SQL transformations and expression evaluation, e.g. configuration of ExecutionModes.
Create a spark session using settings from this global config
enable hive for spark session
classes to register for spark kryo serialization
enable periodic memory usage logging, see detailed configuration MemoryLogTimerConfig
enable shutdown hook logger to trace shutdown cause
spark options
Define UDFs to be registered in spark session.
Define UDFs to be registered in spark session. The registered UDFs are available in Spark SQL transformations and expression evaluation, e.g. configuration of ExecutionModes.
Define state listeners to be registered for receiving events of the execution of SmartDataLake job
Global configuration options
classes to register for spark kryo serialization
spark options
enable hive for spark session
enable periodic memory usage logging, see detailed configuration MemoryLogTimerConfig
enable shutdown hook logger to trace shutdown cause
Define state listeners to be registered for receiving events of the execution of SmartDataLake job
Define UDFs to be registered in spark session. The registered UDFs are available in Spark SQL transformations and expression evaluation, e.g. configuration of ExecutionModes.