Default Smart Data Lake Command Line Application.
Global configuration options
Global configuration options
classes to register for spark kryo serialization
spark options
enable hive for spark session
enable periodic memory usage logging, see detailled configuration MemoryLogTimerConfig
enable shutdown hook logger to trace shutdown cause
Configuration for periodic memory usage logging
Configuration for periodic memory usage logging
interval in seconds between memory usage logs
enable logging linux memory
enable logging details about linux cgroup memory
enable logging details about different jvm buffers
Abstract Smart Data Lake Command Line Application.
This case class represents a default configuration for the App.
This case class represents a default configuration for the App. It is populated by parsing command-line arguments. It also specifies default values.
Regex pattern to select the feed to execute.
Application name.
A configuration file or a directory containing configuration files.
The Spark master URL passed to SparkContext when in local mode.
The Spark deploy mode passed to SparkContext when in local mode.
Kerberos user name (username
@kerberosDomain
) for local mode.
Kerberos domain (username
@kerberosDomain
) for local mode.
Path to Kerberos keytab file for local mode.
Run in test mode:
Interface to notify interested parties about action results & metric
Configuration to notify interested parties about action results & metric
Configuration to notify interested parties about action results & metric
fully qualified class name of class implementing StateListener interface. The class needs a constructor with one parameter options: Map[String,String]
.
Options are passed to StateListener constructor.
Databricks Smart Data Lake Command Line Application.
Databricks Smart Data Lake Command Line Application.
As there is an old version of config-*.jar deployed on Databricks, this special App uses a ChildFirstClassLoader to override it in the classpath.
Smart Data Lake Builder application for local mode.
Smart Data Lake Builder application for local mode.
Sets master to local[*] and deployMode to client by default.
Default Smart Data Lake Command Line Application.
Implementation Note: This must be a class and not an object in order to be found by reflection in DatabricksSmartDataLakeBuilder