Create a new SDL configuration.
Create a new SDL configuration.
Could be used in the future to set default values.
a new, initialized SmartDataLakeBuilderConfig.
InstanceRegistry instance
InstanceRegistry instance
Entry-Point of the application.
Entry-Point of the application.
Command-line arguments.
Parses the supplied (command line) arguments.
Parses the supplied (command line) arguments.
This method parses command line arguments and creates the corresponding SmartDataLakeBuilderConfig
an Array of command line arguments.
a configuration initialized with default values.
a new configuration with default values overwritten from the supplied command line arguments.
The Parser defines how to extract the options from the command line args.
The Parser defines how to extract the options from the command line args. Subclasses SmartDataLakeBuilder can define additional options to be extracted.
Run the application with the provided configuration.
Run the application with the provided configuration.
Application configuration (parsed from command line).
Start a simulation run.
Start a simulation run. This executes the DAG and returns all subfeeds including the transformed DataFrames. Only prepare and init are executed. All initial subfeeds must be provided as input.
Note: this only works with SparkActions for now
application configuration
initial subfeeds for DataObjects at the beginning of the DAG
tuple of list of final subfeeds and statistics (action count per RuntimeEventState)
Databricks Smart Data Lake Command Line Application.
As there is an old version of config-*.jar deployed on Databricks, this special App uses a ChildFirstClassLoader to override it in the classpath.