DefaultSmartDataLakeBuilder

Value Members

1. final def !=(arg0: Any): Boolean

   

   Definition Classes

   AnyRef → Any

2. final def ##(): Int

   

   Definition Classes

   AnyRef → Any

3. final def ==(arg0: Any): Boolean

   

   Definition Classes

   AnyRef → Any

4. val appType: String

   

   Definition Classes

   SmartDataLakeBuilder

5. val appVersion: String

   

   Definition Classes

   SmartDataLakeBuilder

6. final def asInstanceOf[T0]: T0

   

   Definition Classes

   Any

7. def clone(): AnyRef

   

   Attributes

   protected[java.lang]

   Definition Classes

   AnyRef

   Annotations

   @throws( ... )

8. final def eq(arg0: AnyRef): Boolean

   

   Definition Classes

   AnyRef

9. def equals(arg0: Any): Boolean

   

   Definition Classes

   AnyRef → Any

10. def finalize(): Unit

    

    Attributes

    protected[java.lang]

    Definition Classes

    AnyRef

    Annotations

    @throws( classOf[java.lang.Throwable] )

11. final def getClass(): Class[_]

    

    Definition Classes

    AnyRef → Any

12. def hashCode(): Int

    

    Definition Classes

    AnyRef → Any

13. def initConfigFromEnvironment: SmartDataLakeBuilderConfig

    

    Create a new SDL configuration.

    Create a new SDL configuration.

    Could be used in the future to set default values.

    returns

    a new, initialized SmartDataLakeBuilderConfig.

    Definition Classes

    SmartDataLakeBuilder

14. val instanceRegistry: InstanceRegistry

    

    InstanceRegistry instance

    InstanceRegistry instance

    Definition Classes

    SmartDataLakeBuilder

15. final def isInstanceOf[T0]: Boolean

    

    Definition Classes

    Any

16. lazy val logger: Logger

    

    Attributes

    protected

    Definition Classes

    SmartDataLakeLogger

17. final def ne(arg0: AnyRef): Boolean

    

    Definition Classes

    AnyRef

18. final def notify(): Unit

    

    Definition Classes

    AnyRef

19. final def notifyAll(): Unit

    

    Definition Classes

    AnyRef

20. def parseAndRun(args: Array[String], ignoreOverrideJars: Boolean = false): Unit

    

21. def parseCommandLineArguments(args: Array[String], config: SmartDataLakeBuilderConfig): Option[SmartDataLakeBuilderConfig]

    

    Parses the supplied (command line) arguments.

    Parses the supplied (command line) arguments.

    This method parses command line arguments and creates the corresponding SmartDataLakeBuilderConfig

    args

    an Array of command line arguments.

    config

    a configuration initialized with default values.

    returns

    a new configuration with default values overwritten from the supplied command line arguments.

    Definition Classes

    SmartDataLakeBuilder

22. val parser: OptionParser[SmartDataLakeBuilderConfig]

    

    The Parser defines how to extract the options from the command line args.

    The Parser defines how to extract the options from the command line args. Subclasses SmartDataLakeBuilder can define additional options to be extracted.

    Attributes

    protected

    Definition Classes

    SmartDataLakeBuilder

23. def run(appConfig: SmartDataLakeBuilderConfig): Map[RuntimeEventState, Int]

    

    Run the application with the provided configuration.

    Run the application with the provided configuration.

    appConfig

    Application configuration (parsed from command line).

    Definition Classes

    SmartDataLakeBuilder

24. def startSimulation(appConfig: SmartDataLakeBuilderConfig, initialSubFeeds: Seq[SparkSubFeed])(implicit instanceRegistry: InstanceRegistry, session: SparkSession): (Seq[SparkSubFeed], Map[RuntimeEventState, Int])

    

    Start a simulation run.

    Start a simulation run. This executes the DAG and returns all subfeeds including the transformed DataFrames. Only prepare and init are executed. All initial subfeeds must be provided as input.

    Note: this only works with SparkActions for now

    appConfig

    application configuration

    initialSubFeeds

    initial subfeeds for DataObjects at the beginning of the DAG

    returns

    tuple of list of final subfeeds and statistics (action count per RuntimeEventState)

    Definition Classes

    SmartDataLakeBuilder

25. final def synchronized[T0](arg0: ⇒ T0): T0

    

    Definition Classes

    AnyRef

26. def toString(): String

    

    Definition Classes

    AnyRef → Any

27. final def wait(): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

28. final def wait(arg0: Long, arg1: Int): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

29. final def wait(arg0: Long): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )