Class

io.smartdatalake.app

SmartDataLakeBuilder

Related Doc: package app

Permalink

abstract class SmartDataLakeBuilder extends SmartDataLakeLogger

Abstract Smart Data Lake Command Line Application.

Linear Supertypes
SmartDataLakeLogger, AnyRef, Any
Known Subclasses
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. SmartDataLakeBuilder
  2. SmartDataLakeLogger
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new SmartDataLakeBuilder()

    Permalink

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  4. val appType: String

    Permalink
  5. val appVersion: String

    Permalink
  6. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  7. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  8. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  9. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  10. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  11. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  12. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  13. def initConfigFromEnvironment: SmartDataLakeBuilderConfig

    Permalink

    Create a new SDL configuration.

    Create a new SDL configuration.

    Could be used in the future to set default values.

    returns

    a new, initialized SmartDataLakeBuilderConfig.

  14. val instanceRegistry: InstanceRegistry

    Permalink

    InstanceRegistry instance

  15. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  16. lazy val logger: Logger

    Permalink
    Attributes
    protected
    Definition Classes
    SmartDataLakeLogger
  17. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  18. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  19. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  20. def parseCommandLineArguments(args: Array[String], config: SmartDataLakeBuilderConfig): Option[SmartDataLakeBuilderConfig]

    Permalink

    Parses the supplied (command line) arguments.

    Parses the supplied (command line) arguments.

    This method parses command line arguments and creates the corresponding SmartDataLakeBuilderConfig

    args

    an Array of command line arguments.

    config

    a configuration initialized with default values.

    returns

    a new configuration with default values overwritten from the supplied command line arguments.

  21. val parser: OptionParser[SmartDataLakeBuilderConfig]

    Permalink

    The Parser defines how to extract the options from the command line args.

    The Parser defines how to extract the options from the command line args. Subclasses SmartDataLakeBuilder can define additional options to be extracted.

    Attributes
    protected
  22. def run(appConfig: SmartDataLakeBuilderConfig): Map[RuntimeEventState, Int]

    Permalink

    Run the application with the provided configuration.

    Run the application with the provided configuration.

    appConfig

    Application configuration (parsed from command line).

  23. def startSimulation(appConfig: SmartDataLakeBuilderConfig, initialSubFeeds: Seq[SparkSubFeed])(implicit instanceRegistry: InstanceRegistry, session: SparkSession): (Seq[SparkSubFeed], Map[RuntimeEventState, Int])

    Permalink

    Start a simulation run.

    Start a simulation run. This executes the DAG and returns all subfeeds including the transformed DataFrames. Only prepare and init are executed. All initial subfeeds must be provided as input.

    Note: this only works with SparkActions for now

    appConfig

    application configuration

    initialSubFeeds

    initial subfeeds for DataObjects at the beginning of the DAG

    returns

    tuple of list of final subfeeds and statistics (action count per RuntimeEventState)

  24. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  25. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any
  26. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  27. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  28. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from SmartDataLakeLogger

Inherited from AnyRef

Inherited from Any

Ungrouped