app

Type Members

1. class DefaultSmartDataLakeBuilder extends SmartDataLakeBuilder

   

   Default Smart Data Lake Command Line Application.

   Default Smart Data Lake Command Line Application.

   Implementation Note: This must be a class and not an object in order to be found by reflection in DatabricksSmartDataLakeBuilder

2. case class GlobalConfig(kryoClasses: Option[Seq[String]] = None, sparkOptions: Option[Map[String, String]] = None, enableHive: Boolean = true, memoryLogTimer: Option[MemoryLogTimerConfig] = None, shutdownHookLogger: Boolean = false, stateListeners: Seq[StateListenerConfig] = Seq()) extends SmartDataLakeLogger with Product with Serializable

   

   Global configuration options

   Global configuration options

   kryoClasses

   classes to register for spark kryo serialization

   sparkOptions

   spark options

   enableHive

   enable hive for spark session

   memoryLogTimer

   enable periodic memory usage logging, see detailled configuration MemoryLogTimerConfig

   shutdownHookLogger

   enable shutdown hook logger to trace shutdown cause

3. case class MemoryLogTimerConfig(intervalSec: Int, logLinuxMem: Boolean = true, logLinuxCGroupMem: Boolean = false, logBuffers: Boolean = false) extends Product with Serializable

   

   Configuration for periodic memory usage logging

   Configuration for periodic memory usage logging

   intervalSec

   interval in seconds between memory usage logs

   logLinuxMem

   enable logging linux memory

   logLinuxCGroupMem

   enable logging details about linux cgroup memory

   logBuffers

   enable logging details about different jvm buffers

4. abstract class SmartDataLakeBuilder extends SmartDataLakeLogger

   

   Abstract Smart Data Lake Command Line Application.

5. case class SmartDataLakeBuilderConfig(feedSel: String = null, applicationName: Option[String] = None, configuration: Option[String] = None, master: Option[String] = None, deployMode: Option[String] = None, username: Option[String] = None, kerberosDomain: Option[String] = None, keytabPath: Option[File] = None, partitionValues: Option[Seq[PartitionValues]] = None, multiPartitionValues: Option[Seq[PartitionValues]] = None, parallelism: Int = 1, statePath: Option[String] = None, overrideJars: Option[Seq[String]] = None, test: Option[TestMode.Value] = None) extends Product with Serializable

   

   This case class represents a default configuration for the App.

   This case class represents a default configuration for the App. It is populated by parsing command-line arguments. It also specifies default values.

   feedSel

   Regex pattern to select the feed to execute.

   applicationName

   Application name.

   configuration

   A configuration file or a directory containing configuration files.

   master

   The Spark master URL passed to SparkContext when in local mode.

   deployMode

   The Spark deploy mode passed to SparkContext when in local mode.

   username

   Kerberos user name (username@kerberosDomain) for local mode.

   kerberosDomain

   Kerberos domain (username@kerberosDomain) for local mode.

   keytabPath

   Path to Kerberos keytab file for local mode.

   test

   Run in test mode:

   • "config": validate configuration
   • "dry-run": execute "prepare" and "init" phase to check environment

6. trait StateListener extends AnyRef

   

   Interface to notify interested parties about action results & metric

7. case class StateListenerConfig(className: String, options: Option[Map[String, String]] = None) extends Product with Serializable

   

   Configuration to notify interested parties about action results & metric

   Configuration to notify interested parties about action results & metric

   className

   fully qualified class name of class implementing StateListener interface. The class needs a constructor with one parameter options: Map[String,String].

   options

   Options are passed to StateListener constructor.