definitions

Type Members

sealed trait AuthMode extends AnyRef

Authentication modes define how an application authenticates itself to a given data object/connection
case class BasicAuthMode(userVariable: String, passwordVariable: String) extends AuthMode with Product with Serializable

Derive options for various connection types to connect by basic authentication
sealed trait ExecutionMode extends AnyRef

Execution mode's defines how data is selected when running a data pipeline.
case class PartitionDiffMode(partitionColNb: Option[Int] = None, mainInputId: Option[String] = None, mainOutputId: Option[String] = None, nbOfPartitionValuesPerRun: Option[Int] = None) extends ExecutionMode with Product with Serializable

Partition difference execution mode lists partitions on input & output DataObject and starts loading all missing partitions.
Partition difference execution mode lists partitions on input & output DataObject and starts loading all missing partitions. Partition columns to be used for comparision need to be a common 'init' of input and output partition columns.
partitionColNb
optional number of partition columns to use as a common 'init'.
mainInputId
optional selection of inputId to be used for partition comparision. Only needed if there are multiple input DataObject's.
mainOutputId
optional selection of outputId to be used for partition comparision. Only needed if there are multiple output DataObject's.
nbOfPartitionValuesPerRun
optional restriction of the number of partition values per run.
case class PublicKeyAuthMode(userVariable: String) extends AuthMode with Product with Serializable

Validate by user and private/public key Private key is read from .ssh
case class TokenAuthMode(tokenVariable: String) extends AuthMode with Product with Serializable

Derive options for various connection types to connect by token

Value Members

object DateColumnType extends Enumeration

Datatype for date columns in Hive
object Environment

Environment dependent configurations.
Environment dependent configurations. They can be set - by Java system properties (prefixed with "sdl.", e.g. "sdl.hadoopAuthoritiesWithAclsRequired") - by Environment variables (prefixed with "SDL_" and camelCase converted to uppercase, e.g. "SDL_HADOOP_AUTHORITIES_WITH_ACLS_REQUIRED") - by a custom io.smartdatalake.app.SmartDataLakeBuilder implementation for your environment, which sets these variables directly.
object HiveConventions

Hive conventions
object HiveTableLocationSuffix extends Enumeration

Suffix used for alternating parquet HDFS paths (usually in TickTockHiveTableDataObject for integration layer)
object OutputType extends Enumeration

Options for HDFS output
object TechnicalTableColumn extends Enumeration

Column names specific to historization of Hive tables

package definitions

Type Members

sealed trait AuthMode extends AnyRef

case class BasicAuthMode(userVariable: String, passwordVariable: String) extends AuthMode with Product with Serializable

sealed trait ExecutionMode extends AnyRef

case class PartitionDiffMode(partitionColNb: Option[Int] = None, mainInputId: Option[String] = None, mainOutputId: Option[String] = None, nbOfPartitionValuesPerRun: Option[Int] = None) extends ExecutionMode with Product with Serializable

case class PublicKeyAuthMode(userVariable: String) extends AuthMode with Product with Serializable

case class TokenAuthMode(tokenVariable: String) extends AuthMode with Product with Serializable

Value Members

object DateColumnType extends Enumeration

object Environment

object HiveConventions

object HiveTableLocationSuffix extends Enumeration

object OutputType extends Enumeration

object TechnicalTableColumn extends Enumeration

Ungrouped