Object

io.smartdatalake.definitions

Environment

Related Doc: package definitions

Permalink

object Environment

Environment dependent configurations. They can be set - by Java system properties (prefixed with "sdl.", e.g. "sdl.hadoopAuthoritiesWithAclsRequired") - by Environment variables (prefixed with "SDL_" and camelCase converted to uppercase, e.g. "SDL_HADOOP_AUTHORITIES_WITH_ACLS_REQUIRED") - by a custom io.smartdatalake.app.SmartDataLakeBuilder implementation for your environment, which sets these variables directly.

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. Environment
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  5. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  6. val configPathsForLocalSubstitution: Seq[String]

    Permalink
  7. val defaultPathSeparator: Char

    Permalink
  8. var enableAutomaticDataFrameCaching: Boolean

    Permalink

    Set to true if you want to enable automatic caching of DataFrames that are used multiple times (default=true).

  9. var enableCheckConfigDuplicates: Boolean

    Permalink

    Set to true to enable check for duplicate first class object definitions when loading configuration (default=true).

    Set to true to enable check for duplicate first class object definitions when loading configuration (default=true). The check fails if Connections, DataObjects or Actions are defined in multiple locations.

  10. var enableOverwriteUnpartitionedSparkFileDataObjectAdls: Boolean

    Permalink

    Set to true if you want to enable workaround to overwrite unpartitioned SparkFileDataObject on Azure ADLSv2 (default=false).

  11. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  12. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  13. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  14. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  15. def globalConfig: GlobalConfig

    Permalink
  16. var hadoopAuthoritiesWithAclsRequired: Seq[String]

    Permalink

    List of hadoop authorities for which acls must be configured The environment parameter can contain multiple authorities separated by comma.

    List of hadoop authorities for which acls must be configured The environment parameter can contain multiple authorities separated by comma. An authority is compared against the filesystem URI with contains(...)

  17. var hadoopDefaultSchemeAuthority: Option[URI]

    Permalink

    Set default hadoop schema and authority for path

  18. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  19. var hdfsAclsLimitToBasedir: Boolean

    Permalink

    Limit setting ACL's to Basedir (default=true) See hdfsAclsUserHomeLevel or hdfsBasedir on how the basedir is determined

  20. var hdfsAclsMinLevelPermissionModify: Int

    Permalink

    Modifying ACL's is only allowed below and including the following level (default=2) See also io.smartdatalake.util.misc.AclUtil

  21. var hdfsAclsMinLevelPermissionOverwrite: Int

    Permalink

    Overwriting ACL's is only allowed below and including the following level (default=5) See also io.smartdatalake.util.misc.AclUtil

  22. var hdfsAclsUserHomeLevel: Int

    Permalink

    Set path level of user home to determine basedir automatically (Default=2 -> /user/myUserHome)

  23. var hdfsBasedir: Option[URI]

    Permalink

    Set basedir explicitly.

    Set basedir explicitly. This overrides automatically detected user home for acl constraints by hdfsAclsUserHomeLevel.

  24. def instanceRegistry: InstanceRegistry

    Permalink
  25. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  26. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  27. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  28. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  29. val runIdPartitionColumnName: String

    Permalink
  30. var schemaEvolutionNewColumnsLast: Boolean

    Permalink

    ordering of columns in SchemaEvolution result - true: result schema is ordered according to existing schema, new columns are appended - false: result schema is ordered according to new schema, deleted columns are appended

  31. var schemaValidationDeepComarison: Boolean

    Permalink

    If true, schema validation inspects the whole hierarchy of structured data types.

    If true, schema validation inspects the whole hierarchy of structured data types. This allows partial matches for schemaMin validation. If false, structural data types must match exactly to validate.

    Example:
    1. Using io.smartdatalake.workflow.dataobject.SchemaValidation.validateSchemaMin: val schema = StructType.fromDDL("c1 STRING, c2 STRUCT(c2_1 INT, c2_2 STRING)") validates against StructType.fromDDL("c1 STRING, c2 STRUCT(c2_1 INT)") only if schemaValidationDeepComarison == true.

  32. var schemaValidationIgnoresNullability: Boolean

    Permalink

    If true, schema validation does not consider nullability of columns/fields when checking for equality.

    If true, schema validation does not consider nullability of columns/fields when checking for equality. If false, schema validation considers two columns/fields different when their nullability property is not equal.

  33. def sparkSession: SparkSession

    Permalink
  34. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  35. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any
  36. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  37. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  38. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from AnyRef

Inherited from Any

Ungrouped