CustomDfDataObject

Value Members

1. final def !=(arg0: Any): Boolean

   

   Definition Classes

   AnyRef → Any

2. final def ##(): Int

   

   Definition Classes

   AnyRef → Any

3. final def ==(arg0: Any): Boolean

   

   Definition Classes

   AnyRef → Any

4. final def asInstanceOf[T0]: T0

   

   Definition Classes

   Any

5. def clone(): AnyRef

   

   Attributes

   protected[java.lang]

   Definition Classes

   AnyRef

   Annotations

   @throws( ... )

6. val creator: CustomDfCreatorConfig

   

7. final def eq(arg0: AnyRef): Boolean

   

   Definition Classes

   AnyRef

8. def factory: FromConfigFactory[DataObject]

   

   Returns the factory that can parse this type (that is, type CO).

   Returns the factory that can parse this type (that is, type CO).

   Typically, implementations of this method should return the companion object of the implementing class. The companion object in turn should implement FromConfigFactory.

   returns

   the factory (object) for this class.

   Definition Classes

   CustomDfDataObject → ParsableFromConfig

9. def finalize(): Unit

   

   Attributes

   protected[java.lang]

   Definition Classes

   AnyRef

   Annotations

   @throws( classOf[java.lang.Throwable] )

10. final def getClass(): Class[_]

    

    Definition Classes

    AnyRef → Any

11. def getConnection[T <: Connection](connectionId: ConnectionId)(implicit registry: InstanceRegistry, ct: ClassTag[T], tt: scala.reflect.api.JavaUniverse.TypeTag[T]): T

    

    Handle class cast exception when getting objects from instance registry

    Handle class cast exception when getting objects from instance registry

    Attributes

    protected

    Definition Classes

    DataObject

12. def getConnectionReg[T <: Connection](connectionId: ConnectionId, registry: InstanceRegistry)(implicit ct: ClassTag[T], tt: scala.reflect.api.JavaUniverse.TypeTag[T]): T

    

    Attributes

    protected

    Definition Classes

    DataObject

13. def getDataFrame(partitionValues: Seq[PartitionValues] = Seq())(implicit session: SparkSession): DataFrame

    

    Definition Classes

    CustomDfDataObject → CanCreateDataFrame

14. val id: DataObjectId

    

    A unique identifier for this instance.

    A unique identifier for this instance.

    Definition Classes

    CustomDfDataObject → DataObject → SdlConfigObject

15. implicit val instanceRegistry: InstanceRegistry

    

16. final def isInstanceOf[T0]: Boolean

    

    Definition Classes

    Any

17. lazy val logger: Logger

    

    Attributes

    protected

    Definition Classes

    SmartDataLakeLogger

18. val metadata: Option[DataObjectMetadata]

    

    Additional metadata for the DataObject

    Additional metadata for the DataObject

    Definition Classes

    CustomDfDataObject → DataObject

19. final def ne(arg0: AnyRef): Boolean

    

    Definition Classes

    AnyRef

20. final def notify(): Unit

    

    Definition Classes

    AnyRef

21. final def notifyAll(): Unit

    

    Definition Classes

    AnyRef

22. def postRead(implicit session: SparkSession): Unit

    

    Runs operations after reading from DataObject

    Runs operations after reading from DataObject

    Definition Classes

    DataObject

23. def postWrite(implicit session: SparkSession): Unit

    

    Runs operations after writing to DataObject

    Runs operations after writing to DataObject

    Definition Classes

    DataObject

24. def preRead(implicit session: SparkSession): Unit

    

    Runs operations before reading from DataObject

    Runs operations before reading from DataObject

    Definition Classes

    DataObject

25. def preWrite(implicit session: SparkSession): Unit

    

    Runs operations before writing to DataObject

    Runs operations before writing to DataObject

    Definition Classes

    DataObject

26. def prepare(implicit session: SparkSession): Unit

    

    Prepare & test DataObject's prerequisits

    Prepare & test DataObject's prerequisits

    This runs during the "prepare" operation of the DAG.

    Definition Classes

    DataObject

27. val schemaMin: Option[StructType]

    

    An optional, minimal schema that a DataObject schema must have to pass schema validation.

    An optional, minimal schema that a DataObject schema must have to pass schema validation.

    The schema validation semantics are: - Schema A is valid in respect to a minimal schema B when B is a subset of A. This means: the whole column set of B is contained in the column set of A.

    • A column of B is contained in A when A contains a column with equal name and data type.
    • Column order is ignored.
    • Column nullability is ignored.
    • Duplicate columns in terms of name and data type are eliminated (set semantics).

    Note: This is only used by the functionality defined in CanCreateDataFrame and CanWriteDataFrame, that is, when reading or writing Spark data frames from/to the underlying data container. io.smartdatalake.workflow.action.Actions that bypass Spark data frames ignore the schemaMin attribute if it is defined.

    Definition Classes

    CustomDfDataObject → SchemaValidation

28. final def synchronized[T0](arg0: ⇒ T0): T0

    

    Definition Classes

    AnyRef

29. def toStringShort: String

    

    Definition Classes

    DataObject

30. def validateSchemaMin(df: DataFrame): Unit

    

    Validate the schema of a given Spark Data Frame df against schemaMin.

    Validate the schema of a given Spark Data Frame df against schemaMin.

    df

    The data frame to validate.

    Definition Classes

    SchemaValidation

    Exceptions thrown

    SchemaViolationException is the schemaMin does not validate.

31. final def wait(): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

32. final def wait(arg0: Long, arg1: Int): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

33. final def wait(arg0: Long): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )