io.smartdatalake.workflow.dataobject
unique name of this data object
DDL-statement to be executed in prepare phase
SQL-statement to be executed before writing to table
SQL-statement to be executed after writing to table
An optional, minimal schema that this DataObject must have to pass schema validation on reading and writing.
The jdbc table to be read
DDL-statement to be executed in prepare phase
Returns the factory that can parse this type (that is, type CO
).
Returns the factory that can parse this type (that is, type CO
).
Typically, implementations of this method should return the companion object of the implementing class. The companion object in turn should implement FromConfigFactory.
the factory (object) for this class.
Handle class cast exception when getting objects from instance registry
Handle class cast exception when getting objects from instance registry
unique name of this data object
unique name of this data object
Initialize callback before writing data out to disk/sinks.
Initialize callback before writing data out to disk/sinks.
Additional metadata for the DataObject
Additional metadata for the DataObject
Runs operations after reading from DataObject
Runs operations after reading from DataObject
SQL-statement to be executed after writing to table
Runs operations after writing to DataObject
Runs operations after writing to DataObject
Runs operations before reading from DataObject
Runs operations before reading from DataObject
SQL-statement to be executed before writing to table
Runs operations before writing to DataObject
Runs operations before writing to DataObject
Prepare & test DataObject's prerequisits
Prepare & test DataObject's prerequisits
This runs during the "prepare" operation of the DAG.
An optional, minimal schema that this DataObject must have to pass schema validation on reading and writing.
An optional, minimal schema that this DataObject must have to pass schema validation on reading and writing.
The jdbc table to be read
The jdbc table to be read
Validate the schema of a given Spark Data Frame df
against schemaMin
.
Validate the schema of a given Spark Data Frame df
against schemaMin
.
The data frame to validate.
SchemaViolationException
is the schemaMin
does not validate.
DataObject of type JDBC. Provides details for an action to access tables in a database through JDBC.
unique name of this data object
DDL-statement to be executed in prepare phase
SQL-statement to be executed before writing to table
SQL-statement to be executed after writing to table
An optional, minimal schema that this DataObject must have to pass schema validation on reading and writing.
The jdbc table to be read