io.smartdatalake.workflow.dataobject
ParquetFileDataObject
Related Docs:
object ParquetFileDataObject
| package dataobject
io.smartdatalake.workflow.dataobject
unique name of this data object
Hadoop directory where this data object reads/writes it's files. If it doesn't contain scheme and authority, the connections pathPrefix is applied. If pathPrefix is not defined or doesn't define scheme and authority, default schema and authority is applied. Optionally defined partitions are appended with hadoop standard partition layout to this path. Only files ending with *.parquet* are considered as data for this DataObject.
partition columns for this data object
spark SaveMode to use when writing files, default is "overwrite"
Optional definition of repartition operation before writing DataFrame with Spark to Hadoop.
override connections permissions for files created with this connection
optional id of io.smartdatalake.workflow.connection.HadoopFileConnection
Metadata describing this data object.
override connections permissions for files created with this connection
override connections permissions for files created with this connection
Callback that enables potential transformation to be applied to df after the data is read.
Callback that enables potential transformation to be applied to df after the data is read.
Default is to validate the schemaMin and not apply any modification.
Callback that enables potential transformation to be applied to df before the data is written.
Callback that enables potential transformation to be applied to df before the data is written.
Default is to validate the schemaMin and not apply any modification.
Check if the input files exist.
Check if the input files exist.
IllegalArgumentException if failIfFilesMissing = true and no files found at path.
optional id of io.smartdatalake.workflow.connection.HadoopFileConnection
optional id of io.smartdatalake.workflow.connection.HadoopFileConnection
create empty partition
create empty partition
Create empty partitions for partition values not yet existing
Create empty partitions for partition values not yet existing
Delete all data.
Delete all data. This is used to implement SaveMode.Overwrite.
Delete given files.
Delete given files. This is used to cleanup files after they are processed.
Delete Hadoop Partitions.
Delete Hadoop Partitions.
Note that this is only possible, if every set of column names in partitionValues are valid "inits"
of this DataObject's partitions.
Every valid "init" can be produced by repeatedly removing the last element of a collection. Example: - a,b of a,b,c -> OK - a,c of a,b,c -> NOK
scala.collection.TraversableLike.init
Extract partition values from a given file path
Extract partition values from a given file path
Returns the factory that can parse this type (that is, type CO).
Returns the factory that can parse this type (that is, type CO).
Typically, implementations of this method should return the companion object of the implementing class. The companion object in turn should implement FromConfigFactory.
the factory (object) for this class.
Configure whether io.smartdatalake.workflow.action.Actions should fail if the input file(s) are missing on the file system.
Configure whether io.smartdatalake.workflow.action.Actions should fail if the input file(s) are missing on the file system.
Default is false.
Definition of fileName.
Definition of fileName. Default is an asterix to match everything. This is concatenated with the partition layout to search for files.
Create a hadoop FileSystem API handle for the provided SparkSession.
Create a hadoop FileSystem API handle for the provided SparkSession.
The Spark-Format provider to be used
The Spark-Format provider to be used
Handle class cast exception when getting objects from instance registry
Handle class cast exception when getting objects from instance registry
Constructs an Apache Spark DataFrame from the underlying file content.
Constructs an Apache Spark DataFrame from the underlying file content.
the current SparkSession.
a new DataFrame containing the data stored in the file at path
DataFrameReader
List files for given partition values
List files for given partition values
List of partition values to be filtered. If empty all files in root path of DataObject will be listed.
List of FileRefs
get partition values formatted by partition layout
get partition values formatted by partition layout
prepare paths to be searched
prepare paths to be searched
unique name of this data object
unique name of this data object
Initialize callback before writing data out to disk/sinks.
Initialize callback before writing data out to disk/sinks.
Return the InstanceRegistry parsed from the SDL configuration used for this run.
Return the InstanceRegistry parsed from the SDL configuration used for this run.
the current InstanceRegistry.
List partitions on data object's root path
List partitions on data object's root path
Metadata describing this data object.
Metadata describing this data object.
Returns the configured options for the Spark DataFrameReader/DataFrameWriter.
Returns the configured options for the Spark DataFrameReader/DataFrameWriter.
DataFrameWriter
DataFrameReader
Return a String specifying the partition layout.
Return a String specifying the partition layout.
For Hadoop the default partition layout is colname1=<value1>/colname2=<value2>/.../
partition columns for this data object
partition columns for this data object
Hadoop directory where this data object reads/writes it's files.
Hadoop directory where this data object reads/writes it's files. If it doesn't contain scheme and authority, the connections pathPrefix is applied. If pathPrefix is not defined or doesn't define scheme and authority, default schema and authority is applied. Optionally defined partitions are appended with hadoop standard partition layout to this path. Only files ending with *.parquet* are considered as data for this DataObject.
Runs operations after reading from DataObject
Runs operations after reading from DataObject
Runs operations after writing to DataObject
Runs operations after writing to DataObject
Runs operations before reading from DataObject
Runs operations before reading from DataObject
Runs operations before writing to DataObject
Runs operations before writing to DataObject
Prepare & test DataObject's prerequisits
Prepare & test DataObject's prerequisits
This runs during the "prepare" operation of the DAG.
Returns the user-defined schema for reading from the data source.
Returns the user-defined schema for reading from the data source. By default, this should return schema but it
may be customized by data objects that have a source schema and ignore the user-defined schema on read operations.
If a user-defined schema is returned, it overrides any schema inference. If no user-defined schema is set, the schema may be inferred depending on the configuration and type of data frame reader.
The schema to use for the data frame reader when reading from the source.
spark SaveMode to use when writing files, default is "overwrite"
spark SaveMode to use when writing files, default is "overwrite"
An optional DataObject user-defined schema definition.
An optional DataObject user-defined schema definition.
Some DataObjects support optional schema inference.
Specifying this attribute disables automatic schema inference. When the wrapped data source contains a source
schema, this schema attribute is ignored.
Note: This is only used by the functionality defined in CanCreateDataFrame, that is,
when reading Spark data frames from the underlying data container.
io.smartdatalake.workflow.action.Actions that bypass Spark data frames ignore the schema attribute
if it is defined.
An optional, minimal schema that a DataObject schema must have to pass schema validation.
An optional, minimal schema that a DataObject schema must have to pass schema validation.
The schema validation semantics are: - Schema A is valid in respect to a minimal schema B when B is a subset of A. This means: the whole column set of B is contained in the column set of A.
Note: This is only used by the functionality defined in CanCreateDataFrame and CanWriteDataFrame, that is,
when reading or writing Spark data frames from/to the underlying data container.
io.smartdatalake.workflow.action.Actions that bypass Spark data frames ignore the schemaMin attribute
if it is defined.
default separator for paths
default separator for paths
Optional definition of repartition operation before writing DataFrame with Spark to Hadoop.
Optional definition of repartition operation before writing DataFrame with Spark to Hadoop.
Given some FileRefs for another DataObject, translate the paths to the root path of this DataObject
Given some FileRefs for another DataObject, translate the paths to the root path of this DataObject
Validate the schema of a given Spark Data Frame df against schemaMin.
Validate the schema of a given Spark Data Frame df against schemaMin.
The data frame to validate.
SchemaViolationException is the schemaMin does not validate.
Writes the provided DataFrame to the filesystem.
Writes the provided DataFrame to the filesystem.
The partitionValues attribute is used to partition the output by the given columns on the file system.
the DataFrame to write to the file system.
The partition layout to write.
the current SparkSession.
DataFrameWriter.partitionBy
A io.smartdatalake.workflow.dataobject.DataObject backed by an Apache Hive data source.
It manages read and write access and configurations required for io.smartdatalake.workflow.action.Actions to work on Parquet formatted files.
Reading and writing details are delegated to Apache Spark org.apache.spark.sql.DataFrameReader and org.apache.spark.sql.DataFrameWriter respectively.
unique name of this data object
Hadoop directory where this data object reads/writes it's files. If it doesn't contain scheme and authority, the connections pathPrefix is applied. If pathPrefix is not defined or doesn't define scheme and authority, default schema and authority is applied. Optionally defined partitions are appended with hadoop standard partition layout to this path. Only files ending with *.parquet* are considered as data for this DataObject.
partition columns for this data object
spark SaveMode to use when writing files, default is "overwrite"
Optional definition of repartition operation before writing DataFrame with Spark to Hadoop.
override connections permissions for files created with this connection
optional id of io.smartdatalake.workflow.connection.HadoopFileConnection
Metadata describing this data object.
org.apache.spark.sql.DataFrameWriter
org.apache.spark.sql.DataFrameReader