Spark DataFrame to be processed. DataFrame should not be saved to state (@transient).
id of the DataObject this SubFeed corresponds to
Values of Partitions transported by this SubFeed
true if this subfeed is a start node of the dag
true if this subfeed only contains a dummy DataFrame. Dummy DataFrames can be used for validating the lineage in init phase, but not for the exec phase.
a spark sql filter expression. This is used by SparkIncrementalMode.
Break lineage.
Break lineage. This means to discard an existing DataFrame or List of FileRefs, so that it is requested again from the DataObject. This is usable to break long DataFrame Lineages over multiple Actions and instead reread the data from an intermediate table
Spark DataFrame to be processed.
Spark DataFrame to be processed. DataFrame should not be saved to state (@transient).
id of the DataObject this SubFeed corresponds to
id of the DataObject this SubFeed corresponds to
a spark sql filter expression.
a spark sql filter expression. This is used by SparkIncrementalMode.
true if this subfeed is a start node of the dag
true if this subfeed is a start node of the dag
true if this subfeed only contains a dummy DataFrame.
true if this subfeed only contains a dummy DataFrame. Dummy DataFrames can be used for validating the lineage in init phase, but not for the exec phase.
Values of Partitions transported by this SubFeed
Values of Partitions transported by this SubFeed
A SparkSubFeed is used to transport DataFrame's between Actions.
Spark DataFrame to be processed. DataFrame should not be saved to state (@transient).
id of the DataObject this SubFeed corresponds to
Values of Partitions transported by this SubFeed
true if this subfeed is a start node of the dag
true if this subfeed only contains a dummy DataFrame. Dummy DataFrames can be used for validating the lineage in init phase, but not for the exec phase.
a spark sql filter expression. This is used by SparkIncrementalMode.