path to files to be processed
id of the DataObject this SubFeed corresponds to
Values of Partitions transported by this SubFeed
true if this subfeed is a start node of the dag
true if this subfeed is the result of a skipped action
store mapping of input to output file references. This is also used for post processing (e.g. delete after read).
Break lineage.
Break lineage. This means to discard an existing DataFrame or List of FileRefs, so that it is requested again from the DataObject. On one side this is usable to break long DataFrame Lineages over multiple Actions and instead reread the data from an intermediate table. On the other side it is needed if partition values or filter condition are changed.
id of the DataObject this SubFeed corresponds to
id of the DataObject this SubFeed corresponds to
store mapping of input to output file references.
store mapping of input to output file references. This is also used for post processing (e.g. delete after read).
path to files to be processed
true if this subfeed is a start node of the dag
true if this subfeed is a start node of the dag
true if this subfeed is the result of a skipped action
true if this subfeed is the result of a skipped action
Values of Partitions transported by this SubFeed
Values of Partitions transported by this SubFeed
A FileSubFeed is used to transport references to files between Actions.
path to files to be processed
id of the DataObject this SubFeed corresponds to
Values of Partitions transported by this SubFeed
true if this subfeed is a start node of the dag
true if this subfeed is the result of a skipped action
store mapping of input to output file references. This is also used for post processing (e.g. delete after read).