A generic directed acyclic graph (DAG) consisting of DAGNodes interconnected with directed DAGEdges.
A FileSubFeed is used to transport references to files between Actions.
A FileSubFeed is used to transport references to files between Actions.
path to files to be processed
id of the DataObject this SubFeed corresponds to
Values of Partitions transported by this SubFeed
used to remember processed input FileRef's for post processing (e.g. delete after read)
A InitSubFeed is used to initialize first Nodes of a DAG.
A InitSubFeed is used to initialize first Nodes of a DAG.
id of the DataObject this SubFeed corresponds to
Values of Partitions transported by this SubFeed
Exception to signal that a configured pipeline can't be executed properly
A SparkSubFeed is used to transport DataFrame's between Actions.
A SparkSubFeed is used to transport DataFrame's between Actions.
Spark DataFrame to be processed. DataFrame should not be saved to state (@transient).
id of the DataObject this SubFeed corresponds to
Values of Partitions transported by this SubFeed
A SubFeed transports references to data between Actions.
A SubFeed transports references to data between Actions. Data can be represented by different technologies like Files or DataFrame.
A generic directed acyclic graph (DAG) consisting of DAGNodes interconnected with directed DAGEdges.
This DAG can have multiple start nodes and multiple end nodes as well as disconnected parts.