An ETL Pipeline submits each of its stages in order to Spark.
An ETL Pipeline submits each of its stages in order to Spark. The submission order will match the order declared in the pipeline configuration. Spark may alter the order of evaulation once it has analyzed the DAG. The run method is designed to mimic a basic interpreter, we new stage types are created they need to be added here in order for them to be executed.
It would be possible to extend this process to support other compute engines as the submitted stages are not specific to Spark.