experimental

Type Members

1. final case class Pipeline[Source, Output, Result](load: SIO[Dataset[Source]], transform: (Dataset[Source]) ⇒ Dataset[Output], action: (Dataset[Output]) ⇒ Task[Result]) extends Product with Serializable

   

   A class representing a Spark pipeline.

   A class representing a Spark pipeline. Generally speaking, a Spark pipeline can be divided in three components:

   • We load a dataframe from an external source => The load effect
   • We transform this dataframe using Spark => The transform function
   • We do something with the output => The action effect

   Source

   The input type of the dataset

   Output

   The output type of the dataset

   Result

   The result type of the pipeline

   load

   The effect to create an dataset in input

   transform

   The whole dataset processing

   action

   The effect to extract a result from the dataset transformation