Pipeline

A class representing a Spark pipeline. Generally speaking, a Spark pipeline can be divided in three components:

We load a dataframe from an external source => The load effect
We transform this dataframe using Spark => The transform function
We do something with the output => The action effect

Source: The input type of the dataset
Output: The output type of the dataset
Result: The result type of the pipeline
load: The effect to create an dataset in input
transform: The whole dataset processing
action: The effect to extract a result from the dataset transformation

Linear Supertypes

Serializable, Serializable, Product, Equals, AnyRef, Any

Instance Constructors

new Pipeline(load: SIO[Dataset[Source]], transform: (Dataset[Source]) ⇒ Dataset[Output], action: (Dataset[Output]) ⇒ Task[Result])

load
The effect to create an dataset in input
transform
The whole dataset processing
action
The effect to extract a result from the dataset transformation

Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
val action: (Dataset[Output]) ⇒ Task[Result]

The effect to extract a result from the dataset transformation
final def asInstanceOf[T0]: T0

Definition Classes
Any
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
val load: SIO[Dataset[Source]]

The effect to create an dataset in input
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
def run(implicit trace: Trace): SIO[Result]

Runs the pipeline computation as a ZIO effect.
Runs the pipeline computation as a ZIO effect. You must provide a SparkSession layer to actually run the effect.
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
val transform: (Dataset[Source]) ⇒ Dataset[Output]

The whole dataset processing
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )

Related Docs: object Pipeline | package experimental

final case class Pipeline[Source, Output, Result](load: SIO[Dataset[Source]], transform: (Dataset[Source]) ⇒ Dataset[Output], action: (Dataset[Output]) ⇒ Task[Result]) extends Product with Serializable

Instance Constructors

new Pipeline(load: SIO[Dataset[Source]], transform: (Dataset[Source]) ⇒ Dataset[Output], action: (Dataset[Output]) ⇒ Task[Result])

Value Members

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: Any): Boolean

val action: (Dataset[Output]) ⇒ Task[Result]

final def asInstanceOf[T0]: T0

def clone(): AnyRef

final def eq(arg0: AnyRef): Boolean

def finalize(): Unit

final def getClass(): Class[_]

final def isInstanceOf[T0]: Boolean

val load: SIO[Dataset[Source]]

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

def run(implicit trace: Trace): SIO[Result]

final def synchronized[T0](arg0: ⇒ T0): T0

val transform: (Dataset[Source]) ⇒ Dataset[Output]

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from AnyRef

Inherited from Any

Ungrouped