class PipelineBuilder extends Logging
Mutable class to build a Pipeline. A pipeline represents a set of stages interconnected with a org.codefeedr.buffer.Buffer. This pipeline gets translated into a DirectedAcyclicGraph.
This builder allows for setting the following properties: - Name of the pipeline. - Type of buffer (e.g. Kafka) and properties for this buffer. - Type of pipeline: sequential or DAG (nonsequential). - Properties for all the stages. - KeyManager for all the stages.
- Alphabetic
- By Inheritance
- PipelineBuilder
- Logging
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Instance Constructors
- new PipelineBuilder()
Value Members
-
def
addParents[In <: Serializable, Mid1 <: Serializable, Mid2 <: Serializable, Out <: Serializable](stage: Stage[In, Mid1], parent: Stage[Mid2, Out]): PipelineBuilder
Add a parent to a node.
Add a parent to a node.
- In
Incoming type from 'start' stage.
- Mid1
Intermediate type which is output for 'start' stage.
- Mid2
Intermediate type which is output for 'end' stage.
- Out
Outgoing type for 'end' stage.
- stage
The child stage.
- parent
The parent stage.
- returns
The builder instance.
-
def
addParents[In <: Serializable, Out <: Serializable](stage: Stage[In, Out], parents: StageList): PipelineBuilder
Add multiple parents (stages) in given ordered list.
Add multiple parents (stages) in given ordered list.
- In
Incoming type of the stage.
- Out
Outgoing type of the stage.
- stage
Stage to add the parents to.
- parents
Parents to connect to the child
stage.- returns
The builder instance.
-
def
append[In <: Serializable](trans: (DataStream[In]) ⇒ Any)(implicit arg0: ClassTag[In], arg1: scala.reflect.api.JavaUniverse.TypeTag[In]): PipelineBuilder
Append a stage created from an output function.
Append a stage created from an output function.
- In
Type of the DataStream.
- trans
Output function.
- returns
The builder instance.
-
def
append[In <: Serializable, Out <: Serializable](trans: (DataStream[In]) ⇒ DataStream[Out])(implicit arg0: ClassTag[In], arg1: scala.reflect.api.JavaUniverse.TypeTag[In], arg2: ClassTag[Out], arg3: scala.reflect.api.JavaUniverse.TypeTag[Out]): PipelineBuilder
Append a stage created from a transform function.
Append a stage created from a transform function.
- In
Incoming type of the DataStream.
- Out
Outgoing type of the DataStream.
- trans
Transform function from one type to another.
- returns
The builder instance.
-
def
append[IN <: Serializable, OUT <: Serializable](stage: Stage[IN, OUT]): PipelineBuilder
Append a Stage in a sequential pipeline.
Append a Stage in a sequential pipeline.
- IN
Incoming type of the stage.
- OUT
Outgoing type of the stage.
- stage
The new stage to add.
- returns
This builder instance.
-
def
appendSource[In <: Serializable](input: (StreamExecutionEnvironment) ⇒ DataStream[In])(implicit arg0: ClassTag[In], arg1: scala.reflect.api.JavaUniverse.TypeTag[In]): PipelineBuilder
Append a stage created from an input function.
Append a stage created from an input function.
- In
Type of the DataStream.
- input
Input function.
- returns
The builder instance.
-
def
build(): Pipeline
Builds a pipeline from the builder configuration.
Builds a pipeline from the builder configuration.
- returns
A new Pipeline.
- Exceptions thrown
EmptyPipelineExceptionWhen no pipeline is defined (graph is empty).
-
def
disablePipelineVerification(): PipelineBuilder
Disables the pipeline verification.
Disables the pipeline verification. This is not recommended, it allows for nasty pipelines.
- returns
The builder instance.
-
def
edge(from: List[Stage[_ <: Serializable, _ <: Serializable]], to: List[Stage[_ <: Serializable, _ <: Serializable]]): Unit
Links multiple stages to multiple stages.
Links multiple stages to multiple stages.
- from
The list of stages to start from.
- to
The list of stages to direct to.
-
def
edge[In <: Serializable, Out <: Serializable](from: List[Stage[_ <: Serializable, _ <: Serializable]], to: Stage[In, Out]): PipelineBuilder
Directs multiple stages to one stage.
Directs multiple stages to one stage.
- from
The list of stages to start from.
- to
The stage to end.
- returns
The builder instance.
-
def
edge[In <: Serializable, Out <: Serializable](from: Stage[In, Out], to: List[Stage[_ <: Serializable, _ <: Serializable]]): PipelineBuilder
Directs a stage to multiple other stages.
Directs a stage to multiple other stages.
- from
The stage to start from.
- to
The list of stages to direct to.
- returns
The builder instance.
-
def
edge[In <: Serializable, Mid1 <: Serializable, Mid2 <: Serializable, Out <: Serializable](from: Stage[In, Mid1], to: Stage[Mid2, Out]): PipelineBuilder
Creates an edge between two stages.
Creates an edge between two stages. The 'to' must not already have a parent.
If the graph is not configured yet (has no nodes), the graph is switched to a DAG automatically. If it was already configured as sequential, it will throw an IllegalStateException.
- In
Incoming type from 'start' stage.
- Mid1
Intermediate type which is output for 'start' stage.
- Mid2
Intermediate type which is output for 'end' stage.
- Out
Outgoing type for 'end' stage.
- from
The stage to start from.
- to
The stage to end at.
-
def
enableCheckpointing(interval: Long): PipelineBuilder
Enable checkpointing for this pipeline.
Enable checkpointing for this pipeline.
- interval
The interval to checkpoint on.
- returns
This builder instance.
-
def
enableCheckpointing(interval: Long, checkpointingMode: CheckpointingMode): PipelineBuilder
Enable checkpointing for this pipeline.
Enable checkpointing for this pipeline.
- interval
The interval to checkpoint on.
- checkpointingMode
The checkpointingmode (exactly once or at least once).
- returns
This builder instance.
-
def
getBufferType: BufferType
Get the type of the buffer.
Get the type of the buffer.
- returns
The buffer type.
-
def
getPipelineType: PipelineType
Get the type of the pipeline.
Get the type of the pipeline.
- returns
Type of pipeline: Sequential or DAG.
-
def
setBufferProperty(key: String, value: String): PipelineBuilder
Set a buffer property.
Set a buffer property. A buffer property is generic for all buffers (like the serializer).
- key
Key of the property.
- value
Value of the property.
- returns
This builder instance.
-
def
setBufferType(bufferType: BufferType): PipelineBuilder
Set the type of the buffer.
Set the type of the buffer.
- bufferType
The new buffer type.
- returns
The builder instance.
-
def
setCheckpointingMode(checkpointingMode: CheckpointingMode): PipelineBuilder
Sets the CheckpointMode for this pipeline.
Sets the CheckpointMode for this pipeline. Note: this method does not enable checkpointing.
- checkpointingMode
The checkpointingmode (exactly once or at least once).
-
def
setKeyManager(km: KeyManager): PipelineBuilder
Set a KeyManager.
Set a KeyManager. A key manager handles API key management and can be accessed by every stage.
- km
The key manager instance.
- returns
This builder instance.
-
def
setPipelineName(name: String): PipelineBuilder
Set name of the pipeline.
Set name of the pipeline.
- name
Name of the pipeline.
- returns
This builder instance.
-
def
setPipelineType(pipelineType: PipelineType): PipelineBuilder
Set the type of the pipeline.
Set the type of the pipeline.
- pipelineType
Type of the pipeline.
- returns
This builder instance.
-
def
setRestartStrategy(strategy: RestartStrategyConfiguration): PipelineBuilder
Set the RestartStrategy of the whole pipeline.
Set the RestartStrategy of the whole pipeline.
- strategy
The strategy.
- returns
The builder instance.
-
def
setSerializer(serializer: SerializerType): PipelineBuilder
Sets the serializer type for the buffer.
Sets the serializer type for the buffer.
- serializer
The serializer type (which is basically a string).
- returns
This builder instance.
-
def
setStageProperty(stage: Stage[_, _], key: String, value: String): PipelineBuilder
Set a stage property.
Set a stage property.
- key
Key of the property.
- value
Value of the property.
- returns
This builder instance.
-
def
setStateBackend(stateBackend: StateBackend): PipelineBuilder
Sets the StateBackend of the whole pipeline.
Sets the StateBackend of the whole pipeline.
- stateBackend
the statebackend.
- returns
The builder instance.
-
def
setStreamTimeCharacteristic(timeCharacteristic: TimeCharacteristic): PipelineBuilder
Set TimeCharacteristic of the whole pipeline.
Set TimeCharacteristic of the whole pipeline.
- timeCharacteristic
The TimeCharacterisic.
- returns
This builder instance.