class Pipeline extends Logging with HasRegistry[Stage] with HasDescription with Identifiable with HasBenchmark with HasDiagram
Pipeline is a complete data transformation workflow.
- Annotations
- @Evolving()
- Alphabetic
- By Inheritance
- Pipeline
- HasDiagram
- HasBenchmark
- Identifiable
- HasDescription
- HasRegistry
- Logging
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Instance Constructors
- new Pipeline()
Value Members
-
final
def
!=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
##(): Int
- Definition Classes
- AnyRef → Any
-
final
def
==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
val
_benchmark: Option[Boolean]
- Attributes
- protected
- Definition Classes
- HasBenchmark
-
def
addStage(stage: Stage): Pipeline.this.type
Add a new stage to the pipeline
Add a new stage to the pipeline
- stage
stage object
-
def
addStage[T <: Factory[_]](constructorArgs: Array[Any] = Array.empty, persistence: Boolean = true)(implicit arg0: ClassTag[T]): Pipeline.this.type
Add a new stage containing the given factory
Add a new stage containing the given factory
- T
Class of the Factory
- constructorArgs
Arguments of the primary constructor of the factory
- persistence
if set to true, then the
write
method of the factory will be called to persist the output.
- Annotations
- @throws( ... )
- Exceptions thrown
IllegalArgumentException
this will be thrown if arguments don't match
-
def
addStage(factory: Class[_ <: Factory[_]], constructorArgs: Any*): Pipeline.this.type
Add a new stage containing the given factory
Add a new stage containing the given factory
- factory
Factory to be executed
- constructorArgs
Arguments of the primary constructor of the factory
- Annotations
- @throws( ... )
- Exceptions thrown
IllegalArgumentException
this will be thrown if arguments don't match
-
def
addStage(factory: Factory[_]): Pipeline.this.type
Add a new stage containing the given factory
Add a new stage containing the given factory
- factory
Factory to be executed
-
final
def
asInstanceOf[T0]: T0
- Definition Classes
- Any
-
def
benchmark(boo: Boolean): Pipeline.this.type
Set to true to enable the benchmarking
Set to true to enable the benchmarking
- boo
true to enable benchmarking
- returns
this object
- Definition Classes
- HasBenchmark
-
def
benchmark: Option[Boolean]
True if the benchmark will be measured, otherwise false
True if the benchmark will be measured, otherwise false
- returns
boolean
- Definition Classes
- HasBenchmark
-
def
clearRegistry(): Unit
Clear the registry
Clear the registry
- Attributes
- protected
- Definition Classes
- HasRegistry
-
def
clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
def
deliverableDispatcher: DeliverableDispatcher
Get the deliverable dispatcher of this pipeline
-
def
describe(): Pipeline.this.type
Describe the pipeline
Describe the pipeline
- Definition Classes
- Pipeline → HasDescription
-
def
diagramId: String
Get the diagram ID
Get the diagram ID
- Definition Classes
- Pipeline → HasDiagram
-
final
def
eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
equals(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
def
finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( classOf[java.lang.Throwable] )
-
def
formatDiagramId(prettyName: String, deliveryId: String, suffix: String): String
- Attributes
- protected
- Definition Classes
- HasDiagram
-
def
getBenchmarkResult: Array[BenchmarkResult]
Get the aggregated benchmark result.
Get the aggregated benchmark result.
- returns
an array of BenchmarkResult
- Definition Classes
- Pipeline → HasBenchmark
-
def
getCanonicalName: String
- Definition Classes
- Identifiable
-
final
def
getClass(): Class[_]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
def
getDeliverable(t: scala.reflect.api.JavaUniverse.Type): Array[Deliverable[_]]
Get the Deliverable of the given runtime Type
Get the Deliverable of the given runtime Type
- t
runtime type of the Deliverable's payload
-
def
getLastOutput: Any
Get the output of the last factory of the last stage
Get the output of the last factory of the last stage
- returns
an object. it has to be convert to the according type manually.
-
def
getOutput[A](cls: Class[_ <: Factory[_]]): A
Get the output of a specific Factory
Get the output of a specific Factory
- cls
class of the Factory
-
def
getPrettyName: String
- Definition Classes
- HasDescription
-
def
getRegisteredItem(uuid: UUID): Option[Stage]
For a given UUID, return the corresponding registered item
-
def
getRegistry: ListMap[UUID, Stage]
Return the registry
Return the registry
- Definition Classes
- HasRegistry
-
def
getRegistryLength: Long
Return the number of items in the current registry
Return the number of items in the current registry
- Definition Classes
- HasRegistry
- def getStage(id: Int): Option[Stage]
-
def
getTypeArgList(tpe: scala.reflect.api.JavaUniverse.Type): List[scala.reflect.api.JavaUniverse.Symbol]
- Attributes
- protected
- Definition Classes
- HasDiagram
-
def
getUUID: UUID
- Definition Classes
- Identifiable
-
def
hasRegisteredItem(uuid: UUID): Boolean
Check if the UUID exists in the registry
Check if the UUID exists in the registry
- uuid
an UUID
- returns
true if it already exists in the registry, false otherwise
- Definition Classes
- HasRegistry
-
def
hasRegisteredItem(item: Identifiable): Boolean
Check if the Identifiable exists in the registry
Check if the Identifiable exists in the registry
- item
an object that inherit io.github.setl.internal.Identifiable
- returns
true if it already exists in the registry, false otherwise
- Definition Classes
- HasRegistry
-
def
hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
final
def
isInstanceOf[T0]: Boolean
- Definition Classes
- Any
-
def
isRegistryEmpty: Boolean
Return true if the registry is empty, false otherwise
Return true if the registry is empty, false otherwise
- Definition Classes
- HasRegistry
-
def
lastRegisteredItem: Option[Stage]
Return the last registered item
Return the last registered item
- returns
if the registry is empty, None will be returned
- Definition Classes
- HasRegistry
-
def
log: Logger
- Attributes
- protected
- Definition Classes
- Logging
-
def
logDebug(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logError(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logInfo(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logName: String
- Attributes
- protected
- Definition Classes
- Logging
-
def
logTrace(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logWarning(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
final
def
ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
final
def
notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
final
def
notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
def
optimization(boolean: Boolean): Pipeline.this.type
Set to true to allow auto optimization of this pipeline.
Set to true to allow auto optimization of this pipeline. The default SimplePipelineOptimizer will be used.
- boolean
true to allow optimization, otherwise false
- returns
this pipeline
-
def
optimization(optimizer: PipelineOptimizer): Pipeline.this.type
Optimise execution with an optimizer
Optimise execution with an optimizer
- optimizer
an implementation of pipeline optimizer
- returns
this pipeline
-
def
optimization: Boolean
Boolean indicating if the pipeline will be optimized
-
def
pipelineInspector: PipelineInspector
Get the inspector of this pipeline
-
def
registerNewItem(item: Stage): Unit
Register a new io.github.setl.internal.Identifiable in registry
Register a new io.github.setl.internal.Identifiable in registry
- item
an object that inherit io.github.setl.internal.Identifiable
- returns
true if the given item is registered, false otherwise
- Attributes
- protected
- Definition Classes
- HasRegistry
- Annotations
- @throws( ... )
-
def
registerNewItems(items: Iterable[Stage]): Unit
Register multiple items
Register multiple items
- items
an io.github.setl.internal.Identifiable object
- Attributes
- protected
- Definition Classes
- HasRegistry
-
def
run(): Pipeline.this.type
Execute the pipeline
-
def
setInput[T](payload: T, deliveryId: String)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Pipeline.this.type
Set input of the pipeline
Set input of the pipeline
- T
type of payload
- payload
object to be delivered
- deliveryId
id of this delivery
-
def
setInput[T](payload: T)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Pipeline.this.type
Set input of the pipeline
Set input of the pipeline
- T
type of payload
- payload
object to be delivered
-
def
setInput[T](payload: T, consumer: Class[_ <: Factory[_]], deliveryId: String)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Pipeline.this.type
Set input of the pipeline
Set input of the pipeline
- T
type of payload
- payload
object to be delivered
- consumer
consumer of this payload
- deliveryId
id of this delivery
-
def
setInput[T](payload: T, consumer: Class[_ <: Factory[_]])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Pipeline.this.type
Set input of the pipeline
Set input of the pipeline
- T
type of payload
- payload
object to be delivered
- consumer
consumer of this payload
-
def
setInput[T](payload: T, consumer: Class[_ <: Factory[_]], consumers: Class[_ <: Factory[_]]*)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Pipeline.this.type
Set input of the pipeline
Set input of the pipeline
- T
type of payload
- payload
object to be delivered
- consumer
consumer of this payload
- consumers
other consumers of the payload
-
def
setInput[T](payload: T, deliveryId: String, consumer: Class[_ <: Factory[_]], consumers: Class[_ <: Factory[_]]*)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Pipeline.this.type
Set input of the pipeline
Set input of the pipeline
- T
type of payload
- payload
object to be delivered
- deliveryId
id of this delivery
- consumer
consumer of this payload
- consumers
other consumers of the payload
-
def
setInput[T](payload: T, consumer: Seq[Class[_ <: Factory[_]]], deliveryId: String)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Pipeline.this.type
Set input of the pipeline
Set input of the pipeline
- T
type of payload
- payload
object to be delivered
- consumer
consumer of this payload
- deliveryId
id of this delivery
-
def
setInput(deliverable: Deliverable[_]): Pipeline.this.type
Put the Deliverable to the input pool of the pipeline
Put the Deliverable to the input pool of the pipeline
- deliverable
Deliverable object
-
def
showDiagram(): Unit
Display the diagram
Display the diagram
- Definition Classes
- Pipeline → HasDiagram
-
def
stages: List[Stage]
Return all the stages of this pipeline
-
final
def
synchronized[T0](arg0: ⇒ T0): T0
- Definition Classes
- AnyRef
-
def
toDiagram: String
Generate the diagram
Generate the diagram
- Definition Classes
- Pipeline → HasDiagram
-
def
toString(): String
- Definition Classes
- AnyRef → Any
-
final
def
wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()