Class

io.smartdatalake.workflow

ActionPipelineContext

Related Doc: package workflow

Permalink

case class ActionPipelineContext(feed: String, application: String, runId: Int, attemptId: Int, instanceRegistry: InstanceRegistry, referenceTimestamp: Option[LocalDateTime] = None, appConfig: SmartDataLakeBuilderConfig, runStartTime: LocalDateTime = LocalDateTime.now(), attemptStartTime: LocalDateTime = LocalDateTime.now(), simulation: Boolean = false, phase: ExecutionPhase = ExecutionPhase.Prepare, dataFrameReuseStatistics: Map[(DataObjectId, Seq[PartitionValues]), Seq[ActionId]] = mutable.Map()) extends SmartDataLakeLogger with Product with Serializable

ActionPipelineContext contains start and runtime information about a SmartDataLake run.

feed

feed selector of the run

application

application name of the run

runId

runId of the run. Stays 1 if recovery is not enabled.

attemptId

attemptId of the run. Stays 1 if recovery is not enabled.

instanceRegistry

registry of all SmartDataLake objects parsed from the config

referenceTimestamp

timestamp used as reference in certain actions (e.g. HistorizeAction)

appConfig

the command line parameters parsed into a SmartDataLakeBuilderConfig object

runStartTime

start time of the run

attemptStartTime

start time of attempt

simulation

true if this is a simulation run

phase

current execution phase

dataFrameReuseStatistics

Counter how many times a DataFrame of a SparkSubFeed is reused by an Action later in the pipeline. The counter is increased during ExecutionPhase.Init when preparing the SubFeeds for an Action and it is decreased in ExecutionPhase.Exec to unpersist the DataFrame after there is no need for it anymore.

Annotations
@DeveloperApi()
Linear Supertypes
Serializable, Serializable, Product, Equals, SmartDataLakeLogger, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. ActionPipelineContext
  2. Serializable
  3. Serializable
  4. Product
  5. Equals
  6. SmartDataLakeLogger
  7. AnyRef
  8. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new ActionPipelineContext(feed: String, application: String, runId: Int, attemptId: Int, instanceRegistry: InstanceRegistry, referenceTimestamp: Option[LocalDateTime] = None, appConfig: SmartDataLakeBuilderConfig, runStartTime: LocalDateTime = LocalDateTime.now(), attemptStartTime: LocalDateTime = LocalDateTime.now(), simulation: Boolean = false, phase: ExecutionPhase = ExecutionPhase.Prepare, dataFrameReuseStatistics: Map[(DataObjectId, Seq[PartitionValues]), Seq[ActionId]] = mutable.Map())

    Permalink

    feed

    feed selector of the run

    application

    application name of the run

    runId

    runId of the run. Stays 1 if recovery is not enabled.

    attemptId

    attemptId of the run. Stays 1 if recovery is not enabled.

    instanceRegistry

    registry of all SmartDataLake objects parsed from the config

    referenceTimestamp

    timestamp used as reference in certain actions (e.g. HistorizeAction)

    appConfig

    the command line parameters parsed into a SmartDataLakeBuilderConfig object

    runStartTime

    start time of the run

    attemptStartTime

    start time of attempt

    simulation

    true if this is a simulation run

    phase

    current execution phase

    dataFrameReuseStatistics

    Counter how many times a DataFrame of a SparkSubFeed is reused by an Action later in the pipeline. The counter is increased during ExecutionPhase.Init when preparing the SubFeeds for an Action and it is decreased in ExecutionPhase.Exec to unpersist the DataFrame after there is no need for it anymore.

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  4. val appConfig: SmartDataLakeBuilderConfig

    Permalink

    the command line parameters parsed into a SmartDataLakeBuilderConfig object

  5. val application: String

    Permalink

    application name of the run

  6. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any

  7. val attemptId: Int

    Permalink

    attemptId of the run.

    attemptId of the run. Stays 1 if recovery is not enabled.

  8. val attemptStartTime: LocalDateTime

    Permalink

    start time of attempt

  9. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  10. val dataFrameReuseStatistics: Map[(DataObjectId, Seq[PartitionValues]), Seq[ActionId]]

    Permalink

    Counter how many times a DataFrame of a SparkSubFeed is reused by an Action later in the pipeline.

    Counter how many times a DataFrame of a SparkSubFeed is reused by an Action later in the pipeline. The counter is increased during ExecutionPhase.Init when preparing the SubFeeds for an Action and it is decreased in ExecutionPhase.Exec to unpersist the DataFrame after there is no need for it anymore.

  11. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  12. val feed: String

    Permalink

    feed selector of the run

  13. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  14. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any

  15. val instanceRegistry: InstanceRegistry

    Permalink

    registry of all SmartDataLake objects parsed from the config

  16. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any

  17. lazy val logger: Logger

    Permalink
    Attributes
    protected
    Definition Classes
    SmartDataLakeLogger

  18. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  19. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef

  20. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef

  21. var phase: ExecutionPhase

    Permalink

    current execution phase

  22. val referenceTimestamp: Option[LocalDateTime]

    Permalink

    timestamp used as reference in certain actions (e.g.

    timestamp used as reference in certain actions (e.g. HistorizeAction)

  23. val runId: Int

    Permalink

    runId of the run.

    runId of the run. Stays 1 if recovery is not enabled.

  24. val runStartTime: LocalDateTime

    Permalink

    start time of the run

  25. val simulation: Boolean

    Permalink

    true if this is a simulation run

  26. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef

  27. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  28. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  29. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from SmartDataLakeLogger

Inherited from AnyRef

Inherited from Any

Ungrouped