Class/Object

io.smartdatalake.workflow

DAG

Related Docs: object DAG | package workflow

Permalink

case class DAG[N <: DAGNode] extends SmartDataLakeLogger with Product with Serializable

A generic directed acyclic graph (DAG) consisting of DAGNodes interconnected with directed DAGEdges.

This DAG can have multiple start nodes and multiple end nodes as well as disconnected parts.

Linear Supertypes
Serializable, Serializable, Product, Equals, SmartDataLakeLogger, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. DAG
  2. Serializable
  3. Serializable
  4. Product
  5. Equals
  6. SmartDataLakeLogger
  7. AnyRef
  8. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  5. def buildTaskGraph[A <: DAGResult](eventListener: DAGEventListener[N])(operation: (DAGNode, Seq[A]) ⇒ Seq[A])(implicit scheduler: Scheduler): Task[Seq[Try[A]]]

    Permalink

    Build a single task that is a combination of node computations (node tasks) executed in the topological order defined by the DAG.

    Build a single task that is a combination of node computations (node tasks) executed in the topological order defined by the DAG.

    Monix tasks is a library for lazy cancelable futures

    eventListener

    A instance of DAGEventListener to be notified about progress of DAG execution

    operation

    A function that computes the result (DAGResult) for the current node, given the result of its predecessors given.

    scheduler

    The Scheduler to use for Tasks.

    Note

    This method does not trigger any execution but builds a complex collection of tasks and synchronization boundaries that specify a correct order of execution as defined by the DAG. The computation only runs when the returning task is scheduled for execution.

    See also

    https://medium.com/@sderosiaux/are-scala-futures-the-past-69bd62b9c001

  6. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  7. val endNodes: Seq[DAGNode]

    Permalink

    End points for DAG execution.

  8. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  9. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  10. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  11. def getNodes: Seq[N]

    Permalink

    Get the nodes of this DAG Note: we must filter nodes which are not of the preliminary node type of this DAG, e.g.

    Get the nodes of this DAG Note: we must filter nodes which are not of the preliminary node type of this DAG, e.g. InitDAGNode

    returns

    list of nodes of the preliminary node type of this DAG

  12. def getResultTask[A <: DAGResult](tasks: Map[NodeId, Task[Try[Seq[A]]]], nodeId: NodeId, resultId: String): Task[Try[A]]

    Permalink

    Create a task that fetches a specific DAGResult produced by the node with id nodeId.

    Create a task that fetches a specific DAGResult produced by the node with id nodeId.

    A

    The result type - supertype DAGResult ensures it has a resultId defined.

    tasks

    A map of tasks that compute (future) results indexed by node.

    nodeId

    The id of the producing node.

    resultId

    The id of the result to search among all nodes results.

    returns

    The task that computes the result specified by nodeId and resultId.

  13. val incomingEdgesMap: Map[NodeId, Seq[DAGEdge]]

    Permalink

    A lookup table for incoming edges indexed by node id.

  14. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  15. lazy val logger: Logger

    Permalink
    Attributes
    protected
    Definition Classes
    SmartDataLakeLogger
  16. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  17. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  18. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  19. val sortedNodes: Seq[DAGNode]

    Permalink

    All nodes of the DAG sorted in topological order.

  20. val startNodes: Seq[DAGNode]

    Permalink

    Starting points for DAG execution.

  21. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  22. def toString(): String

    Permalink

    Create text representation of the graph by using an ASCII graph layout library

    Create text representation of the graph by using an ASCII graph layout library

    Definition Classes
    DAG → AnyRef → Any
  23. def trySeqToSeqTry[A](trySeq: Try[Seq[A]]): Seq[Try[A]]

    Permalink

    Convert a Try of a Result-List to a List of Result-Try's

  24. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  25. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  26. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from SmartDataLakeLogger

Inherited from AnyRef

Inherited from Any

Ungrouped