Class/Object

io.smartdatalake.workflow

FileSubFeed

Related Docs: object FileSubFeed | package workflow

Permalink

case class FileSubFeed(fileRefs: Option[Seq[FileRef]], dataObjectId: DataObjectId, partitionValues: Seq[PartitionValues], processedInputFileRefs: Option[Seq[FileRef]] = None) extends SubFeed with Product with Serializable

A FileSubFeed is used to transport references to files between Actions.

fileRefs

path to files to be processed

dataObjectId

id of the DataObject this SubFeed corresponds to

partitionValues

Values of Partitions transported by this SubFeed

processedInputFileRefs

used to remember processed input FileRef's for post processing (e.g. delete after read)

Linear Supertypes
Serializable, Serializable, Product, Equals, SubFeed, DAGResult, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. FileSubFeed
  2. Serializable
  3. Serializable
  4. Product
  5. Equals
  6. SubFeed
  7. DAGResult
  8. AnyRef
  9. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new FileSubFeed(fileRefs: Option[Seq[FileRef]], dataObjectId: DataObjectId, partitionValues: Seq[PartitionValues], processedInputFileRefs: Option[Seq[FileRef]] = None)

    Permalink

    fileRefs

    path to files to be processed

    dataObjectId

    id of the DataObject this SubFeed corresponds to

    partitionValues

    Values of Partitions transported by this SubFeed

    processedInputFileRefs

    used to remember processed input FileRef's for post processing (e.g. delete after read)

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  4. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any

  5. def breakLineage(): FileSubFeed

    Permalink

    Break lineage.

    Break lineage. This means to discard an existing DataFrame or List of FileRefs, so that it is requested again from the DataObject. This is usable to break long DataFrame Lineages over multiple Actions and instead reread the data from an intermediate table

    Definition Classes
    FileSubFeed → SubFeed

  6. def clearPartitionValues(): FileSubFeed

    Permalink
    Definition Classes
    FileSubFeed → SubFeed

  7. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  8. val dataObjectId: DataObjectId

    Permalink

    id of the DataObject this SubFeed corresponds to

    id of the DataObject this SubFeed corresponds to

    Definition Classes
    FileSubFeed → SubFeed

  9. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  10. val fileRefs: Option[Seq[FileRef]]

    Permalink

    path to files to be processed

  11. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  12. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any

  13. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any

  14. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  15. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef

  16. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef

  17. val partitionValues: Seq[PartitionValues]

    Permalink

    Values of Partitions transported by this SubFeed

    Values of Partitions transported by this SubFeed

    Definition Classes
    FileSubFeed → SubFeed

  18. val processedInputFileRefs: Option[Seq[FileRef]]

    Permalink

    used to remember processed input FileRef's for post processing (e.g.

    used to remember processed input FileRef's for post processing (e.g. delete after read)

  19. def resultId: String

    Permalink
    Definition Classes
    SubFeed → DAGResult

  20. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef

  21. def updatePartitionValues(partitions: Seq[String]): FileSubFeed

    Permalink
    Definition Classes
    FileSubFeed → SubFeed

  22. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  23. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  24. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from SubFeed

Inherited from DAGResult

Inherited from AnyRef

Inherited from Any

Ungrouped