Trait/Object

com.spotify.scio.io

ScioIO

Related Docs: object ScioIO | package io

Permalink

trait ScioIO[T] extends AnyRef

Base trait for all Read/Write IO classes. Every IO connector must implement this. This trait has two abstract implicit methods #read, #write that need to be implemented in every subtype. Look at the com.spotify.scio.io.TextIO subclass for a reference implementation. IO connectors can choose to override #readTest and #writeTest if custom test logic is necessary.

Linear Supertypes
AnyRef, Any
Known Subclasses
BinaryIO, CustomIO, DatastoreIO, PubsubIO, ReadIO, TestIO, TextIO
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. ScioIO
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Type Members

  1. abstract type ReadP

    Permalink

  2. abstract type WriteP

    Permalink

Abstract Value Members

  1. abstract def read(sc: ScioContext, params: ReadP): SCollection[T]

    Permalink
    Attributes
    protected

  2. abstract def tap(read: ReadP): Tap[T]

    Permalink

    Write options also return a ClosedTap.

    Write options also return a ClosedTap. Once the job completes you can open the Tap. Tap abstracts away the logic of reading the dataset directly as an Iterator[T] or re-opening it in another ScioContext. The Future is complete once the job finishes. This can be used to do light weight pipeline orchestration e.g. WordCountOrchestration.scala.

  3. abstract val tapT: TapT[T]

    Permalink

  4. abstract def write(data: SCollection[T], params: WriteP): Tap[T]

    Permalink
    Attributes
    protected

Concrete Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  4. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any

  5. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  6. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  7. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  8. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  9. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any

  10. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  11. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any

  12. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  13. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef

  14. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef

  15. def readTest(sc: ScioContext, params: ReadP)(implicit coder: Coder[T]): SCollection[T]

    Permalink
    Attributes
    protected

  16. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef

  17. def testId: String

    Permalink

  18. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any

  19. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  20. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  21. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  22. def writeTest(data: SCollection[T], params: WriteP)(implicit coder: Coder[T]): Tap[T]

    Permalink
    Attributes
    protected

Inherited from AnyRef

Inherited from Any

Ungrouped