io.gearpump.streaming.source

DataSource

Related Doc: package source

trait DataSource extends Serializable

interface to implement custom source where data is read into the system. a DataSource could be a message queue like kafka or simply data generation source.

an example would be like

GenStringSource extends DataSource {

  def open(context: TaskContext, startTime: Option[TimeStamp]): Unit = {}

  def read(batchSize: Int): List[Message] = {
    List.fill(batchSize)(Message("message"))
  }

  def close(): Unit = {}
}

subclass is required to be serializable

Linear Supertypes
Serializable, AnyRef, Any
Known Subclasses
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. DataSource
  2. Serializable
  3. AnyRef
  4. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Abstract Value Members

  1. abstract def close(): Unit

    close connection to data source invoked in onStop() method of Task

  2. abstract def open(context: TaskContext, startTime: Option[TimeStamp]): Unit

    open connection to data source invoked in onStart() method of Task

    open connection to data source invoked in onStart() method of Task

    context

    is the task context at runtime

    startTime

    is the start time of system

  3. abstract def read(batchSize: Int): List[Message]

    read a number of messages from data source.

    read a number of messages from data source. invoked in each onNext() method of Task

    batchSize

    max number of messages to read

    returns

    a list of messages wrapped in Message

Concrete Value Members

  1. final def !=(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  5. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  6. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  7. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  8. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  9. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any
  10. def hashCode(): Int

    Definition Classes
    AnyRef → Any
  11. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  12. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  13. final def notify(): Unit

    Definition Classes
    AnyRef
  14. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  15. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  16. def toString(): String

    Definition Classes
    AnyRef → Any
  17. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  18. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  19. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped