Package

org.apache.gearpump.streaming

source

Permalink

package source

Visibility

Public
All

Type Members

trait DataSource extends Serializable

Interface to implement custom source where data is read into the system.
Interface to implement custom source where data is read into the system. a DataSource could be a message queue like kafka or simply data generation source.
An example would be like
```
GenMsgSource extends DataSource {

  def open(context: TaskContext, startTime: TimeStamp): Unit = {}

  def read(context: TaskContext): Message = {
    Message("message")
  }

  def close(): Unit = {}
}
```
subclass is required to be serializable
class DataSourceTask[IN, OUT] extends Task

Default Task container for org.apache.gearpump.streaming.source.DataSource that reads from DataSource in batch See org.apache.gearpump.streaming.source.DataSourceProcessor for its usage
Default Task container for org.apache.gearpump.streaming.source.DataSource that reads from DataSource in batch See org.apache.gearpump.streaming.source.DataSourceProcessor for its usage
DataSourceTask calls:
- DataSource.open() in onStart and pass in org.apache.gearpump.streaming.task.TaskContext and application start time
- DataSource.read() in each onNext, which reads a batch of messages
- DataSource.close() in onStop
case class Watermark(instant: Instant) extends Product with Serializable

message used by source task to report source watermark.

Value Members

object DataSourceConfig
object DataSourceProcessor

Utility that helps user to create a DAG starting with DataSourceTask user should pass in a DataSource
Utility that helps user to create a DAG starting with DataSourceTask user should pass in a DataSource
Here is an example to build a DAG that reads from Kafka source followed by word count
```
val source = new KafkaSource()
val sourceProcessor =  DataSourceProcessor(source, 1)
val split = Processor[Split](1)
val sum = Processor[Sum](1)
val dag = sourceProcessor ~> split ~> sum
```
object Watermark extends Serializable

Ungrouped