RawKinesisStreamConsumer

A Consumer of data from a Kinesis Stream.

If multiple shards exist, the processBatch function here will be executed from different threads, possibly concurrently.

streamName: The name of the Kinesis stream to consume
config: The configuration
executorService: to use for processing records - this is passed to, and used by, the underlying Java Amazon library. Defaults to Executors.newCachedThreadPool()
processBatch: A function that is called for each "batch" of records received from the kinesis stream. This function may be called from different threads (for different shards), and possibly even concurrently. Note this function should complete in less time than the configured KinesisConsumerConfig.leaseFailoverTime as otherwise the lease could expire while processing is proceeding.

Linear Supertypes

Serializable, Serializable, Product, Equals, Loggable, AnyRef, Any

Instance Constructors

new RawKinesisStreamConsumer(streamName: String, config: KinesisConsumerConfig, executorService: ExecutorService = Executors.newCachedThreadPool())(processBatch: (Seq[Record], Checkpoint) ⇒ Unit)

streamName
The name of the Kinesis stream to consume
config
The configuration
executorService
to use for processing records - this is passed to, and used by, the underlying Java Amazon library. Defaults to Executors.newCachedThreadPool()
processBatch
A function that is called for each "batch" of records received from the kinesis stream. This function may be called from different threads (for different shards), and possibly even concurrently. Note this function should complete in less time than the configured KinesisConsumerConfig.leaseFailoverTime as otherwise the lease could expire while processing is proceeding.

Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def asInstanceOf[T0]: T0

Definition Classes
Any
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
val config: KinesisConsumerConfig

The configuration
def debug(message: ⇒ String, ex: Throwable): Unit

Attributes
protected
Definition Classes
Loggable
def debug(message: ⇒ String): Unit

Attributes
protected
Definition Classes
Loggable
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def error(message: ⇒ String, ex: Throwable): Unit

Attributes
protected
Definition Classes
Loggable
def error(message: ⇒ String): Unit

Attributes
protected
Definition Classes
Loggable
def error(ex: Throwable): Unit

Attributes
protected
Definition Classes
Loggable
val executorService: ExecutorService

to use for processing records - this is passed to, and used by, the underlying Java Amazon library.
to use for processing records - this is passed to, and used by, the underlying Java Amazon library. Defaults to Executors.newCachedThreadPool()
def fatal(message: ⇒ String, ex: Throwable): Unit

Attributes
protected
Definition Classes
Loggable
def fatal(message: ⇒ String): Unit

Attributes
protected
Definition Classes
Loggable
def fatal(ex: Throwable): Unit

Attributes
protected
Definition Classes
Loggable
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
def info(message: ⇒ String, ex: Throwable): Unit

Attributes
protected
Definition Classes
Loggable
def info(message: ⇒ String): Unit

Attributes
protected
Definition Classes
Loggable
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
def onShardShutdown(func: (Checkpoint, ShardId) ⇒ Unit): Unit

Register a function to be called when a shard is being terminated.
Register a function to be called when a shard is being terminated.
This can be useful to allow applications decide their own checkpoint workflow, rather than checkpointing every "batch".
If no function is registered here, then an automatic checkpoint is issued on shard termination.
Only a single registration is allowed.
func
A function that will be called on shard termination - this is passed a Checkpoint and the ID of the shard being terminated.
def shutdown(): Unit

Initiate the shutdown of processing the stream.
def start(): Unit

Start reading data from the stream.
val streamName: String

The name of the Kinesis stream to consume
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
def trace(message: ⇒ String, ex: Throwable): Unit

Attributes
protected
Definition Classes
Loggable
def trace(message: ⇒ String): Unit

Attributes
protected
Definition Classes
Loggable
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
def warn(message: ⇒ String, ex: Throwable): Unit

Attributes
protected
Definition Classes
Loggable
def warn(message: ⇒ String): Unit

Attributes
protected
Definition Classes
Loggable

case class RawKinesisStreamConsumer(streamName: String, config: KinesisConsumerConfig, executorService: ExecutorService = Executors.newCachedThreadPool())(processBatch: (Seq[Record], Checkpoint) ⇒ Unit) extends Loggable with Product with Serializable

Instance Constructors

new RawKinesisStreamConsumer(streamName: String, config: KinesisConsumerConfig, executorService: ExecutorService = Executors.newCachedThreadPool())(processBatch: (Seq[Record], Checkpoint) ⇒ Unit)

Value Members

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: Any): Boolean

final def asInstanceOf[T0]: T0

def clone(): AnyRef

val config: KinesisConsumerConfig

def debug(message: ⇒ String, ex: Throwable): Unit

def debug(message: ⇒ String): Unit

final def eq(arg0: AnyRef): Boolean

def error(message: ⇒ String, ex: Throwable): Unit

def error(message: ⇒ String): Unit

def error(ex: Throwable): Unit

val executorService: ExecutorService

def fatal(message: ⇒ String, ex: Throwable): Unit

def fatal(message: ⇒ String): Unit

def fatal(ex: Throwable): Unit

def finalize(): Unit

final def getClass(): Class[_]

def info(message: ⇒ String, ex: Throwable): Unit

def info(message: ⇒ String): Unit

final def isInstanceOf[T0]: Boolean

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

def onShardShutdown(func: (Checkpoint, ShardId) ⇒ Unit): Unit

def shutdown(): Unit

def start(): Unit

val streamName: String

final def synchronized[T0](arg0: ⇒ T0): T0

def trace(message: ⇒ String, ex: Throwable): Unit

def trace(message: ⇒ String): Unit

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

def warn(message: ⇒ String, ex: Throwable): Unit

def warn(message: ⇒ String): Unit

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from Loggable

Inherited from AnyRef

Inherited from Any

Ungrouped