Offers a ZStream based interface to the Amazon Kinesis Client Library (KCL)
Ensures proper resource shutdown and failure handling
- Companion:
- class
Type members
Classlikes
Staging area for checkpoints
Staging area for checkpoints
Guarantees that the last staged record is checkpointed upon stream shutdown / interruption
Value members
Concrete methods
Similar to shardedStream
accessor but provides the recordProcessor
callback function for processing records and
takes care of checkpointing. The other difference is that it returns a ZIO of unit rather than a ZStream.
Similar to shardedStream
accessor but provides the recordProcessor
callback function for processing records and
takes care of checkpointing. The other difference is that it returns a ZIO of unit rather than a ZStream.
- Type parameters:
- R
ZIO environment type required by the
deserializer
and therecordProcessor
- T
Type of record values
- Value parameters:
- applicationName
Application name for coordinating shard leases
- checkpointBatchSize
Maximum number of records before checkpointing
- checkpointDuration
Maximum interval before checkpointing
- configureKcl
Make additional KCL Scheduler configurations
- deserializer
Deserializer for record values
- initialPosition
Position in stream to start at when there is no previous checkpoint for this application
- leaseTableName
Optionally set the lease table name - defaults to None. If not specified the
applicationName
will be used.- maxShardBufferSize
The maximum number of records per shard to store in a queue before blocking the KCL record processor until records have been dequeued. Note that the stream returned from this method will have internal chunk buffers as well.
- metricsNamespace
CloudWatch metrics namespace
- recordProcessor
A function for processing a
Record[T]
- requestShutdown
Effect that when completed will trigger a graceful shutdown of the KCL and the streams.
- streamName
Name of the Kinesis stream
- workerIdentifier
Identifier used for the worker in this application group. Used in logging and written to the lease table.
- Returns:
A ZIO that completes with Unit when record processing is stopped via requestShutdown or fails when the consumer stream fails
Implements a fake DynamicConsumer
that also offers fake checkpointing functionality that can be tracked using the
refCheckpointedList
parameter.
Implements a fake DynamicConsumer
that also offers fake checkpointing functionality that can be tracked using the
refCheckpointedList
parameter.
- Value parameters:
- refCheckpointedList
A Ref that will be used to store the checkpointed records
- shards
A ZStream that is a fake representation of a Kinesis shard. There are helper constructors to create these - see nl.vroste.zio.kinesis.client.dynamicconsumer.fake.DynamicConsumerFake.shardsFromIterables and nl.vroste.zio.kinesis.client.dynamicconsumer.fake.DynamicConsumerFake.shardsFromStreams
- Returns:
A ZLayer of the fake
DynamicConsumer
implementation
Overloaded version of above but without fake checkpointing functionality
Overloaded version of above but without fake checkpointing functionality
- Value parameters:
- shards
A ZStream that is a fake representation of a Kinesis shard. There are helper constructors to create these - see nl.vroste.zio.kinesis.client.dynamicconsumer.fake.DynamicConsumerFake.shardsFromIterables and nl.vroste.zio.kinesis.client.dynamicconsumer.fake.DynamicConsumerFake.shardsFromStreams
- Returns:
A ZLayer of the fake
DynamicConsumer
implementation