zio-kinesis-dynamic-consumer
API
Generated with
zio-kinesis-dynamic-consumer/nl.vroste.zio.kinesis.client.dynamicconsumer/DynamicConsumer

DynamicConsumer

nl.vroste.zio.kinesis.client.dynamicconsumer.DynamicConsumer
See theDynamicConsumer companion object
trait DynamicConsumer

Attributes

Companion
object
Graph
Supertypes
class Object
trait Matchable
class Any

Members list

Value members

Abstract methods

def shardedStream[R, T](streamName: String, applicationName: String, deserializer: Deserializer[R, T], requestShutdown: UIO[Unit], initialPosition: InitialPositionInStreamExtended, leaseTableName: Option[String], metricsNamespace: Option[String], workerIdentifier: String, maxShardBufferSize: Int, configureKcl: SchedulerConfig => SchedulerConfig): ZStream[R, Throwable, (String, ZStream[Any, Throwable, Record[T]], Checkpointer)]

Create a ZStream of records on a Kinesis stream with substreams per shard

Create a ZStream of records on a Kinesis stream with substreams per shard

Uses DynamoDB for lease coordination between different instances of consumers with the same application name and for offset checkpointing.

Type parameters

R

ZIO environment type required by the deserializer

T

Type of record values

Value parameters

applicationName

Application name for coordinating shard leases

configureKcl

Make additional KCL Scheduler configurations

deserializer

Deserializer for record values

initialPosition

Position in stream to start at when there is no previous checkpoint for this application

leaseTableName

Optionally set the lease table name - defaults to None. If not specified the applicationName will be used.

maxShardBufferSize

The maximum number of records per shard to store in a queue before blocking the KCL record processor until records have been dequeued. Note that the stream returned from this method will have internal chunk buffers as well.

metricsNamespace

CloudWatch metrics namespace

requestShutdown

Effect that when completed will trigger a graceful shutdown of the KCL and the streams.

streamIdentifier

Stream to consume from. Either just the name or the whole arn.

workerIdentifier

Identifier used for the worker in this application group. Used in logging and written to the lease table.

Attributes

Returns

A nested ZStream - the outer ZStream represents the collection of shards, and the inner ZStream represents the individual shard

Generated with