zio-kinesis-dynamic-consumer/nl.vroste.zio.kinesis.client.dynamicconsumer/DynamicConsumer/Service

Service

trait Service
class Object
trait Matchable
class Any

Value members

Abstract methods

def shardedStream[R, T](streamName: String, applicationName: String, deserializer: Deserializer[R, T], requestShutdown: UIO[Unit], initialPosition: InitialPositionInStreamExtended, leaseTableName: Option[String], metricsNamespace: Option[String], workerIdentifier: String, maxShardBufferSize: Int, configureKcl: SchedulerConfig => SchedulerConfig): ZStream[R, Throwable, (String, ZStream[Any, Throwable, Record[T]], Checkpointer)]

Create a ZStream of records on a Kinesis stream with substreams per shard

Create a ZStream of records on a Kinesis stream with substreams per shard

Uses DynamoDB for lease coordination between different instances of consumers with the same application name and for offset checkpointing.

Type parameters:
R

ZIO environment type required by the deserializer

T

Type of record values

Value parameters:
applicationName

Application name for coordinating shard leases

configureKcl

Make additional KCL Scheduler configurations

deserializer

Deserializer for record values

initialPosition

Position in stream to start at when there is no previous checkpoint for this application

leaseTableName

Optionally set the lease table name - defaults to None. If not specified the applicationName will be used.

maxShardBufferSize

The maximum number of records per shard to store in a queue before blocking the KCL record processor until records have been dequeued. Note that the stream returned from this method will have internal chunk buffers as well. Prefer powers of 2 for efficiency.

metricsNamespace

CloudWatch metrics namespace

requestShutdown

Effect that when completed will trigger a graceful shutdown of the KCL and the streams.

streamName

Name of the Kinesis stream

workerIdentifier

Identifier used for the worker in this application group. Used in logging and written to the lease table.

Returns:

A nested ZStream - the outer ZStream represents the collection of shards, and the inner ZStream represents the individual shard