zio-kinesis-dynamic-consumer/nl.vroste.zio.kinesis.client.dynamicconsumer/DynamicConsumer

DynamicConsumer

object DynamicConsumer

Offers a ZStream based interface to the Amazon Kinesis Client Library (KCL)

Ensures proper resource shutdown and failure handling

class Object
trait Matchable
class Any

Type members

Classlikes

trait Checkpointer

Staging area for checkpoints

Staging area for checkpoints

Guarantees that the last staged record is checkpointed upon stream shutdown / interruption

final case class Record[+T](shardId: String, sequenceNumber: String, approximateArrivalTimestamp: Instant, data: T, partitionKey: String, encryptionType: EncryptionType, subSequenceNumber: Option[Long], explicitHashKey: Option[String], aggregated: Boolean)
trait Service

Value members

Concrete methods

def consumeWith[R, RC, T](streamName: String, applicationName: String, deserializer: Deserializer[R, T], requestShutdown: UIO[Unit], initialPosition: InitialPositionInStreamExtended, leaseTableName: Option[String], metricsNamespace: Option[String], workerIdentifier: String, maxShardBufferSize: Int, checkpointBatchSize: Long, checkpointDuration: Duration, configureKcl: SchedulerConfig => SchedulerConfig)(recordProcessor: Record[T] => RIO[RC, Unit]): ZIO[R & RC & Blocking & Logging & Clock & DynamicConsumer, Throwable, Unit]

Similar to shardedStream accessor but provides the recordProcessor callback function for processing records and takes care of checkpointing. The other difference is that it returns a ZIO of unit rather than a ZStream.

Similar to shardedStream accessor but provides the recordProcessor callback function for processing records and takes care of checkpointing. The other difference is that it returns a ZIO of unit rather than a ZStream.

Type parameters:
R

ZIO environment type required by the deserializer and the recordProcessor

T

Type of record values

Value parameters:
applicationName

Application name for coordinating shard leases

checkpointBatchSize

Maximum number of records before checkpointing

checkpointDuration

Maximum interval before checkpointing

configureKcl

Make additional KCL Scheduler configurations

deserializer

Deserializer for record values

initialPosition

Position in stream to start at when there is no previous checkpoint for this application

leaseTableName

Optionally set the lease table name - defaults to None. If not specified the applicationName will be used.

maxShardBufferSize

The maximum number of records per shard to store in a queue before blocking the KCL record processor until records have been dequeued. Note that the stream returned from this method will have internal chunk buffers as well.

metricsNamespace

CloudWatch metrics namespace

recordProcessor

A function for processing a Record[T]

requestShutdown

Effect that when completed will trigger a graceful shutdown of the KCL and the streams.

streamName

Name of the Kinesis stream

workerIdentifier

Identifier used for the worker in this application group. Used in logging and written to the lease table.

Returns:

A ZIO that completes with Unit when record processing is stopped via requestShutdown or fails when the consumer stream fails

def fake(shards: ZStream[Any, Throwable, (String, ZStream[Any, Throwable, Chunk[Byte]])], refCheckpointedList: Ref[Seq[Record[Any]]]): ZLayer[Clock, Nothing, Has[Service]]

Implements a fake DynamicConsumer that also offers fake checkpointing functionality that can be tracked using the refCheckpointedList parameter.

Implements a fake DynamicConsumer that also offers fake checkpointing functionality that can be tracked using the refCheckpointedList parameter.

Value parameters:
refCheckpointedList

A Ref that will be used to store the checkpointed records

shards

A ZStream that is a fake representation of a Kinesis shard. There are helper constructors to create these - see nl.vroste.zio.kinesis.client.dynamicconsumer.fake.DynamicConsumerFake.shardsFromIterables and nl.vroste.zio.kinesis.client.dynamicconsumer.fake.DynamicConsumerFake.shardsFromStreams

Returns:

A ZLayer of the fake DynamicConsumer implementation

def fake(shards: ZStream[Any, Throwable, (String, ZStream[Any, Throwable, Chunk[Byte]])]): ZLayer[Clock, Nothing, Has[Service]]

Overloaded version of above but without fake checkpointing functionality

Overloaded version of above but without fake checkpointing functionality

Value parameters:
shards

A ZStream that is a fake representation of a Kinesis shard. There are helper constructors to create these - see nl.vroste.zio.kinesis.client.dynamicconsumer.fake.DynamicConsumerFake.shardsFromIterables and nl.vroste.zio.kinesis.client.dynamicconsumer.fake.DynamicConsumerFake.shardsFromStreams

Returns:

A ZLayer of the fake DynamicConsumer implementation

def shardedStream[R, T](streamName: String, applicationName: String, deserializer: Deserializer[R, T], requestShutdown: UIO[Unit], initialPosition: InitialPositionInStreamExtended, leaseTableName: Option[String], metricsNamespace: Option[String], workerIdentifier: String, maxShardBufferSize: Int, configureKcl: SchedulerConfig => SchedulerConfig): ZStream[DynamicConsumer & R, Throwable, (String, ZStream[Any, Throwable, Record[T]], Checkpointer)]

Concrete fields

val live: ZLayer[Logging & Blocking & Kinesis & CloudWatch & DynamoDb, Nothing, DynamicConsumer]