zio-kinesis-dynamic-consumer
API
Generated with
zio-kinesis-dynamic-consumer/nl.vroste.zio.kinesis.client.dynamicconsumer/DynamicConsumer

DynamicConsumer

nl.vroste.zio.kinesis.client.dynamicconsumer.DynamicConsumer
See theDynamicConsumer companion trait
object DynamicConsumer

Offers a ZStream based interface to the Amazon Kinesis Client Library (KCL)

Ensures proper resource shutdown and failure handling

Attributes

Companion
trait
Graph
Supertypes
class Object
trait Matchable
class Any
Self type

Members list

Type members

Classlikes

trait Checkpointer

Staging area for checkpoints

Staging area for checkpoints

Guarantees that the last staged record is checkpointed upon stream shutdown / interruption

Attributes

Supertypes
class Object
trait Matchable
class Any
final case class Record[+T](shardId: String, sequenceNumber: String, approximateArrivalTimestamp: Instant, data: T, partitionKey: String, encryptionType: EncryptionType, subSequenceNumber: Option[Long], explicitHashKey: Option[String], aggregated: Boolean)

Attributes

Supertypes
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all

Value members

Concrete methods

def consumeWith[R, RC, T](streamName: String, applicationName: String, deserializer: Deserializer[R, T], requestShutdown: UIO[Unit], initialPosition: InitialPositionInStreamExtended, leaseTableName: Option[String], metricsNamespace: Option[String], workerIdentifier: String, maxShardBufferSize: Int, checkpointBatchSize: Long, checkpointDuration: Duration, configureKcl: SchedulerConfig => SchedulerConfig)(recordProcessor: (Record[T]) => RIO[RC, Unit]): ZIO[R & RC & DynamicConsumer, Throwable, Unit]

Similar to shardedStream accessor but provides the recordProcessor callback function for processing records and takes care of checkpointing. The other difference is that it returns a ZIO of unit rather than a ZStream.

Similar to shardedStream accessor but provides the recordProcessor callback function for processing records and takes care of checkpointing. The other difference is that it returns a ZIO of unit rather than a ZStream.

Type parameters

R

ZIO environment type required by the deserializer and the recordProcessor

T

Type of record values

Value parameters

applicationName

Application name for coordinating shard leases

checkpointBatchSize

Maximum number of records before checkpointing

checkpointDuration

Maximum interval before checkpointing

configureKcl

Make additional KCL Scheduler configurations

deserializer

Deserializer for record values

initialPosition

Position in stream to start at when there is no previous checkpoint for this application

leaseTableName

Optionally set the lease table name - defaults to None. If not specified the applicationName will be used.

maxShardBufferSize

The maximum number of records per shard to store in a queue before blocking the KCL record processor until records have been dequeued. Note that the stream returned from this method will have internal chunk buffers as well.

metricsNamespace

CloudWatch metrics namespace

recordProcessor

A function for processing a Record[T]

requestShutdown

Effect that when completed will trigger a graceful shutdown of the KCL and the streams.

streamName

Name of the Kinesis stream

workerIdentifier

Identifier used for the worker in this application group. Used in logging and written to the lease table.

Attributes

Returns

A ZIO that completes with Unit when record processing is stopped via requestShutdown or fails when the consumer stream fails

def fake(shards: ZStream[Any, Throwable, (String, ZStream[Any, Throwable, Chunk[Byte]])], refCheckpointedList: Ref[Seq[Record[Any]]]): ZLayer[Any, Nothing, DynamicConsumer]

Implements a fake DynamicConsumer that also offers fake checkpointing functionality that can be tracked using the refCheckpointedList parameter.

Implements a fake DynamicConsumer that also offers fake checkpointing functionality that can be tracked using the refCheckpointedList parameter.

Value parameters

refCheckpointedList

A Ref that will be used to store the checkpointed records

shards

A ZStream that is a fake representation of a Kinesis shard. There are helper constructors to create these - see nl.vroste.zio.kinesis.client.dynamicconsumer.fake.DynamicConsumerFake.shardsFromIterables and nl.vroste.zio.kinesis.client.dynamicconsumer.fake.DynamicConsumerFake.shardsFromStreams

Attributes

Returns

A ZLayer of the fake DynamicConsumer implementation

def fake(shards: ZStream[Any, Throwable, (String, ZStream[Any, Throwable, Chunk[Byte]])]): ZLayer[Any, Nothing, DynamicConsumer]

Overloaded version of above but without fake checkpointing functionality

Overloaded version of above but without fake checkpointing functionality

Value parameters

shards

A ZStream that is a fake representation of a Kinesis shard. There are helper constructors to create these - see nl.vroste.zio.kinesis.client.dynamicconsumer.fake.DynamicConsumerFake.shardsFromIterables and nl.vroste.zio.kinesis.client.dynamicconsumer.fake.DynamicConsumerFake.shardsFromStreams

Attributes

Returns

A ZLayer of the fake DynamicConsumer implementation

def shardedStream[R, T](streamName: String, applicationName: String, deserializer: Deserializer[R, T], requestShutdown: UIO[Unit], initialPosition: InitialPositionInStreamExtended, leaseTableName: Option[String], metricsNamespace: Option[String], workerIdentifier: String, maxShardBufferSize: Int, configureKcl: SchedulerConfig => SchedulerConfig): ZStream[DynamicConsumer & R, Throwable, (String, ZStream[Any, Throwable, Record[T]], Checkpointer)]

Concrete fields

val live: ZLayer[Kinesis & CloudWatch & DynamoDb, Nothing, DynamicConsumer]
Generated with