object KCLConsumerFS2
- Source
- KCLConsumerFS2.scala
- Alphabetic
- By Inheritance
- KCLConsumerFS2
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Type Members
- final case class Config[F[_]](underlying: KCLConsumer.Config[F], queue: Queue[F, CommittableRecord[F]], fs2Config: FS2Config) extends Product with Serializable
Configuration for the KCLConsumerFS2
Configuration for the KCLConsumerFS2
- underlying
- queue
- final case class FS2Config(queueSize: Int, commitMaxChunk: Int, commitMaxWait: FiniteDuration, commitMaxRetries: Int, commitRetryInterval: FiniteDuration) extends Product with Serializable
Configuration for the FS2 implementation
Configuration for the FS2 implementation
- queueSize
Size of the underlying queue for the FS2 stream. If the queue fills up, backpressure on the processors will occur. Default 100
- commitMaxChunk
Max records to be received in the commitRecords Pipe before a commit is run. Default is 1000
- commitMaxWait
Max duration to wait in commitRecords Pipe before a commit is run. Default is 10 seconds
- commitMaxRetries
Max number of retries for a commit operation
- commitRetryInterval
Interval to wait between commit retries
- final case class StreamAndDeferred[F[_]](stream: Stream[F, CommittableRecord[F]], deferred: Deferred[F, Unit]) extends Product with Serializable
Helper class that holds both an fs2.Stream and a Deferred
Helper class that holds both an fs2.Stream and a Deferred
- stream
fs2.Stream of kinesis4cats.kcl.CommittableRecord values to process
- deferred
Deferred which will complete when a defined WorkerState is recognized.
- final case class StreamAndRef[F[_]](stream: Stream[F, CommittableRecord[F]], ref: Ref[F, WorkerState]) extends Product with Serializable
Helper class that holds both an fs2.Stream and a Ref of the WorkerState
Helper class that holds both an fs2.Stream and a Ref of the WorkerState
- stream
fs2.Stream of kinesis4cats.kcl.CommittableRecord values to process
- ref
Ref that contains the current WorkerState
Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- def apply[F[_]](checkpointConfig: CheckpointConfig, coordinatorConfig: CoordinatorConfig, leaseManagementConfig: LeaseManagementConfig, lifecycleConfig: LifecycleConfig, metricsConfig: MetricsConfig, retrievalConfig: RetrievalConfig, fs2Config: FS2Config = FS2Config.default, processConfig: ProcessConfig = defaultProcessConfig)(implicit F: Async[F], P: Parallel[F], encoders: LogEncoders): Resource[F, KCLConsumerFS2[F]]
Low-level constructor for KCLConsumerFS2.
Low-level constructor for KCLConsumerFS2.
- checkpointConfig
- coordinatorConfig
- leaseManagementConfig
- lifecycleConfig
- metricsConfig
- retrievalConfig
- processConfig
- F
Async instance
- encoders
RecordProcessor.LogEncoders for encoding structured logs
- returns
Resource containing the KCLConsumerFS2
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @native() @IntrinsicCandidate()
- def configsBuilder[F[_]](kinesisClient: KinesisAsyncClient, dynamoClient: DynamoDbAsyncClient, cloudWatchClient: CloudWatchAsyncClient, streamName: String, appName: String, fs2Config: FS2Config = FS2Config.default, workerId: String = Utils.randomUUIDString, processConfig: ProcessConfig = defaultProcessConfig)(tfn: (KCLConsumer.Config[F]) => KCLConsumer.Config[F] = (x: kinesis4cats.kcl.KCLConsumer.Config[F]) => x)(implicit F: Async[F], P: Parallel[F], encoders: LogEncoders): Resource[F, KCLConsumerFS2[F]]
Constructor for the KCLConsumerFS2 that leverages the ConfigsBuilder from the KCL.
Constructor for the KCLConsumerFS2 that leverages the ConfigsBuilder from the KCL. This is a simpler entry-point for creating the configuration, and provides a transform function to add any custom configuration that was not covered by the default
- kinesisClient
- dynamoClient
- cloudWatchClient
- streamName
Name of the Kinesis stream to consume from
- appName
Name of the application. Usually also the dynamo table name for checkpoints
- workerId
Unique identifier for a single instance of this consumer. Default is a random UUID.
- processConfig
- tfn
Function to update the KCLConsumer.Config. Useful for overriding defaults.
- F
Async instance
- encoders
RecordProcessor.LogEncoders for encoding structured logs
- returns
Resource containing the KCLConsumerFS2.Config
- def configsBuilderMultiStream[F[_]](kinesisClient: KinesisAsyncClient, dynamoClient: DynamoDbAsyncClient, cloudWatchClient: CloudWatchAsyncClient, tracker: MultiStreamTracker, appName: String, fs2Config: FS2Config = FS2Config.default, workerId: String = Utils.randomUUIDString, processConfig: ProcessConfig = defaultProcessConfig)(tfn: (KCLConsumer.Config[F]) => KCLConsumer.Config[F] = (x: kinesis4cats.kcl.KCLConsumer.Config[F]) => x)(implicit F: Async[F], P: Parallel[F], encoders: LogEncoders): Resource[F, KCLConsumerFS2[F]]
Constructor for the KCLConsumerFS2 that leverages the ConfigsBuilder from the KCL.
Constructor for the KCLConsumerFS2 that leverages the ConfigsBuilder from the KCL. This is a simpler entry-point for creating the configuration, and provides a transform function to add any custom configuration that was not covered by the default. This constructor specifically leverages the MultiStreamTracker to allow for consumption from multiple streams.
- kinesisClient
- dynamoClient
- cloudWatchClient
- tracker
- appName
Name of the application. Usually also the dynamo table name for checkpoints
- workerId
Unique identifier for a single instance of this consumer. Default is a random UUID.
- processConfig
- tfn
Function to update the KCLConsumer.Config. Useful for overriding defaults.
- F
Async instance
- encoders
RecordProcessor.LogEncoders for encoding structured logs
- returns
Resource containing the KCLConsumerFS2.Config
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @native() @IntrinsicCandidate()
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native() @IntrinsicCandidate()
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native() @IntrinsicCandidate()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native() @IntrinsicCandidate()
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- object Config extends Serializable
- object FS2Config extends Serializable