Consumer

Accessor method for Consumer.assignment

Accessor method for Consumer.beginningOffsets

Accessor method for Consumer.committed

Execute an effect for each record and commit the offset after processing

This method is the easiest way of processing messages on a Kafka topic.

Messages on a single partition are processed sequentially, while the processing of multiple partitions happens in parallel.

Offsets are committed after execution of the effect. They are batched when a commit action is in progress to avoid backpressuring the stream. When commits fail due to a org.apache.kafka.clients.consumer.RetriableCommitFailedException they are retried according to commitRetryPolicy

The effect should absorb any failures. Failures should be handled by retries or ignoring the error, which will result in the Kafka message being skipped.

Messages are processed with 'at least once' consistency: it is not guaranteed that every message that is processed by the effect has a corresponding offset commit before stream termination.

Usage example:

val settings: ConsumerSettings = ???
val subscription = Subscription.Topics(Set("my-kafka-topic"))

val consumerIO = Consumer.consumeWith(settings, subscription, Serdes.string, Serdes.string) { case (key, value) =>
 // Process the received record here
 putStrLn(s"Received record: ${key}: ${value}")
}

Type parameters:

K: Type of keys (an implicit Deserializer should be in scope)
R: Environment for the consuming effect
R1: Environment for the deserializers
V: Type of values (an implicit Deserializer should be in scope)

Value parameters:

commitRetryPolicy: Retry commits that failed due to a RetriableCommitFailedException according to this schedule
f: Function that returns the effect to execute for each message. It is passed the key and value
keyDeserializer: Deserializer for the key of the messages
settings: Settings for creating a Consumer
subscription: Topic subscription parameters
valueDeserializer: Deserializer for the value of the messages

Returns: