Members list
Type members
Classlikes
Attributes
- Companion:
- trait
- Graph
- Supertypes
- Self type
- AutoOffsetStrategy.type
Attributes
- Companion:
- trait
- Graph
- Supertypes
- Self type
- OffsetRetrieval.type
Attributes
- Graph
- Supertypes
- trait Singletontrait Producttrait Mirrortrait Producttrait Equalstrait NoStackTraceclass RuntimeExceptionclass Exceptionclass Throwabletrait Serializableclass Objecttrait Matchableclass Any
- Self type
- RunloopTimeout.type
Value members
Concrete methods
Accessor method for Consumer.assignment
Accessor method for Consumer.beginningOffsets
Accessor method for Consumer.committed
Execute an effect for each record and commit the offset after processing
Execute an effect for each record and commit the offset after processing
This method is the easiest way of processing messages on a Kafka topic.
Messages on a single partition are processed sequentially, while the processing of multiple partitions happens in parallel.
Offsets are committed after execution of the effect. They are batched when a commit action is in progress to avoid backpressuring the stream. When commits fail due to a org.apache.kafka.clients.consumer.RetriableCommitFailedException they are retried according to commitRetryPolicy
The effect should absorb any failures. Failures should be handled by retries or ignoring the error, which will result in the Kafka message being skipped.
Messages are processed with 'at least once' consistency: it is not guaranteed that every message that is processed by the effect has a corresponding offset commit before stream termination.
Usage example:
val settings: ConsumerSettings = ???
val subscription = Subscription.Topics(Set("my-kafka-topic"))
val consumerIO = Consumer.consumeWith(settings, subscription, Serdes.string, Serdes.string) { record =>
// Process the received record here
putStrLn(s"Received record: ${record.key()}: ${record.value()}")
}
Attributes
- K
Type of keys (an implicit
Deserializer
should be in scope)- R
Environment for the consuming effect
- R1
Environment for the deserializers
- V
Type of values (an implicit
Deserializer
should be in scope)- commitRetryPolicy
Retry commits that failed due to a RetriableCommitFailedException according to this schedule
- f
Function that returns the effect to execute for each message. It is passed the org.apache.kafka.clients.consumer.ConsumerRecord.
- keyDeserializer
Deserializer for the key of the messages
- settings
Settings for creating a Consumer
- subscription
Topic subscription parameters
- valueDeserializer
Deserializer for the value of the messages
- Returns:
Effect that completes with a unit value only when interrupted. May fail when the Consumer fails.
Accessor method for Consumer.endOffsets
Accessor method for Consumer.listTopics
Accessor method for Consumer.metrics
Accessor method for Consumer.offsetsForTimes
Accessor method for Consumer.partitionedStream
Accessor method for Consumer.partitionsFor
Accessor method for Consumer.plainStream
Accessor method for Consumer.position
Accessor method for Consumer.stopConsumption
Accessor method for Consumer.subscription