docs
API
Generated with
docs/zio.kafka/zio.kafka.consumer/Consumer

Consumer

zio.kafka.consumer.Consumer
See theConsumer companion trait
object Consumer

Attributes

Companion
trait
Graph
Supertypes
class Object
trait Matchable
class Any
Self type
Consumer.type

Members list

Type members

Classlikes

object AutoOffsetStrategy

Attributes

Companion
trait
Supertypes
trait Sum
trait Mirror
class Object
trait Matchable
class Any
Self type
sealed trait AutoOffsetStrategy

Attributes

Companion
object
Supertypes
class Object
trait Matchable
class Any
Known subtypes
object Earliest.type
object Latest.type
object None.type
Self type
case object CommitTimeout extends RuntimeException, NoStackTrace

Attributes

Supertypes
trait Singleton
trait Product
trait Mirror
trait Product
trait Equals
trait NoStackTrace
class Exception
class Throwable
trait Serializable
class Object
trait Matchable
class Any
Show all
Self type
object OffsetRetrieval

Attributes

Companion
trait
Supertypes
trait Sum
trait Mirror
class Object
trait Matchable
class Any
Self type
sealed trait OffsetRetrieval

Attributes

Companion
object
Supertypes
class Object
trait Matchable
class Any
Known subtypes
class Auto
class Manual
case object RunloopTimeout extends RuntimeException, NoStackTrace

Attributes

Supertypes
trait Singleton
trait Product
trait Mirror
trait Product
trait Equals
trait NoStackTrace
class Exception
class Throwable
trait Serializable
class Object
trait Matchable
class Any
Show all
Self type

Value members

Concrete methods

def assignment: RIO[Consumer, Set[TopicPartition]]

Accessor method

Accessor method

Attributes

def beginningOffsets(partitions: Set[TopicPartition], timeout: Duration): RIO[Consumer, Map[TopicPartition, Long]]

Accessor method

Accessor method

Attributes

def committed(partitions: Set[TopicPartition], timeout: Duration): RIO[Consumer, Map[TopicPartition, Option[OffsetAndMetadata]]]

Accessor method

Accessor method

Attributes

def consumeWith[R : Tag, R1 : Tag, K, V](settings: ConsumerSettings, subscription: Subscription, keyDeserializer: Deserializer[R, K], valueDeserializer: Deserializer[R, V], commitRetryPolicy: Schedule[Any, Any, Any])(f: ConsumerRecord[K, V] => URIO[R1, Unit]): RIO[R & R1, Unit]

Execute an effect for each record and commit the offset after processing

Execute an effect for each record and commit the offset after processing

This method is the easiest way of processing messages on a Kafka topic.

Messages on a single partition are processed sequentially, while the processing of multiple partitions happens in parallel.

Offsets are committed after execution of the effect. They are batched when a commit action is in progress to avoid backpressuring the stream. When commits fail due to a org.apache.kafka.clients.consumer.RetriableCommitFailedException they are retried according to commitRetryPolicy

The effect should absorb any failures. Failures should be handled by retries or ignoring the error, which will result in the Kafka message being skipped.

Messages are processed with 'at least once' consistency: it is not guaranteed that every message that is processed by the effect has a corresponding offset commit before stream termination.

Usage example:

val settings: ConsumerSettings = ???
val subscription = Subscription.Topics(Set("my-kafka-topic"))

val consumerIO = Consumer.consumeWith(settings, subscription, Serdes.string, Serdes.string) { record =>
 // Process the received record here
 putStrLn(s"Received record: ${record.key()}: ${record.value()}")
}

Type parameters

K

Type of keys (an implicit Deserializer should be in scope)

R

Environment for the consuming effect

R1

Environment for the deserializers

V

Type of values (an implicit Deserializer should be in scope)

Value parameters

commitRetryPolicy

Retry commits that failed due to a RetriableCommitFailedException according to this schedule

f

Function that returns the effect to execute for each message. It is passed the org.apache.kafka.clients.consumer.ConsumerRecord.

keyDeserializer

Deserializer for the key of the messages

settings

Settings for creating a Consumer

subscription

Topic subscription parameters

valueDeserializer

Deserializer for the value of the messages

Attributes

Returns

Effect that completes with a unit value only when interrupted. May fail when the Consumer fails.

def endOffsets(partitions: Set[TopicPartition], timeout: Duration): RIO[Consumer, Map[TopicPartition, Long]]

Accessor method

Accessor method

Attributes

def fromJavaConsumer(javaConsumer: Consumer[Array[Byte], Array[Byte]], settings: ConsumerSettings, diagnostics: Diagnostics): ZIO[Scope, Throwable, Consumer]

Create a zio-kafka Consumer from an org.apache.kafka KafkaConsumer

Create a zio-kafka Consumer from an org.apache.kafka KafkaConsumer

You are responsible for creating and closing the KafkaConsumer

Attributes

def listTopics(timeout: Duration): RIO[Consumer, Map[String, List[PartitionInfo]]]

Accessor method

Accessor method

Attributes

def make(settings: ConsumerSettings, diagnostics: Diagnostics): ZIO[Scope, Throwable, Consumer]
def metrics: RIO[Consumer, Map[MetricName, Metric]]

Accessor method

Accessor method

Attributes

def offsetsForTimes(timestamps: Map[TopicPartition, Long], timeout: Duration): RIO[Consumer, Map[TopicPartition, OffsetAndTimestamp]]

Accessor method

Accessor method

Attributes

def partitionedAssignmentStream[R, K, V](subscription: Subscription, keyDeserializer: Deserializer[R, K], valueDeserializer: Deserializer[R, V]): ZStream[Consumer, Throwable, Chunk[(TopicPartition, ZStream[R, Throwable, CommittableRecord[K, V]])]]

Accessor method

Accessor method

Attributes

def partitionedStream[R, K, V](subscription: Subscription, keyDeserializer: Deserializer[R, K], valueDeserializer: Deserializer[R, V]): ZStream[Consumer, Throwable, (TopicPartition, ZStream[R, Throwable, CommittableRecord[K, V]])]

Accessor method

Accessor method

Attributes

def partitionsFor(topic: String, timeout: Duration): RIO[Consumer, List[PartitionInfo]]

Accessor method

Accessor method

Attributes

def plainStream[R, K, V](subscription: Subscription, keyDeserializer: Deserializer[R, K], valueDeserializer: Deserializer[R, V], bufferSize: Int): ZStream[R & Consumer, Throwable, CommittableRecord[K, V]]

Accessor method

Accessor method

Attributes

def position(partition: TopicPartition, timeout: Duration): RIO[Consumer, Long]

Accessor method

Accessor method

Attributes

def stopConsumption: RIO[Consumer, Unit]

Accessor method

Accessor method

Attributes

def subscription: RIO[Consumer, Set[String]]

Accessor method

Accessor method

Attributes

Concrete fields

val offsetBatches: ZSink[Any, Nothing, Offset, Nothing, OffsetBatch]
Generated with