KafkaConsumer

Manually assigns all partitions for the specified topic to the consumer.

Manually assigns the specified list of partitions for the specified topic to the consumer. This function does not allow for incremental assignment and will replace the previous assignment (if there is one).

Manual topic assignment through this method does not use the consumer's group management functionality. As such, there will be no rebalance operation triggered when group membership or cluster and topic metadata change. Note that it is not possible to use both manual partition assignment with assign and group assignment with subscribe.

If auto-commit is enabled, an async commit (based on the old assignment) will be triggered before the new assignment replaces the old one.

To unassign all partitions, use KafkaConsumer#unsubscribe.

Manually assigns the specified list of topic partitions to the consumer. This function does not allow for incremental assignment and will replace the previous assignment (if there is one).

Manual topic assignment through this method does not use the consumer's group management functionality. As such, there will be no rebalance operation triggered when group membership or cluster and topic metadata change. Note that it is not possible to use both manual partition assignment with assign and group assigment with subscribe.

If auto-commit is enabled, an async commit (based on the old assignment) will be triggered before the new assignment replaces the old one.

To unassign all partitions, use KafkaConsumer#unsubscribe.

Returns the set of partitions currently assigned to this consumer.

Stream where the elements are the set of TopicPartitions currently assigned to this consumer. The stream emits whenever a rebalance changes partition assignments.

Wait for consumer to shut down. Note that awaitTermination is guaranteed to complete after consumer shutdown, even when the consumer is cancelled with terminate.

This method will not initiate shutdown. To initiate shutdown and wait for it to complete, you can use terminate >> awaitTermination.

Returns the first offset for the specified partitions.

Returns the first offset for the specified partitions.

Timeout is determined by default.api.timeout.ms, which is set using ConsumerSettings#withDefaultApiTimeout.

Commit the specified offsets for the specified list of topics and partitions to Kafka.

This commits offsets to Kafka. The offsets committed using this API will be used on the first fetch after every rebalance and also on startup. As such, if you need to store offsets in anything other than Kafka, this API should not be used. The committed offset should be the next message your application will consume, i.e. lastProcessedMessageOffset + 1. If automatic group management with subscribe is used, then the committed offsets must belong to the currently auto-assigned partitions.

Offsets committed through multiple calls to this API are guaranteed to be sent in the same order as the invocations. Additionally note that offsets committed through this API are guaranteed to complete before a subsequent call to commitSync (and variants) returns.

Note, that the recommended way for committing offsets in fs2-kafka is to use commit on CommittableConsumerRecord, CommittableOffset or CommittableOffsetBatch. commitAsync and commitSync usually needs only for some custom scenarios.

Commit the specified offsets for the specified list of topics and partitions.

This commits offsets to Kafka. The offsets committed using this API will be used on the first fetch after every rebalance and also on startup. As such, if you need to store offsets in anything other than Kafka, this API should not be used. The committed offset should be the next message your application will consume, i.e. lastProcessedMessageOffset + 1. If automatic group management with subscribe is used, then the committed offsets must belong to the currently auto-assigned partitions.

Despite of it's name, this method is not blocking. But it's based on a blocking org.apache.kafka.clients.consumer.KafkaConsumer#commitSync method.

Note, that the recommended way for committing offsets in fs2-kafka is to use commit on CommittableConsumerRecord, CommittableOffset or CommittableOffsetBatch. commitAsync and commitSync usually needs only for some custom scenarios.

Returns the last offset for the specified partitions.

Returns the last offset for the specified partitions.

Timeout is determined by request.timeout.ms, which is set using ConsumerSettings#withRequestTimeout.

Returns consumer metrics.

Stream where the elements themselves are Streams which continually request records for a single partition. These Streams will have to be processed in parallel, using parJoin or parJoinUnbounded. Note that when using parJoin(n) and n is smaller than the number of currently assigned partitions, then there will be assigned partitions which won't be processed. For that reason, prefer parJoinUnbounded and the actual limit will be the number of assigned partitions.

If you do not want to process all partitions in parallel, then you can use stream instead, where records for all partitions are in a single Stream.

Returns the partitions for the specified topic.

Returns the partitions for the specified topic.

Timeout is determined by default.api.timeout.ms, which is set using ConsumerSettings#withDefaultApiTimeout.

Stream where each element contains a current assignment. The current assignment is the Map, where keys is a TopicPartition, and values are streams with records for a particular TopicPartition.

New assignments will be received on each rebalance. On rebalance, Kafka revoke all previously assigned partitions, and after that assigned new partitions all at once. partitionsMapStream reflects this process in a streaming manner.

Note, that partition streams for revoked partitions will be closed after the new assignment comes.

This is the most generic Stream method. If you don't need such control, consider using partitionedStream or stream methods. They are both based on a partitionsMapStream.

Returns the offset of the next record that will be fetched.

Returns the offset of the next record that will be fetched.

Timeout is determined by default.api.timeout.ms, which is set using ConsumerSettings#withDefaultApiTimeout.

Overrides the fetch offsets that the consumer will use when reading the next record. If this API is invoked for the same partition more than once, the latest offset will be used. Note that you may lose data if this API is arbitrarily used in the middle of consumption to reset the fetch offsets.

Seeks to the first offset for each of the specified partitions. If no partitions are provided, seeks to the first offset for all currently assigned partitions.

Note that this seek evaluates lazily, and only on the next call to poll or position.

Seeks to the first offset for each currently assigned partition. This is equivalent to using seekToBeginning with an empty set of partitions.

Note that this seek evaluates lazily, and only on the next call to poll or position.

Seeks to the last offset for each of the specified partitions. If no partitions are provided, seeks to the last offset for all currently assigned partitions.

Note that this seek evaluates lazily, and only on the next call to poll or position.

Seeks to the last offset for each currently assigned partition. This is equivalent to using seekToEnd with an empty set of partitions.

Note that this seek evaluates lazily, and only on the next call to poll or position.

Stops consuming new messages from Kafka. This method could be used to implement a graceful shutdown.

This method has a few effects:

1. After this call no more data will be fetched from Kafka through the poll method.
2. All currently running streams will continue to run until all in-flight messages will be processed. It means that streams will be completed when all fetched messages will be processed.
   

Alias for partitionedStream.parJoinUnbounded. See partitionedStream for more information.

Subscribes the consumer to the topics matching the specified Regex. Note that you have to use one of the subscribe functions before you can use any of the provided Streams, or a NotSubscribedException will be raised in the Streams.

Subscribes the consumer to the specified topics. Note that you have to use one of the subscribe functions to subscribe to one or more topics before using any of the provided Streams, or a NotSubscribedException will be raised in the Streams.

Subscribes the consumer to the specified topics. Note that you have to use one of the subscribe functions to subscribe to one or more topics before using any of the provided Streams, or a NotSubscribedException will be raised in the Streams.

Whenever terminate is invoked, an attempt will be made to stop the underlying consumer. The terminate operation will not wait for the consumer to shutdown. If you also want to wait for the shutdown to complete, you can use terminate >> awaitTermination.

Unsubscribes the consumer from all topics and partitions assigned by subscribe or assign.

A Fiber that can be used to cancel the underlying consumer, or wait for it to complete. If you're using KafkaConsumer.stream, or any other provided stream in KafkaConsumer, these will be automatically interrupted when the underlying consumer has been cancelled or when it finishes with an exception.

Whenever cancel is invoked, an attempt will be made to stop the underlying consumer. The cancel operation will not wait for the consumer to shutdown. If you also want to wait for the shutdown to complete, you can use join. Note that join is guaranteed to complete after consumer shutdown, even when the consumer is cancelled with cancel.

This Fiber instance is usually only required if the consumer needs to be cancelled due to some external condition, or when an external process needs to be cancelled whenever the consumer has shut down. Most of the time, when you're only using the streams provided by KafkaConsumer, there is no need to use this.