fs2.kafka
Members list
Packages
Type members
Classlikes
The available options for ProducerSettings#withAcks.
Available options include:
- Acks#Zero to not wait for any acknowledgement from the server,
- Acks#One to only wait for acknowledgement from the leader node,
- Acks#All to wait for acknowledgement from all in-sync replicas.
The available options for ProducerSettings#withAcks.
Available options include:
- Acks#Zero to not wait for any acknowledgement from the server,
- Acks#One to only wait for acknowledgement from the leader node,
- Acks#All to wait for acknowledgement from all in-sync replicas.
Attributes
- Companion
- object
- Source
- Acks.scala
- Supertypes
Attributes
- Companion
- class
- Source
- Acks.scala
- Supertypes
- Self type
-
Acks.type
AdminClientSettings contain settings necessary to create a KafkaAdminClient.
AdminClientSettings contain settings necessary to create a KafkaAdminClient. Several convenience functions are provided so that you don't have to work with String
values and keys from AdminClientConfig
. It's still possible to set AdminClientConfig
values with functions like withProperty.
AdminClientSettings instances are immutable and all modification functions return a new AdminClientSettings instance.
Use AdminClientSettings#apply for the default settings, and then apply any desired modifications on top of that instance.
Attributes
- Companion
- object
- Source
- AdminClientSettings.scala
- Supertypes
Attributes
- Companion
- class
- Source
- AdminClientSettings.scala
- Supertypes
- Self type
-
AdminClientSettings.type
The available options for ConsumerSettings#withAutoOffsetReset.
Available options include:
- AutoOffsetReset#Earliest to reset to the earliest offsets,
- AutoOffsetReset#Latest to reset to the latest offsets,
- AutoOffsetReset#None to fail if no offsets are available.
The available options for ConsumerSettings#withAutoOffsetReset.
Available options include:
- AutoOffsetReset#Earliest to reset to the earliest offsets,
- AutoOffsetReset#Latest to reset to the latest offsets,
- AutoOffsetReset#None to fail if no offsets are available.
Attributes
- Companion
- object
- Source
- AutoOffsetReset.scala
- Supertypes
Attributes
- Companion
- class
- Source
- AutoOffsetReset.scala
- Supertypes
- Self type
-
AutoOffsetReset.type
CommitRecovery describes how to recover from exceptions raised while trying to commit offsets.
CommitRecovery describes how to recover from exceptions raised while trying to commit offsets. See CommitRecovery#Default for the default recovery strategy. If you do not wish to recover from any exceptions, you can use CommitRecovery#None.
To create a new CommitRecovery, simply create a new instance and implement the recoverCommitWith function with the wanted recovery strategy. To use the CommitRecovery, you can simply set it with ConsumerSettings#withCommitRecovery.
Attributes
- Companion
- object
- Source
- CommitRecovery.scala
- Supertypes
Attributes
- Companion
- class
- Source
- CommitRecovery.scala
- Supertypes
- Self type
-
CommitRecovery.type
CommitRecoveryException indicates that offset commit recovery was attempted attempts
times for offsets
, but that it wasn't able to complete successfully.
CommitRecoveryException indicates that offset commit recovery was attempted attempts
times for offsets
, but that it wasn't able to complete successfully. The last encountered exception is provided as lastException
.
Use CommitRecoveryException#apply to create a new instance.
Attributes
- Companion
- object
- Source
- CommitRecoveryException.scala
- Supertypes
-
class KafkaExceptionclass RuntimeExceptionclass Exceptionclass Throwabletrait Serializableclass Objecttrait Matchableclass AnyShow all
Attributes
- Companion
- class
- Source
- CommitRecoveryException.scala
- Supertypes
- Self type
CommitTimeoutException indicates that offset commit took longer than the configured ConsumerSettings#commitTimeout.
CommitTimeoutException indicates that offset commit took longer than the configured ConsumerSettings#commitTimeout. The timeout and offsets are included in the exception message.
Attributes
- Source
- CommitTimeoutException.scala
- Supertypes
-
class KafkaExceptionclass RuntimeExceptionclass Exceptionclass Throwabletrait Serializableclass Objecttrait Matchableclass AnyShow all
CommittableConsumerRecord is a Kafka record along with an instance of CommittableOffset, which can be used commit the record offset to Kafka.
CommittableConsumerRecord is a Kafka record along with an instance of CommittableOffset, which can be used commit the record offset to Kafka. Offsets are normally committed in batches, either using CommittableOffsetBatch or via pipes, like commitBatchWithin. If you are not committing offsets to Kafka then you can use record to get the underlying record and also discard the offset.
While normally not necessary, CommittableConsumerRecord#apply can be used to create a new instance.
Attributes
- Companion
- object
- Source
- CommittableConsumerRecord.scala
- Supertypes
Attributes
- Companion
- class
- Source
- CommittableConsumerRecord.scala
- Supertypes
- Self type
CommittableOffset represents an offsetAndMetadata for a topicPartition, along with the ability to commit that offset to Kafka with commit.
CommittableOffset represents an offsetAndMetadata for a topicPartition, along with the ability to commit that offset to Kafka with commit. Note that offsets are normally committed in batches for performance reasons. Pipes like commitBatchWithin use CommittableOffsetBatch to commit the offsets in batches.
While normally not necessary, CommittableOffset#apply can be used to create a new instance.
Attributes
- Companion
- object
- Source
- CommittableOffset.scala
- Supertypes
Attributes
- Companion
- class
- Source
- CommittableOffset.scala
- Supertypes
- Self type
-
CommittableOffset.type
CommittableOffsetBatch represents a batch of Kafka offsets which can be committed together using commit.
CommittableOffsetBatch represents a batch of Kafka offsets which can be committed together using commit. An offset, or one more batch, can be added an existing batch using updated
. Note that this requires the offsets per topic-partition to be included in-order, since offset commits in general require it.
Use CommittableOffsetBatch#empty to create an empty batch. The CommittableOffset#batch function can be used to create a batch from an existing CommittableOffset.
If you have some offsets in-order per topic-partition, you can fold them together using CommittableOffsetBatch#empty and updated
, or you can use CommittableOffsetBatch#fromFoldable. Generally, prefer to use fromFoldable
, as it has better performance. Provided pipes like commitBatchWithin are also to be preferred, as they also achieve better performance.
Attributes
- Companion
- object
- Source
- CommittableOffsetBatch.scala
- Supertypes
Attributes
- Companion
- class
- Source
- CommittableOffsetBatch.scala
- Supertypes
- Self type
CommittableProducerRecords represents zero or more ProducerRecords and a CommittableOffset, used by TransactionalKafkaProducer to produce the records and commit the offset atomically.
CommittableProducerRecordss can be created using one of the following options.
- CommittableProducerRecords#apply
to produce zero or more records within the same transaction as the offset is committed.
- CommittableProducerRecords#one
to produce exactly one record within the same transaction as the offset is committed.
CommittableProducerRecords represents zero or more ProducerRecords and a CommittableOffset, used by TransactionalKafkaProducer to produce the records and commit the offset atomically.
CommittableProducerRecordss can be created using one of the following options.
- CommittableProducerRecords#apply
to produce zero or more records within the same transaction as the offset is committed.
- CommittableProducerRecords#one
to produce exactly one record within the same transaction as the offset is committed.
Attributes
- Companion
- object
- Source
- CommittableProducerRecords.scala
- Supertypes
Attributes
- Companion
- class
- Source
- CommittableProducerRecords.scala
- Supertypes
- Self type
Indicates that one or more of the following conditions occurred while attempting to commit offsets.
- There were CommittableOffsets without a consumer group ID.
- There were CommittableOffsets for multiple consumer group IDs.
Indicates that one or more of the following conditions occurred while attempting to commit offsets.
- There were CommittableOffsets without a consumer group ID.
- There were CommittableOffsets for multiple consumer group IDs.
Attributes
- Source
- ConsumerGroupException.scala
- Supertypes
-
class KafkaExceptionclass RuntimeExceptionclass Exceptionclass Throwabletrait Serializableclass Objecttrait Matchableclass AnyShow all
ConsumerRecord represents a record which has been consumed from Kafka.
ConsumerRecord represents a record which has been consumed from Kafka. At the very least, this includes a key of type K
, value of type V
, and the topic, partition, and offset of the consumed record.
To create a new instance, use ConsumerRecord#apply
Attributes
- Companion
- object
- Source
- ConsumerRecord.scala
- Supertypes
Attributes
- Companion
- class
- Source
- ConsumerRecord.scala
- Supertypes
- Self type
-
ConsumerRecord.type
ConsumerSettings contain settings necessary to create a KafkaConsumer.
ConsumerSettings contain settings necessary to create a KafkaConsumer. At the very least, this includes key and value deserializers.
The following consumer configuration defaults are used.
- auto.offset.reset
is set to none
to avoid the surprise of the otherwise default latest
setting.
- enable.auto.commit
is set to false
since offset commits are managed manually.
Several convenience functions are provided so that you don't have to work with String
values and ConsumerConfig
for configuration. It's still possible to specify ConsumerConfig
values with functions like withProperty.
ConsumerSettings instances are immutable and all modification functions return a new ConsumerSettings instance.
Use ConsumerSettings#apply
to create a new instance.
Attributes
- Companion
- object
- Source
- ConsumerSettings.scala
- Supertypes
Attributes
- Companion
- class
- Source
- ConsumerSettings.scala
- Supertypes
- Self type
-
ConsumerSettings.type
ConsumerShutdownException indicates that a request could not be completed because the consumer has already shutdown.
ConsumerShutdownException indicates that a request could not be completed because the consumer has already shutdown.
Attributes
- Source
- ConsumerShutdownException.scala
- Supertypes
-
class KafkaExceptionclass RuntimeExceptionclass Exceptionclass Throwabletrait Serializableclass Objecttrait Matchableclass AnyShow all
Exception raised with Deserializer#failWith when deserialization was unable to complete successfully.
Exception raised with Deserializer#failWith when deserialization was unable to complete successfully.
Attributes
- Source
- DeserializationException.scala
- Supertypes
-
class KafkaExceptionclass RuntimeExceptionclass Exceptionclass Throwabletrait Serializableclass Objecttrait Matchableclass AnyShow all
Functional composable Kafka key- and record deserializer with support for effect types.
Functional composable Kafka key- and record deserializer with support for effect types.
Attributes
- Companion
- object
- Source
- Deserializer.scala
- Supertypes
Attributes
- Companion
- class
- Source
- Deserializer.scala
- Supertypes
- Self type
-
GenericDeserializer.type
Attributes
- Companion
- object
- Source
- Serializer.scala
- Supertypes
Functional composable Kafka key- and record serializer with support for effect types.
Functional composable Kafka key- and record serializer with support for effect types.
Attributes
- Companion
- class
- Source
- Serializer.scala
- Supertypes
- Self type
-
GenericSerializer.type
Header represents a String
key and Array[Byte]
value which can be included as part of Headers when creating a ProducerRecord.
Header represents a String
key and Array[Byte]
value which can be included as part of Headers when creating a ProducerRecord. Headers are included together with a record once produced, and can be used by consumers.
To create a new Header, use Header#apply.
Attributes
- Companion
- object
- Source
- Header.scala
- Supertypes
Attributes
- Companion
- class
- Source
- Header.scala
- Supertypes
- Self type
-
Header.type
HeaderDeserializer is a functional deserializer for Kafka record header values.
HeaderDeserializer is a functional deserializer for Kafka record header values. It's similar to Deserializer, except it only has access to the header bytes, and it does not interoperate with the Kafka Deserializer
interface.
Attributes
- Companion
- object
- Source
- HeaderDeserializer.scala
- Supertypes
Attributes
- Companion
- class
- Source
- HeaderDeserializer.scala
- Supertypes
- Self type
-
HeaderDeserializer.type
HeaderSerializer is a functional serializer for Kafka record header values.
HeaderSerializer is a functional serializer for Kafka record header values. It's similar to Serializer, except it only has access to the value, and it does not interoperate with the Kafka Serializer
interface.
Attributes
- Companion
- object
- Source
- HeaderSerializer.scala
- Supertypes
Attributes
- Companion
- class
- Source
- HeaderSerializer.scala
- Supertypes
- Self type
-
HeaderSerializer.type
Headers represent an immutable append-only collection of Headers. To create a new Headers instance, you can use Headers#apply or Headers#empty and add an instance of Header using append
.
Attributes
- Companion
- object
- Source
- Headers.scala
- Supertypes
Attributes
- Companion
- class
- Source
- Headers.scala
- Supertypes
- Self type
-
Headers.type
The available options for ConsumerSettings#withIsolationLevel.
Available options include:
- IsolationLevel#ReadCommitted to only read committed records,
- IsolationLevel#ReadUncommitted to also read uncommitted records.
The available options for ConsumerSettings#withIsolationLevel.
Available options include:
- IsolationLevel#ReadCommitted to only read committed records,
- IsolationLevel#ReadUncommitted to also read uncommitted records.
Attributes
- Companion
- object
- Source
- IsolationLevel.scala
- Supertypes
Attributes
- Companion
- class
- Source
- IsolationLevel.scala
- Supertypes
- Self type
-
IsolationLevel.type
Jitter represents the ability to apply jitter to an existing value n
, effectively multiplying n
with a pseudorandom value between 0
and 1
(both inclusive, although implementation dependent).
The default Jitter#default uses java.util.Random
for pseudorandom values and always applies jitter with a value between 0
(inclusive) and 1
(exclusive).
Jitter represents the ability to apply jitter to an existing value n
, effectively multiplying n
with a pseudorandom value between 0
and 1
(both inclusive, although implementation dependent).
The default Jitter#default uses java.util.Random
for pseudorandom values and always applies jitter with a value between 0
(inclusive) and 1
(exclusive). If no jitter is desired, use Jitter#none.
Attributes
- Companion
- object
- Source
- Jitter.scala
- Supertypes
Attributes
- Companion
- class
- Source
- Jitter.scala
- Supertypes
- Self type
-
Jitter.type
KafkaAdminClient represents an admin client for Kafka, which is able to describe queries about topics, consumer groups, offsets, and other entities related to Kafka.
Use KafkaAdminClient.resource or KafkaAdminClient.stream to create an instance.
KafkaAdminClient represents an admin client for Kafka, which is able to describe queries about topics, consumer groups, offsets, and other entities related to Kafka.
Use KafkaAdminClient.resource or KafkaAdminClient.stream to create an instance.
Attributes
- Companion
- object
- Source
- KafkaAdminClient.scala
- Supertypes
Attributes
- Companion
- class
- Source
- KafkaAdminClient.scala
- Supertypes
- Self type
-
KafkaAdminClient.type
KafkaConsumer represents a consumer of Kafka records, with the ability to subscribe
to topics, start a single top-level stream, and optionally control it via the provided fiber instance.
The following top-level streams are provided.
- stream provides a single stream of records, where the order of records is guaranteed per topic-partition.
- partitionedStream provides a stream with elements as streams that continually request records for a single partition.
KafkaConsumer represents a consumer of Kafka records, with the ability to subscribe
to topics, start a single top-level stream, and optionally control it via the provided fiber instance.
The following top-level streams are provided.
- stream provides a single stream of records, where the order of records is guaranteed per topic-partition.
- partitionedStream provides a stream with elements as streams that continually request records for a single partition. Order is guaranteed per topic-partition, but all assigned partitions will have to be processed in parallel.
- partitionsMapStream provides a stream where each element contains a current assignment. The current assignment is the Map
, where keys is a TopicPartition
, and values are streams with records for a particular TopicPartition
.
For the streams, records are wrapped in CommittableConsumerRecords which provide CommittableOffsets with the ability to commit record offsets to Kafka. For performance reasons, offsets are usually committed in batches using CommittableOffsetBatch. Provided Pipe
s, like commitBatchWithin are available for batch committing offsets. If you are not committing offsets to Kafka, you can simply discard the CommittableOffset, and only make use of the record.
While it's technically possible to start more than one stream from a single KafkaConsumer, it is generally not recommended as there is no guarantee which stream will receive which records, and there might be an overlap, in terms of duplicate records, between the two streams. If a first stream completes, possibly with error, there's no guarantee the stream has processed all of the records it received, and a second stream from the same KafkaConsumer might not be able to pick up where the first one left off. Therefore, only create a single top-level stream per KafkaConsumer, and if you want to start a new stream if the first one finishes, let the KafkaConsumer shutdown and create a new one.
Attributes
- Companion
- object
- Source
- KafkaConsumer.scala
- Supertypes
-
trait KafkaConsumerLifecycle[F]trait KafkaMetrics[F]trait KafkaCommit[F]trait KafkaTopics[F]trait KafkaSubscription[F]trait KafkaOffsetsV2[F]trait KafkaOffsets[F]trait KafkaAssignment[F]class Objecttrait Matchableclass AnyShow all
Attributes
- Companion
- class
- Source
- KafkaConsumer.scala
- Supertypes
- Self type
-
KafkaConsumer.type
KafkaProducer represents a producer of Kafka records, with the ability to produce ProducerRecord
s using produce.
KafkaProducer represents a producer of Kafka records, with the ability to produce ProducerRecord
s using produce.
Attributes
- Companion
- object
- Source
- KafkaProducer.scala
- Supertypes
- Known subtypes
-
Attributes
- Companion
- class
- Source
- KafkaProducer.scala
- Supertypes
- Self type
-
KafkaProducer.type
KafkaProducerConnection represents a connection to a Kafka broker that can be used to create KafkaProducer instances.
KafkaProducerConnection represents a connection to a Kafka broker that can be used to create KafkaProducer instances. All KafkaProducer instances created from an given KafkaProducerConnection share a single underlying connection.
Attributes
- Companion
- object
- Source
- KafkaProducerConnection.scala
- Supertypes
Attributes
- Companion
- class
- Source
- KafkaProducerConnection.scala
- Supertypes
- Self type
Attributes
- Source
- package.scala
- Supertypes
Phantom types to indicate whether a Serializer/Deserializer if for keys, values, or both
Phantom types to indicate whether a Serializer/Deserializer if for keys, values, or both
Attributes
- Source
- package.scala
- Supertypes
- Known subtypes
NotSubscribedException indicates that a Stream
was started in KafkaConsumer even though the consumer had not been subscribed to any topics before starting.
NotSubscribedException indicates that a Stream
was started in KafkaConsumer even though the consumer had not been subscribed to any topics before starting.
Attributes
- Source
- NotSubscribedException.scala
- Supertypes
-
class KafkaExceptionclass RuntimeExceptionclass Exceptionclass Throwabletrait Serializableclass Objecttrait Matchableclass AnyShow all
ProducerRecord represents a record which can be produced to Kafka.
ProducerRecord represents a record which can be produced to Kafka. At the very least, this includes a key of type K
, a value of type V
, and to which topic the record should be produced. The partition, timestamp, and headers can be set by using the withPartition, withTimestamp, and withHeaders functions, respectively.
To create a new instance, use ProducerRecord#apply.
Attributes
- Companion
- object
- Source
- ProducerRecord.scala
- Supertypes
Attributes
- Companion
- class
- Source
- ProducerRecord.scala
- Supertypes
- Self type
-
ProducerRecord.type
Attributes
- Source
- package.scala
- Supertypes
- Self type
-
ProducerRecords.type
ProducerSettings contain settings necessary to create a KafkaProducer.
ProducerSettings contain settings necessary to create a KafkaProducer. At the very least, this includes a key serializer and a value serializer.
Several convenience functions are provided so that you don't have to work with String
values and ProducerConfig
for configuration. It's still possible to specify ProducerConfig
values with functions like withProperty.
ProducerSettings instances are immutable and all modification functions return a new ProducerSettings instance.
Use ProducerSettings#apply
to create a new instance.
Attributes
- Companion
- object
- Source
- ProducerSettings.scala
- Supertypes
Attributes
- Companion
- class
- Source
- ProducerSettings.scala
- Supertypes
- Self type
-
ProducerSettings.type
Exception raised with Serializer#failWith when serialization was unable to complete successfully.
Exception raised with Serializer#failWith when serialization was unable to complete successfully.
Attributes
- Source
- SerializationException.scala
- Supertypes
-
class KafkaExceptionclass RuntimeExceptionclass Exceptionclass Throwabletrait Serializableclass Objecttrait Matchableclass AnyShow all
Timestamp is an optional timestamp value representing a createTime, logAppendTime, unknownTime, or no timestamp at all.
Timestamp is an optional timestamp value representing a createTime, logAppendTime, unknownTime, or no timestamp at all.
Attributes
- Companion
- object
- Source
- Timestamp.scala
- Supertypes
Attributes
- Companion
- class
- Source
- Timestamp.scala
- Supertypes
- Self type
-
Timestamp.type
Represents a producer of Kafka records specialized for 'read-process-write' streams, with the ability to atomically produce ProducerRecord
s and commit corresponding CommittableOffsets using produce.
Records are wrapped in TransactionalProducerRecords, which is a chunk of CommittableProducerRecord which wrap zero or more records together with a CommittableOffset.
Represents a producer of Kafka records specialized for 'read-process-write' streams, with the ability to atomically produce ProducerRecord
s and commit corresponding CommittableOffsets using produce.
Records are wrapped in TransactionalProducerRecords, which is a chunk of CommittableProducerRecord which wrap zero or more records together with a CommittableOffset.
Attributes
- Companion
- object
- Source
- TransactionalKafkaProducer.scala
- Supertypes
- Known subtypes
-
Attributes
- Companion
- class
- Source
- TransactionalKafkaProducer.scala
- Supertypes
- Self type
Attributes
- Source
- package.scala
- Supertypes
- Self type
TransactionalProducerSettings contain settings necessary to create a TransactionalKafkaProducer.
TransactionalProducerSettings contain settings necessary to create a TransactionalKafkaProducer. This includes a transactional ID and any other ProducerSettings.
TransactionalProducerSettings instances are immutable and modification functions return a new TransactionalProducerSettings instance.
Use TransactionalProducerSettings.apply to create a new instance.
Attributes
- Companion
- object
- Source
- TransactionalProducerSettings.scala
- Supertypes
Attributes
- Companion
- class
- Source
- TransactionalProducerSettings.scala
- Supertypes
- Self type
UnexpectedTopicException is raised when serialization or deserialization occurred for an unexpected topic which isn't supported by the Serializer or Deserializer.
UnexpectedTopicException is raised when serialization or deserialization occurred for an unexpected topic which isn't supported by the Serializer or Deserializer.
Attributes
- Source
- UnexpectedTopicException.scala
- Supertypes
-
class KafkaExceptionclass RuntimeExceptionclass Exceptionclass Throwabletrait Serializableclass Objecttrait Matchableclass AnyShow all
Attributes
- Source
- package.scala
- Supertypes
Attributes
- Source
- instances.scala
- Supertypes
- Self type
-
instances.type
Types
Attributes
- Source
- package.scala
Attributes
- Source
- package.scala
Alias for Java Kafka Consumer[Array[Byte], Array[Byte]]
.
Alias for Java Kafka ConsumerRecord[Array[Byte], Array[Byte]]
.
Alias for Java Kafka ConsumerRecords[Array[Byte], Array[Byte]]
.
Alias for Java Kafka Producer[Array[Byte], Array[Byte]]
.
Alias for Java Kafka ProducerRecord[Array[Byte], Array[Byte]]
.
Alias for Java Kafka Deserializer[A]
.
Alias for Java Kafka Header
.
Alias for Java Kafka Headers
.
Alias for Java Kafka Serializer[A]
.
Attributes
- Source
- package.scala
Attributes
- Source
- package.scala
Attributes
- Source
- package.scala
Attributes
- Source
- package.scala
Attributes
- Source
- package.scala
Attributes
- Source
- package.scala
Attributes
- Source
- package.scala
Attributes
- Source
- package.scala
Value members
Concrete methods
Commits offsets in batches of every n
offsets or time window of length d
, whichever happens first.
Commits offsets in batches of every n
offsets or time window of length d
, whichever happens first. If there are no offsets to commit within a time window, no attempt will be made to commit offsets for that time window.
Attributes
- Source
- package.scala