Package

monix

kafka

Permalink

package kafka

Visibility
  1. Public
  2. All

Type Members

  1. final case class Deserializer[A](className: String, classType: Class[_ <: org.apache.kafka.common.serialization.Deserializer[A]]) extends Product with Serializable

    Permalink

    Wraps a Kafka Deserializer, provided for convenience, since it can be implicitly fetched from the context.

  2. final case class KafkaConsumerConfig(bootstrapServers: List[String], fetchMinBytes: Int, groupId: String, heartbeatInterval: FiniteDuration, maxPartitionFetchBytes: Int, sessionTimeout: FiniteDuration, sslKeyPassword: Option[String], sslKeyStorePassword: Option[String], sslKeyStoreLocation: Option[String], sslTrustStoreLocation: Option[String], sslTrustStorePassword: Option[String], autoOffsetReset: AutoOffsetReset, connectionsMaxIdleTime: FiniteDuration, enableAutoCommit: Boolean, excludeInternalTopics: Boolean, maxPollRecords: Int, receiveBufferInBytes: Int, requestTimeout: FiniteDuration, saslKerberosServiceName: Option[String], saslMechanism: String, securityProtocol: SecurityProtocol, sendBufferInBytes: Int, sslEnabledProtocols: List[SSLProtocol], sslKeystoreType: String, sslProtocol: SSLProtocol, sslProvider: Option[String], sslTruststoreType: String, checkCRCs: Boolean, clientId: String, fetchMaxWaitTime: FiniteDuration, metadataMaxAge: FiniteDuration, reconnectBackoffTime: FiniteDuration, retryBackoffTime: FiniteDuration, observableCommitType: ObservableCommitType, observableCommitOrder: ObservableCommitOrder, observableSeekToEndOnStart: Boolean) extends Product with Serializable

    Permalink

    Configuration for Kafka Consumer.

    Configuration for Kafka Consumer.

    For the official documentation on the available configuration options, see Consumer Configs on kafka.apache.org.

    bootstrapServers

    is the bootstrap.servers setting, a list of host/port pairs to use for establishing the initial connection to the Kafka cluster.

    fetchMinBytes

    is the fetch.min.bytes setting, the minimum amount of data the server should return for a fetch request.

    groupId

    is the group.id setting, a unique string that identifies the consumer group this consumer belongs to.

    heartbeatInterval

    is the heartbeat.interval.ms setting, the expected time between heartbeats to the consumer coordinator when using Kafka's group management facilities.

    maxPartitionFetchBytes

    is the max.partition.fetch.bytes setting, the maximum amount of data per-partition the server will return.

    sessionTimeout

    is the session.timeout.ms setting, the timeout used to detect failures when using Kafka's group management facilities.

    sslKeyPassword

    is the ssl.key.password setting and represents the password of the private key in the key store file. This is optional for client.

    sslKeyStorePassword

    is the ssl.keystore.password setting, being the password of the private key in the key store file. This is optional for client.

    sslKeyStoreLocation

    is the ssl.keystore.location setting and represents the location of the key store file. This is optional for client and can be used for two-way authentication for client.

    sslTrustStoreLocation

    is the ssl.truststore.location setting and is the location of the trust store file.

    sslTrustStorePassword

    is the ssl.truststore.password setting and is the password for the trust store file.

    autoOffsetReset

    is the auto.offset.reset setting, specifying what to do when there is no initial offset in Kafka or if the current offset does not exist any more on the server (e.g. because that data has been deleted).

    connectionsMaxIdleTime

    is the connections.max.idle.ms setting and specifies how much time to wait before closing idle connections.

    enableAutoCommit

    is the enable.auto.commit setting. If true the consumer's offset will be periodically committed in the background.

    excludeInternalTopics

    is the exclude.internal.topics setting. Whether records from internal topics (such as offsets) should be exposed to the consumer. If set to true the only way to receive records from an internal topic is subscribing to it.

    maxPollRecords

    is the max.poll.records setting, the maximum number of records returned in a single call to poll().

    receiveBufferInBytes

    is the receive.buffer.bytes setting, the size of the TCP receive buffer (SO_RCVBUF) to use when reading data.

    requestTimeout

    is the request.timeout.ms setting, The configuration controls the maximum amount of time the client will wait for the response of a request. If the response is not received before the timeout elapses the client will resend the request if necessary or fail the request if retries are exhausted.

    saslKerberosServiceName

    is the sasl.kerberos.service.name setting, being the Kerberos principal name that Kafka runs as.

    saslMechanism

    is the sasl.mechanism setting, being the SASL mechanism used for client connections. This may be any mechanism for which a security provider is available.

    securityProtocol

    is the security.protocol setting, being the protocol used to communicate with brokers.

    sendBufferInBytes

    is the send.buffer.bytes setting, being the size of the TCP send buffer (SO_SNDBUF) to use when sending data.

    sslEnabledProtocols

    is the ssl.enabled.protocols setting, being the list of protocols enabled for SSL connections.

    sslKeystoreType

    is the ssl.keystore.type setting, being the file format of the key store file.

    sslProtocol

    is the ssl.protocol setting, being the SSL protocol used to generate the SSLContext. Default setting is TLS, which is fine for most cases. Allowed values in recent JVMs are TLS, TLSv1.1 and TLSv1.2. SSL, SSLv2 and SSLv3 may be supported in older JVMs, but their usage is discouraged due to known security vulnerabilities.

    sslProvider

    is the ssl.provider setting, being the name of the security provider used for SSL connections. Default value is the default security provider of the JVM.

    sslTruststoreType

    is the ssl.truststore.type setting, being the file format of the trust store file.

    checkCRCs

    is the check.crcs setting, specifying to automatically check the CRC32 of the records consumed. This ensures no on-the-wire or on-disk corruption to the messages occurred. This check adds some overhead, so it may be disabled in cases seeking extreme performance.

    clientId

    is the client.id setting, an id string to pass to the server when making requests. The purpose of this is to be able to track the source of requests beyond just ip/port by allowing a logical application name to be included in server-side request logging.

    fetchMaxWaitTime

    is the fetch.max.wait.ms setting, the maximum amount of time the server will block before answering the fetch request if there isn't sufficient data to immediately satisfy the requirement given by fetch.min.bytes.

    metadataMaxAge

    is the metadata.max.age.ms setting. The period of time in milliseconds after which we force a refresh of metadata even if we haven't seen any partition leadership changes to proactively discover any new brokers or partitions.

    reconnectBackoffTime

    is the reconnect.backoff.ms setting. The amount of time to wait before attempting to reconnect to a given host. This avoids repeatedly connecting to a host in a tight loop. This backoff applies to all requests sent by the consumer to the broker.

    retryBackoffTime

    is the retry.backoff.ms setting. The amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

    observableCommitType

    is the monix.observable.commit.type setting. Represents the type of commit to make when the enableAutoCommit setting is set to false, in which case the observable has to commit on every batch.

    observableCommitOrder

    is the monix.observable.commit.order setting. Specifies when the commit should happen, like before we receive the acknowledgement from downstream, or afterwards.

  3. final class KafkaConsumerObservable[K, V] extends Observable[ConsumerRecord[K, V]]

    Permalink

    Exposes an Observable that consumes a Kafka stream by means of a Kafka Consumer client.

    Exposes an Observable that consumes a Kafka stream by means of a Kafka Consumer client.

    In order to get initialized, it needs a configuration. See the KafkaConsumerConfig needed and see monix/kafka/default.conf, (in the resource files) that is exposing all default values.

  4. trait KafkaProducer[K, V] extends Serializable

    Permalink

    Wraps the Kafka Producer.

  5. case class KafkaProducerConfig(bootstrapServers: List[String], acks: Acks, bufferMemoryInBytes: Int, compressionType: CompressionType, retries: Int, sslKeyPassword: Option[String], sslKeyStorePassword: Option[String], sslKeyStoreLocation: Option[String], sslTrustStoreLocation: Option[String], sslTrustStorePassword: Option[String], batchSizeInBytes: Int, clientId: String, connectionsMaxIdleTime: FiniteDuration, lingerTime: FiniteDuration, maxBlockTime: FiniteDuration, maxRequestSizeInBytes: Int, partitionerClass: Option[PartitionerName], receiveBufferInBytes: Int, requestTimeout: FiniteDuration, saslKerberosServiceName: Option[String], saslMechanism: String, securityProtocol: SecurityProtocol, sendBufferInBytes: Int, sslEnabledProtocols: List[SSLProtocol], sslKeystoreType: String, sslProtocol: SSLProtocol, sslProvider: Option[String], sslTruststoreType: String, reconnectBackoffTime: FiniteDuration, retryBackoffTime: FiniteDuration, metadataMaxAge: FiniteDuration, monixSinkParallelism: Int) extends Product with Serializable

    Permalink

    The Kafka Producer config.

    The Kafka Producer config.

    For the official documentation on the available configuration options, see Producer Configs on kafka.apache.org.

    bootstrapServers

    is the bootstrap.servers setting and represents the list of servers to connect to.

    acks

    is the acks setting and represents the number of acknowledgments the producer requires the leader to have received before considering a request complete. See Acks.

    bufferMemoryInBytes

    is the buffer.memory setting and represents the total bytes of memory the producer can use to buffer records waiting to be sent to the server.

    compressionType

    is the compression.type setting and specifies what compression algorithm to apply to all the generated data by the producer. The default is none (no compression applied).

    retries

    is the retries setting. A value greater than zero will cause the client to resend any record whose send fails with a potentially transient error.

    sslKeyPassword

    is the ssl.key.password setting and represents the password of the private key in the key store file. This is optional for client.

    sslKeyStorePassword

    is the ssl.keystore.password setting, being the password of the private key in the key store file. This is optional for client.

    sslKeyStoreLocation

    is the ssl.keystore.location setting and represents the location of the key store file. This is optional for client and can be used for two-way authentication for client.

    sslTrustStoreLocation

    is the ssl.truststore.location setting and is the location of the trust store file.

    sslTrustStorePassword

    is the ssl.truststore.password setting and is the password for the trust store file.

    batchSizeInBytes

    is the batch.size setting. The producer will attempt to batch records together into fewer requests whenever multiple records are being sent to the same partition. This setting specifies the maximum number of records to batch together.

    clientId

    is the client.id setting, an id string to pass to the server when making requests. The purpose of this is to be able to track the source of requests beyond just ip/port by allowing a logical application name to be included in server-side request logging.

    connectionsMaxIdleTime

    is the connections.max.idle.ms setting and specifies how much time to wait before closing idle connections.

    lingerTime

    is the linger.ms setting and specifies to buffer records for more efficient batching, up to the maximum batch size or for the maximum lingerTime. If zero, then no buffering will happen, but if different from zero, then records will be delayed in absence of load.

    maxBlockTime

    is the max.block.ms setting. The configuration controls how long KafkaProducer.send() and KafkaProducer.partitionsFor() will block. These methods can be blocked either because the buffer is full or metadata unavailable.

    maxRequestSizeInBytes

    is the max.request.size setting and represents the maximum size of a request in bytes. This is also effectively a cap on the maximum record size.

    partitionerClass

    is the partitioner.class setting and represents a class that implements the org.apache.kafka.clients.producer.Partitioner interface.

    receiveBufferInBytes

    is the receive.buffer.bytes setting being the size of the TCP receive buffer (SO_RCVBUF) to use when reading data.

    requestTimeout

    is the request.timeout.ms setting, a configuration the controls the maximum amount of time the client will wait for the response of a request.

    saslKerberosServiceName

    is the sasl.kerberos.service.name setting, being the Kerberos principal name that Kafka runs as.

    saslMechanism

    is the sasl.mechanism setting, being the SASL mechanism used for client connections. This may be any mechanism for which a security provider is available.

    securityProtocol

    is the security.protocol setting, being the protocol used to communicate with brokers.

    sendBufferInBytes

    is the send.buffer.bytes setting, being the size of the TCP send buffer (SO_SNDBUF) to use when sending data.

    sslEnabledProtocols

    is the ssl.enabled.protocols setting, being the list of protocols enabled for SSL connections.

    sslKeystoreType

    is the ssl.keystore.type setting, being the file format of the key store file.

    sslProtocol

    is the ssl.protocol setting, being the SSL protocol used to generate the SSLContext. Default setting is TLS, which is fine for most cases. Allowed values in recent JVMs are TLS, TLSv1.1 and TLSv1.2. SSL, SSLv2 and SSLv3 may be supported in older JVMs, but their usage is discouraged due to known security vulnerabilities.

    sslProvider

    is the ssl.provider setting, being the name of the security provider used for SSL connections. Default value is the default security provider of the JVM.

    sslTruststoreType

    is the ssl.truststore.type setting, being the file format of the trust store file.

    reconnectBackoffTime

    is the reconnect.backoff.ms setting. The amount of time to wait before attempting to reconnect to a given host. This avoids repeatedly connecting to a host in a tight loop. This backoff applies to all requests sent by the consumer to the broker.

    retryBackoffTime

    is the retry.backoff.ms setting. The amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

    metadataMaxAge

    is the metadata.max.age.ms setting. The period of time in milliseconds after which we force a refresh of metadata even if we haven't seen any partition leadership changes to proactively discover any new brokers or partitions.

    monixSinkParallelism

    is the monix.producer.sink.parallelism setting indicating how many requests the KafkaProducerSink can execute in parallel.

  6. final class KafkaProducerSink[K, V] extends Consumer[Seq[ProducerRecord[K, V]], Unit] with StrictLogging with Serializable

    Permalink

    A monix.reactive.Consumer that pushes incoming messages into a KafkaProducer.

  7. final case class Serializer[A](className: String, classType: Class[_ <: org.apache.kafka.common.serialization.Serializer[A]]) extends Product with Serializable

    Permalink

    Wraps a Kafka Serializer, provided for convenience, since it can be implicitly fetched from the context.

Ungrouped