KafkaClient
Related Docs:
object KafkaClient
| package kafka
Like leaders but queries only for supplied topics
Creates discrete signal of leaders that is queried from periodical query of metadata from brokers.
Creates discrete signal of leaders that is queried from periodical query of metadata from brokers.
While this stream is consumed, this will keep connection with very first broker that have answered request for metadata succesfully.
If there is no broker available to server metadata request (all brokers was failing recently w/o providing valid response), this will fail as NoBrokerAvailable.
If the broker from which metadata are queried will fail, this will try next broker in supplied seed.
Delay to refresh new metadata from last known good broker
Queries offsets for given topic and partition.
Queries offsets for given topic and partition. Returns offset of first message kept (head) and offset of next message that will arrive to topic. When numbers are equal, then the topic does not include any messages at all.
Id of the topic
Id of the partition
Publishes Chunk of messages to the ensemble.
Publishes Chunk of messages to the ensemble. The messages are published as a whole batch, so when this terminates, all messages are guaranteed to be processed by kafka server.
Returns offset of very first message published.
Id of the topic to publish to.
Partition to publish to.
If true, this requires quorum of ISR to commit message before leader will reply. If false, only leader is required to confirm this publish request.
Defines how long to wait for server to confirm that messages were published. Note that this will fail the resulting task if timeout is exceeded, but that won't guarantee that messages were not published.
When defined, messages will be compressed by supplied algorithm.
Chunk of messages to publish. First is id of the topic, second is partition, then key and message itself.
Additionally A may be passed to pair the offset of the message in resulting chunk.
Like publishN except this won't await for messages to be confirmed to be published successfully.
Like publishN except this won't await for messages to be confirmed to be published successfully.
Note that this does not guarantee that message was even sent to server. It will get queued and will be delivered to server within earliest opportunity (once server will be ready to accept it).
Subscribes to specified topic to receive messages published to that topic.
Subscribes to specified topic to receive messages published to that topic.
Essentially this acts sort of unix tail command.
Note that user can fine-tune reads from topic by specifying minChunkByteSize, maxChunkByteSize and maxWaitTime parameters
to optimize chunking and flow control of reads from Kafka. Default values provide polling each 1 minute whenever at least one message is available.
User can by fine-tuning the maxWaitTime and leaderFailureMaxAttempts recovery in case of leadership changes in kafka cluster.
For example, when leader fails, the stream will stop for about leaderFailureTimeout and then tries to continue where the last fetch ended.
However wehn there are leaderFailureMaxAttempts successive failures, then the stream will fail.
Setting leaderFailureTimeout to 0 and leaderFailureMaxAttempts to 0 will cause resulting stream to fail immediatelly when any failure occurs.
Name of the topic to subscribe to
Partition to subscribe to
Offset of the topic to start to read from. First received message may have offset larger than supplied offset only if the oldest message has offset higher than supplied offset. Otherwise this will always return first message with this offset. -1 specified start from tail (new message arriving to topic)
When true, the implementation will prefetch next chunk of messages from kafka while processing last chunk of messages.
Min size of bytes to read from kafka in single attempt. That number of bytes must be available, in order for read to succeed.
Max number of bytes to include in reply. Should be always > than max siz of single message including key.
Maximum time to wait before reply, even when minChunkByteSize is not satisfied.
When fetch from Kafka leader fails, this will try to recover connection every this period up to leaderFailureMaxAttempts attempt count is exhausted
Maximum attempts to recover from leader failure, then this will fail.
Publishes single message to the supplied topic.
Publishes single message to the supplied topic. Returns None, if the message was not published due topic/partition not existent or Some(offset) of published message.
When F finishes its evaluation, message is guaranteed to be seen by the ensemble.
Topic to publish to
Partition to publish to
Key of the message
Message itself
If true, this requires quorum of ISR to commit message before leader will reply. If false, only leader is required to confirm this publish request.
Timeout server waits for replicas to ack the request. If the publish request won't be acked by server in this time, then the request fails to be published.
Like publish except this won't wait for the confirmation that message was published (fire'n forget).
Like publish except this won't wait for the confirmation that message was published (fire'n forget).
Note that this does not guarantee that message was even sent to server. It will get queued and will be delivered to server within earliest opportunity (once server will be ready to accept it).
Client that binds to kafka broker. Usually application need only one client.
Client lives until the emitted process is interrupted, or fails.