Use custom interval for auto-commit or commit flushing on manual commit.
Consumer Timeout Throw a timeout exception to the consumer if no message is available for consumption after the specified interval
Dump current props for debugging
Store offsets in Kafka and/or ZooKeeper.
Store offsets in Kafka and/or ZooKeeper. NOTE: Server instance must be 8.2 or higher
dualCommit = true means store in both ZooKeeper(legacy) and Kafka(new) places.
What to do when there is no initial offset in Zookeeper or if an offset is out of range: 1) smallest : automatically reset the offset to the smallest offset 2) largest : automatically reset the offset to the largest offset 3) anything else: throw exception to the consumer.
What to do when there is no initial offset in Zookeeper or if an offset is out of range: 1) smallest : automatically reset the offset to the smallest offset 2) largest : automatically reset the offset to the largest offset 3) anything else: throw exception to the consumer. If this is set to largest, the consumer may lose some messages when the number of partitions, for the topics it subscribes to, changes on the broker.
*************************************************************************************** To prevent data loss during partition addition, set auto.offset.reset to smallest
This make sense to change to true if you know you are listening for new data only as of after you connect to the stream new things are coming out. you can audit/reconcile in another consumer which this flag allows you to toggle if it is catch-up and new stuff or just new stuff coming out of the stream. This will also block waiting for new stuff so it makes a good listener.
//readFromStartOfStream: Boolean = true readFromStartOfStream: Boolean = false ***************************************************************************************
Set any additional properties as needed
Generate the Kafka ConsumerConfig object