Class KafkaIdempotentRepository
java.lang.Object
org.apache.camel.support.service.BaseService
org.apache.camel.support.service.ServiceSupport
org.apache.camel.processor.idempotent.kafka.KafkaIdempotentRepository
- All Implemented Interfaces:
AutoCloseable
,org.apache.camel.CamelContextAware
,org.apache.camel.Service
,org.apache.camel.ShutdownableService
,org.apache.camel.spi.HasCamelContext
,org.apache.camel.spi.IdempotentRepository
,org.apache.camel.StatefulService
,org.apache.camel.SuspendableService
@ManagedResource(description="Kafka IdempotentRepository")
public class KafkaIdempotentRepository
extends org.apache.camel.support.service.ServiceSupport
implements org.apache.camel.spi.IdempotentRepository, org.apache.camel.CamelContextAware
A Kafka topic-based implementation of
IdempotentRepository
. Uses a local cache of
previously seen Message IDs. Mutations that come in via the (add(String)
), or remove(String)
method
will update the local cache and broadcast the change in state on a Kafka topic to other instances. The cache is
back-filled from the topic by a Kafka consumer. The topic used must be unique per logical repository (i.e. two routes
de-duplicate using different repositories, and different topics). This class makes no assumptions about the number of
partitions (it is designed to consume from all at the same time), or replication factor of the topic. Each repository
instance that uses the topic (e.g. typically on different machines running in parallel) controls its own consumer
group, so in a cluster of 10 Camel processes using the same topic each will control its own offset. On startup, the
instance subscribes to the topic and rewinds the offset to the beginning, rebuilding the cache to the latest state.
The cache will not be considered warmed up until one poll of pollDurationMs
in length returns 0 records.
Startup will not be completed until either the cache has warmed up, or 30 seconds go by; if the latter happens the
idempotent repository may be in an inconsistent state until its consumer catches up to the end of the topic. To use,
this repository must be placed in the Camel registry, either manually or by registration as a bean in
Spring/Blueprint, as it is CamelContext aware.-
Field Summary
Fields inherited from class org.apache.camel.support.service.BaseService
BUILT, FAILED, INITIALIZED, INITIALIZING, lock, NEW, SHUTDOWN, SHUTTING_DOWN, STARTED, STARTING, status, STOPPED, STOPPING, SUSPENDED, SUSPENDING
-
Constructor Summary
ConstructorsConstructorDescriptionNo-op constructor for XML/property-based object initialisation.KafkaIdempotentRepository
(String topic, String bootstrapServers) KafkaIdempotentRepository
(String topic, String bootstrapServers, int maxCacheSize, int pollDurationMs) KafkaIdempotentRepository
(String topic, String bootstrapServers, int maxCacheSize, int pollDurationMs, String groupId) KafkaIdempotentRepository
(String topic, String bootstrapServers, String groupId) KafkaIdempotentRepository
(String topic, Properties consumerConfig, Properties producerConfig) KafkaIdempotentRepository
(String topic, Properties consumerConfig, Properties producerConfig, int maxCacheSize, int pollDurationMs) KafkaIdempotentRepository
(String topic, Properties consumerConfig, Properties producerConfig, int maxCacheSize, int pollDurationMs, String groupId) KafkaIdempotentRepository
(String topic, Properties consumerConfig, Properties producerConfig, String groupId) -
Method Summary
Modifier and TypeMethodDescriptionboolean
void
clear()
boolean
boolean
protected void
doStart()
protected void
doStop()
org.apache.camel.CamelContext
int
int
getTopic()
boolean
boolean
boolean
void
setBootstrapServers
(String bootstrapServers) Sets thevoid
setCamelContext
(org.apache.camel.CamelContext camelContext) void
setConsumerConfig
(Properties consumerConfig) Sets the properties that will be used by the Kafka consumer.void
setGroupId
(String groupId) Sets the group id of the Kafka consumer.void
setMaxCacheSize
(int maxCacheSize) Sets the maximum size of the local key cache.void
setPollDurationMs
(int pollDurationMs) Sets the poll duration of the Kafka consumer.void
setProducerConfig
(Properties producerConfig) Sets the properties that will be used by the Kafka producer.void
Sets the name of the Kafka topic used by this idempotent repository.Methods inherited from class org.apache.camel.support.service.BaseService
build, doBuild, doFail, doInit, doLifecycleChange, doResume, doShutdown, doSuspend, fail, getStatus, init, isBuild, isInit, isNew, isRunAllowed, isShutdown, isStarted, isStarting, isStartingOrStarted, isStopped, isStopping, isStoppingOrStopped, isSuspended, isSuspending, isSuspendingOrSuspended, resume, shutdown, start, stop, suspend
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
Methods inherited from interface org.apache.camel.spi.IdempotentRepository
add, confirm, contains, remove
Methods inherited from interface org.apache.camel.Service
build, close, init, start, stop
Methods inherited from interface org.apache.camel.ShutdownableService
shutdown
Methods inherited from interface org.apache.camel.StatefulService
getStatus, isRunAllowed, isStarted, isStarting, isStopped, isStopping, isSuspending
Methods inherited from interface org.apache.camel.SuspendableService
isSuspended, resume, suspend
-
Constructor Details
-
KafkaIdempotentRepository
public KafkaIdempotentRepository()No-op constructor for XML/property-based object initialisation. From Java, prefer one of the other constructors. -
KafkaIdempotentRepository
-
KafkaIdempotentRepository
-
KafkaIdempotentRepository
-
KafkaIdempotentRepository
public KafkaIdempotentRepository(String topic, Properties consumerConfig, Properties producerConfig) -
KafkaIdempotentRepository
public KafkaIdempotentRepository(String topic, Properties consumerConfig, Properties producerConfig, String groupId) -
KafkaIdempotentRepository
public KafkaIdempotentRepository(String topic, Properties consumerConfig, Properties producerConfig, int maxCacheSize, int pollDurationMs) -
KafkaIdempotentRepository
-
KafkaIdempotentRepository
public KafkaIdempotentRepository(String topic, Properties consumerConfig, Properties producerConfig, int maxCacheSize, int pollDurationMs, String groupId)
-
-
Method Details
-
getTopic
-
setTopic
Sets the name of the Kafka topic used by this idempotent repository. Each functionally-separate repository should use a different topic.- Parameters:
topic
- The topic name.
-
getBootstrapServers
-
setBootstrapServers
Sets thebootstrap.servers
property on the internal Kafka producer and consumer. Use this as shorthand if not settingconsumerConfig
andproducerConfig
. If used, this component will apply sensible default configurations for the producer and consumer.- Parameters:
bootstrapServers
- Thebootstrap.servers
value to use.
-
getProducerConfig
-
setProducerConfig
Sets the properties that will be used by the Kafka producer. OverridesbootstrapServers
, so must define thebootstrap.servers
property itself. Prefer usingbootstrapServers
for default configuration unless you specifically need non-standard configuration options such as SSL/SASL.- Parameters:
producerConfig
- The producer configuration properties.
-
getConsumerConfig
-
setConsumerConfig
Sets the properties that will be used by the Kafka consumer. OverridesbootstrapServers
, so must define thebootstrap.servers
property itself. Prefer usingbootstrapServers
for default configuration unless you specifically need non-standard configuration options such as SSL/SASL.- Parameters:
consumerConfig
- The consumer configuration properties.
-
getMaxCacheSize
public int getMaxCacheSize() -
setMaxCacheSize
public void setMaxCacheSize(int maxCacheSize) Sets the maximum size of the local key cache.- Parameters:
maxCacheSize
- The maximum key cache size.
-
getPollDurationMs
public int getPollDurationMs() -
setPollDurationMs
public void setPollDurationMs(int pollDurationMs) Sets the poll duration of the Kafka consumer. The local caches are updated immediately; this value will affect how far behind other peers in the cluster are, which are updating their caches from the topic, relative to the idempotent consumer instance issued the cache action message. The default value of this isDEFAULT_POLL_DURATION_MS
. If setting this value explicitly, be aware that there is a tradeoff between the remote cache liveness and the volume of network traffic between this repository's consumer and the Kafka brokers. The cache warmup process also depends on there being one poll that fetches nothing - this indicates that the stream has been consumed up to the current point. If the poll duration is excessively long for the rate at which messages are sent on the topic, there exists a possibility that the cache cannot be warmed up and will operate in an inconsistent state relative to its peers until it catches up.- Parameters:
pollDurationMs
- The poll duration in milliseconds.
-
getGroupId
-
setGroupId
Sets the group id of the Kafka consumer.- Parameters:
groupId
- The poll duration in milliseconds.
-
setCamelContext
public void setCamelContext(org.apache.camel.CamelContext camelContext) - Specified by:
setCamelContext
in interfaceorg.apache.camel.CamelContextAware
-
getCamelContext
public org.apache.camel.CamelContext getCamelContext()- Specified by:
getCamelContext
in interfaceorg.apache.camel.spi.HasCamelContext
-
doStart
- Overrides:
doStart
in classorg.apache.camel.support.service.BaseService
- Throws:
Exception
-
doStop
protected void doStop()- Overrides:
doStop
in classorg.apache.camel.support.service.BaseService
-
add
- Specified by:
add
in interfaceorg.apache.camel.spi.IdempotentRepository
-
contains
@ManagedOperation(description="Does the store contain the given key") public boolean contains(String key) - Specified by:
contains
in interfaceorg.apache.camel.spi.IdempotentRepository
-
remove
- Specified by:
remove
in interfaceorg.apache.camel.spi.IdempotentRepository
-
confirm
- Specified by:
confirm
in interfaceorg.apache.camel.spi.IdempotentRepository
-
clear
public void clear()- Specified by:
clear
in interfaceorg.apache.camel.spi.IdempotentRepository
-
isPollerRunning
@ManagedOperation(description="Number of times duplicate messages have been detected") public boolean isPollerRunning() -
isCacheReady
public boolean isCacheReady()
-