Package org.springframework.integration.kafka.inbound

Class KafkaMessageSource<K,​V>

java.lang.Object
org.springframework.integration.util.AbstractExpressionEvaluator
org.springframework.integration.endpoint.AbstractMessageSource<java.lang.Object>
org.springframework.integration.kafka.inbound.KafkaMessageSource<K,​V>
Type Parameters:
K - the key type.
V - the value type.
All Implemented Interfaces:
org.springframework.beans.factory.Aware, org.springframework.beans.factory.BeanFactoryAware, org.springframework.beans.factory.BeanNameAware, org.springframework.beans.factory.DisposableBean, org.springframework.beans.factory.InitializingBean, org.springframework.context.Lifecycle, org.springframework.integration.core.MessageSource<java.lang.Object>, org.springframework.integration.core.Pausable, org.springframework.integration.IntegrationPattern, org.springframework.integration.support.context.NamedComponent, org.springframework.integration.support.management.IntegrationInboundManagement, org.springframework.integration.support.management.IntegrationManagement, org.springframework.integration.support.management.ManageableLifecycle
public class KafkaMessageSource<K,​V>
extends org.springframework.integration.endpoint.AbstractMessageSource<java.lang.Object>
implements org.springframework.integration.core.Pausable
Polled message source for Apache Kafka. Only one thread can poll for data (or acknowledge a message) at a time.

NOTE: If the application acknowledges messages out of order, the acks will be deferred until all messages prior to the offset are ack'd. If multiple records are retrieved and an earlier offset is requeued, records from the subsequent offsets will be redelivered - even if they were processed successfully. Applications should therefore implement idempotency.

Starting with version 3.1.2, this source implements Pausable which allows you to pause and resume the Consumer. While the consumer is paused, you must continue to call AbstractMessageSource.receive() within max.poll.interval.ms, to prevent a rebalance.

Since:
5.4

  • Nested Class Summary

    Nested Classes 
    Modifier and Type Class Description
    static class  KafkaMessageSource.KafkaAckCallback<K,​V>
    AcknowledgmentCallback for Kafka.
    static class  KafkaMessageSource.KafkaAckCallbackFactory<K,​V>
    AcknowledgmentCallbackFactory for KafkaAckInfo.
    static interface  KafkaMessageSource.KafkaAckInfo<K,​V>
    Information for building an KafkaAckCallback.
    class  KafkaMessageSource.KafkaAckInfoImpl
    Information for building an KafkaAckCallback.

    Nested classes/interfaces inherited from interface org.springframework.integration.support.management.IntegrationManagement

    org.springframework.integration.support.management.IntegrationManagement.ManagementOverrides

  • Field Summary

    Fields 
    Modifier and Type Field Description
    static java.lang.String REMAINING_RECORDS
    The number of records remaining from the previous poll.

    Fields inherited from class org.springframework.integration.util.AbstractExpressionEvaluator

    EXPRESSION_PARSER, logger

    Fields inherited from interface org.springframework.integration.support.management.IntegrationManagement

    METER_PREFIX, RECEIVE_COUNTER_NAME, SEND_TIMER_NAME

  • Constructor Summary

    Constructors 
    Constructor Description
    KafkaMessageSource​(org.springframework.kafka.core.ConsumerFactory<K,​V> consumerFactory, org.springframework.kafka.listener.ConsumerProperties consumerProperties)
    Construct an instance with the supplied parameters.
    KafkaMessageSource​(org.springframework.kafka.core.ConsumerFactory<K,​V> consumerFactory, org.springframework.kafka.listener.ConsumerProperties consumerProperties, boolean allowMultiFetch)
    Construct an instance with the supplied parameters.
    KafkaMessageSource​(org.springframework.kafka.core.ConsumerFactory<K,​V> consumerFactory, org.springframework.kafka.listener.ConsumerProperties consumerProperties, KafkaMessageSource.KafkaAckCallbackFactory<K,​V> ackCallbackFactory)
    Construct an instance with the supplied parameters.
    KafkaMessageSource​(org.springframework.kafka.core.ConsumerFactory<K,​V> consumerFactory, org.springframework.kafka.listener.ConsumerProperties consumerProperties, KafkaMessageSource.KafkaAckCallbackFactory<K,​V> ackCallbackFactory, boolean allowMultiFetch)
    Construct an instance with the supplied parameters.

  • Method Summary

    Modifier and Type Method Description
    protected void createConsumer()  
    void destroy()  
    protected java.lang.Object doReceive()  
    java.util.Collection<org.apache.kafka.common.TopicPartition> getAssignedPartitions()
    Return the currently assigned partitions.
    protected java.lang.String getClientId()  
    protected java.time.Duration getCommitTimeout()  
    java.lang.String getComponentType()  
    org.springframework.kafka.listener.ConsumerProperties getConsumerProperties()
    Get a reference to the configured consumer properties; allows further customization of the properties before the source is started.
    protected java.lang.String getGroupId()  
    protected org.springframework.kafka.support.converter.RecordMessageConverter getMessageConverter()  
    protected java.lang.Class<?> getPayloadType()  
    protected long getPollTimeout()  
    protected org.apache.kafka.clients.consumer.ConsumerRebalanceListener getRebalanceListener()  
    boolean isPaused()  
    protected boolean isRawMessageHeader()  
    boolean isRunning()  
    protected void onInit()  
    void pause()  
    void resume()  
    void setCloseTimeout​(java.time.Duration closeTimeout)
    Set the close timeout - default 30 seconds.
    void setMessageConverter​(org.springframework.kafka.support.converter.RecordMessageConverter messageConverter)
    Set the message converter to replace the default MessagingMessageConverter.
    void setPayloadType​(java.lang.Class<?> payloadType)
    Set the payload type.
    void setRawMessageHeader​(boolean rawMessageHeader)
    Set to true to include the raw ConsumerRecord as headers with keys KafkaHeaders.RAW_DATA and IntegrationMessageHeaderAccessor.SOURCE_DATA.
    void start()  
    void stop()  

    Methods inherited from class org.springframework.integration.endpoint.AbstractMessageSource

    buildMessage, getBeanName, getComponentName, getManagedName, getManagedType, getOverrides, isLoggingEnabled, receive, registerMetricsCaptor, setBeanName, setHeaderExpressions, setLoggingEnabled, setManagedName, setManagedType

    Methods inherited from class org.springframework.integration.util.AbstractExpressionEvaluator

    afterPropertiesSet, evaluateExpression, evaluateExpression, evaluateExpression, evaluateExpression, evaluateExpression, evaluateExpression, evaluateExpression, getBeanFactory, getEvaluationContext, getEvaluationContext, getMessageBuilderFactory, setBeanFactory, setConversionService

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    Methods inherited from interface org.springframework.integration.support.management.IntegrationManagement

    getThisAs

    Methods inherited from interface org.springframework.integration.core.MessageSource

    getIntegrationPatternType

  • Field Details

  • Constructor Details

    • KafkaMessageSource

      public KafkaMessageSource​(org.springframework.kafka.core.ConsumerFactory<K,​V> consumerFactory, org.springframework.kafka.listener.ConsumerProperties consumerProperties)
      Construct an instance with the supplied parameters. Fetching multiple records per poll will be disabled.
      Parameters:
      consumerFactory - the consumer factory.
      consumerProperties - the consumer properties.
      Since:
      3.2
      See Also:
      KafkaMessageSource(ConsumerFactory, ConsumerProperties, KafkaAckCallbackFactory, boolean)

    • KafkaMessageSource

      public KafkaMessageSource​(org.springframework.kafka.core.ConsumerFactory<K,​V> consumerFactory, org.springframework.kafka.listener.ConsumerProperties consumerProperties, boolean allowMultiFetch)
      Construct an instance with the supplied parameters. Set 'allowMultiFetch' to true to allow up to max.poll.records to be fetched on each poll. When false (default) max.poll.records is coerced to 1 if the consumer factory is a DefaultKafkaConsumerFactory or otherwise rejected with an IllegalArgumentException. IMPORTANT: When true, you must call AbstractMessageSource.receive() at a sufficient rate to consume the number of records received within max.poll.interval.ms. When false, you must call AbstractMessageSource.receive() within max.poll.interval.ms. pause() will not take effect until the records from the previous poll are consumed.
      Parameters:
      consumerFactory - the consumer factory.
      consumerProperties - the consumer properties.
      allowMultiFetch - true to allow max.poll.records > 1.
      Since:
      3.2

    • KafkaMessageSource

      public KafkaMessageSource​(org.springframework.kafka.core.ConsumerFactory<K,​V> consumerFactory, org.springframework.kafka.listener.ConsumerProperties consumerProperties, KafkaMessageSource.KafkaAckCallbackFactory<K,​V> ackCallbackFactory)
      Construct an instance with the supplied parameters. Fetching multiple records per poll will be disabled.
      Parameters:
      consumerFactory - the consumer factory.
      consumerProperties - the consumer properties.
      ackCallbackFactory - the ack callback factory.
      Since:
      3.2
      See Also:
      KafkaMessageSource(ConsumerFactory, ConsumerProperties, KafkaAckCallbackFactory, boolean)

    • KafkaMessageSource

      public KafkaMessageSource​(org.springframework.kafka.core.ConsumerFactory<K,​V> consumerFactory, org.springframework.kafka.listener.ConsumerProperties consumerProperties, KafkaMessageSource.KafkaAckCallbackFactory<K,​V> ackCallbackFactory, boolean allowMultiFetch)
      Construct an instance with the supplied parameters. Set 'allowMultiFetch' to true to allow up to max.poll.records to be fetched on each poll. When false (default) max.poll.records is coerced to 1 if the consumer factory is a DefaultKafkaConsumerFactory or otherwise rejected with an IllegalArgumentException. IMPORTANT: When true, you must call AbstractMessageSource.receive() at a sufficient rate to consume the number of records received within max.poll.interval.ms. When false, you must call AbstractMessageSource.receive() within max.poll.interval.ms. pause() will not take effect until the records from the previous poll are consumed.
      Parameters:
      consumerFactory - the consumer factory.
      consumerProperties - the consumer properties.
      ackCallbackFactory - the ack callback factory.
      allowMultiFetch - true to allow max.poll.records > 1.
      Since:
      3.2

  • Method Details

    • getAssignedPartitions

      public java.util.Collection<org.apache.kafka.common.TopicPartition> getAssignedPartitions()
      Return the currently assigned partitions.
      Returns:
      the partitions.
      Since:
      3.2.2

    • onInit

      protected void onInit()
      Overrides:
      onInit in class org.springframework.integration.util.AbstractExpressionEvaluator

    • getConsumerProperties

      public org.springframework.kafka.listener.ConsumerProperties getConsumerProperties()
      Get a reference to the configured consumer properties; allows further customization of the properties before the source is started.
      Returns:
      the properties.
      Since:
      3.2

    • getGroupId

      protected java.lang.String getGroupId()

    • getClientId

      protected java.lang.String getClientId()

    • getPollTimeout

      protected long getPollTimeout()

    • getMessageConverter

      protected org.springframework.kafka.support.converter.RecordMessageConverter getMessageConverter()

    • setMessageConverter

      public void setMessageConverter​(org.springframework.kafka.support.converter.RecordMessageConverter messageConverter)
      Set the message converter to replace the default MessagingMessageConverter.
      Parameters:
      messageConverter - the converter.

    • getPayloadType

      protected java.lang.Class<?> getPayloadType()

    • setPayloadType

      public void setPayloadType​(java.lang.Class<?> payloadType)
      Set the payload type. Only applies if a type-aware message converter is provided.
      Parameters:
      payloadType - the type to convert to.

    • getRebalanceListener

      protected org.apache.kafka.clients.consumer.ConsumerRebalanceListener getRebalanceListener()

    • getComponentType

      public java.lang.String getComponentType()
      Specified by:
      getComponentType in interface org.springframework.integration.support.context.NamedComponent

    • isRawMessageHeader

      protected boolean isRawMessageHeader()

    • setRawMessageHeader

      public void setRawMessageHeader​(boolean rawMessageHeader)
      Set to true to include the raw ConsumerRecord as headers with keys KafkaHeaders.RAW_DATA and IntegrationMessageHeaderAccessor.SOURCE_DATA. enabling callers to have access to the record to process errors.
      Parameters:
      rawMessageHeader - true to include the header.

    • getCommitTimeout

      protected java.time.Duration getCommitTimeout()

    • setCloseTimeout

      public void setCloseTimeout​(java.time.Duration closeTimeout)
      Set the close timeout - default 30 seconds.
      Parameters:
      closeTimeout - the close timeout.

    • isRunning

      public boolean isRunning()
      Specified by:
      isRunning in interface org.springframework.context.Lifecycle
      Specified by:
      isRunning in interface org.springframework.integration.support.management.ManageableLifecycle

    • start

      public void start()
      Specified by:
      start in interface org.springframework.context.Lifecycle
      Specified by:
      start in interface org.springframework.integration.support.management.ManageableLifecycle

    • stop

      public void stop()
      Specified by:
      stop in interface org.springframework.context.Lifecycle
      Specified by:
      stop in interface org.springframework.integration.support.management.ManageableLifecycle

    • pause

      public void pause()
      Specified by:
      pause in interface org.springframework.integration.core.Pausable

    • resume

      public void resume()
      Specified by:
      resume in interface org.springframework.integration.core.Pausable

    • isPaused

      public boolean isPaused()
      Specified by:
      isPaused in interface org.springframework.integration.core.Pausable

    • doReceive

      protected java.lang.Object doReceive()
      Specified by:
      doReceive in class org.springframework.integration.endpoint.AbstractMessageSource<java.lang.Object>

    • createConsumer

      protected void createConsumer()

    • destroy

      public void destroy()
      Specified by:
      destroy in interface org.springframework.beans.factory.DisposableBean
      Specified by:
      destroy in interface org.springframework.integration.support.management.IntegrationManagement
      Overrides:
      destroy in class org.springframework.integration.endpoint.AbstractMessageSource<java.lang.Object>