Package kafka.log.remote

Class RemoteLogManager

java.lang.Object
kafka.log.remote.RemoteLogManager
All Implemented Interfaces:
Closeable, AutoCloseable
public class RemoteLogManager extends Object implements Closeable
This class is responsible for - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances - receives any leader and follower replica events and partition stop events and act on them - also provides APIs to fetch indexes, metadata about remote log segments - copying log segments to the remote storage - cleaning up segments that are expired based on retention size or retention time

  • Nested Class Summary

    Nested Classes
    Modifier and Type
    Class
    Description
    static class 
    RemoteLogManager.RetentionSizeData
     
    static class 
    RemoteLogManager.RetentionTimeData
     

  • Constructor Summary

    Constructors
    Constructor
    Description
    RemoteLogManager(org.apache.kafka.server.log.remote.storage.RemoteLogManagerConfig rlmConfig, int brokerId, String logDir, String clusterId, org.apache.kafka.common.utils.Time time, Function<org.apache.kafka.common.TopicPartition,Optional<kafka.log.UnifiedLog>> fetchLog, BiConsumer<org.apache.kafka.common.TopicPartition,Long> updateRemoteLogStartOffset, org.apache.kafka.storage.log.metrics.BrokerTopicStats brokerTopicStats, org.apache.kafka.common.metrics.Metrics metrics)
    Creates RemoteLogManager instance with the given arguments.

  • Method Summary

    Modifier and Type
    Method
    Description
    org.apache.kafka.storage.internals.log.AsyncOffsetReadFutureHolder<org.apache.kafka.storage.internals.log.OffsetResultHolder.FileRecordsOrError>
    asyncOffsetRead(org.apache.kafka.common.TopicPartition topicPartition, Long timestamp, Long startingOffset, org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache leaderEpochCache, Supplier<scala.Option<org.apache.kafka.common.record.FileRecords.TimestampAndOffset>> searchLocalLog)
     
    Future<Void>
    asyncRead(org.apache.kafka.storage.internals.log.RemoteStorageFetchInfo fetchInfo, Consumer<org.apache.kafka.storage.internals.log.RemoteLogReadResult> callback)
    Submit a remote log read task.
    void
    close()
    Closes and releases all the resources like RemoterStorageManager and RemoteLogMetadataManager.
    Optional<org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata>
    fetchNextSegmentWithTxnIndex(org.apache.kafka.common.TopicPartition topicPartition, int epochForOffset, long offset)
    Returns the next segment that may contain the aborted transaction entries.
    Optional<org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata>
    fetchRemoteLogSegmentMetadata(org.apache.kafka.common.TopicPartition topicPartition, int epochForOffset, long offset)
     
    org.apache.kafka.common.metrics.Sensor
    fetchThrottleTimeSensor()
     
    Optional<org.apache.kafka.common.record.FileRecords.TimestampAndOffset>
    findOffsetByTimestamp(org.apache.kafka.common.TopicPartition tp, long timestamp, long startingOffset, org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache leaderEpochCache)
    Search the message offset in the remote storage for the given timestamp and starting-offset.
    long
    getFetchThrottleTimeMs()
     
    void
    onEndPointCreated(org.apache.kafka.common.Endpoint endpoint)
     
    void
    onLeadershipChange(Set<kafka.cluster.Partition> partitionsBecomeLeader, Set<kafka.cluster.Partition> partitionsBecomeFollower, Map<String,org.apache.kafka.common.Uuid> topicIds)
    Callback to receive any leadership changes for the topic partitions assigned to this broker.
    org.apache.kafka.storage.internals.log.FetchDataInfo
    read(org.apache.kafka.storage.internals.log.RemoteStorageFetchInfo remoteStorageFetchInfo)
     
    void
    resizeCacheSize(long remoteLogIndexFileCacheSize)
     
    void
    resizeCopierThreadPool(int newSize)
     
    void
    resizeExpirationThreadPool(int newSize)
     
    void
    resizeReaderThreadPool(int newSize)
     
    void
    setDelayedOperationPurgatory(org.apache.kafka.server.purgatory.DelayedOperationPurgatory<kafka.server.DelayedRemoteListOffsets> delayedRemoteListOffsetsPurgatory)
     
    void
    startup()
     
    void
    stopLeaderCopyRLMTasks(Set<kafka.cluster.Partition> partitions)
     
    void
    stopPartitions(Set<org.apache.kafka.server.common.StopPartition> stopPartitions, BiConsumer<org.apache.kafka.common.TopicPartition,Throwable> errorHandler)
    Stop the remote-log-manager task for the given partitions.
    org.apache.kafka.server.log.remote.storage.RemoteStorageManager
    storageManager()
     
    void
    updateCopyQuota(long quota)
     
    void
    updateFetchQuota(long quota)
     

    Methods inherited from class java.lang.Object

    equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Constructor Details

    • RemoteLogManager

      public RemoteLogManager(org.apache.kafka.server.log.remote.storage.RemoteLogManagerConfig rlmConfig, int brokerId, String logDir, String clusterId, org.apache.kafka.common.utils.Time time, Function<org.apache.kafka.common.TopicPartition,Optional<kafka.log.UnifiedLog>> fetchLog, BiConsumer<org.apache.kafka.common.TopicPartition,Long> updateRemoteLogStartOffset, org.apache.kafka.storage.log.metrics.BrokerTopicStats brokerTopicStats, org.apache.kafka.common.metrics.Metrics metrics) throws IOException
      Creates RemoteLogManager instance with the given arguments.
      Parameters:
      rlmConfig - Configuration required for remote logging subsystem(tiered storage) at the broker level.
      brokerId - id of the current broker.
      logDir - directory of Kafka log segments.
      clusterId - The cluster id.
      time - Time instance.
      fetchLog - function to get UnifiedLog instance for a given topic.
      updateRemoteLogStartOffset - function to update the log-start-offset for a given topic partition.
      brokerTopicStats - BrokerTopicStats instance to update the respective metrics.
      metrics - Metrics instance
      Throws:
      IOException

  • Method Details

    • setDelayedOperationPurgatory

      public void setDelayedOperationPurgatory(org.apache.kafka.server.purgatory.DelayedOperationPurgatory<kafka.server.DelayedRemoteListOffsets> delayedRemoteListOffsetsPurgatory)

    • resizeCacheSize

      public void resizeCacheSize(long remoteLogIndexFileCacheSize)

    • updateCopyQuota

      public void updateCopyQuota(long quota)

    • updateFetchQuota

      public void updateFetchQuota(long quota)

    • resizeCopierThreadPool

      public void resizeCopierThreadPool(int newSize)

    • resizeExpirationThreadPool

      public void resizeExpirationThreadPool(int newSize)

    • resizeReaderThreadPool

      public void resizeReaderThreadPool(int newSize)

    • getFetchThrottleTimeMs

      public long getFetchThrottleTimeMs()

    • fetchThrottleTimeSensor

      public org.apache.kafka.common.metrics.Sensor fetchThrottleTimeSensor()

    • onEndPointCreated

      public void onEndPointCreated(org.apache.kafka.common.Endpoint endpoint)

    • startup

      public void startup()

    • storageManager

      public org.apache.kafka.server.log.remote.storage.RemoteStorageManager storageManager()

    • onLeadershipChange

      public void onLeadershipChange(Set<kafka.cluster.Partition> partitionsBecomeLeader, Set<kafka.cluster.Partition> partitionsBecomeFollower, Map<String,org.apache.kafka.common.Uuid> topicIds)
      Callback to receive any leadership changes for the topic partitions assigned to this broker. If there are no existing tasks for a given topic partition then it will assign new leader or follower task else it will convert the task to respective target state(leader or follower).
      Parameters:
      partitionsBecomeLeader - partitions that have become leaders on this broker.
      partitionsBecomeFollower - partitions that have become followers on this broker.
      topicIds - topic name to topic id mappings.

    • stopLeaderCopyRLMTasks

      public void stopLeaderCopyRLMTasks(Set<kafka.cluster.Partition> partitions)

    • stopPartitions

      public void stopPartitions(Set<org.apache.kafka.server.common.StopPartition> stopPartitions, BiConsumer<org.apache.kafka.common.TopicPartition,Throwable> errorHandler)
      Stop the remote-log-manager task for the given partitions. And, calls the RemoteLogMetadataManager.onStopPartitions(Set) when StopPartition.deleteLocalLog is true. Deletes the partitions from the remote storage when StopPartition.deleteRemoteLog is true.
      Parameters:
      stopPartitions - topic partitions that needs to be stopped.
      errorHandler - callback to handle any errors while stopping the partitions.

    • fetchRemoteLogSegmentMetadata

      public Optional<org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata> fetchRemoteLogSegmentMetadata(org.apache.kafka.common.TopicPartition topicPartition, int epochForOffset, long offset) throws org.apache.kafka.server.log.remote.storage.RemoteStorageException
      Throws:
      org.apache.kafka.server.log.remote.storage.RemoteStorageException

    • fetchNextSegmentWithTxnIndex

      public Optional<org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata> fetchNextSegmentWithTxnIndex(org.apache.kafka.common.TopicPartition topicPartition, int epochForOffset, long offset) throws org.apache.kafka.server.log.remote.storage.RemoteStorageException
      Returns the next segment that may contain the aborted transaction entries. The search ensures that the returned segment offsets are greater than or equal to the given offset and in the same epoch.
      Parameters:
      topicPartition - topic partition to search
      epochForOffset - the epoch
      offset - the offset
      Returns:
      The next segment that contains the transaction index in the same epoch.
      Throws:
      org.apache.kafka.server.log.remote.storage.RemoteStorageException - If an error occurs while fetching the remote log segment metadata.

    • asyncOffsetRead

      public org.apache.kafka.storage.internals.log.AsyncOffsetReadFutureHolder<org.apache.kafka.storage.internals.log.OffsetResultHolder.FileRecordsOrError> asyncOffsetRead(org.apache.kafka.common.TopicPartition topicPartition, Long timestamp, Long startingOffset, org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache leaderEpochCache, Supplier<scala.Option<org.apache.kafka.common.record.FileRecords.TimestampAndOffset>> searchLocalLog)

    • findOffsetByTimestamp

      public Optional<org.apache.kafka.common.record.FileRecords.TimestampAndOffset> findOffsetByTimestamp(org.apache.kafka.common.TopicPartition tp, long timestamp, long startingOffset, org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache leaderEpochCache) throws org.apache.kafka.server.log.remote.storage.RemoteStorageException, IOException
      Search the message offset in the remote storage for the given timestamp and starting-offset. Once the target segment where the search to be performed is found: 1. If the target segment lies in the local storage (common segments that lies in both remote and local storage), then the search will be performed in the local storage. 2. If the target segment is found only in the remote storage, then the search will be performed in the remote storage.

      This method returns an option of TimestampOffset. The returned value is determined using the following ordered list of rules:

      - If there are no messages in the remote storage, return Empty - If all the messages in the remote storage have smaller offsets, return Empty - If all the messages in the remote storage have smaller timestamps, return Empty - Otherwise, return an option of TimestampOffset. The offset is the offset of the first message whose timestamp is greater than or equals to the target timestamp and whose offset is greater than or equals to the startingOffset.

      Parameters:
      tp - topic partition in which the offset to be found.
      timestamp - The timestamp to search for.
      startingOffset - The starting offset to search.
      leaderEpochCache - LeaderEpochFileCache of the topic partition.
      Returns:
      the timestamp and offset of the first message that meets the requirements. Empty will be returned if there is no such message.
      Throws:
      org.apache.kafka.server.log.remote.storage.RemoteStorageException
      IOException

    • read

      public org.apache.kafka.storage.internals.log.FetchDataInfo read(org.apache.kafka.storage.internals.log.RemoteStorageFetchInfo remoteStorageFetchInfo) throws org.apache.kafka.server.log.remote.storage.RemoteStorageException, IOException
      Throws:
      org.apache.kafka.server.log.remote.storage.RemoteStorageException
      IOException

    • asyncRead

      public Future<Void> asyncRead(org.apache.kafka.storage.internals.log.RemoteStorageFetchInfo fetchInfo, Consumer<org.apache.kafka.storage.internals.log.RemoteLogReadResult> callback)
      Submit a remote log read task. This method returns immediately. The read operation is executed in a thread pool. The callback will be called when the task is done.
      Throws:
      RejectedExecutionException - if the task cannot be accepted for execution (task queue is full)

    • close

      public void close()
      Closes and releases all the resources like RemoterStorageManager and RemoteLogMetadataManager.
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable