kafka.log.remote

Class RemoteLogManager

java.lang.Object

kafka.log.remote.RemoteLogManager

All Implemented Interfaces:

Closeable, AutoCloseable

public class RemoteLogManager
extends Object
implements Closeable

This class is responsible for - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances - receives any leader and follower replica events and partition stop events and act on them - also provides APIs to fetch indexes, metadata about remote log segments - copying log segments to the remote storage - cleaning up segments that are expired based on retention size or retention time

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	RemoteLogManager.RetentionSizeData
static class	RemoteLogManager.RetentionTimeData

Constructor Summary

Constructors

Constructor and Description

RemoteLogManager(org.apache.kafka.server.log.remote.storage.RemoteLogManagerConfig rlmConfig, int brokerId, String logDir, String clusterId, org.apache.kafka.common.utils.Time time, Function<org.apache.kafka.common.TopicPartition,Optional<kafka.log.UnifiedLog>> fetchLog, BiConsumer<org.apache.kafka.common.TopicPartition,Long> updateRemoteLogStartOffset, kafka.server.BrokerTopicStats brokerTopicStats, org.apache.kafka.common.metrics.Metrics metrics) Creates RemoteLogManager instance with the given arguments.

Method Summary

Modifier and Type	Method and Description
Future<Void>	asyncRead(org.apache.kafka.storage.internals.log.RemoteStorageFetchInfo fetchInfo, Consumer<org.apache.kafka.storage.internals.log.RemoteLogReadResult> callback) Submit a remote log read task.
void	close() Closes and releases all the resources like RemoterStorageManager and RemoteLogMetadataManager.
Optional<org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata>	fetchRemoteLogSegmentMetadata(org.apache.kafka.common.TopicPartition topicPartition, int epochForOffset, long offset)
org.apache.kafka.common.metrics.Sensor	fetchThrottleTimeSensor()
Optional<org.apache.kafka.common.record.FileRecords.TimestampAndOffset>	findOffsetByTimestamp(org.apache.kafka.common.TopicPartition tp, long timestamp, long startingOffset, org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache leaderEpochCache) Search the message offset in the remote storage based on timestamp and offset.
long	getFetchThrottleTimeMs()
void	onEndPointCreated(kafka.cluster.EndPoint endpoint)
void	onLeadershipChange(Set<kafka.cluster.Partition> partitionsBecomeLeader, Set<kafka.cluster.Partition> partitionsBecomeFollower, Map<String,org.apache.kafka.common.Uuid> topicIds) Callback to receive any leadership changes for the topic partitions assigned to this broker.
org.apache.kafka.storage.internals.log.FetchDataInfo	read(org.apache.kafka.storage.internals.log.RemoteStorageFetchInfo remoteStorageFetchInfo)
void	resizeCacheSize(long remoteLogIndexFileCacheSize)
void	startup()
void	stopLeaderCopyRLMTasks(Set<kafka.cluster.Partition> partitions)
void	stopPartitions(Set<kafka.server.StopPartition> stopPartitions, BiConsumer<org.apache.kafka.common.TopicPartition,Throwable> errorHandler) Stop the remote-log-manager task for the given partitions.
org.apache.kafka.server.log.remote.storage.RemoteStorageManager	storageManager()
void	updateCopyQuota(long quota)
void	updateFetchQuota(long quota)

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

RemoteLogManager

public RemoteLogManager(org.apache.kafka.server.log.remote.storage.RemoteLogManagerConfig rlmConfig,
                        int brokerId,
                        String logDir,
                        String clusterId,
                        org.apache.kafka.common.utils.Time time,
                        Function<org.apache.kafka.common.TopicPartition,Optional<kafka.log.UnifiedLog>> fetchLog,
                        BiConsumer<org.apache.kafka.common.TopicPartition,Long> updateRemoteLogStartOffset,
                        kafka.server.BrokerTopicStats brokerTopicStats,
                        org.apache.kafka.common.metrics.Metrics metrics)
                 throws IOException

Creates RemoteLogManager instance with the given arguments.

Parameters:

rlmConfig - Configuration required for remote logging subsystem(tiered storage) at the broker level.

brokerId - id of the current broker.

logDir - directory of Kafka log segments.

time - Time instance.

clusterId - The cluster id.

fetchLog - function to get UnifiedLog instance for a given topic.

updateRemoteLogStartOffset - function to update the log-start-offset for a given topic partition.

brokerTopicStats - BrokerTopicStats instance to update the respective metrics.

metrics - Metrics instance

Throws:

IOException

Method Detail

resizeCacheSize

public void resizeCacheSize(long remoteLogIndexFileCacheSize)

updateCopyQuota

public void updateCopyQuota(long quota)

updateFetchQuota

public void updateFetchQuota(long quota)

getFetchThrottleTimeMs

public long getFetchThrottleTimeMs()

fetchThrottleTimeSensor

public org.apache.kafka.common.metrics.Sensor fetchThrottleTimeSensor()

onEndPointCreated

public void onEndPointCreated(kafka.cluster.EndPoint endpoint)

startup

public void startup()

storageManager

public org.apache.kafka.server.log.remote.storage.RemoteStorageManager storageManager()

onLeadershipChange

public void onLeadershipChange(Set<kafka.cluster.Partition> partitionsBecomeLeader,
                               Set<kafka.cluster.Partition> partitionsBecomeFollower,
                               Map<String,org.apache.kafka.common.Uuid> topicIds)

Callback to receive any leadership changes for the topic partitions assigned to this broker. If there are no existing tasks for a given topic partition then it will assign new leader or follower task else it will convert the task to respective target state(leader or follower).

Parameters:

partitionsBecomeLeader - partitions that have become leaders on this broker.

partitionsBecomeFollower - partitions that have become followers on this broker.

topicIds - topic name to topic id mappings.

stopLeaderCopyRLMTasks

public void stopLeaderCopyRLMTasks(Set<kafka.cluster.Partition> partitions)

stopPartitions

public void stopPartitions(Set<kafka.server.StopPartition> stopPartitions,
                           BiConsumer<org.apache.kafka.common.TopicPartition,Throwable> errorHandler)

Stop the remote-log-manager task for the given partitions. And, calls the RemoteLogMetadataManager.onStopPartitions(Set) when StopPartition.deleteLocalLog() is true. Deletes the partitions from the remote storage when StopPartition.deleteRemoteLog() is true.

Parameters:

stopPartitions - topic partitions that needs to be stopped.

errorHandler - callback to handle any errors while stopping the partitions.

fetchRemoteLogSegmentMetadata

public Optional<org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata> fetchRemoteLogSegmentMetadata(org.apache.kafka.common.TopicPartition topicPartition,
                                                                                                                   int epochForOffset,
                                                                                                                   long offset)
                                                                                                            throws org.apache.kafka.server.log.remote.storage.RemoteStorageException

Throws:

org.apache.kafka.server.log.remote.storage.RemoteStorageException

findOffsetByTimestamp

public Optional<org.apache.kafka.common.record.FileRecords.TimestampAndOffset> findOffsetByTimestamp(org.apache.kafka.common.TopicPartition tp,
                                                                                                     long timestamp,
                                                                                                     long startingOffset,
                                                                                                     org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache leaderEpochCache)
                                                                                              throws org.apache.kafka.server.log.remote.storage.RemoteStorageException,
                                                                                                     IOException

Search the message offset in the remote storage based on timestamp and offset.

This method returns an option of TimestampOffset. The returned value is determined using the following ordered list of rules:

- If there are no messages in the remote storage, return Empty - If all the messages in the remote storage have smaller offsets, return Empty - If all the messages in the remote storage have smaller timestamps, return Empty - Otherwise, return an option of TimestampOffset. The offset is the offset of the first message whose timestamp is greater than or equals to the target timestamp and whose offset is greater than or equals to the startingOffset.

Parameters:

tp - topic partition in which the offset to be found.

timestamp - The timestamp to search for.

startingOffset - The starting offset to search.

leaderEpochCache - LeaderEpochFileCache of the topic partition.

Returns:

the timestamp and offset of the first message that meets the requirements. Empty will be returned if there is no such message.

Throws:

org.apache.kafka.server.log.remote.storage.RemoteStorageException

IOException

read

public org.apache.kafka.storage.internals.log.FetchDataInfo read(org.apache.kafka.storage.internals.log.RemoteStorageFetchInfo remoteStorageFetchInfo)
                                                          throws org.apache.kafka.server.log.remote.storage.RemoteStorageException,
                                                                 IOException

Throws:

org.apache.kafka.server.log.remote.storage.RemoteStorageException

IOException

asyncRead

public Future<Void> asyncRead(org.apache.kafka.storage.internals.log.RemoteStorageFetchInfo fetchInfo,
                              Consumer<org.apache.kafka.storage.internals.log.RemoteLogReadResult> callback)

Submit a remote log read task. This method returns immediately. The read operation is executed in a thread pool. The callback will be called when the task is done.

Throws:

RejectedExecutionException - if the task cannot be accepted for execution (task queue is full)

close

public void close()

Closes and releases all the resources like RemoterStorageManager and RemoteLogMetadataManager.

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable