kafka.log.remote

Class RemoteLogManager

java.lang.Object

kafka.log.remote.RemoteLogManager

All Implemented Interfaces:

Closeable, AutoCloseable

public class RemoteLogManager
extends Object
implements Closeable

This class is responsible for - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances - receives any leader and follower replica events and partition stop events and act on them - also provides APIs to fetch indexes, metadata about remote log segments - copying log segments to remote storage

Constructor Summary

Constructors

Constructor and Description
RemoteLogManager(org.apache.kafka.server.log.remote.storage.RemoteLogManagerConfig rlmConfig, int brokerId, String logDir, org.apache.kafka.common.utils.Time time, Function<org.apache.kafka.common.TopicPartition,Optional<kafka.log.UnifiedLog>> fetchLog) Creates RemoteLogManager instance with the given arguments.

Method Summary

Modifier and Type	Method and Description
void	close() Closes and releases all the resources like RemoterStorageManager and RemoteLogMetadataManager.
Optional<org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata>	fetchRemoteLogSegmentMetadata(org.apache.kafka.common.TopicPartition topicPartition, int epochForOffset, long offset)
Optional<org.apache.kafka.common.record.FileRecords.TimestampAndOffset>	findOffsetByTimestamp(org.apache.kafka.common.TopicPartition tp, long timestamp, long startingOffset, org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache leaderEpochCache) Search the message offset in the remote storage based on timestamp and offset.
void	onLeadershipChange(Set<kafka.cluster.Partition> partitionsBecomeLeader, Set<kafka.cluster.Partition> partitionsBecomeFollower, Map<String,org.apache.kafka.common.Uuid> topicIds) Callback to receive any leadership changes for the topic partitions assigned to this broker.
kafka.log.remote.RemoteLogManager.RLMScheduledThreadPool	rlmScheduledThreadPool()
void	startup()
void	stopPartitions(org.apache.kafka.common.TopicPartition topicPartition, boolean delete) Deletes the internal topic partition info if delete flag is set as true.
org.apache.kafka.server.log.remote.storage.RemoteStorageManager	storageManager()

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

RemoteLogManager

public RemoteLogManager(org.apache.kafka.server.log.remote.storage.RemoteLogManagerConfig rlmConfig,
                        int brokerId,
                        String logDir,
                        org.apache.kafka.common.utils.Time time,
                        Function<org.apache.kafka.common.TopicPartition,Optional<kafka.log.UnifiedLog>> fetchLog)

Creates RemoteLogManager instance with the given arguments.

Parameters:

rlmConfig - Configuration required for remote logging subsystem(tiered storage) at the broker level.

brokerId - id of the current broker.

logDir - directory of Kafka log segments.

time - Time instance.

fetchLog - function to get UnifiedLog instance for a given topic.

Method Detail

startup

public void startup()

storageManager

public org.apache.kafka.server.log.remote.storage.RemoteStorageManager storageManager()

rlmScheduledThreadPool

public kafka.log.remote.RemoteLogManager.RLMScheduledThreadPool rlmScheduledThreadPool()

onLeadershipChange

public void onLeadershipChange(Set<kafka.cluster.Partition> partitionsBecomeLeader,
                               Set<kafka.cluster.Partition> partitionsBecomeFollower,
                               Map<String,org.apache.kafka.common.Uuid> topicIds)

Callback to receive any leadership changes for the topic partitions assigned to this broker. If there are no existing tasks for a given topic partition then it will assign new leader or follower task else it will convert the task to respective target state(leader or follower).

Parameters:

partitionsBecomeLeader - partitions that have become leaders on this broker.

partitionsBecomeFollower - partitions that have become followers on this broker.

topicIds - topic name to topic id mappings.

stopPartitions

public void stopPartitions(org.apache.kafka.common.TopicPartition topicPartition,
                           boolean delete)

Deletes the internal topic partition info if delete flag is set as true.

Parameters:

topicPartition - topic partition to be stopped.

delete - flag to indicate whether the given topic partitions to be deleted or not.

fetchRemoteLogSegmentMetadata

public Optional<org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata> fetchRemoteLogSegmentMetadata(org.apache.kafka.common.TopicPartition topicPartition,
                                                                                                                   int epochForOffset,
                                                                                                                   long offset)
                                                                                                            throws org.apache.kafka.server.log.remote.storage.RemoteStorageException

Throws:

org.apache.kafka.server.log.remote.storage.RemoteStorageException

findOffsetByTimestamp

public Optional<org.apache.kafka.common.record.FileRecords.TimestampAndOffset> findOffsetByTimestamp(org.apache.kafka.common.TopicPartition tp,
                                                                                                     long timestamp,
                                                                                                     long startingOffset,
                                                                                                     org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache leaderEpochCache)
                                                                                              throws org.apache.kafka.server.log.remote.storage.RemoteStorageException,
                                                                                                     IOException

Search the message offset in the remote storage based on timestamp and offset.

This method returns an option of TimestampOffset. The returned value is determined using the following ordered list of rules:

- If there are no messages in the remote storage, return None - If all the messages in the remote storage have smaller offsets, return None - If all the messages in the remote storage have smaller timestamps, return None - Otherwise, return an option of TimestampOffset. The offset is the offset of the first message whose timestamp is greater than or equals to the target timestamp and whose offset is greater than or equals to the startingOffset.

Parameters:

tp - topic partition in which the offset to be found.

timestamp - The timestamp to search for.

startingOffset - The starting offset to search.

leaderEpochCache - LeaderEpochFileCache of the topic partition.

Returns:

the timestamp and offset of the first message that meets the requirements. None will be returned if there is no such message.

Throws:

org.apache.kafka.server.log.remote.storage.RemoteStorageException

IOException

close

public void close()

Closes and releases all the resources like RemoterStorageManager and RemoteLogMetadataManager.

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable