Class Translog
- All Implemented Interfaces:
java.io.Closeable
,java.lang.AutoCloseable
,IndexShardComponent
public class Translog extends AbstractIndexShardComponent implements IndexShardComponent, java.io.Closeable
InternalEngine
.
Additionally, since Elasticsearch 2.0 the engine also records a TRANSLOG_UUID_KEY
with each commit to ensure a strong
association between the lucene index an the transaction log file. This UUID is used to prevent accidental recovery from a transaction
log that belongs to a
different engine.
Each Translog has only one translog file open for writes at any time referenced by a translog generation ID. This ID is written to a
translog.ckp
file that is designed to fit in a single disk block such that a write of the file is atomic. The checkpoint file
is written on each fsync operation of the translog and records the number of operations written, the current translog's file generation,
its fsynced offset in bytes, and other important statistics.
When the current translog file reaches a certain size (IndexSettings.INDEX_TRANSLOG_GENERATION_THRESHOLD_SIZE_SETTING
, or when
a clear separation between old and new operations (upon change in primary term), the current file is reopened for read only and a new
write only file is created. Any non-current, read only translog file always has a translog-${gen}.ckp
associated with it
which is an fsynced copy of its last translog.ckp
such that in disaster recovery last fsynced offsets, number of
operation etc. are still preserved.
-
Nested Class Summary
Nested Classes Modifier and Type Class Description static class
Translog.Delete
static class
Translog.Durability
static class
Translog.Index
static class
Translog.Location
static class
Translog.NoOp
static interface
Translog.Operation
A generic interface representing an operation performed on the transaction log.static interface
Translog.Snapshot
A snapshot of the transaction log, allows to iterate over all the transaction log operations.static class
Translog.Source
static class
Translog.TranslogGeneration
References a transaction log generation -
Field Summary
Fields Modifier and Type Field Description static java.lang.String
CHECKPOINT_FILE_NAME
static java.lang.String
CHECKPOINT_SUFFIX
static int
DEFAULT_HEADER_SIZE_IN_BYTES
protected ReleasableLock
readLock
protected TragicExceptionHolder
tragedy
static java.lang.String
TRANSLOG_FILE_PREFIX
static java.lang.String
TRANSLOG_FILE_SUFFIX
static java.lang.String
TRANSLOG_UUID_KEY
protected ReleasableLock
writeLock
Fields inherited from class org.elasticsearch.index.shard.AbstractIndexShardComponent
indexSettings, logger, shardId
-
Constructor Summary
Constructors Constructor Description Translog(TranslogConfig config, java.lang.String translogUUID, TranslogDeletionPolicy deletionPolicy, java.util.function.LongSupplier globalCheckpointSupplier, java.util.function.LongSupplier primaryTermSupplier, java.util.function.LongConsumer persistedSequenceNumberConsumer)
Creates a new Translog instance. -
Method Summary
Modifier and Type Method Description java.io.Closeable
acquireRetentionLock()
Acquires a lock on the translog files, preventing them from being trimmedTranslog.Location
add(Translog.Operation operation)
Adds an operation to the transaction log.void
close()
protected void
closeOnTragicEvent(java.lang.Exception ex)
Closes the translog if the current translog writer experienced a tragic exception.static java.lang.String
createEmptyTranslog(java.nio.file.Path location, long initialGlobalCheckpoint, ShardId shardId, long primaryTerm)
static java.lang.String
createEmptyTranslog(java.nio.file.Path location, ShardId shardId, long initialGlobalCheckpoint, long primaryTerm, java.lang.String translogUUID, ChannelFactory factory)
Creates a new empty translog within the specifiedlocation
that contains the giveninitialGlobalCheckpoint
,primaryTerm
andtranslogUUID
.long
currentFileGeneration()
Returns the generation of the current transaction log.boolean
ensureSynced(java.util.stream.Stream<Translog.Location> locations)
Ensures that all locations in the given stream have been synced / written to the underlying storage.boolean
ensureSynced(Translog.Location location)
Ensures that the given location has be synced / written to the underlying storage.int
estimateTotalOperationsFromMinSeq(long minSeqNo)
Returns the number of operations in the transaction files that contain operations with seq# above the given number.TranslogConfig
getConfig()
TranslogDeletionPolicy
getDeletionPolicy()
static java.lang.String
getFilename(long generation)
package private for testingTranslog.TranslogGeneration
getGeneration()
Returns the current generation of this translog.long
getLastSyncedGlobalCheckpoint()
The last synced checkpoint for this translog.Translog.Location
getLastWriteLocation()
The a Translog.Location that will sort after the Translog.Location returned by the last write but before any locations which can be returned by the next write.long
getMaxSeqNo()
Returns the max seq_no of translog operations found in this translog.long
getMinFileGeneration()
Returns the minimum file generation referenced by the translogTranslog.TranslogGeneration
getMinGenerationForSeqNo(long seqNo)
Gets the minimum generation that could contain any sequence number after the specified sequence number, or the current generation if there is no generation that could any such sequence number.java.lang.Exception
getTragicException()
If thisTranslog
was closed as a side-effect of a tragic exception, e.g.java.lang.String
getTranslogUUID()
Returns the translog uuid used to associate a lucene index with a translog.boolean
isOpen()
Returnstrue
if thisTranslog
is still open.java.nio.file.Path
location()
Returns all translog locations as absolute paths.Translog.Snapshot
newSnapshot()
Translog.Snapshot
newSnapshot(long fromSeqNo, long toSeqNo)
Creates a new translog snapshot containing operations from the given range.static long
parseIdFromFileName(java.nio.file.Path translogFile)
Extracts the translog generation from a file name.static long
readGlobalCheckpoint(java.nio.file.Path location, java.lang.String expectedTranslogUUID)
Reads the sequence numbers global checkpoint from the translog checkpoint.static long
readMinTranslogGeneration(java.nio.file.Path location, java.lang.String expectedTranslogUUID)
Returns the minimum translog generation retained by the translog at the given location.Translog.Operation
readOperation(Translog.Location location)
Reads and returns the operation from the given location if the generation it references is still available.static java.util.List<Translog.Operation>
readOperations(StreamInput input, java.lang.String source)
Reads a list of operations written withwriteOperations(StreamOutput, List)
void
rollGeneration()
Roll the current translog generation into a new generation.boolean
shouldRollGeneration()
Tests whether or not the translog generation should be rolled to a new generation.long
sizeInBytes()
Returns the size in bytes of the v fileslong
sizeInBytesByMinGen(long minGeneration)
Returns the size in bytes of the translog files at least the given generationTranslogStats
stats()
return statsvoid
sync()
Sync's the translog.boolean
syncNeeded()
Returnstrue
if an fsync is required to ensure durability of the translogs operations or it's metadata.int
totalOperations()
Returns the number of operations in the translog filesint
totalOperationsByMinGen(long minGeneration)
Returns the number of operations in the translog files at least the given generationvoid
trimOperations(long belowTerm, long aboveSeqNo)
Trims translog for terms of files belowbelowTerm
and seq# aboveaboveSeqNo
.void
trimUnreferencedReaders()
Trims unreferenced translog generations by askingTranslogDeletionPolicy
for the minimum required generationstatic void
writeOperationNoSize(BufferedChecksumStreamOutput out, Translog.Operation op)
static void
writeOperations(StreamOutput outStream, java.util.List<Translog.Operation> toWrite)
Writes all operations in the given iterable to the given output stream including the size of the array usereadOperations(StreamInput, String)
to read it back.Methods inherited from class org.elasticsearch.index.shard.AbstractIndexShardComponent
indexSettings, shardId
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
Methods inherited from interface org.elasticsearch.index.shard.IndexShardComponent
indexSettings, shardId
-
Field Details
-
TRANSLOG_UUID_KEY
public static final java.lang.String TRANSLOG_UUID_KEY- See Also:
- Constant Field Values
-
TRANSLOG_FILE_PREFIX
public static final java.lang.String TRANSLOG_FILE_PREFIX- See Also:
- Constant Field Values
-
TRANSLOG_FILE_SUFFIX
public static final java.lang.String TRANSLOG_FILE_SUFFIX- See Also:
- Constant Field Values
-
CHECKPOINT_SUFFIX
public static final java.lang.String CHECKPOINT_SUFFIX- See Also:
- Constant Field Values
-
CHECKPOINT_FILE_NAME
public static final java.lang.String CHECKPOINT_FILE_NAME- See Also:
- Constant Field Values
-
DEFAULT_HEADER_SIZE_IN_BYTES
public static final int DEFAULT_HEADER_SIZE_IN_BYTES -
readLock
-
writeLock
-
tragedy
-
-
Constructor Details
-
Translog
public Translog(TranslogConfig config, java.lang.String translogUUID, TranslogDeletionPolicy deletionPolicy, java.util.function.LongSupplier globalCheckpointSupplier, java.util.function.LongSupplier primaryTermSupplier, java.util.function.LongConsumer persistedSequenceNumberConsumer) throws java.io.IOExceptionCreates a new Translog instance. This method will create a new transaction log unless the givenTranslog.TranslogGeneration
isnull
. If the generation isnull
this method is destructive and will delete all files in the translog path given. If the generation is notnull
, this method tries to open the given translog generation. The generation is treated as the last generation referenced from already committed data. This means all operations that have not yet been committed should be in the translog file referenced by this generation. The translog creation will fail if this generation can't be opened.- Parameters:
config
- the configuration of this translogtranslogUUID
- the translog uuid to open, null for a new translogdeletionPolicy
- an instance ofTranslogDeletionPolicy
that controls when a translog file can be safely deletedglobalCheckpointSupplier
- a supplier for the global checkpointprimaryTermSupplier
- a supplier for the latest value of primary term of the owning index shard. The latest term value is examined and stored in the header whenever a new generation is rolled. It's guaranteed from outside that a new generation is rolled when the term is increased. This guarantee allows to us to validate and reject operation whose term is higher than the primary term stored in the translog header.persistedSequenceNumberConsumer
- a callback that's called whenever an operation with a given sequence number is successfully persisted.- Throws:
java.io.IOException
-
-
Method Details
-
parseIdFromFileName
public static long parseIdFromFileName(java.nio.file.Path translogFile)Extracts the translog generation from a file name.- Throws:
java.lang.IllegalArgumentException
- if the path doesn't match the expected pattern.
-
isOpen
public boolean isOpen()Returnstrue
if thisTranslog
is still open. -
close
public void close() throws java.io.IOException- Specified by:
close
in interfacejava.lang.AutoCloseable
- Specified by:
close
in interfacejava.io.Closeable
- Throws:
java.io.IOException
-
location
public java.nio.file.Path location()Returns all translog locations as absolute paths. These paths don't contain actual translog files they are directories holding the transaction logs. -
currentFileGeneration
public long currentFileGeneration()Returns the generation of the current transaction log. -
getMinFileGeneration
public long getMinFileGeneration()Returns the minimum file generation referenced by the translog -
totalOperations
public int totalOperations()Returns the number of operations in the translog files -
sizeInBytes
public long sizeInBytes()Returns the size in bytes of the v files -
totalOperationsByMinGen
public int totalOperationsByMinGen(long minGeneration)Returns the number of operations in the translog files at least the given generation -
estimateTotalOperationsFromMinSeq
public int estimateTotalOperationsFromMinSeq(long minSeqNo)Returns the number of operations in the transaction files that contain operations with seq# above the given number. -
sizeInBytesByMinGen
public long sizeInBytesByMinGen(long minGeneration)Returns the size in bytes of the translog files at least the given generation -
add
Adds an operation to the transaction log.- Parameters:
operation
- the operation to add- Returns:
- the location of the operation in the translog
- Throws:
java.io.IOException
- if adding the operation to the translog resulted in an I/O exception
-
shouldRollGeneration
public boolean shouldRollGeneration()Tests whether or not the translog generation should be rolled to a new generation. This test is based on the size of the current generation compared to the configured generation threshold size.- Returns:
true
if the current generation should be rolled to a new generation
-
getLastWriteLocation
The a Translog.Location that will sort after the Translog.Location returned by the last write but before any locations which can be returned by the next write. -
getLastSyncedGlobalCheckpoint
public long getLastSyncedGlobalCheckpoint()The last synced checkpoint for this translog.- Returns:
- the last synced checkpoint
-
newSnapshot
- Throws:
java.io.IOException
-
newSnapshot
Creates a new translog snapshot containing operations from the given range.- Parameters:
fromSeqNo
- the lower bound of the range (inclusive)toSeqNo
- the upper bound of the range (inclusive)- Returns:
- the new snapshot
- Throws:
java.io.IOException
-
readOperation
Reads and returns the operation from the given location if the generation it references is still available. Otherwise this method will returnnull
.- Throws:
java.io.IOException
-
acquireRetentionLock
public java.io.Closeable acquireRetentionLock()Acquires a lock on the translog files, preventing them from being trimmed -
sync
public void sync() throws java.io.IOExceptionSync's the translog.- Throws:
java.io.IOException
-
syncNeeded
public boolean syncNeeded()Returnstrue
if an fsync is required to ensure durability of the translogs operations or it's metadata. -
getFilename
public static java.lang.String getFilename(long generation)package private for testing -
trimOperations
public void trimOperations(long belowTerm, long aboveSeqNo) throws java.io.IOExceptionTrims translog for terms of files belowbelowTerm
and seq# aboveaboveSeqNo
. Effectively it moves max visible seq#Checkpoint.trimmedAboveSeqNo
thereforeTranslogSnapshot
skips those operations.- Throws:
java.io.IOException
-
ensureSynced
Ensures that the given location has be synced / written to the underlying storage.- Returns:
- Returns
true
iff this call caused an actual sync operation otherwisefalse
- Throws:
java.io.IOException
-
ensureSynced
public boolean ensureSynced(java.util.stream.Stream<Translog.Location> locations) throws java.io.IOExceptionEnsures that all locations in the given stream have been synced / written to the underlying storage. This method allows for internal optimization to minimize the amount of fsync operations if multiple locations must be synced.- Returns:
- Returns
true
iff this call caused an actual sync operation otherwisefalse
- Throws:
java.io.IOException
-
closeOnTragicEvent
protected void closeOnTragicEvent(java.lang.Exception ex)Closes the translog if the current translog writer experienced a tragic exception. Note that in case this thread closes the translog it must not already be holding a read lock on the translog as it will acquire a write lock in the course of closing the translog- Parameters:
ex
- if an exception occurs closing the translog, it will be suppressed into the provided exception
-
stats
return stats -
getConfig
-
getDeletionPolicy
-
readOperations
public static java.util.List<Translog.Operation> readOperations(StreamInput input, java.lang.String source) throws java.io.IOExceptionReads a list of operations written withwriteOperations(StreamOutput, List)
- Throws:
java.io.IOException
-
writeOperations
public static void writeOperations(StreamOutput outStream, java.util.List<Translog.Operation> toWrite) throws java.io.IOExceptionWrites all operations in the given iterable to the given output stream including the size of the array usereadOperations(StreamInput, String)
to read it back.- Throws:
java.io.IOException
-
writeOperationNoSize
public static void writeOperationNoSize(BufferedChecksumStreamOutput out, Translog.Operation op) throws java.io.IOException- Throws:
java.io.IOException
-
getMinGenerationForSeqNo
Gets the minimum generation that could contain any sequence number after the specified sequence number, or the current generation if there is no generation that could any such sequence number.- Parameters:
seqNo
- the sequence number- Returns:
- the minimum generation for the sequence number
-
rollGeneration
public void rollGeneration() throws java.io.IOExceptionRoll the current translog generation into a new generation. This does not commit the translog.- Throws:
java.io.IOException
- if an I/O exception occurred during any file operations
-
trimUnreferencedReaders
public void trimUnreferencedReaders() throws java.io.IOExceptionTrims unreferenced translog generations by askingTranslogDeletionPolicy
for the minimum required generation- Throws:
java.io.IOException
-
getGeneration
Returns the current generation of this translog. This corresponds to the latest uncommitted translog generation -
getTragicException
public java.lang.Exception getTragicException()If thisTranslog
was closed as a side-effect of a tragic exception, e.g. disk full while flushing a new segment, this returns the root cause exception. Otherwise (no tragic exception has occurred) it returns null. -
readGlobalCheckpoint
public static long readGlobalCheckpoint(java.nio.file.Path location, java.lang.String expectedTranslogUUID) throws java.io.IOExceptionReads the sequence numbers global checkpoint from the translog checkpoint. This ensures that the translogUUID from this translog matches with the provided translogUUID.- Parameters:
location
- the location of the translog- Returns:
- the global checkpoint
- Throws:
java.io.IOException
- if an I/O exception occurred reading the checkpointTranslogCorruptedException
- if the translog is corrupted or mismatched with the given uuid
-
readMinTranslogGeneration
public static long readMinTranslogGeneration(java.nio.file.Path location, java.lang.String expectedTranslogUUID) throws java.io.IOExceptionReturns the minimum translog generation retained by the translog at the given location. This ensures that the translogUUID from this translog matches with the provided translogUUID.- Parameters:
location
- the location of the translog- Returns:
- the minimum translog generation
- Throws:
java.io.IOException
- if an I/O exception occurred reading the checkpointTranslogCorruptedException
- if the translog is corrupted or mismatched with the given uuid
-
getTranslogUUID
public java.lang.String getTranslogUUID()Returns the translog uuid used to associate a lucene index with a translog. -
getMaxSeqNo
public long getMaxSeqNo()Returns the max seq_no of translog operations found in this translog. Since this value is calculated based on the current existing readers, this value is not necessary to be the max seq_no of all operations have been stored in this translog. -
createEmptyTranslog
public static java.lang.String createEmptyTranslog(java.nio.file.Path location, long initialGlobalCheckpoint, ShardId shardId, long primaryTerm) throws java.io.IOException- Throws:
java.io.IOException
-
createEmptyTranslog
public static java.lang.String createEmptyTranslog(java.nio.file.Path location, ShardId shardId, long initialGlobalCheckpoint, long primaryTerm, @Nullable java.lang.String translogUUID, @Nullable ChannelFactory factory) throws java.io.IOExceptionCreates a new empty translog within the specifiedlocation
that contains the giveninitialGlobalCheckpoint
,primaryTerm
andtranslogUUID
. This method should be used directly under specific circumstances like for shards that will see no indexing. Specifying a non-unique translog UUID could cause a lot of issues and that's why in all (but one) cases the methodcreateEmptyTranslog(Path, long, ShardId, long)
should be used instead.- Parameters:
location
- aPath
to the directory that will contains the translog files (translog + translog checkpoint)shardId
- theShardId
initialGlobalCheckpoint
- the global checkpoint to initialize the translog withprimaryTerm
- the shard's primary term to initialize the translog withtranslogUUID
- the unique identifier to initialize the translog withfactory
- aChannelFactory
used to open translog files- Returns:
- the translog's unique identifier
- Throws:
java.io.IOException
- if something went wrong during translog creation
-