public class Translog extends AbstractIndexShardComponent implements IndexShardComponent, Closeable, org.apache.lucene.index.TwoPhaseCommit
InternalEngine
. The engine
records the current translog generation getGeneration()
in it's commit metadata using TRANSLOG_GENERATION_KEY
to reference the generation that contains all operations that have not yet successfully been committed to the engines lucene index.
Additionally, since Elasticsearch 2.0 the engine also records a TRANSLOG_UUID_KEY
with each commit to ensure a strong association
between the lucene index an the transaction log file. This UUID is used to prevent accidential recovery from a transaction log that belongs to a
different engine.
Each Translog has only one translog file open at any time referenced by a translog generation ID. This ID is written to a translog.ckp file that is designed to fit in a single disk block such that a write of the file is atomic. The checkpoint file is written on each fsync operation of the translog and records the number of operations written, the current tranlogs file generation and it's fsynced offset in bytes.
When a translog is opened the checkpoint is use to retrieve the latest translog file generation and subsequently to open the last written file to recovery operations.
The Translog.TranslogGeneration
on TranslogConfig.getTranslogGeneration()
given when the translog is opened is compared against
the latest generation and all consecutive translog files singe the given generation and the last generation in the checkpoint will be recovered and preserved until the next
generation is committed using commit()
. In the common case the translog file generation in the checkpoint and the generation passed to the translog on creation are
the same. The only situation when they can be different is when an actual translog commit fails in between prepareCommit()
and commit()
. In such a case
the currently being committed translog file will not be deleted since it's commit was not successful. Yet, a new/current translog file is already opened at that point such that there is more than
one translog file present. Such an uncommitted translog file always has a translog-${gen}.ckp associated with it which is an fsynced copy of the it's last translog.ckp such that in
disaster recovery last fsynced offsets, number of operation etc. are still preserved.
Modifier and Type | Class and Description |
---|---|
static class |
Translog.Create |
static class |
Translog.Delete |
static class |
Translog.DeleteByQuery
Deprecated.
Delete-by-query is removed in 2.0, but we keep this so translog can replay on upgrade.
|
static class |
Translog.Durabilty |
static class |
Translog.Index |
static class |
Translog.Location |
static interface |
Translog.Operation
A generic interface representing an operation performed on the transaction log.
|
static interface |
Translog.Snapshot
A snapshot of the transaction log, allows to iterate over all the transaction log operations.
|
static class |
Translog.Source |
static class |
Translog.TranslogGeneration
References a transaction log generation
|
static class |
Translog.View
a view into the translog, capturing all translog file at the moment of creation
and updated with any future translog.
|
Modifier and Type | Field and Description |
---|---|
static String |
CHECKPOINT_FILE_NAME |
static String |
CHECKPOINT_SUFFIX |
protected ReleasableLock |
readLock |
static String |
TRANSLOG_FILE_PREFIX |
static String |
TRANSLOG_FILE_SUFFIX |
static String |
TRANSLOG_GENERATION_KEY |
static String |
TRANSLOG_UUID_KEY |
protected ReleasableLock |
writeLock |
deprecationLogger, indexSettings, logger, shardId
Constructor and Description |
---|
Translog(TranslogConfig config)
Creates a new Translog instance.
|
Modifier and Type | Method and Description |
---|---|
Translog.Location |
add(Translog.Operation operation)
Adds a created / delete / index operations to the transaction log.
|
void |
close() |
void |
commit() |
long |
currentFileGeneration()
Returns the generation of the current transaction log.
|
boolean |
ensureSynced(Translog.Location location)
Ensures that the given location has be synced / written to the underlying storage.
|
TranslogConfig |
getConfig() |
static String |
getFilename(long generation)
package private for testing
|
Translog.TranslogGeneration |
getGeneration()
Returns the current generation of this translog.
|
Throwable |
getTragicException()
If this
Translog was closed as a side-effect of a tragic exception,
e.g. |
boolean |
isCurrent(Translog.TranslogGeneration generation)
Returns
true iff the given generation is the current gbeneration of this translog |
boolean |
isOpen()
Returns
true if this Translog is still open. |
Path |
location()
Returns all translog locations as absolute paths.
|
Translog.Snapshot |
newSnapshot()
Snapshots the current transaction log allowing to safely iterate over the snapshot.
|
Translog.View |
newView()
Returns a view into the current translog that is guaranteed to retain all current operations
while receiving future ones as well
|
static long |
parseIdFromFileName(Path translogFile)
Extracts the translog generation from a file name.
|
void |
prepareCommit() |
Translog.Operation |
read(Translog.Location location)
Read the Operation object from the given location.
|
static List<Translog.Operation> |
readOperations(StreamInput input)
Reads a list of operations written with
writeOperations(StreamOutput, List) |
void |
rollback() |
long |
sizeInBytes()
Returns the size in bytes of the translog files that aren't committed to lucene.
|
TranslogStats |
stats()
return stats
|
void |
sync()
Sync's the translog.
|
boolean |
syncNeeded() |
int |
totalOperations()
Returns the number of operations in the transaction files that aren't committed to lucene..
|
void |
updateBuffer(ByteSizeValue bufferSize) |
static void |
upgradeLegacyTranslog(ESLogger logger,
TranslogConfig config)
This method is used to upgarde a pre 2.0 translog structure to the new checkpoint based structure.
|
static void |
writeOperationNoSize(BufferedChecksumStreamOutput out,
Translog.Operation op) |
static void |
writeOperations(StreamOutput outStream,
List<Translog.Operation> toWrite)
Writes all operations in the given iterable to the given output stream including the size of the array
use
readOperations(StreamInput) to read it back. |
indexSettings, nodeName, shardId
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
indexSettings, shardId
public static final String TRANSLOG_GENERATION_KEY
public static final String TRANSLOG_UUID_KEY
public static final String TRANSLOG_FILE_PREFIX
public static final String TRANSLOG_FILE_SUFFIX
public static final String CHECKPOINT_SUFFIX
public static final String CHECKPOINT_FILE_NAME
protected final ReleasableLock readLock
protected final ReleasableLock writeLock
public Translog(TranslogConfig config) throws IOException
TranslogConfig
has
a non-null Translog.TranslogGeneration
. If the generation is null this method
us destructive and will delete all files in the translog path given.IOException
TranslogConfig.getTranslogPath()
public static void upgradeLegacyTranslog(ESLogger logger, TranslogConfig config) throws IOException
Translog.TranslogGeneration
in the given config is
used to determine the smallest file generation to upgrade. The procedure will travers the translog
directory to find all files that have a generation greater or equal to the translog generation and
renames the files to the new .tlog file format.
For each of the files a ${filename}.ckp
file is written containing the size of the translog in bytes, it's ID and the number of operations. Since
these files are all relying on the pre 2.0 truncation feature where we read operations until hitting an EOFException
the number of operations are recoreded as -1. Later once these files are opened for reading legacy readers will
allow for unknown number of operations and mimic the old behavior.
IOException
public static long parseIdFromFileName(Path translogFile)
IllegalArgumentException
- if the path doesn't match the expected pattern.public void updateBuffer(ByteSizeValue bufferSize)
public boolean isOpen()
true
if this Translog
is still open.public void close() throws IOException
close
in interface Closeable
close
in interface AutoCloseable
IOException
public Path location()
public long currentFileGeneration()
public int totalOperations()
public long sizeInBytes()
public Translog.Operation read(Translog.Location location)
null
instead.public Translog.Location add(Translog.Operation operation) throws IOException
IOException
Translog.Operation
,
Translog.Create
,
Translog.Index
,
Translog.Delete
public Translog.Snapshot newSnapshot()
public Translog.View newView()
public void sync() throws IOException
IOException
public boolean syncNeeded()
public static String getFilename(long generation)
public boolean ensureSynced(Translog.Location location) throws IOException
true
iff this call caused an actual sync operation otherwise false
IOException
public TranslogStats stats()
public TranslogConfig getConfig()
public static List<Translog.Operation> readOperations(StreamInput input) throws IOException
writeOperations(StreamOutput, List)
IOException
public static void writeOperations(StreamOutput outStream, List<Translog.Operation> toWrite) throws IOException
readOperations(StreamInput)
to read it back.IOException
public static void writeOperationNoSize(BufferedChecksumStreamOutput out, Translog.Operation op) throws IOException
IOException
public void prepareCommit() throws IOException
prepareCommit
in interface org.apache.lucene.index.TwoPhaseCommit
IOException
public void commit() throws IOException
commit
in interface org.apache.lucene.index.TwoPhaseCommit
IOException
public void rollback() throws IOException
rollback
in interface org.apache.lucene.index.TwoPhaseCommit
IOException
public Translog.TranslogGeneration getGeneration()
public boolean isCurrent(Translog.TranslogGeneration generation)
true
iff the given generation is the current gbeneration of this translogpublic Throwable getTragicException()
Translog
was closed as a side-effect of a tragic exception,
e.g. disk full while flushing a new segment, this returns the root cause exception.
Otherwise (no tragic exception has occurred) it returns null.Copyright © 2009–2016. All rights reserved.