Class Engine
- All Implemented Interfaces:
java.io.Closeable
,java.lang.AutoCloseable
- Direct Known Subclasses:
InternalEngine
,ReadOnlyEngine
public abstract class Engine
extends java.lang.Object
implements java.io.Closeable
-
Nested Class Summary
Nested Classes Modifier and Type Class Description static class
Engine.CommitId
static class
Engine.Delete
static class
Engine.DeleteResult
static interface
Engine.EventListener
static class
Engine.Get
static class
Engine.GetResult
static class
Engine.HistorySource
Whether we should read history operations from translog or Lucene indexstatic class
Engine.Index
static class
Engine.IndexCommitRef
static class
Engine.IndexResult
protected static class
Engine.IndexThrottle
A throttling class that can be activated, causing theacquireThrottle
method to block on a lock when throttling is enabledstatic class
Engine.NoOp
protected static class
Engine.NoOpLock
A Lock implementation that always allows the lock to be acquiredstatic class
Engine.NoOpResult
static class
Engine.Operation
static class
Engine.Result
Base class for index and delete operation results Holds result meta data (e.g.static class
Engine.Searcher
static class
Engine.SearcherScope
static class
Engine.SyncedFlushResult
static interface
Engine.TranslogRecoveryRunner
static interface
Engine.Warmer
Called for each new opened engine reader to warm new segments -
Field Summary
Fields Modifier and Type Field Description protected java.lang.String
allocationId
protected EngineConfig
engineConfig
protected Engine.EventListener
eventListener
protected org.apache.lucene.util.SetOnce<java.lang.Exception>
failedEngine
protected java.util.concurrent.locks.ReentrantLock
failEngineLock
static java.lang.String
FORCE_MERGE_UUID_KEY
static java.lang.String
HISTORY_UUID_KEY
protected java.util.concurrent.atomic.AtomicBoolean
isClosed
protected long
lastWriteNanos
protected org.apache.logging.log4j.Logger
logger
static java.lang.String
MAX_UNSAFE_AUTO_ID_TIMESTAMP_COMMIT_ID
static java.lang.String
MIN_RETAINED_SEQNO
protected ReleasableLock
readLock
protected java.util.concurrent.locks.ReentrantReadWriteLock
rwl
protected ShardId
shardId
protected Store
store
static java.lang.String
SYNC_COMMIT_ID
protected ReleasableLock
writeLock
-
Constructor Summary
Constructors Modifier Constructor Description protected
Engine(EngineConfig engineConfig)
-
Method Summary
Modifier and Type Method Description abstract java.io.Closeable
acquireHistoryRetentionLock(Engine.HistorySource historySource)
Acquires a lock on the translog files and Lucene soft-deleted documents to prevent them from being trimmedabstract Engine.IndexCommitRef
acquireLastIndexCommit(boolean flushFirst)
Snapshots the most recent index and returns a handle to it.abstract Engine.IndexCommitRef
acquireSafeIndexCommit()
Snapshots the most recent safe index commit from the engine.Engine.Searcher
acquireSearcher(java.lang.String source)
Returns a new searcher instance.Engine.Searcher
acquireSearcher(java.lang.String source, Engine.SearcherScope scope)
Returns a new searcher instance.abstract void
activateThrottling()
Request that this engine throttle incoming indexing requests to one thread.abstract void
advanceMaxSeqNoOfUpdatesOrDeletes(long maxSeqNoOfUpdatesOnPrimary)
A replica shard receives a new max_seq_no_of_updates from its primary shard, then calls this method to advance this marker to at least the given sequence number.void
close()
protected abstract void
closeNoLock(java.lang.String reason, java.util.concurrent.CountDownLatch closedLatch)
Method to close the engine while the write lock is held.CommitStats
commitStats()
get commits stats for the last commitabstract CompletionStats
completionStats(java.lang.String... fieldNamePatterns)
Returns theCompletionStats
for this engineEngineConfig
config()
abstract void
deactivateThrottling()
Reverses a previousactivateThrottling()
call.abstract Engine.DeleteResult
delete(Engine.Delete delete)
Perform document delete operation on the engineprotected DocsStats
docsStats(org.apache.lucene.index.IndexReader indexReader)
DocsStats
docStats()
Returns theDocsStats
for this engineprotected void
ensureOpen()
protected void
ensureOpen(java.lang.Exception suppressed)
abstract boolean
ensureTranslogSynced(java.util.stream.Stream<Translog.Location> locations)
Ensures that all locations in the given stream have been written to the underlying storage.abstract int
estimateNumberOfHistoryOperations(java.lang.String reason, Engine.HistorySource historySource, MapperService mapperService, long startingSeqNo)
Returns the estimated number of history operations whose seq# at leaststartingSeqNo
(inclusive) in this engine.void
failEngine(java.lang.String reason, java.lang.Exception failure)
fail engine due to some error.protected void
fillSegmentStats(org.apache.lucene.index.SegmentReader segmentReader, boolean includeSegmentFileSizes, SegmentsStats stats)
abstract int
fillSeqNoGaps(long primaryTerm)
Fills up the local checkpoints history with no-ops until the local checkpoint and the max seen sequence ID are identical.Engine.CommitId
flush()
Flushes the state of the engine including the transaction log, clearing memory and persisting documents in the lucene index to disk including a potentially heavy and durable fsync operation.abstract Engine.CommitId
flush(boolean force, boolean waitIfOngoing)
Flushes the state of the engine including the transaction log, clearing memory.void
flushAndClose()
Flush the engine (committing segments to disk and truncating the translog) and close it.abstract void
forceMerge(boolean flush, int maxNumSegments, boolean onlyExpungeDeletes, boolean upgrade, boolean upgradeOnlyAncientSegments, java.lang.String forceMergeUUID)
Triggers a forced merge on this engineabstract Engine.GetResult
get(Engine.Get get, java.util.function.BiFunction<java.lang.String,Engine.SearcherScope,Engine.Searcher> searcherFactory)
protected Engine.GetResult
getFromSearcher(Engine.Get get, java.util.function.BiFunction<java.lang.String,Engine.SearcherScope,Engine.Searcher> searcherFactory, Engine.SearcherScope scope)
abstract java.lang.String
getHistoryUUID()
returns the history uuid for the engineabstract long
getIndexBufferRAMBytesUsed()
How much heap is used that would be freed by a refresh.abstract long
getIndexThrottleTimeInMillis()
Returns the number of milliseconds this engine was under index throttling.protected abstract org.apache.lucene.index.SegmentInfos
getLastCommittedSegmentInfos()
abstract long
getLastSyncedGlobalCheckpoint()
Returns the latest global checkpoint value that has been persisted in the underlying storage (i.e.long
getLastWriteNanos()
Returns the timestamp of the last write in nanoseconds.long
getMaxSeenAutoIdTimestamp()
Returns the maximum auto_id_timestamp of all append-only index requests have been processed by this engine or the auto_id_timestamp received from its primary shard viaupdateMaxUnsafeAutoIdTimestamp(long)
.abstract long
getMaxSeqNoOfUpdatesOrDeletes()
Returns the maximum sequence number of either update or delete operations have been processed in this engine or the sequence number fromadvanceMaxSeqNoOfUpdatesOrDeletes(long)
.MergeStats
getMergeStats()
abstract long
getMinRetainedSeqNo()
Gets the minimum retained sequence number for this engine.abstract long
getPersistedLocalCheckpoint()
protected abstract org.apache.lucene.search.ReferenceManager<ElasticsearchDirectoryReader>
getReferenceManager(Engine.SearcherScope scope)
abstract SafeCommitInfo
getSafeCommitInfo()
abstract SeqNoStats
getSeqNoStats(long globalCheckpoint)
abstract Translog.Location
getTranslogLastWriteLocation()
Returns the last location that the translog of this engine has written into.abstract TranslogStats
getTranslogStats()
abstract long
getWritingBytes()
Returns how many bytes we are currently moving from heap to diskprotected static long
guardedRamBytesUsed(org.apache.lucene.util.Accountable a)
Returns 0 in the case where accountable is null, otherwise returnsramBytesUsed()
abstract boolean
hasCompleteOperationHistory(java.lang.String reason, Engine.HistorySource historySource, MapperService mapperService, long startingSeqNo)
Checks if this engine has every operations sincestartingSeqNo
(inclusive) in its history (either Lucene or translog)abstract Engine.IndexResult
index(Engine.Index index)
Perform document index operation on the engineabstract boolean
isThrottled()
Returns thetrue
iff this engine is currently under index throttling.abstract boolean
isTranslogSyncNeeded()
Checks if the underlying storage sync is required.protected boolean
maybeFailEngine(java.lang.String source, java.lang.Exception e)
Check whether the engine should be failedabstract void
maybePruneDeletes()
Tries to prune buffered deletes from the version map.abstract boolean
maybeRefresh(java.lang.String source)
Synchronously refreshes the engine for new search operations to reflect the latest changes unless another thread is already refreshing the engine concurrently.abstract Translog.Snapshot
newChangesSnapshot(java.lang.String source, MapperService mapperService, long fromSeqNo, long toSeqNo, boolean requiredFullRange)
Creates a new history snapshot from Lucene for reading operations whose seqno in the requesting seqno range (both inclusive).abstract Engine.NoOpResult
noOp(Engine.NoOp noOp)
void
onSettingsChanged(org.elasticsearch.common.unit.TimeValue translogRetentionAge, ByteSizeValue translogRetentionSize, long softDeletesRetentionOps)
abstract Translog.Snapshot
readHistoryOperations(java.lang.String reason, Engine.HistorySource historySource, MapperService mapperService, long startingSeqNo)
Creates a new history snapshot for reading operations sincestartingSeqNo
(inclusive).abstract Engine
recoverFromTranslog(Engine.TranslogRecoveryRunner translogRecoveryRunner, long recoverUpToSeqNo)
Performs recovery from the transaction log up torecoverUpToSeqNo
(inclusive).abstract void
refresh(java.lang.String source)
Synchronously refreshes the engine for new search operations to reflect the latest changes.boolean
refreshNeeded()
abstract int
restoreLocalHistoryFromTranslog(Engine.TranslogRecoveryRunner translogRecoveryRunner)
This method replays translog to restore the Lucene index which might be reverted previously.abstract void
rollTranslogGeneration()
Rolls the translog generation and cleans unneeded.abstract java.util.List<Segment>
segments(boolean verbose)
The list of segments in the engine.SegmentsStats
segmentsStats(boolean includeSegmentFileSizes, boolean includeUnloadedSegments)
Global stats on segments.abstract boolean
shouldPeriodicallyFlush()
Checks if this engine should be flushed periodically.abstract boolean
shouldRollTranslogGeneration()
Tests whether or not the translog generation should be rolled to a new generation.abstract void
skipTranslogRecovery()
Do not replay translog operations, but make the engine be ready.abstract Engine.SyncedFlushResult
syncFlush(java.lang.String syncId, Engine.CommitId expectedCommitId)
Attempts to do a special commit where the given syncID is put into the commit data.abstract void
syncTranslog()
abstract void
trimOperationsFromTranslog(long belowTerm, long aboveSeqNo)
Trims translog for terms belowbelowTerm
and seq# aboveaboveSeqNo
abstract void
trimUnreferencedTranslogFiles()
checks and removes translog files that no longer need to be retained.abstract void
updateMaxUnsafeAutoIdTimestamp(long newTimestamp)
Forces this engine to advance its max_unsafe_auto_id_timestamp marker to at least the given timestamp.void
verifyEngineBeforeIndexClosing()
Performs the pre-closing checks on theEngine
.abstract void
writeIndexingBuffer()
Called when our engine is using too much heap and should move buffered indexed/deleted documents to disk.protected void
writerSegmentStats(SegmentsStats stats)
-
Field Details
-
SYNC_COMMIT_ID
public static final java.lang.String SYNC_COMMIT_ID- See Also:
- Constant Field Values
-
HISTORY_UUID_KEY
public static final java.lang.String HISTORY_UUID_KEY- See Also:
- Constant Field Values
-
FORCE_MERGE_UUID_KEY
public static final java.lang.String FORCE_MERGE_UUID_KEY- See Also:
- Constant Field Values
-
MIN_RETAINED_SEQNO
public static final java.lang.String MIN_RETAINED_SEQNO- See Also:
- Constant Field Values
-
MAX_UNSAFE_AUTO_ID_TIMESTAMP_COMMIT_ID
public static final java.lang.String MAX_UNSAFE_AUTO_ID_TIMESTAMP_COMMIT_ID- See Also:
- Constant Field Values
-
shardId
-
allocationId
protected final java.lang.String allocationId -
logger
protected final org.apache.logging.log4j.Logger logger -
engineConfig
-
store
-
isClosed
protected final java.util.concurrent.atomic.AtomicBoolean isClosed -
eventListener
-
failEngineLock
protected final java.util.concurrent.locks.ReentrantLock failEngineLock -
rwl
protected final java.util.concurrent.locks.ReentrantReadWriteLock rwl -
readLock
-
writeLock
-
failedEngine
protected final org.apache.lucene.util.SetOnce<java.lang.Exception> failedEngine -
lastWriteNanos
protected volatile long lastWriteNanos
-
-
Constructor Details
-
Method Details
-
guardedRamBytesUsed
protected static long guardedRamBytesUsed(org.apache.lucene.util.Accountable a)Returns 0 in the case where accountable is null, otherwise returnsramBytesUsed()
-
config
-
getLastCommittedSegmentInfos
protected abstract org.apache.lucene.index.SegmentInfos getLastCommittedSegmentInfos() -
getMergeStats
-
getHistoryUUID
public abstract java.lang.String getHistoryUUID()returns the history uuid for the engine -
getWritingBytes
public abstract long getWritingBytes()Returns how many bytes we are currently moving from heap to disk -
completionStats
Returns theCompletionStats
for this engine -
docStats
Returns theDocsStats
for this engine -
docsStats
-
verifyEngineBeforeIndexClosing
public void verifyEngineBeforeIndexClosing() throws java.lang.IllegalStateExceptionPerforms the pre-closing checks on theEngine
.- Throws:
java.lang.IllegalStateException
- if the sanity checks failed
-
getIndexThrottleTimeInMillis
public abstract long getIndexThrottleTimeInMillis()Returns the number of milliseconds this engine was under index throttling. -
isThrottled
public abstract boolean isThrottled()Returns thetrue
iff this engine is currently under index throttling.- See Also:
getIndexThrottleTimeInMillis()
-
trimOperationsFromTranslog
public abstract void trimOperationsFromTranslog(long belowTerm, long aboveSeqNo) throws EngineExceptionTrims translog for terms belowbelowTerm
and seq# aboveaboveSeqNo
- Throws:
EngineException
- See Also:
Translog.trimOperations(long, long)
-
index
Perform document index operation on the engine- Parameters:
index
- operation to perform- Returns:
Engine.IndexResult
containing updated translog location, version and document specific failures Note: engine level failures (i.e. persistent engine failures) are thrown- Throws:
java.io.IOException
-
delete
Perform document delete operation on the engine- Parameters:
delete
- operation to perform- Returns:
Engine.DeleteResult
containing updated translog location, version and document specific failures Note: engine level failures (i.e. persistent engine failures) are thrown- Throws:
java.io.IOException
-
noOp
- Throws:
java.io.IOException
-
syncFlush
public abstract Engine.SyncedFlushResult syncFlush(java.lang.String syncId, Engine.CommitId expectedCommitId) throws EngineExceptionAttempts to do a special commit where the given syncID is put into the commit data. The attempt succeeds if there are not pending writes in lucene and the current point is equal to the expected one.- Parameters:
syncId
- id of this syncexpectedCommitId
- the expected value of- Returns:
- true if the sync commit was made, false o.w.
- Throws:
EngineException
-
getFromSearcher
protected final Engine.GetResult getFromSearcher(Engine.Get get, java.util.function.BiFunction<java.lang.String,Engine.SearcherScope,Engine.Searcher> searcherFactory, Engine.SearcherScope scope) throws EngineException- Throws:
EngineException
-
get
public abstract Engine.GetResult get(Engine.Get get, java.util.function.BiFunction<java.lang.String,Engine.SearcherScope,Engine.Searcher> searcherFactory) throws EngineException- Throws:
EngineException
-
acquireSearcher
Returns a new searcher instance. The consumer of this API is responsible for releasing the returned searcher in a safe manner, preferably in a try/finally block.- Parameters:
source
- the source API or routing that triggers this searcher acquire- Throws:
EngineException
- See Also:
Engine.Searcher.close()
-
acquireSearcher
public Engine.Searcher acquireSearcher(java.lang.String source, Engine.SearcherScope scope) throws EngineExceptionReturns a new searcher instance. The consumer of this API is responsible for releasing the returned searcher in a safe manner, preferably in a try/finally block.- Parameters:
source
- the source API or routing that triggers this searcher acquirescope
- the scope of this searcher ie. if the searcher will be used for get or search purposes- Throws:
EngineException
- See Also:
Engine.Searcher.close()
-
getReferenceManager
protected abstract org.apache.lucene.search.ReferenceManager<ElasticsearchDirectoryReader> getReferenceManager(Engine.SearcherScope scope) -
isTranslogSyncNeeded
public abstract boolean isTranslogSyncNeeded()Checks if the underlying storage sync is required. -
ensureTranslogSynced
public abstract boolean ensureTranslogSynced(java.util.stream.Stream<Translog.Location> locations) throws java.io.IOExceptionEnsures that all locations in the given stream have been written to the underlying storage.- Throws:
java.io.IOException
-
syncTranslog
public abstract void syncTranslog() throws java.io.IOException- Throws:
java.io.IOException
-
acquireHistoryRetentionLock
Acquires a lock on the translog files and Lucene soft-deleted documents to prevent them from being trimmed -
newChangesSnapshot
public abstract Translog.Snapshot newChangesSnapshot(java.lang.String source, MapperService mapperService, long fromSeqNo, long toSeqNo, boolean requiredFullRange) throws java.io.IOExceptionCreates a new history snapshot from Lucene for reading operations whose seqno in the requesting seqno range (both inclusive). This feature requires soft-deletes enabled. If soft-deletes are disabled, this method will throw anIllegalStateException
.- Throws:
java.io.IOException
-
readHistoryOperations
public abstract Translog.Snapshot readHistoryOperations(java.lang.String reason, Engine.HistorySource historySource, MapperService mapperService, long startingSeqNo) throws java.io.IOExceptionCreates a new history snapshot for reading operations sincestartingSeqNo
(inclusive). The returned snapshot can be retrieved from either Lucene index or translog files.- Throws:
java.io.IOException
-
estimateNumberOfHistoryOperations
public abstract int estimateNumberOfHistoryOperations(java.lang.String reason, Engine.HistorySource historySource, MapperService mapperService, long startingSeqNo) throws java.io.IOExceptionReturns the estimated number of history operations whose seq# at leaststartingSeqNo
(inclusive) in this engine.- Throws:
java.io.IOException
-
hasCompleteOperationHistory
public abstract boolean hasCompleteOperationHistory(java.lang.String reason, Engine.HistorySource historySource, MapperService mapperService, long startingSeqNo) throws java.io.IOExceptionChecks if this engine has every operations sincestartingSeqNo
(inclusive) in its history (either Lucene or translog)- Throws:
java.io.IOException
-
getMinRetainedSeqNo
public abstract long getMinRetainedSeqNo()Gets the minimum retained sequence number for this engine.- Returns:
- the minimum retained sequence number
-
getTranslogStats
-
getTranslogLastWriteLocation
Returns the last location that the translog of this engine has written into. -
ensureOpen
protected final void ensureOpen(java.lang.Exception suppressed) -
ensureOpen
protected final void ensureOpen() -
commitStats
get commits stats for the last commit -
getPersistedLocalCheckpoint
public abstract long getPersistedLocalCheckpoint()- Returns:
- the persisted local checkpoint for this Engine
-
getSeqNoStats
- Returns:
- a
SeqNoStats
object, using local state and the supplied global checkpoint
-
getLastSyncedGlobalCheckpoint
public abstract long getLastSyncedGlobalCheckpoint()Returns the latest global checkpoint value that has been persisted in the underlying storage (i.e. translog's checkpoint) -
segmentsStats
public SegmentsStats segmentsStats(boolean includeSegmentFileSizes, boolean includeUnloadedSegments)Global stats on segments. -
fillSegmentStats
protected void fillSegmentStats(org.apache.lucene.index.SegmentReader segmentReader, boolean includeSegmentFileSizes, SegmentsStats stats) -
writerSegmentStats
-
getIndexBufferRAMBytesUsed
public abstract long getIndexBufferRAMBytesUsed()How much heap is used that would be freed by a refresh. Note that this may throwAlreadyClosedException
. -
segments
The list of segments in the engine. -
refreshNeeded
public boolean refreshNeeded() -
refresh
Synchronously refreshes the engine for new search operations to reflect the latest changes.- Throws:
EngineException
-
maybeRefresh
Synchronously refreshes the engine for new search operations to reflect the latest changes unless another thread is already refreshing the engine concurrently.- Returns:
true
if the a refresh happened. Otherwisefalse
- Throws:
EngineException
-
writeIndexingBuffer
Called when our engine is using too much heap and should move buffered indexed/deleted documents to disk.- Throws:
EngineException
-
shouldPeriodicallyFlush
public abstract boolean shouldPeriodicallyFlush()Checks if this engine should be flushed periodically. This check is mainly based on the uncommitted translog size and the translog flush threshold setting. -
flush
Flushes the state of the engine including the transaction log, clearing memory.- Parameters:
force
- iftrue
a lucene commit is executed even if no changes need to be committed.waitIfOngoing
- iftrue
this call will block until all currently running flushes have finished. Otherwise this call will return without blocking.- Returns:
- the commit Id for the resulting commit
- Throws:
EngineException
-
flush
Flushes the state of the engine including the transaction log, clearing memory and persisting documents in the lucene index to disk including a potentially heavy and durable fsync operation. This operation is not going to block if another flush operation is currently running and won't write a lucene commit if nothing needs to be committed.- Returns:
- the commit Id for the resulting commit
- Throws:
EngineException
-
trimUnreferencedTranslogFiles
checks and removes translog files that no longer need to be retained. SeeTranslogDeletionPolicy
for details- Throws:
EngineException
-
shouldRollTranslogGeneration
public abstract boolean shouldRollTranslogGeneration()Tests whether or not the translog generation should be rolled to a new generation. This test is based on the size of the current generation compared to the configured generation threshold size.- Returns:
true
if the current generation should be rolled to a new generation
-
rollTranslogGeneration
Rolls the translog generation and cleans unneeded.- Throws:
EngineException
-
forceMerge
public abstract void forceMerge(boolean flush, int maxNumSegments, boolean onlyExpungeDeletes, boolean upgrade, boolean upgradeOnlyAncientSegments, @Nullable java.lang.String forceMergeUUID) throws EngineException, java.io.IOExceptionTriggers a forced merge on this engine- Throws:
EngineException
java.io.IOException
-
acquireLastIndexCommit
public abstract Engine.IndexCommitRef acquireLastIndexCommit(boolean flushFirst) throws EngineExceptionSnapshots the most recent index and returns a handle to it. If needed will try and "commit" the lucene index to make sure we have a "fresh" copy of the files to snapshot.- Parameters:
flushFirst
- indicates whether the engine should flush before returning the snapshot- Throws:
EngineException
-
acquireSafeIndexCommit
Snapshots the most recent safe index commit from the engine.- Throws:
EngineException
-
getSafeCommitInfo
- Returns:
- a summary of the contents of the current safe commit
-
failEngine
public void failEngine(java.lang.String reason, @Nullable java.lang.Exception failure)fail engine due to some error. the engine will also be closed. The underlying store is marked corrupted iff failure is caused by index corruption -
maybeFailEngine
protected boolean maybeFailEngine(java.lang.String source, java.lang.Exception e)Check whether the engine should be failed -
closeNoLock
protected abstract void closeNoLock(java.lang.String reason, java.util.concurrent.CountDownLatch closedLatch)Method to close the engine while the write lock is held. Must decrement the supplied when closing work is done and resources are freed. -
flushAndClose
public void flushAndClose() throws java.io.IOExceptionFlush the engine (committing segments to disk and truncating the translog) and close it.- Throws:
java.io.IOException
-
close
public void close() throws java.io.IOException- Specified by:
close
in interfacejava.lang.AutoCloseable
- Specified by:
close
in interfacejava.io.Closeable
- Throws:
java.io.IOException
-
onSettingsChanged
public void onSettingsChanged(org.elasticsearch.common.unit.TimeValue translogRetentionAge, ByteSizeValue translogRetentionSize, long softDeletesRetentionOps) -
getLastWriteNanos
public long getLastWriteNanos()Returns the timestamp of the last write in nanoseconds. Note: this time might not be absolutely accurate since theEngine.Operation.startTime()
is used which might be slightly inaccurate.- See Also:
System.nanoTime()
,Engine.Operation.startTime()
-
activateThrottling
public abstract void activateThrottling()Request that this engine throttle incoming indexing requests to one thread. Must be matched by a later call todeactivateThrottling()
. -
deactivateThrottling
public abstract void deactivateThrottling()Reverses a previousactivateThrottling()
call. -
restoreLocalHistoryFromTranslog
public abstract int restoreLocalHistoryFromTranslog(Engine.TranslogRecoveryRunner translogRecoveryRunner) throws java.io.IOExceptionThis method replays translog to restore the Lucene index which might be reverted previously. This ensures that all acknowledged writes are restored correctly when this engine is promoted.- Returns:
- the number of translog operations have been recovered
- Throws:
java.io.IOException
-
fillSeqNoGaps
public abstract int fillSeqNoGaps(long primaryTerm) throws java.io.IOExceptionFills up the local checkpoints history with no-ops until the local checkpoint and the max seen sequence ID are identical.- Parameters:
primaryTerm
- the shards primary term this engine was created for- Returns:
- the number of no-ops added
- Throws:
java.io.IOException
-
recoverFromTranslog
public abstract Engine recoverFromTranslog(Engine.TranslogRecoveryRunner translogRecoveryRunner, long recoverUpToSeqNo) throws java.io.IOExceptionPerforms recovery from the transaction log up torecoverUpToSeqNo
(inclusive). This operation will close the engine if the recovery fails.- Parameters:
translogRecoveryRunner
- the translog recovery runnerrecoverUpToSeqNo
- the upper bound, inclusive, of sequence number to be recovered- Throws:
java.io.IOException
-
skipTranslogRecovery
public abstract void skipTranslogRecovery()Do not replay translog operations, but make the engine be ready. -
maybePruneDeletes
public abstract void maybePruneDeletes()Tries to prune buffered deletes from the version map. -
getMaxSeenAutoIdTimestamp
public long getMaxSeenAutoIdTimestamp()Returns the maximum auto_id_timestamp of all append-only index requests have been processed by this engine or the auto_id_timestamp received from its primary shard viaupdateMaxUnsafeAutoIdTimestamp(long)
. Notes this method returns the auto_id_timestamp of all append-only requests, not max_unsafe_auto_id_timestamp. -
updateMaxUnsafeAutoIdTimestamp
public abstract void updateMaxUnsafeAutoIdTimestamp(long newTimestamp)Forces this engine to advance its max_unsafe_auto_id_timestamp marker to at least the given timestamp. The engine will disable optimization for all append-only whose timestamp at mostnewTimestamp
. -
getMaxSeqNoOfUpdatesOrDeletes
public abstract long getMaxSeqNoOfUpdatesOrDeletes()Returns the maximum sequence number of either update or delete operations have been processed in this engine or the sequence number fromadvanceMaxSeqNoOfUpdatesOrDeletes(long)
. An index request is considered as an update operation if it overwrites the existing documents in Lucene index with the same document id.A note on the optimization using max_seq_no_of_updates_or_deletes: For each operation O, the key invariants are:
- I1: There is no operation on docID(O) with seqno that is > MSU(O) and < seqno(O)
- I2: If MSU(O) < seqno(O) then docID(O) did not exist when O was applied; more precisely, if there is any O' with seqno(O') < seqno(O) and docID(O') = docID(O) then the one with the greatest seqno is a delete.
When a receiving shard (either a replica or a follower) receives an operation O, it must first ensure its own MSU at least MSU(O), and then compares its MSU to its local checkpoint (LCP). If LCP < MSU then there's a gap: there may be some operations that act on docID(O) about which we do not yet know, so we cannot perform an add. Note this also covers the case where a future operation O' with seqNo(O') > seqNo(O) and docId(O') = docID(O) is processed before O. In that case MSU(O') is at least seqno(O') and this means MSU >= seqNo(O') > seqNo(O) > LCP (because O wasn't processed yet).
However, if MSU <= LCP then there is no gap: we have processed every operation <= LCP, and no operation O' with seqno(O') > LCP and seqno(O') < seqno(O) also has docID(O') = docID(O), because such an operation would have seqno(O') > LCP >= MSU >= MSU(O) which contradicts the first invariant. Furthermore in this case we immediately know that docID(O) has been deleted (or never existed) without needing to check Lucene for the following reason. If there's no earlier operation on docID(O) then this is clear, so suppose instead that the preceding operation on docID(O) is O': 1. The first invariant above tells us that seqno(O') <= MSU(O) <= LCP so we have already applied O' to Lucene. 2. Also MSU(O) <= MSU <= LCP < seqno(O) (we discard O if seqno(O) <= LCP) so the second invariant applies, meaning that the O' was a delete.
Therefore, if MSU <= LCP < seqno(O) we know that O can safely be optimized with and added to lucene with addDocument. Moreover, operations that are optimized using the MSU optimization must not be processed twice as this will create duplicates in Lucene. To avoid this we check the local checkpoint tracker to see if an operation was already processed.
- See Also:
advanceMaxSeqNoOfUpdatesOrDeletes(long)
-
advanceMaxSeqNoOfUpdatesOrDeletes
public abstract void advanceMaxSeqNoOfUpdatesOrDeletes(long maxSeqNoOfUpdatesOnPrimary)A replica shard receives a new max_seq_no_of_updates from its primary shard, then calls this method to advance this marker to at least the given sequence number.
-