- All Implemented Interfaces:
Closeable
,AutoCloseable
-
Nested Class Summary
Nested ClassesModifier and TypeClassDescriptionprotected static final class
protected static final class
Nested classes/interfaces inherited from class org.elasticsearch.index.engine.Engine
Engine.Delete, Engine.DeleteResult, Engine.EventListener, Engine.FlushResult, Engine.Get, Engine.GetResult, Engine.Index, Engine.IndexCommitListener, Engine.IndexCommitRef, Engine.IndexResult, Engine.IndexThrottle, Engine.NoOp, Engine.NoOpLock, Engine.NoOpResult, Engine.Operation, Engine.RefreshResult, Engine.Result, Engine.Searcher, Engine.SearcherScope, Engine.SearcherSupplier, Engine.TranslogRecoveryRunner, Engine.Warmer
-
Field Summary
FieldsModifier and TypeFieldDescriptionprotected static final String
final AtomicLong
final AtomicLong
protected static final String
Fields inherited from class org.elasticsearch.index.engine.Engine
CAN_MATCH_SEARCH_SOURCE, DOC_STATS_SOURCE, engineConfig, ES_VERSION, eventListener, failedEngine, failEngineLock, FORCE_MERGE_UUID_KEY, HISTORY_UUID_KEY, isClosed, lastWriteNanos, logger, MAX_UNSAFE_AUTO_ID_TIMESTAMP_COMMIT_ID, MIN_RETAINED_SEQNO, readLock, rwl, SEARCH_SOURCE, shardId, store, SYNC_COMMIT_ID, writeLock
-
Constructor Summary
Constructors -
Method Summary
Modifier and TypeMethodDescriptionAcquires a lock on the translog files and Lucene soft-deleted documents to prevent them from being trimmedacquireLastIndexCommit
(boolean flushFirst) Snapshots the most recent index and returns a handle to it.Snapshots the most recent safe index commit from the engine.void
Request that this engine throttle incoming indexing requests to one thread.void
addFlushListener
(Translog.Location location, ActionListener<Long> listener) protected void
advanceMaxSeqNoOfDeletesOnPrimary
(long seqNo) protected void
advanceMaxSeqNoOfUpdatesOnPrimary
(long seqNo) void
advanceMaxSeqNoOfUpdatesOrDeletes
(long maxSeqNoOfUpdatesOnPrimary) A replica shard receives a new max_seq_no_of_updates from its primary shard, then calls this method to advance this marker to at least the given sequence number.protected boolean
assertNonPrimaryOrigin
(Engine.Operation operation) protected boolean
protected boolean
assertPrimaryIncomingSequenceNumber
(Engine.Operation.Origin origin, long seqNo) void
asyncEnsureGlobalCheckpointSynced
(long globalCheckpoint, Consumer<Exception> listener) Ensures that the global checkpoint has been persisted to the underlying storage.void
asyncEnsureTranslogSynced
(Translog.Location location, Consumer<Exception> listener) Ensures that the location has been written to the underlying storage.protected final void
closeNoLock
(String reason, CountDownLatch closedLatch) Closes the engine without acquiring the write lock.protected void
commitIndexWriter
(org.apache.lucene.index.IndexWriter writer, Translog translog) Commits the specified index writer.completionStats
(String... fieldNamePatterns) Returns theCompletionStats
for this engineint
countChanges
(String source, long fromSeqNo, long toSeqNo) Counts the number of operations in the range of the given sequence numbers.protected ElasticsearchReaderManager
createInternalReaderManager
(ElasticsearchDirectoryReader directoryReader) protected LiveVersionMapArchive
void
Reverses a previousEngine.activateThrottling()
call.delete
(Engine.Delete delete) Perform document delete operation on the engineprotected InternalEngine.DeletionStrategy
int
fillSeqNoGaps
(long primaryTerm) Fills up the local checkpoints history with no-ops until the local checkpoint and the max seen sequence ID are identical.void
flush
(boolean force, boolean waitIfOngoing, ActionListener<Engine.FlushResult> listener) Flushes the state of the engine including the transaction log, clearing memory, and writing the documents in the Lucene index to disk.void
forceMerge
(boolean flush, int maxNumSegments, boolean onlyExpungeDeletes, String forceMergeUUID) Triggers a forced merge on this engineprotected long
get
(Engine.Get get, MappingLookup mappingLookup, DocumentParser documentParser, Function<Engine.Searcher, Engine.Searcher> searcherWrapper) Allows InternalEngine extenders to return custom key-value pairs which will be included in the Lucene commit user-data.returns the force merge uuid for the enginegetFromTranslog
(Engine.Get get, MappingLookup mappingLookup, DocumentParser documentParser, Function<Engine.Searcher, Engine.Searcher> searcherWrapper) Similar toEngine.get(org.elasticsearch.index.engine.Engine.Get, org.elasticsearch.index.mapper.MappingLookup, org.elasticsearch.index.mapper.DocumentParser, java.util.function.Function<org.elasticsearch.index.engine.Engine.Searcher, org.elasticsearch.index.engine.Engine.Searcher>)
, but it only attempts to serve the get from the translog.returns the history uuid for the enginelong
How much heap is used that would be freed by a refresh.long
Returns the number of milliseconds this engine was under index throttling.org.apache.lucene.index.SegmentInfos
long
Returns the latest global checkpoint value that has been persisted in the underlying storage (i.e.long
protected LiveVersionMapArchive
final long
Returns the maximum auto_id_timestamp of all append-only index requests have been processed by this engine or the auto_id_timestamp received from its primary shard viaEngine.updateMaxUnsafeAutoIdTimestamp(long)
.long
long
Returns the maximum sequence number of either update or delete operations have been processed in this engine or the sequence number fromEngine.advanceMaxSeqNoOfUpdatesOrDeletes(long)
.final long
Returns the minimum seqno that is retained in the Lucene index.long
protected long
long
getRawFieldRange
(String field) protected final org.apache.lucene.search.ReferenceManager<ElasticsearchDirectoryReader>
getSeqNoStats
(long globalCheckpoint) Returns the last location that the translog of this engine has written into.long
Returns how many bytes we are currently moving from indexing buffer to segments on diskprotected final boolean
Checks if the given operation has been processed in this engine or not.boolean
hasCompleteOperationHistory
(String reason, long startingSeqNo) Checks if this engine has every operations sincestartingSeqNo
(inclusive) in its history (either Lucene or translog)protected boolean
index
(Engine.Index index) Perform document index operation on the engineprotected InternalEngine.IndexingStrategy
boolean
Returns thetrue
iff this engine is currently under index throttling.boolean
Checks if the underlying storage sync is required.protected final void
markSeqNoAsSeen
(long seqNo) Marks the given seq_no as seen and advances the max_seq_no of this engine to at least that value.protected boolean
maybeFailEngine
(String source, Exception e) Check whether the engine should be failedvoid
Tries to prune buffered deletes from the version map.void
maybeRefresh
(String source, ActionListener<Engine.RefreshResult> listener) Asynchronously refreshes the engine for new search operations to reflect the latest changes unless another thread is already refreshing the engine concurrently.newChangesSnapshot
(String source, long fromSeqNo, long toSeqNo, boolean requiredFullRange, boolean singleConsumer, boolean accessStats) Creates a new history snapshot from Lucene for reading operations whose seqno in the requesting seqno range (both inclusive).protected Translog.Snapshot
newTranslogSnapshot
(long fromSeqNo, long toSeqNo) noOp
(Engine.NoOp noOp) void
protected final InternalEngine.DeletionStrategy
protected final InternalEngine.IndexingStrategy
Executes a pre-flight check for a given NoOp.protected Engine.GetResult
realtimeGetUnderLock
(Engine.Get get, MappingLookup mappingLookup, DocumentParser documentParser, Function<Engine.Searcher, Engine.Searcher> searcherWrapper, boolean getFromSearcher) void
recoverFromTranslog
(Engine.TranslogRecoveryRunner translogRecoveryRunner, long recoverUpToSeqNo, ActionListener<Void> listener) Performs recovery from the transaction log up torecoverUpToSeqNo
(inclusive).Synchronously refreshes the engine for new search operations to reflect the latest changes.protected final Engine.RefreshResult
refresh
(String source, Engine.SearcherScope scope, boolean block) protected final void
refreshIfNeeded
(String source, long requestingSeqNo) Refresh this engine **internally** iff the requesting seq_no is greater than the last refreshed checkpoint.protected Engine.RefreshResult
refreshInternalSearcher
(String source, boolean block) int
restoreLocalHistoryFromTranslog
(Engine.TranslogRecoveryRunner translogRecoveryRunner) This method replays translog to restore the Lucene index which might be reverted previously.void
Rolls the translog generation and cleans unneeded.segments()
The list of segments in the engine.boolean
Checks if this engine should be flushed periodically.boolean
Tests whether or not the translog generation should be rolled to a new generation.void
Do not replay translog operations, but make the engine be ready.void
void
trimOperationsFromTranslog
(long belowTerm, long aboveSeqNo) Trims translog for terms belowbelowTerm
and seq# aboveaboveSeqNo
void
checks and removes translog files that no longer need to be retained.final void
updateMaxUnsafeAutoIdTimestamp
(long newTimestamp) Forces this engine to advance its max_unsafe_auto_id_timestamp marker to at least the given timestamp.protected void
waitForCommitDurability
(long generation, ActionListener<Void> listener) void
Called when our engine is using too much heap and should move buffered indexed/deleted documents to disk.protected final void
writerSegmentStats
(SegmentsStats stats) Methods inherited from class org.elasticsearch.index.engine.Engine
acquireIndexCommitForSnapshot, acquireSearcher, acquireSearcher, acquireSearcher, acquireSearcherSupplier, acquireSearcherSupplier, addSegmentGenerationListener, allowSearchIdleOptimization, close, commitStats, config, denseVectorStats, denseVectorStats, docsStats, docStats, ensureOpen, ensureOpen, externalRefresh, failEngine, fillSegmentStats, flush, flush, flushAndClose, getEngineConfig, getFromSearcher, getLastWriteNanos, readIndexVersion, recoverFromTranslog, refreshNeeded, segmentsStats, verifyEngineBeforeIndexClosing
-
Field Details
-
REAL_TIME_GET_REFRESH_SOURCE
- See Also:
-
UNSAFE_VERSION_MAP_REFRESH_SOURCE
- See Also:
-
translogGetCount
-
translogInMemorySegmentsCount
-
-
Constructor Details
-
InternalEngine
-
-
Method Details
-
completionStats
Description copied from class:Engine
Returns theCompletionStats
for this engine- Specified by:
completionStats
in classEngine
-
restoreLocalHistoryFromTranslog
public int restoreLocalHistoryFromTranslog(Engine.TranslogRecoveryRunner translogRecoveryRunner) throws IOException Description copied from class:Engine
This method replays translog to restore the Lucene index which might be reverted previously. This ensures that all acknowledged writes are restored correctly when this engine is promoted.- Specified by:
restoreLocalHistoryFromTranslog
in classEngine
- Returns:
- the number of translog operations have been recovered
- Throws:
IOException
-
fillSeqNoGaps
Description copied from class:Engine
Fills up the local checkpoints history with no-ops until the local checkpoint and the max seen sequence ID are identical.- Specified by:
fillSeqNoGaps
in classEngine
- Parameters:
primaryTerm
- the shards primary term this engine was created for- Returns:
- the number of no-ops added
- Throws:
IOException
-
recoverFromTranslog
public void recoverFromTranslog(Engine.TranslogRecoveryRunner translogRecoveryRunner, long recoverUpToSeqNo, ActionListener<Void> listener) Description copied from class:Engine
Performs recovery from the transaction log up torecoverUpToSeqNo
(inclusive). This operation will close the engine if the recovery fails.- Specified by:
recoverFromTranslog
in classEngine
- Parameters:
translogRecoveryRunner
- the translog recovery runnerrecoverUpToSeqNo
- the upper bound, inclusive, of sequence number to be recoveredlistener
- listener notified on completion of the recovery, whether successful or otherwise
-
skipTranslogRecovery
public void skipTranslogRecovery()Description copied from class:Engine
Do not replay translog operations, but make the engine be ready.- Specified by:
skipTranslogRecovery
in classEngine
-
newTranslogSnapshot
- Throws:
IOException
-
isTranslogSyncNeeded
public boolean isTranslogSyncNeeded()Description copied from class:Engine
Checks if the underlying storage sync is required.- Specified by:
isTranslogSyncNeeded
in classEngine
-
asyncEnsureTranslogSynced
Description copied from class:Engine
Ensures that the location has been written to the underlying storage.- Specified by:
asyncEnsureTranslogSynced
in classEngine
-
asyncEnsureGlobalCheckpointSynced
Description copied from class:Engine
Ensures that the global checkpoint has been persisted to the underlying storage.- Specified by:
asyncEnsureGlobalCheckpointSynced
in classEngine
-
syncTranslog
- Specified by:
syncTranslog
in classEngine
- Throws:
IOException
-
getTranslogStats
- Specified by:
getTranslogStats
in classEngine
-
getTranslogLastWriteLocation
Description copied from class:Engine
Returns the last location that the translog of this engine has written into.- Specified by:
getTranslogLastWriteLocation
in classEngine
-
getHistoryUUID
Description copied from class:Engine
returns the history uuid for the engine- Specified by:
getHistoryUUID
in classEngine
-
getForceMergeUUID
returns the force merge uuid for the engine -
getWritingBytes
public long getWritingBytes()Returns how many bytes we are currently moving from indexing buffer to segments on disk- Specified by:
getWritingBytes
in classEngine
-
createInternalReaderManager
protected ElasticsearchReaderManager createInternalReaderManager(ElasticsearchDirectoryReader directoryReader) -
get
public Engine.GetResult get(Engine.Get get, MappingLookup mappingLookup, DocumentParser documentParser, Function<Engine.Searcher, Engine.Searcher> searcherWrapper) -
getFromTranslog
public Engine.GetResult getFromTranslog(Engine.Get get, MappingLookup mappingLookup, DocumentParser documentParser, Function<Engine.Searcher, Engine.Searcher> searcherWrapper) Description copied from class:Engine
Similar toEngine.get(org.elasticsearch.index.engine.Engine.Get, org.elasticsearch.index.mapper.MappingLookup, org.elasticsearch.index.mapper.DocumentParser, java.util.function.Function<org.elasticsearch.index.engine.Engine.Searcher, org.elasticsearch.index.engine.Engine.Searcher>)
, but it only attempts to serve the get from the translog. If not found in translog, it returns null, asEngine.GetResult.NOT_EXISTS
could mean deletion.- Overrides:
getFromTranslog
in classEngine
-
realtimeGetUnderLock
protected Engine.GetResult realtimeGetUnderLock(Engine.Get get, MappingLookup mappingLookup, DocumentParser documentParser, Function<Engine.Searcher, Engine.Searcher> searcherWrapper, boolean getFromSearcher) - Parameters:
getFromSearcher
- indicates whether we also try the internal searcher if not found in translog. In the case where we just started tracking locations in the translog, we always use the internal searcher.
-
assertPrimaryCanOptimizeAddDocument
-
assertPrimaryIncomingSequenceNumber
-
generateSeqNoForOperationOnPrimary
-
advanceMaxSeqNoOfUpdatesOnPrimary
protected void advanceMaxSeqNoOfUpdatesOnPrimary(long seqNo) -
advanceMaxSeqNoOfDeletesOnPrimary
protected void advanceMaxSeqNoOfDeletesOnPrimary(long seqNo) -
index
Description copied from class:Engine
Perform document index operation on the engine- Specified by:
index
in classEngine
- Parameters:
index
- operation to perform- Returns:
Engine.IndexResult
containing updated translog location, version and document specific failures Note: engine level failures (i.e. persistent engine failures) are thrown- Throws:
IOException
-
planIndexingAsNonPrimary
protected final InternalEngine.IndexingStrategy planIndexingAsNonPrimary(Engine.Index index) throws IOException - Throws:
IOException
-
indexingStrategyForOperation
protected InternalEngine.IndexingStrategy indexingStrategyForOperation(Engine.Index index) throws IOException - Throws:
IOException
-
delete
Description copied from class:Engine
Perform document delete operation on the engine- Specified by:
delete
in classEngine
- Parameters:
delete
- operation to perform- Returns:
Engine.DeleteResult
containing updated translog location, version and document specific failures Note: engine level failures (i.e. persistent engine failures) are thrown- Throws:
IOException
-
deletionStrategyForOperation
protected InternalEngine.DeletionStrategy deletionStrategyForOperation(Engine.Delete delete) throws IOException - Throws:
IOException
-
planDeletionAsNonPrimary
protected final InternalEngine.DeletionStrategy planDeletionAsNonPrimary(Engine.Delete delete) throws IOException - Throws:
IOException
-
assertNonPrimaryOrigin
-
maybePruneDeletes
public void maybePruneDeletes()Description copied from class:Engine
Tries to prune buffered deletes from the version map.- Specified by:
maybePruneDeletes
in classEngine
-
noOp
- Specified by:
noOp
in classEngine
- Throws:
IOException
-
preFlightCheckForNoOp
Executes a pre-flight check for a given NoOp. If this method returns a non-empty result, the engine won't process this NoOp and returns a failure.- Throws:
IOException
-
refresh
Description copied from class:Engine
Synchronously refreshes the engine for new search operations to reflect the latest changes.- Specified by:
refresh
in classEngine
- Throws:
EngineException
-
maybeRefresh
public void maybeRefresh(String source, ActionListener<Engine.RefreshResult> listener) throws EngineException Description copied from class:Engine
Asynchronously refreshes the engine for new search operations to reflect the latest changes unless another thread is already refreshing the engine concurrently.- Specified by:
maybeRefresh
in classEngine
- Throws:
EngineException
-
refreshInternalSearcher
protected Engine.RefreshResult refreshInternalSearcher(String source, boolean block) throws EngineException - Throws:
EngineException
-
refresh
protected final Engine.RefreshResult refresh(String source, Engine.SearcherScope scope, boolean block) throws EngineException - Throws:
EngineException
-
writeIndexingBuffer
Description copied from class:Engine
Called when our engine is using too much heap and should move buffered indexed/deleted documents to disk.- Specified by:
writeIndexingBuffer
in classEngine
- Throws:
IOException
-
shouldPeriodicallyFlush
public boolean shouldPeriodicallyFlush()Description copied from class:Engine
Checks if this engine should be flushed periodically. This check is mainly based on the uncommitted translog size and the translog flush threshold setting.- Specified by:
shouldPeriodicallyFlush
in classEngine
-
flush
public void flush(boolean force, boolean waitIfOngoing, ActionListener<Engine.FlushResult> listener) throws EngineException Description copied from class:Engine
Flushes the state of the engine including the transaction log, clearing memory, and writing the documents in the Lucene index to disk. This method will synchronously flush on the calling thread. However, depending on engine implementation, full durability will not be guaranteed until the listener is triggered.- Specified by:
flush
in classEngine
- Parameters:
force
- iftrue
a lucene commit is executed even if no changes need to be committed.waitIfOngoing
- iftrue
this call will block until all currently running flushes have finished. Otherwise this call will return without blocking.listener
- to notify after fully durability has been achieved. ifwaitIfOngoing==false
and ongoing request is detected, no flush will have occurred and the listener will be completed with a marker indicating no flush and unknown generation.- Throws:
EngineException
-
hasUncommittedChanges
protected boolean hasUncommittedChanges() -
rollTranslogGeneration
Description copied from class:Engine
Rolls the translog generation and cleans unneeded.- Specified by:
rollTranslogGeneration
in classEngine
- Throws:
EngineException
-
trimUnreferencedTranslogFiles
Description copied from class:Engine
checks and removes translog files that no longer need to be retained. SeeTranslogDeletionPolicy
for details- Specified by:
trimUnreferencedTranslogFiles
in classEngine
- Throws:
EngineException
-
shouldRollTranslogGeneration
public boolean shouldRollTranslogGeneration()Description copied from class:Engine
Tests whether or not the translog generation should be rolled to a new generation. This test is based on the size of the current generation compared to the configured generation threshold size.- Specified by:
shouldRollTranslogGeneration
in classEngine
- Returns:
true
if the current generation should be rolled to a new generation
-
trimOperationsFromTranslog
Description copied from class:Engine
Trims translog for terms belowbelowTerm
and seq# aboveaboveSeqNo
- Specified by:
trimOperationsFromTranslog
in classEngine
- Throws:
EngineException
- See Also:
-
forceMerge
public void forceMerge(boolean flush, int maxNumSegments, boolean onlyExpungeDeletes, String forceMergeUUID) throws EngineException, IOException Description copied from class:Engine
Triggers a forced merge on this engine- Specified by:
forceMerge
in classEngine
- Throws:
EngineException
IOException
-
acquireLastIndexCommit
Description copied from class:Engine
Snapshots the most recent index and returns a handle to it. If needed will try and "commit" the lucene index to make sure we have a "fresh" copy of the files to snapshot.- Specified by:
acquireLastIndexCommit
in classEngine
- Parameters:
flushFirst
- indicates whether the engine should flush before returning the snapshot- Throws:
EngineException
-
acquireSafeIndexCommit
Description copied from class:Engine
Snapshots the most recent safe index commit from the engine.- Specified by:
acquireSafeIndexCommit
in classEngine
- Throws:
EngineException
-
getSafeCommitInfo
- Specified by:
getSafeCommitInfo
in classEngine
- Returns:
- a summary of the contents of the current safe commit
-
maybeFailEngine
Description copied from class:Engine
Check whether the engine should be failed- Overrides:
maybeFailEngine
in classEngine
-
getLastCommittedSegmentInfos
public org.apache.lucene.index.SegmentInfos getLastCommittedSegmentInfos()- Specified by:
getLastCommittedSegmentInfos
in classEngine
-
writerSegmentStats
- Overrides:
writerSegmentStats
in classEngine
-
getIndexBufferRAMBytesUsed
public long getIndexBufferRAMBytesUsed()Description copied from class:Engine
How much heap is used that would be freed by a refresh. Note that this may throwAlreadyClosedException
.- Specified by:
getIndexBufferRAMBytesUsed
in classEngine
-
segments
Description copied from class:Engine
The list of segments in the engine. -
closeNoLock
Closes the engine without acquiring the write lock. This should only be called while the write lock is hold or in a disaster condition ie. if the engine is failed.- Specified by:
closeNoLock
in classEngine
-
getReferenceManager
protected final org.apache.lucene.search.ReferenceManager<ElasticsearchDirectoryReader> getReferenceManager(Engine.SearcherScope scope) - Specified by:
getReferenceManager
in classEngine
-
activateThrottling
public void activateThrottling()Description copied from class:Engine
Request that this engine throttle incoming indexing requests to one thread. Must be matched by a later call toEngine.deactivateThrottling()
.- Specified by:
activateThrottling
in classEngine
-
deactivateThrottling
public void deactivateThrottling()Description copied from class:Engine
Reverses a previousEngine.activateThrottling()
call.- Specified by:
deactivateThrottling
in classEngine
-
isThrottled
public boolean isThrottled()Description copied from class:Engine
Returns thetrue
iff this engine is currently under index throttling.- Specified by:
isThrottled
in classEngine
- See Also:
-
getIndexThrottleTimeInMillis
public long getIndexThrottleTimeInMillis()Description copied from class:Engine
Returns the number of milliseconds this engine was under index throttling.- Specified by:
getIndexThrottleTimeInMillis
in classEngine
-
commitIndexWriter
protected void commitIndexWriter(org.apache.lucene.index.IndexWriter writer, Translog translog) throws IOException Commits the specified index writer.- Parameters:
writer
- the index writer to committranslog
- the translog- Throws:
IOException
-
getCommitExtraUserData
Allows InternalEngine extenders to return custom key-value pairs which will be included in the Lucene commit user-data. Custom user data keys can be overwritten by if their keys conflict keys used by InternalEngine. -
onSettingsChanged
public void onSettingsChanged()- Overrides:
onSettingsChanged
in classEngine
-
getMergeStats
- Overrides:
getMergeStats
in classEngine
-
getLastSyncedGlobalCheckpoint
public long getLastSyncedGlobalCheckpoint()Description copied from class:Engine
Returns the latest global checkpoint value that has been persisted in the underlying storage (i.e. translog's checkpoint)- Specified by:
getLastSyncedGlobalCheckpoint
in classEngine
-
getMaxSeqNo
public long getMaxSeqNo()- Specified by:
getMaxSeqNo
in classEngine
- Returns:
- the max issued or seen seqNo for this Engine
-
getProcessedLocalCheckpoint
public long getProcessedLocalCheckpoint()- Specified by:
getProcessedLocalCheckpoint
in classEngine
- Returns:
- the processed local checkpoint for this Engine
-
getPersistedLocalCheckpoint
public long getPersistedLocalCheckpoint()- Specified by:
getPersistedLocalCheckpoint
in classEngine
- Returns:
- the persisted local checkpoint for this Engine
-
markSeqNoAsSeen
protected final void markSeqNoAsSeen(long seqNo) Marks the given seq_no as seen and advances the max_seq_no of this engine to at least that value. -
hasBeenProcessedBefore
Checks if the given operation has been processed in this engine or not.- Returns:
- true if the given operation was processed; otherwise false.
-
getSeqNoStats
- Specified by:
getSeqNoStats
in classEngine
- Returns:
- a
SeqNoStats
object, using local state and the supplied global checkpoint
-
countChanges
Description copied from class:Engine
Counts the number of operations in the range of the given sequence numbers.- Specified by:
countChanges
in classEngine
- Parameters:
source
- the source of the requestfromSeqNo
- the start sequence number (inclusive)toSeqNo
- the end sequence number (inclusive)- Throws:
IOException
- See Also:
-
newChangesSnapshot
public Translog.Snapshot newChangesSnapshot(String source, long fromSeqNo, long toSeqNo, boolean requiredFullRange, boolean singleConsumer, boolean accessStats) throws IOException Description copied from class:Engine
Creates a new history snapshot from Lucene for reading operations whose seqno in the requesting seqno range (both inclusive). This feature requires soft-deletes enabled. If soft-deletes are disabled, this method will throw anIllegalStateException
.- Specified by:
newChangesSnapshot
in classEngine
- Throws:
IOException
-
hasCompleteOperationHistory
Description copied from class:Engine
Checks if this engine has every operations sincestartingSeqNo
(inclusive) in its history (either Lucene or translog)- Specified by:
hasCompleteOperationHistory
in classEngine
-
getMinRetainedSeqNo
public final long getMinRetainedSeqNo()Returns the minimum seqno that is retained in the Lucene index. Operations whose seq# are at least this value should exist in the Lucene index.- Specified by:
getMinRetainedSeqNo
in classEngine
- Returns:
- the minimum retained sequence number
-
acquireHistoryRetentionLock
Description copied from class:Engine
Acquires a lock on the translog files and Lucene soft-deleted documents to prevent them from being trimmed- Specified by:
acquireHistoryRetentionLock
in classEngine
-
refreshIfNeeded
Refresh this engine **internally** iff the requesting seq_no is greater than the last refreshed checkpoint. -
getMaxSeenAutoIdTimestamp
public final long getMaxSeenAutoIdTimestamp()Description copied from class:Engine
Returns the maximum auto_id_timestamp of all append-only index requests have been processed by this engine or the auto_id_timestamp received from its primary shard viaEngine.updateMaxUnsafeAutoIdTimestamp(long)
. Notes this method returns the auto_id_timestamp of all append-only requests, not max_unsafe_auto_id_timestamp.- Overrides:
getMaxSeenAutoIdTimestamp
in classEngine
-
updateMaxUnsafeAutoIdTimestamp
public final void updateMaxUnsafeAutoIdTimestamp(long newTimestamp) Description copied from class:Engine
Forces this engine to advance its max_unsafe_auto_id_timestamp marker to at least the given timestamp. The engine will disable optimization for all append-only whose timestamp at mostnewTimestamp
.- Specified by:
updateMaxUnsafeAutoIdTimestamp
in classEngine
-
getMaxSeqNoOfUpdatesOrDeletes
public long getMaxSeqNoOfUpdatesOrDeletes()Description copied from class:Engine
Returns the maximum sequence number of either update or delete operations have been processed in this engine or the sequence number fromEngine.advanceMaxSeqNoOfUpdatesOrDeletes(long)
. An index request is considered as an update operation if it overwrites the existing documents in Lucene index with the same document id.A note on the optimization using max_seq_no_of_updates_or_deletes: For each operation O, the key invariants are:
- I1: There is no operation on docID(O) with seqno that is > MSU(O) and < seqno(O)
- I2: If MSU(O) < seqno(O) then docID(O) did not exist when O was applied; more precisely, if there is any O' with seqno(O') < seqno(O) and docID(O') = docID(O) then the one with the greatest seqno is a delete.
When a receiving shard (either a replica or a follower) receives an operation O, it must first ensure its own MSU at least MSU(O), and then compares its MSU to its local checkpoint (LCP). If LCP < MSU then there's a gap: there may be some operations that act on docID(O) about which we do not yet know, so we cannot perform an add. Note this also covers the case where a future operation O' with seqNo(O') > seqNo(O) and docId(O') = docID(O) is processed before O. In that case MSU(O') is at least seqno(O') and this means MSU >= seqNo(O') > seqNo(O) > LCP (because O wasn't processed yet).
However, if MSU <= LCP then there is no gap: we have processed every operation <= LCP, and no operation O' with seqno(O') > LCP and seqno(O') < seqno(O) also has docID(O') = docID(O), because such an operation would have seqno(O') > LCP >= MSU >= MSU(O) which contradicts the first invariant. Furthermore in this case we immediately know that docID(O) has been deleted (or never existed) without needing to check Lucene for the following reason. If there's no earlier operation on docID(O) then this is clear, so suppose instead that the preceding operation on docID(O) is O': 1. The first invariant above tells us that seqno(O') <= MSU(O) <= LCP so we have already applied O' to Lucene. 2. Also MSU(O) <= MSU <= LCP < seqno(O) (we discard O if seqno(O) <= LCP) so the second invariant applies, meaning that the O' was a delete.
Therefore, if MSU <= LCP < seqno(O) we know that O can safely be optimized with and added to lucene with addDocument. Moreover, operations that are optimized using the MSU optimization must not be processed twice as this will create duplicates in Lucene. To avoid this we check the local checkpoint tracker to see if an operation was already processed.
- Specified by:
getMaxSeqNoOfUpdatesOrDeletes
in classEngine
- See Also:
-
advanceMaxSeqNoOfUpdatesOrDeletes
public void advanceMaxSeqNoOfUpdatesOrDeletes(long maxSeqNoOfUpdatesOnPrimary) Description copied from class:Engine
A replica shard receives a new max_seq_no_of_updates from its primary shard, then calls this method to advance this marker to at least the given sequence number.- Specified by:
advanceMaxSeqNoOfUpdatesOrDeletes
in classEngine
-
getRawFieldRange
- Specified by:
getRawFieldRange
in classEngine
- Returns:
- a
ShardLongFieldRange
containing the min and max raw values of the given field for this shard if the engine guarantees these values never to change, orShardLongFieldRange.EMPTY
if this field is empty, orShardLongFieldRange.UNKNOWN
if this field's value range may change in future.
-
addFlushListener
- Overrides:
addFlushListener
in classEngine
-
waitForCommitDurability
-
getLastUnsafeSegmentGenerationForGets
public long getLastUnsafeSegmentGenerationForGets() -
createLiveVersionMapArchive
-
getLiveVersionMapArchive
-
getLiveVersionMap
-
getPreCommitSegmentGeneration
protected long getPreCommitSegmentGeneration()
-