Module org.elasticsearch.server
Package org.elasticsearch.index.engine

Class Engine

java.lang.Object
org.elasticsearch.index.engine.Engine
All Implemented Interfaces:
Closeable, AutoCloseable
Direct Known Subclasses:
InternalEngine, ReadOnlyEngine
public abstract class Engine extends Object implements Closeable

  • Field Details

  • Constructor Details

  • Method Details

    • readIndexVersion

      public static IndexVersion readIndexVersion(String esVersion)
      Reads an IndexVersion from an es_version metadata string

    • config

      public final EngineConfig config()

    • getLastCommittedSegmentInfos

      public abstract org.apache.lucene.index.SegmentInfos getLastCommittedSegmentInfos()

    • getMergeStats

      public MergeStats getMergeStats()

    • getHistoryUUID

      public abstract String getHistoryUUID()
      returns the history uuid for the engine

    • getWritingBytes

      public abstract long getWritingBytes()
      Returns how many bytes we are currently moving from heap to disk

    • completionStats

      public abstract CompletionStats completionStats(String... fieldNamePatterns)
      Returns the CompletionStats for this engine

    • docStats

      public DocsStats docStats()
      Returns the DocsStats for this engine

    • docsStats

      protected final DocsStats docsStats(org.apache.lucene.index.IndexReader indexReader)

    • denseVectorStats

      public DenseVectorStats denseVectorStats(MappingLookup mappingLookup)
      Returns the DenseVectorStats for this engine

    • denseVectorStats

      protected final DenseVectorStats denseVectorStats(org.apache.lucene.index.IndexReader indexReader, List<String> fields)

    • sparseVectorStats

      public SparseVectorStats sparseVectorStats(MappingLookup mappingLookup)
      Returns the SparseVectorStats for this engine

    • sparseVectorStats

      protected final SparseVectorStats sparseVectorStats(org.apache.lucene.index.IndexReader indexReader, List<org.apache.lucene.util.BytesRef> fields)

    • verifyEngineBeforeIndexClosing

      public void verifyEngineBeforeIndexClosing() throws IllegalStateException
      Performs the pre-closing checks on the Engine.
      Throws:
      IllegalStateException - if the sanity checks failed

    • getIndexThrottleTimeInMillis

      public abstract long getIndexThrottleTimeInMillis()
      Returns the number of milliseconds this engine was under index throttling.

    • isThrottled

      public abstract boolean isThrottled()
      Returns the true iff this engine is currently under index throttling.
      See Also:

    • trimOperationsFromTranslog

      public abstract void trimOperationsFromTranslog(long belowTerm, long aboveSeqNo) throws EngineException
      Trims translog for terms below belowTerm and seq# above aboveSeqNo
      Throws:
      EngineException
      See Also:

    • getTotalFlushTimeExcludingWaitingOnLockInMillis

      public long getTotalFlushTimeExcludingWaitingOnLockInMillis()
      Returns the total time flushes have been executed excluding waiting on locks.

    • index

      public abstract Engine.IndexResult index(Engine.Index index) throws IOException
      Perform document index operation on the engine
      Parameters:
      index - operation to perform
      Returns:
      Engine.IndexResult containing updated translog location, version and document specific failures Note: engine level failures (i.e. persistent engine failures) are thrown
      Throws:
      IOException

    • delete

      public abstract Engine.DeleteResult delete(Engine.Delete delete) throws IOException
      Perform document delete operation on the engine
      Parameters:
      delete - operation to perform
      Returns:
      Engine.DeleteResult containing updated translog location, version and document specific failures Note: engine level failures (i.e. persistent engine failures) are thrown
      Throws:
      IOException

    • noOp

      public abstract Engine.NoOpResult noOp(Engine.NoOp noOp) throws IOException
      Throws:
      IOException

    • getFromSearcher

      protected final Engine.GetResult getFromSearcher(Engine.Get get, Engine.Searcher searcher, boolean uncachedLookup) throws EngineException
      Throws:
      EngineException

    • get

      public abstract Engine.GetResult get(Engine.Get get, MappingLookup mappingLookup, DocumentParser documentParser, Function<Engine.Searcher,Engine.Searcher> searcherWrapper)

    • getFromTranslog

      public Engine.GetResult getFromTranslog(Engine.Get get, MappingLookup mappingLookup, DocumentParser documentParser, Function<Engine.Searcher,Engine.Searcher> searcherWrapper)
      Similar to get(org.elasticsearch.index.engine.Engine.Get, org.elasticsearch.index.mapper.MappingLookup, org.elasticsearch.index.mapper.DocumentParser, java.util.function.Function<org.elasticsearch.index.engine.Engine.Searcher, org.elasticsearch.index.engine.Engine.Searcher>), but it only attempts to serve the get from the translog. If not found in translog, it returns null, as Engine.GetResult.NOT_EXISTS could mean deletion.

    • acquireSearcherSupplier

      public final Engine.SearcherSupplier acquireSearcherSupplier(Function<Engine.Searcher,Engine.Searcher> wrapper) throws EngineException
      Acquires a point-in-time reader that can be used to create Engine.Searchers on demand.
      Throws:
      EngineException

    • acquireSearcherSupplier

      public Engine.SearcherSupplier acquireSearcherSupplier(Function<Engine.Searcher,Engine.Searcher> wrapper, Engine.SearcherScope scope) throws EngineException
      Acquires a point-in-time reader that can be used to create Engine.Searchers on demand.
      Throws:
      EngineException

    • acquireSearcher

      public final Engine.Searcher acquireSearcher(String source) throws EngineException
      Throws:
      EngineException

    • acquireSearcher

      public Engine.Searcher acquireSearcher(String source, Engine.SearcherScope scope) throws EngineException
      Throws:
      EngineException

    • acquireSearcher

      public Engine.Searcher acquireSearcher(String source, Engine.SearcherScope scope, Function<Engine.Searcher,Engine.Searcher> wrapper) throws EngineException
      Throws:
      EngineException

    • getReferenceManager

      protected abstract org.apache.lucene.search.ReferenceManager<ElasticsearchDirectoryReader> getReferenceManager(Engine.SearcherScope scope)

    • isTranslogSyncNeeded

      public abstract boolean isTranslogSyncNeeded()
      Checks if the underlying storage sync is required.

    • allowSearchIdleOptimization

      public boolean allowSearchIdleOptimization()
      Whether search idleness may be allowed to be considered for skipping a scheduled refresh.

    • asyncEnsureTranslogSynced

      public abstract void asyncEnsureTranslogSynced(Translog.Location location, Consumer<Exception> listener)
      Ensures that the location has been written to the underlying storage.

    • asyncEnsureGlobalCheckpointSynced

      public abstract void asyncEnsureGlobalCheckpointSynced(long globalCheckpoint, Consumer<Exception> listener)
      Ensures that the global checkpoint has been persisted to the underlying storage.

    • syncTranslog

      public abstract void syncTranslog() throws IOException
      Throws:
      IOException

    • acquireHistoryRetentionLock

      public abstract Closeable acquireHistoryRetentionLock()
      Acquires a lock on the translog files and Lucene soft-deleted documents to prevent them from being trimmed

    • countChanges

      public abstract int countChanges(String source, long fromSeqNo, long toSeqNo) throws IOException
      Counts the number of operations in the range of the given sequence numbers.
      Parameters:
      source - the source of the request
      fromSeqNo - the start sequence number (inclusive)
      toSeqNo - the end sequence number (inclusive)
      Throws:
      IOException
      See Also:

    • newChangesSnapshot

      public abstract Translog.Snapshot newChangesSnapshot(String source, long fromSeqNo, long toSeqNo, boolean requiredFullRange, boolean singleConsumer, boolean accessStats, long maxChunkSize) throws IOException
      Creates a new history snapshot from Lucene for reading operations whose seqno in the requesting seqno range (both inclusive). This feature requires soft-deletes enabled. If soft-deletes are disabled, this method will throw an IllegalStateException.
      Throws:
      IOException

    • hasCompleteOperationHistory

      public abstract boolean hasCompleteOperationHistory(String reason, long startingSeqNo)
      Checks if this engine has every operations since startingSeqNo(inclusive) in its history (either Lucene or translog)

    • getMinRetainedSeqNo

      public abstract long getMinRetainedSeqNo()
      Gets the minimum retained sequence number for this engine.
      Returns:
      the minimum retained sequence number

    • getTranslogStats

      public abstract TranslogStats getTranslogStats()

    • getTranslogLastWriteLocation

      public abstract Translog.Location getTranslogLastWriteLocation()
      Returns the last location that the translog of this engine has written into.

    • ensureOpen

      protected final void ensureOpen(Exception suppressed)

    • ensureOpen

      protected final void ensureOpen()

    • commitStats

      public final CommitStats commitStats()
      get commits stats for the last commit

    • getMaxSeqNo

      public abstract long getMaxSeqNo()
      Returns:
      the max issued or seen seqNo for this Engine

    • getProcessedLocalCheckpoint

      public abstract long getProcessedLocalCheckpoint()
      Returns:
      the processed local checkpoint for this Engine

    • getPersistedLocalCheckpoint

      public abstract long getPersistedLocalCheckpoint()
      Returns:
      the persisted local checkpoint for this Engine

    • getSeqNoStats

      public abstract SeqNoStats getSeqNoStats(long globalCheckpoint)
      Returns:
      a SeqNoStats object, using local state and the supplied global checkpoint

    • getLastSyncedGlobalCheckpoint

      public abstract long getLastSyncedGlobalCheckpoint()
      Returns the latest global checkpoint value that has been persisted in the underlying storage (i.e. translog's checkpoint)

    • segmentsStats

      public SegmentsStats segmentsStats(boolean includeSegmentFileSizes, boolean includeUnloadedSegments)
      Global stats on segments.

    • fillSegmentStats

      protected void fillSegmentStats(org.apache.lucene.index.SegmentReader segmentReader, boolean includeSegmentFileSizes, SegmentsStats stats)

    • writerSegmentStats

      protected void writerSegmentStats(SegmentsStats stats)

    • getIndexBufferRAMBytesUsed

      public abstract long getIndexBufferRAMBytesUsed()
      How much heap is used that would be freed by a refresh. This includes both the current memory being freed and any remaining memory usage that could be freed, e.g., by refreshing. Note that this may throw AlreadyClosedException.

    • segments

      public abstract List<Segment> segments()
      The list of segments in the engine.

    • segments

      public abstract List<Segment> segments(boolean includeVectorFormatsInfo)

    • refreshNeeded

      public boolean refreshNeeded()

    • refresh

      @Nullable public abstract Engine.RefreshResult refresh(String source) throws EngineException
      Synchronously refreshes the engine for new search operations to reflect the latest changes.
      Throws:
      EngineException

    • externalRefresh

      public void externalRefresh(String source, ActionListener<Engine.RefreshResult> listener)
      An async variant of refresh(String) that may apply some rate-limiting.

    • maybeRefresh

      @Nullable public abstract void maybeRefresh(String source, ActionListener<Engine.RefreshResult> listener) throws EngineException
      Asynchronously refreshes the engine for new search operations to reflect the latest changes unless another thread is already refreshing the engine concurrently.
      Throws:
      EngineException

    • writeIndexingBuffer

      public abstract void writeIndexingBuffer() throws IOException
      Called when our engine is using too much heap and should move buffered indexed/deleted documents to disk.
      Throws:
      IOException

    • shouldPeriodicallyFlush

      public abstract boolean shouldPeriodicallyFlush()
      Checks if this engine should be flushed periodically. This check is mainly based on the uncommitted translog size and the translog flush threshold setting.

    • flush

      public void flush(boolean force, boolean waitIfOngoing) throws EngineException
      A Blocking helper method for calling the async flush method.
      Throws:
      EngineException

    • flush

      public final void flush(boolean force, boolean waitIfOngoing, ActionListener<Engine.FlushResult> listener) throws EngineException
      Flushes the state of the engine including the transaction log, clearing memory, and writing the documents in the Lucene index to disk. This method will synchronously flush on the calling thread. However, depending on engine implementation, full durability will not be guaranteed until the listener is triggered.
      Parameters:
      force - if true a lucene commit is executed even if no changes need to be committed.
      waitIfOngoing - if true this call will block until all currently running flushes have finished. Otherwise this call will return without blocking.
      listener - to notify after fully durability has been achieved. if waitIfOngoing==false and ongoing request is detected, no flush will have occurred and the listener will be completed with a marker indicating no flush and unknown generation.
      Throws:
      EngineException

    • flushHoldingLock

      protected abstract void flushHoldingLock(boolean force, boolean waitIfOngoing, ActionListener<Engine.FlushResult> listener) throws EngineException
      The actual implementation of flush(boolean, boolean, ActionListener), to be called either when holding a ref that ensures the engine remains open, or holding IndexShard#engineMutex while closing the engine.
      Throws:
      EngineException

    • flush

      public final void flush() throws EngineException
      Flushes the state of the engine including the transaction log, clearing memory and persisting documents in the lucene index to disk including a potentially heavy and durable fsync operation. This operation is not going to block if another flush operation is currently running and won't write a lucene commit if nothing needs to be committed.
      Throws:
      EngineException

    • trimUnreferencedTranslogFiles

      public abstract void trimUnreferencedTranslogFiles() throws EngineException
      checks and removes translog files that no longer need to be retained. See TranslogDeletionPolicy for details
      Throws:
      EngineException

    • shouldRollTranslogGeneration

      public abstract boolean shouldRollTranslogGeneration()
      Tests whether or not the translog generation should be rolled to a new generation. This test is based on the size of the current generation compared to the configured generation threshold size.
      Returns:
      true if the current generation should be rolled to a new generation

    • rollTranslogGeneration

      public abstract void rollTranslogGeneration() throws EngineException
      Rolls the translog generation and cleans unneeded.
      Throws:
      EngineException

    • forceMerge

      public abstract void forceMerge(boolean flush, int maxNumSegments, boolean onlyExpungeDeletes, String forceMergeUUID) throws EngineException, IOException
      Triggers a forced merge on this engine
      Throws:
      EngineException
      IOException

    • acquireLastIndexCommit

      public abstract Engine.IndexCommitRef acquireLastIndexCommit(boolean flushFirst) throws EngineException
      Snapshots the most recent index and returns a handle to it. If needed will try and "commit" the lucene index to make sure we have a "fresh" copy of the files to snapshot.
      Parameters:
      flushFirst - indicates whether the engine should flush before returning the snapshot
      Throws:
      EngineException

    • acquireSafeIndexCommit

      public abstract Engine.IndexCommitRef acquireSafeIndexCommit() throws EngineException
      Snapshots the most recent safe index commit from the engine.
      Throws:
      EngineException

    • acquireIndexCommitForSnapshot

      public final Engine.IndexCommitRef acquireIndexCommitForSnapshot() throws EngineException
      Acquires the index commit that should be included in a snapshot.
      Throws:
      EngineException

    • getSafeCommitInfo

      public abstract SafeCommitInfo getSafeCommitInfo()
      Returns:
      a summary of the contents of the current safe commit

    • failEngine

      public void failEngine(String reason, @Nullable Exception failure)
      fail engine due to some error. the engine will also be closed. The underlying store is marked corrupted iff failure is caused by index corruption

    • maybeFailEngine

      protected boolean maybeFailEngine(String source, Exception e)
      Check whether the engine should be failed

    • closeNoLock

      protected abstract void closeNoLock(String reason, CountDownLatch closedLatch)
      Closes the engine without acquiring any refs or locks. The caller should either have changed isClosing from false to true or else must hold the failEngineLock. The implementation must decrement the supplied latch when done.

    • isDrainedForClose

      protected final boolean isDrainedForClose()

    • isClosing

      protected final boolean isClosing()

    • acquireEnsureOpenRef

      protected final Releasable acquireEnsureOpenRef()

    • flushAndClose

      public void flushAndClose() throws IOException
      Flush the engine (committing segments to disk and truncating the translog) and close it.
      Throws:
      IOException

    • close

      public void close() throws IOException
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable
      Throws:
      IOException

    • onSettingsChanged

      public void onSettingsChanged()

    • getLastWriteNanos

      public long getLastWriteNanos()
      Returns the timestamp of the last write in nanoseconds. Note: this time might not be absolutely accurate since the Engine.Operation.startTime() is used which might be slightly inaccurate.
      See Also:

    • activateThrottling

      public abstract void activateThrottling()
      Request that this engine throttle incoming indexing requests to one thread. Must be matched by a later call to deactivateThrottling().

    • deactivateThrottling

      public abstract void deactivateThrottling()
      Reverses a previous activateThrottling() call.

    • restoreLocalHistoryFromTranslog

      public abstract int restoreLocalHistoryFromTranslog(Engine.TranslogRecoveryRunner translogRecoveryRunner) throws IOException
      This method replays translog to restore the Lucene index which might be reverted previously. This ensures that all acknowledged writes are restored correctly when this engine is promoted.
      Returns:
      the number of translog operations have been recovered
      Throws:
      IOException

    • fillSeqNoGaps

      public abstract int fillSeqNoGaps(long primaryTerm) throws IOException
      Fills up the local checkpoints history with no-ops until the local checkpoint and the max seen sequence ID are identical.
      Parameters:
      primaryTerm - the shards primary term this engine was created for
      Returns:
      the number of no-ops added
      Throws:
      IOException

    • recoverFromTranslog

      public final void recoverFromTranslog(Engine.TranslogRecoveryRunner translogRecoveryRunner, long recoverUpToSeqNo) throws IOException
      Performs recovery from the transaction log up to recoverUpToSeqNo (inclusive). This operation will close the engine if the recovery fails. Use EngineTestCase#recoverFromTranslog for test usages
      Parameters:
      translogRecoveryRunner - the translog recovery runner
      recoverUpToSeqNo - the upper bound, inclusive, of sequence number to be recovered
      Throws:
      IOException

    • recoverFromTranslog

      public abstract void recoverFromTranslog(Engine.TranslogRecoveryRunner translogRecoveryRunner, long recoverUpToSeqNo, ActionListener<Void> listener)
      Performs recovery from the transaction log up to recoverUpToSeqNo (inclusive). This operation will close the engine if the recovery fails.
      Parameters:
      translogRecoveryRunner - the translog recovery runner
      recoverUpToSeqNo - the upper bound, inclusive, of sequence number to be recovered
      listener - listener notified on completion of the recovery, whether successful or otherwise

    • skipTranslogRecovery

      public abstract void skipTranslogRecovery()
      Do not replay translog operations, but make the engine be ready.

    • maybePruneDeletes

      public abstract void maybePruneDeletes()
      Tries to prune buffered deletes from the version map.

    • getMaxSeenAutoIdTimestamp

      public long getMaxSeenAutoIdTimestamp()
      Returns the maximum auto_id_timestamp of all append-only index requests have been processed by this engine or the auto_id_timestamp received from its primary shard via updateMaxUnsafeAutoIdTimestamp(long). Notes this method returns the auto_id_timestamp of all append-only requests, not max_unsafe_auto_id_timestamp.

    • updateMaxUnsafeAutoIdTimestamp

      public abstract void updateMaxUnsafeAutoIdTimestamp(long newTimestamp)
      Forces this engine to advance its max_unsafe_auto_id_timestamp marker to at least the given timestamp. The engine will disable optimization for all append-only whose timestamp at most newTimestamp.

    • getMaxSeqNoOfUpdatesOrDeletes

      public abstract long getMaxSeqNoOfUpdatesOrDeletes()
      Returns the maximum sequence number of either update or delete operations have been processed in this engine or the sequence number from advanceMaxSeqNoOfUpdatesOrDeletes(long). An index request is considered as an update operation if it overwrites the existing documents in Lucene index with the same document id.

      A note on the optimization using max_seq_no_of_updates_or_deletes: For each operation O, the key invariants are:

      1. I1: There is no operation on docID(O) with seqno that is > MSU(O) and < seqno(O)
      2. I2: If MSU(O) < seqno(O) then docID(O) did not exist when O was applied; more precisely, if there is any O' with seqno(O') < seqno(O) and docID(O') = docID(O) then the one with the greatest seqno is a delete.

      When a receiving shard (either a replica or a follower) receives an operation O, it must first ensure its own MSU at least MSU(O), and then compares its MSU to its local checkpoint (LCP). If LCP < MSU then there's a gap: there may be some operations that act on docID(O) about which we do not yet know, so we cannot perform an add. Note this also covers the case where a future operation O' with seqNo(O') > seqNo(O) and docId(O') = docID(O) is processed before O. In that case MSU(O') is at least seqno(O') and this means MSU >= seqNo(O') > seqNo(O) > LCP (because O wasn't processed yet).

      However, if MSU <= LCP then there is no gap: we have processed every operation <= LCP, and no operation O' with seqno(O') > LCP and seqno(O') < seqno(O) also has docID(O') = docID(O), because such an operation would have seqno(O') > LCP >= MSU >= MSU(O) which contradicts the first invariant. Furthermore in this case we immediately know that docID(O) has been deleted (or never existed) without needing to check Lucene for the following reason. If there's no earlier operation on docID(O) then this is clear, so suppose instead that the preceding operation on docID(O) is O': 1. The first invariant above tells us that seqno(O') <= MSU(O) <= LCP so we have already applied O' to Lucene. 2. Also MSU(O) <= MSU <= LCP < seqno(O) (we discard O if seqno(O) <= LCP) so the second invariant applies, meaning that the O' was a delete.

      Therefore, if MSU <= LCP < seqno(O) we know that O can safely be optimized with and added to lucene with addDocument. Moreover, operations that are optimized using the MSU optimization must not be processed twice as this will create duplicates in Lucene. To avoid this we check the local checkpoint tracker to see if an operation was already processed.

      See Also:

    • advanceMaxSeqNoOfUpdatesOrDeletes

      public abstract void advanceMaxSeqNoOfUpdatesOrDeletes(long maxSeqNoOfUpdatesOnPrimary)
      A replica shard receives a new max_seq_no_of_updates from its primary shard, then calls this method to advance this marker to at least the given sequence number.

    • getRawFieldRange

      public abstract ShardLongFieldRange getRawFieldRange(String field) throws IOException
      Returns:
      a ShardLongFieldRange containing the min and max raw values of the given field for this shard if the engine guarantees these values never to change, or ShardLongFieldRange.EMPTY if this field is empty, or ShardLongFieldRange.UNKNOWN if this field's value range may change in future.
      Throws:
      IOException

    • getEngineConfig

      public final EngineConfig getEngineConfig()

    • addSegmentGenerationListener

      @Deprecated public void addSegmentGenerationListener(long minGeneration, ActionListener<Long> listener)
      Deprecated.
      use addPrimaryTermAndGenerationListener(long, long, ActionListener) instead.
      Allows registering a listener for when the index shard is on a segment generation >= minGeneration.

    • addPrimaryTermAndGenerationListener

      public void addPrimaryTermAndGenerationListener(long minPrimaryTerm, long minGeneration, ActionListener<Long> listener)
      Allows registering a listener for when the index shard is on a primary term >= minPrimaryTerm and a segment generation >= minGeneration.

    • addFlushListener

      public void addFlushListener(Translog.Location location, ActionListener<Long> listener)