Package org.elasticsearch.index.engine

Class Engine

  • All Implemented Interfaces:
    java.io.Closeable, java.lang.AutoCloseable
    Direct Known Subclasses:
    InternalEngine
    public abstract class Engine
extends java.lang.Object
implements java.io.Closeable

    • Field Detail

      • HISTORY_UUID_KEY

        public static final java.lang.String HISTORY_UUID_KEY
        See Also:
        Constant Field Values

      • shardId

        protected final ShardId shardId

      • allocationId

        protected final java.lang.String allocationId

      • logger

        protected final org.apache.logging.log4j.Logger logger

      • store

        protected final Store store

      • isClosed

        protected final java.util.concurrent.atomic.AtomicBoolean isClosed

      • failEngineLock

        protected final java.util.concurrent.locks.ReentrantLock failEngineLock

      • rwl

        protected final java.util.concurrent.locks.ReentrantReadWriteLock rwl

      • failedEngine

        protected final org.apache.lucene.util.SetOnce<java.lang.Exception> failedEngine

      • lastWriteNanos

        protected volatile long lastWriteNanos

    • Constructor Detail

    • Method Detail

      • guardedRamBytesUsed

        protected static long guardedRamBytesUsed​(org.apache.lucene.util.Accountable a)
        Returns 0 in the case where accountable is null, otherwise returns ramBytesUsed()

      • isMergedSegment

        protected static boolean isMergedSegment​(org.apache.lucene.index.LeafReader reader)
        Returns whether a leaf reader comes from a merge (versus flush or addIndexes).

      • getLastCommittedSegmentInfos

        protected abstract org.apache.lucene.index.SegmentInfos getLastCommittedSegmentInfos()

      • getMergeStats

        public MergeStats getMergeStats()

      • getHistoryUUID

        public abstract java.lang.String getHistoryUUID()
        returns the history uuid for the engine

      • getWritingBytes

        public abstract long getWritingBytes()
        Returns how many bytes we are currently moving from heap to disk

      • getIndexThrottleTimeInMillis

        public abstract long getIndexThrottleTimeInMillis()
        Returns the number of milliseconds this engine was under index throttling.

      • isThrottled

        public abstract boolean isThrottled()
        Returns the true iff this engine is currently under index throttling.
        See Also:
        getIndexThrottleTimeInMillis()

      • index

        public abstract Engine.IndexResult index​(Engine.Index index)
                                  throws java.io.IOException
        Perform document index operation on the engine
        Parameters:
        index - operation to perform
        Returns:
        Engine.IndexResult containing updated translog location, version and document specific failures Note: engine level failures (i.e. persistent engine failures) are thrown
        Throws:
        java.io.IOException

      • delete

        public abstract Engine.DeleteResult delete​(Engine.Delete delete)
                                    throws java.io.IOException
        Perform document delete operation on the engine
        Parameters:
        delete - operation to perform
        Returns:
        Engine.DeleteResult containing updated translog location, version and document specific failures Note: engine level failures (i.e. persistent engine failures) are thrown
        Throws:
        java.io.IOException

      • syncFlush

        public abstract Engine.SyncedFlushResult syncFlush​(java.lang.String syncId,
                                                   Engine.CommitId expectedCommitId)
                                            throws EngineException
        Attempts to do a special commit where the given syncID is put into the commit data. The attempt succeeds if there are not pending writes in lucene and the current point is equal to the expected one.
        Parameters:
        syncId - id of this sync
        expectedCommitId - the expected value of
        Returns:
        true if the sync commit was made, false o.w.
        Throws:
        EngineException

      • acquireSearcher

        public final Engine.Searcher acquireSearcher​(java.lang.String source)
                                      throws EngineException
        Returns a new searcher instance. The consumer of this API is responsible for releasing the returned searcher in a safe manner, preferably in a try/finally block.
        Parameters:
        source - the source API or routing that triggers this searcher acquire
        Throws:
        EngineException
        See Also:
        Engine.Searcher.close()

      • acquireSearcher

        public abstract Engine.Searcher acquireSearcher​(java.lang.String source,
                                                Engine.SearcherScope scope)
                                         throws EngineException
        Returns a new searcher instance. The consumer of this API is responsible for releasing the returned searcher in a safe manner, preferably in a try/finally block.
        Parameters:
        source - the source API or routing that triggers this searcher acquire
        scope - the scope of this searcher ie. if the searcher will be used for get or search purposes
        Throws:
        EngineException
        See Also:
        Engine.Searcher.close()

      • isTranslogSyncNeeded

        public abstract boolean isTranslogSyncNeeded()
        Checks if the underlying storage sync is required.

      • ensureTranslogSynced

        public abstract boolean ensureTranslogSynced​(java.util.stream.Stream<Translog.Location> locations)
                                      throws java.io.IOException
        Ensures that all locations in the given stream have been written to the underlying storage.
        Throws:
        java.io.IOException

      • syncTranslog

        public abstract void syncTranslog()
                           throws java.io.IOException
        Throws:
        java.io.IOException

      • acquireTranslogRetentionLock

        public abstract java.io.Closeable acquireTranslogRetentionLock()

      • newTranslogSnapshotFromMinSeqNo

        public abstract Translog.Snapshot newTranslogSnapshotFromMinSeqNo​(long minSeqNo)
                                                           throws java.io.IOException
        Creates a new translog snapshot from this engine for reading translog operations whose seq# at least the provided seq#. The caller has to close the returned snapshot after finishing the reading.
        Throws:
        java.io.IOException

      • estimateTranslogOperationsFromMinSeq

        public abstract int estimateTranslogOperationsFromMinSeq​(long minSeqNo)
        Returns the estimated number of translog operations in this engine whose seq# at least the provided seq#.

      • getTranslogStats

        public abstract TranslogStats getTranslogStats()

      • getTranslogLastWriteLocation

        public abstract Translog.Location getTranslogLastWriteLocation()
        Returns the last location that the translog of this engine has written into.

      • ensureOpen

        protected final void ensureOpen​(java.lang.Exception suppressed)

      • ensureOpen

        protected final void ensureOpen()

      • commitStats

        public CommitStats commitStats()
        get commits stats for the last commit

      • getLocalCheckpoint

        public abstract long getLocalCheckpoint()
        Returns:
        the local checkpoint for this Engine

      • waitForOpsToComplete

        public abstract void waitForOpsToComplete​(long seqNo)
                                   throws java.lang.InterruptedException
        Waits for all operations up to the provided sequence number to complete.
        Parameters:
        seqNo - the sequence number that the checkpoint must advance to before this method returns
        Throws:
        java.lang.InterruptedException - if the thread was interrupted while blocking on the condition

      • resetLocalCheckpoint

        public abstract void resetLocalCheckpoint​(long localCheckpoint)
        Reset the local checkpoint in the tracker to the given local checkpoint
        Parameters:
        localCheckpoint - the new checkpoint to be set

      • getSeqNoStats

        public abstract SeqNoStats getSeqNoStats​(long globalCheckpoint)
        Returns:
        a SeqNoStats object, using local state and the supplied global checkpoint

      • getLastSyncedGlobalCheckpoint

        public abstract long getLastSyncedGlobalCheckpoint()
        Returns the latest global checkpoint value that has been persisted in the underlying storage (i.e. translog's checkpoint)

      • segmentsStats

        public final SegmentsStats segmentsStats​(boolean includeSegmentFileSizes)
        Global stats on segments.

      • writerSegmentStats

        protected void writerSegmentStats​(SegmentsStats stats)

      • getIndexBufferRAMBytesUsed

        public abstract long getIndexBufferRAMBytesUsed()
        How much heap is used that would be freed by a refresh. Note that this may throw AlreadyClosedException.

      • getSegmentInfo

        protected Segment[] getSegmentInfo​(org.apache.lucene.index.SegmentInfos lastCommittedSegmentInfos,
                                   boolean verbose)

      • segments

        public abstract java.util.List<Segment> segments​(boolean verbose)
        The list of segments in the engine.

      • refreshNeeded

        public final boolean refreshNeeded()

      • refresh

        @Nullable
public abstract void refresh​(java.lang.String source)
                      throws EngineException
        Synchronously refreshes the engine for new search operations to reflect the latest changes.
        Throws:
        EngineException

      • writeIndexingBuffer

        public abstract void writeIndexingBuffer()
                                  throws EngineException
        Called when our engine is using too much heap and should move buffered indexed/deleted documents to disk.
        Throws:
        EngineException

      • shouldPeriodicallyFlush

        public abstract boolean shouldPeriodicallyFlush()
        Checks if this engine should be flushed periodically. This check is mainly based on the uncommitted translog size and the translog flush threshold setting.

      • flush

        public abstract Engine.CommitId flush​(boolean force,
                                      boolean waitIfOngoing)
                               throws EngineException
        Flushes the state of the engine including the transaction log, clearing memory.
        Parameters:
        force - if true a lucene commit is executed even if no changes need to be committed.
        waitIfOngoing - if true this call will block until all currently running flushes have finished. Otherwise this call will return without blocking.
        Returns:
        the commit Id for the resulting commit
        Throws:
        EngineException

      • flush

        public abstract Engine.CommitId flush()
                               throws EngineException
        Flushes the state of the engine including the transaction log, clearing memory and persisting documents in the lucene index to disk including a potentially heavy and durable fsync operation. This operation is not going to block if another flush operation is currently running and won't write a lucene commit if nothing needs to be committed.
        Returns:
        the commit Id for the resulting commit
        Throws:
        EngineException

      • shouldRollTranslogGeneration

        public abstract boolean shouldRollTranslogGeneration()
        Tests whether or not the translog generation should be rolled to a new generation. This test is based on the size of the current generation compared to the configured generation threshold size.
        Returns:
        true if the current generation should be rolled to a new generation

      • rollTranslogGeneration

        public abstract void rollTranslogGeneration()
                                     throws EngineException
        Rolls the translog generation and cleans unneeded.
        Throws:
        EngineException

      • forceMerge

        public void forceMerge​(boolean flush)
                throws java.io.IOException
        Force merges to 1 segment
        Throws:
        java.io.IOException

      • forceMerge

        public abstract void forceMerge​(boolean flush,
                                int maxNumSegments,
                                boolean onlyExpungeDeletes,
                                boolean upgrade,
                                boolean upgradeOnlyAncientSegments)
                         throws EngineException,
                                java.io.IOException
        Triggers a forced merge on this engine
        Throws:
        EngineException
        java.io.IOException

      • acquireLastIndexCommit

        public abstract Engine.IndexCommitRef acquireLastIndexCommit​(boolean flushFirst)
                                                      throws EngineException
        Snapshots the most recent index and returns a handle to it. If needed will try and "commit" the lucene index to make sure we have a "fresh" copy of the files to snapshot.
        Parameters:
        flushFirst - indicates whether the engine should flush before returning the snapshot
        Throws:
        EngineException

      • failEngine

        public void failEngine​(java.lang.String reason,
                       @Nullable
                       java.lang.Exception failure)
        fail engine due to some error. the engine will also be closed. The underlying store is marked corrupted iff failure is caused by index corruption

      • maybeFailEngine

        protected boolean maybeFailEngine​(java.lang.String source,
                                  java.lang.Exception e)
        Check whether the engine should be failed

      • closeNoLock

        protected abstract void closeNoLock​(java.lang.String reason,
                                    java.util.concurrent.CountDownLatch closedLatch)
        Method to close the engine while the write lock is held. Must decrement the supplied when closing work is done and resources are freed.

      • flushAndClose

        public void flushAndClose()
                   throws java.io.IOException
        Flush the engine (committing segments to disk and truncating the translog) and close it.
        Throws:
        java.io.IOException

      • close

        public void close()
           throws java.io.IOException
        Specified by:
        close in interface java.lang.AutoCloseable
        Specified by:
        close in interface java.io.Closeable
        Throws:
        java.io.IOException

      • onSettingsChanged

        public void onSettingsChanged()

      • getLastWriteNanos

        public long getLastWriteNanos()
        Returns the timestamp of the last write in nanoseconds. Note: this time might not be absolutely accurate since the Engine.Operation.startTime() is used which might be slightly inaccurate.
        See Also:
        System.nanoTime(), Engine.Operation.startTime()

      • activateThrottling

        public abstract void activateThrottling()
        Request that this engine throttle incoming indexing requests to one thread. Must be matched by a later call to deactivateThrottling().

      • deactivateThrottling

        public abstract void deactivateThrottling()
        Reverses a previous activateThrottling() call.

      • restoreLocalCheckpointFromTranslog

        public abstract void restoreLocalCheckpointFromTranslog()
                                                 throws java.io.IOException
        Marks operations in the translog as completed. This is used to restore the state of the local checkpoint tracker on primary promotion.
        Throws:
        java.io.IOException - if an I/O exception occurred reading the translog

      • fillSeqNoGaps

        public abstract int fillSeqNoGaps​(long primaryTerm)
                           throws java.io.IOException
        Fills up the local checkpoints history with no-ops until the local checkpoint and the max seen sequence ID are identical.
        Parameters:
        primaryTerm - the shards primary term this engine was created for
        Returns:
        the number of no-ops added
        Throws:
        java.io.IOException

      • recoverFromTranslog

        public abstract Engine recoverFromTranslog()
                                    throws java.io.IOException
        Performs recovery from the transaction log. This operation will close the engine if the recovery fails.
        Throws:
        java.io.IOException

      • skipTranslogRecovery

        public abstract void skipTranslogRecovery()
        Do not replay translog operations, but make the engine be ready.

      • isRecovering

        public boolean isRecovering()
        Returns true iff this engine is currently recovering from translog.

      • maybePruneDeletes

        public abstract void maybePruneDeletes()
        Tries to prune buffered deletes from the version map.