Package org.elasticsearch.index.engine

Class ReadOnlyEngine

  • All Implemented Interfaces:
    java.io.Closeable, java.lang.AutoCloseable
    Direct Known Subclasses:
    NoOpEngine
    public class ReadOnlyEngine
extends Engine
    A basic read-only engine that allows switching a shard to be true read-only temporarily or permanently. Note: this engine can be opened side-by-side with a read-write engine but will not reflect any changes made to the read-write engine.
    See Also:
    ReadOnlyEngine(EngineConfig, SeqNoStats, TranslogStats, boolean, Function)

    • Field Detail

      • OFF_HEAP_READER_ATTRIBUTES

        public static final java.util.Map<java.lang.String,​java.lang.String> OFF_HEAP_READER_ATTRIBUTES
        Reader attributes used for read only engines. These attributes prevent loading term dictionaries on-heap even if the field is an ID field if we are reading form memory maps.

    • Constructor Detail

      • ReadOnlyEngine

        public ReadOnlyEngine​(EngineConfig config,
                      SeqNoStats seqNoStats,
                      TranslogStats translogStats,
                      boolean obtainLock,
                      java.util.function.Function<org.apache.lucene.index.DirectoryReader,​org.apache.lucene.index.DirectoryReader> readerWrapperFunction)
        Creates a new ReadOnlyEngine. This ctor can also be used to open a read-only engine on top of an already opened read-write engine. It allows to optionally obtain the writer locks for the shard which would time-out if another engine is still open.
        Parameters:
        config - the engine configuration
        seqNoStats - sequence number statistics for this engine or null if not provided
        translogStats - translog stats for this engine or null if not provided
        obtainLock - if true this engine will try to obtain the IndexWriter.WRITE_LOCK_NAME lock. Otherwise the lock won't be obtained
        readerWrapperFunction - allows to wrap the index-reader for this engine.

    • Method Detail

      • ensureMaxSeqNoEqualsToGlobalCheckpoint

        protected void ensureMaxSeqNoEqualsToGlobalCheckpoint​(SeqNoStats seqNoStats)

      • assertMaxSeqNoEqualsToGlobalCheckpoint

        protected boolean assertMaxSeqNoEqualsToGlobalCheckpoint​(long maxSeqNo,
                                                         long globalCheckpoint)

      • verifyEngineBeforeIndexClosing

        public void verifyEngineBeforeIndexClosing()
                                    throws java.lang.IllegalStateException
        Description copied from class: Engine
        Performs the pre-closing checks on the Engine.
        Overrides:
        verifyEngineBeforeIndexClosing in class Engine
        Throws:
        java.lang.IllegalStateException - if the sanity checks failed

      • wrapReader

        protected final ElasticsearchDirectoryReader wrapReader​(org.apache.lucene.index.DirectoryReader reader,
                                                        java.util.function.Function<org.apache.lucene.index.DirectoryReader,​org.apache.lucene.index.DirectoryReader> readerWrapperFunction)
                                                 throws java.io.IOException
        Throws:
        java.io.IOException

      • open

        protected org.apache.lucene.index.DirectoryReader open​(org.apache.lucene.index.IndexCommit commit)
                                                throws java.io.IOException
        Throws:
        java.io.IOException

      • closeNoLock

        protected void closeNoLock​(java.lang.String reason,
                           java.util.concurrent.CountDownLatch closedLatch)
        Description copied from class: Engine
        Method to close the engine while the write lock is held. Must decrement the supplied when closing work is done and resources are freed.
        Specified by:
        closeNoLock in class Engine

      • getLastCommittedSegmentInfos

        protected org.apache.lucene.index.SegmentInfos getLastCommittedSegmentInfos()
        Specified by:
        getLastCommittedSegmentInfos in class Engine

      • getHistoryUUID

        public java.lang.String getHistoryUUID()
        Description copied from class: Engine
        returns the history uuid for the engine
        Specified by:
        getHistoryUUID in class Engine

      • getWritingBytes

        public long getWritingBytes()
        Description copied from class: Engine
        Returns how many bytes we are currently moving from heap to disk
        Specified by:
        getWritingBytes in class Engine

      • getIndexThrottleTimeInMillis

        public long getIndexThrottleTimeInMillis()
        Description copied from class: Engine
        Returns the number of milliseconds this engine was under index throttling.
        Specified by:
        getIndexThrottleTimeInMillis in class Engine

      • index

        public Engine.IndexResult index​(Engine.Index index)
        Description copied from class: Engine
        Perform document index operation on the engine
        Specified by:
        index in class Engine
        Parameters:
        index - operation to perform
        Returns:
        Engine.IndexResult containing updated translog location, version and document specific failures Note: engine level failures (i.e. persistent engine failures) are thrown

      • delete

        public Engine.DeleteResult delete​(Engine.Delete delete)
        Description copied from class: Engine
        Perform document delete operation on the engine
        Specified by:
        delete in class Engine
        Parameters:
        delete - operation to perform
        Returns:
        Engine.DeleteResult containing updated translog location, version and document specific failures Note: engine level failures (i.e. persistent engine failures) are thrown

      • isTranslogSyncNeeded

        public boolean isTranslogSyncNeeded()
        Description copied from class: Engine
        Checks if the underlying storage sync is required.
        Specified by:
        isTranslogSyncNeeded in class Engine

      • ensureTranslogSynced

        public boolean ensureTranslogSynced​(java.util.stream.Stream<Translog.Location> locations)
        Description copied from class: Engine
        Ensures that all locations in the given stream have been written to the underlying storage.
        Specified by:
        ensureTranslogSynced in class Engine

      • acquireRetentionLock

        public java.io.Closeable acquireRetentionLock()
        Description copied from class: Engine
        Acquires a lock on the translog files and Lucene soft-deleted documents to prevent them from being trimmed
        Specified by:
        acquireRetentionLock in class Engine

      • newChangesSnapshot

        public Translog.Snapshot newChangesSnapshot​(java.lang.String source,
                                            MapperService mapperService,
                                            long fromSeqNo,
                                            long toSeqNo,
                                            boolean requiredFullRange)
                                     throws java.io.IOException
        Description copied from class: Engine
        Creates a new history snapshot from Lucene for reading operations whose seqno in the requesting seqno range (both inclusive). This feature requires soft-deletes enabled. If soft-deletes are disabled, this method will throw an IllegalStateException.
        Specified by:
        newChangesSnapshot in class Engine
        Throws:
        java.io.IOException

      • readHistoryOperations

        public Translog.Snapshot readHistoryOperations​(java.lang.String source,
                                               MapperService mapperService,
                                               long startingSeqNo)
                                        throws java.io.IOException
        Description copied from class: Engine
        Creates a new history snapshot for reading operations since startingSeqNo (inclusive). The returned snapshot can be retrieved from either Lucene index or translog files.
        Specified by:
        readHistoryOperations in class Engine
        Throws:
        java.io.IOException

      • estimateNumberOfHistoryOperations

        public int estimateNumberOfHistoryOperations​(java.lang.String source,
                                             MapperService mapperService,
                                             long startingSeqNo)
                                      throws java.io.IOException
        Description copied from class: Engine
        Returns the estimated number of history operations whose seq# at least startingSeqNo(inclusive) in this engine.
        Specified by:
        estimateNumberOfHistoryOperations in class Engine
        Throws:
        java.io.IOException

      • hasCompleteOperationHistory

        public boolean hasCompleteOperationHistory​(java.lang.String source,
                                           MapperService mapperService,
                                           long startingSeqNo)
                                    throws java.io.IOException
        Description copied from class: Engine
        Checks if this engine has every operations since startingSeqNo(inclusive) in its history (either Lucene or translog)
        Specified by:
        hasCompleteOperationHistory in class Engine
        Throws:
        java.io.IOException

      • getMinRetainedSeqNo

        public long getMinRetainedSeqNo()
        Description copied from class: Engine
        Gets the minimum retained sequence number for this engine.
        Specified by:
        getMinRetainedSeqNo in class Engine
        Returns:
        the minimum retained sequence number

      • getPersistedLocalCheckpoint

        public long getPersistedLocalCheckpoint()
        Specified by:
        getPersistedLocalCheckpoint in class Engine
        Returns:
        the persisted local checkpoint for this Engine

      • getSeqNoStats

        public SeqNoStats getSeqNoStats​(long globalCheckpoint)
        Specified by:
        getSeqNoStats in class Engine
        Returns:
        a SeqNoStats object, using local state and the supplied global checkpoint

      • getLastSyncedGlobalCheckpoint

        public long getLastSyncedGlobalCheckpoint()
        Description copied from class: Engine
        Returns the latest global checkpoint value that has been persisted in the underlying storage (i.e. translog's checkpoint)
        Specified by:
        getLastSyncedGlobalCheckpoint in class Engine

      • getIndexBufferRAMBytesUsed

        public long getIndexBufferRAMBytesUsed()
        Description copied from class: Engine
        How much heap is used that would be freed by a refresh. Note that this may throw AlreadyClosedException.
        Specified by:
        getIndexBufferRAMBytesUsed in class Engine

      • segments

        public java.util.List<Segment> segments​(boolean verbose)
        Description copied from class: Engine
        The list of segments in the engine.
        Specified by:
        segments in class Engine

      • refresh

        public void refresh​(java.lang.String source)
        Description copied from class: Engine
        Synchronously refreshes the engine for new search operations to reflect the latest changes.
        Specified by:
        refresh in class Engine

      • maybeRefresh

        public boolean maybeRefresh​(java.lang.String source)
                     throws EngineException
        Description copied from class: Engine
        Synchronously refreshes the engine for new search operations to reflect the latest changes unless another thread is already refreshing the engine concurrently.
        Specified by:
        maybeRefresh in class Engine
        Returns:
        true if the a refresh happened. Otherwise false
        Throws:
        EngineException

      • shouldPeriodicallyFlush

        public boolean shouldPeriodicallyFlush()
        Description copied from class: Engine
        Checks if this engine should be flushed periodically. This check is mainly based on the uncommitted translog size and the translog flush threshold setting.
        Specified by:
        shouldPeriodicallyFlush in class Engine

      • syncFlush

        public Engine.SyncedFlushResult syncFlush​(java.lang.String syncId,
                                          Engine.CommitId expectedCommitId)
        Description copied from class: Engine
        Attempts to do a special commit where the given syncID is put into the commit data. The attempt succeeds if there are not pending writes in lucene and the current point is equal to the expected one.
        Specified by:
        syncFlush in class Engine
        Parameters:
        syncId - id of this sync
        expectedCommitId - the expected value of
        Returns:
        true if the sync commit was made, false o.w.

      • flush

        public Engine.CommitId flush​(boolean force,
                             boolean waitIfOngoing)
                      throws EngineException
        Description copied from class: Engine
        Flushes the state of the engine including the transaction log, clearing memory.
        Specified by:
        flush in class Engine
        Parameters:
        force - if true a lucene commit is executed even if no changes need to be committed.
        waitIfOngoing - if true this call will block until all currently running flushes have finished. Otherwise this call will return without blocking.
        Returns:
        the commit Id for the resulting commit
        Throws:
        EngineException

      • forceMerge

        public void forceMerge​(boolean flush,
                       int maxNumSegments,
                       boolean onlyExpungeDeletes,
                       boolean upgrade,
                       boolean upgradeOnlyAncientSegments)
        Description copied from class: Engine
        Triggers a forced merge on this engine
        Specified by:
        forceMerge in class Engine

      • acquireLastIndexCommit

        public Engine.IndexCommitRef acquireLastIndexCommit​(boolean flushFirst)
        Description copied from class: Engine
        Snapshots the most recent index and returns a handle to it. If needed will try and "commit" the lucene index to make sure we have a "fresh" copy of the files to snapshot.
        Specified by:
        acquireLastIndexCommit in class Engine
        Parameters:
        flushFirst - indicates whether the engine should flush before returning the snapshot

      • shouldRollTranslogGeneration

        public boolean shouldRollTranslogGeneration()
        Description copied from class: Engine
        Tests whether or not the translog generation should be rolled to a new generation. This test is based on the size of the current generation compared to the configured generation threshold size.
        Specified by:
        shouldRollTranslogGeneration in class Engine
        Returns:
        true if the current generation should be rolled to a new generation

      • rollTranslogGeneration

        public void rollTranslogGeneration()
        Description copied from class: Engine
        Rolls the translog generation and cleans unneeded.
        Specified by:
        rollTranslogGeneration in class Engine

      • restoreLocalHistoryFromTranslog

        public int restoreLocalHistoryFromTranslog​(Engine.TranslogRecoveryRunner translogRecoveryRunner)
        Description copied from class: Engine
        This method replays translog to restore the Lucene index which might be reverted previously. This ensures that all acknowledged writes are restored correctly when this engine is promoted.
        Specified by:
        restoreLocalHistoryFromTranslog in class Engine
        Returns:
        the number of translog operations have been recovered

      • fillSeqNoGaps

        public int fillSeqNoGaps​(long primaryTerm)
        Description copied from class: Engine
        Fills up the local checkpoints history with no-ops until the local checkpoint and the max seen sequence ID are identical.
        Specified by:
        fillSeqNoGaps in class Engine
        Parameters:
        primaryTerm - the shards primary term this engine was created for
        Returns:
        the number of no-ops added

      • recoverFromTranslog

        public Engine recoverFromTranslog​(Engine.TranslogRecoveryRunner translogRecoveryRunner,
                                  long recoverUpToSeqNo)
        Description copied from class: Engine
        Performs recovery from the transaction log up to recoverUpToSeqNo (inclusive). This operation will close the engine if the recovery fails.
        Specified by:
        recoverFromTranslog in class Engine
        Parameters:
        translogRecoveryRunner - the translog recovery runner
        recoverUpToSeqNo - the upper bound, inclusive, of sequence number to be recovered

      • skipTranslogRecovery

        public void skipTranslogRecovery()
        Description copied from class: Engine
        Do not replay translog operations, but make the engine be ready.
        Specified by:
        skipTranslogRecovery in class Engine

      • maybePruneDeletes

        public void maybePruneDeletes()
        Description copied from class: Engine
        Tries to prune buffered deletes from the version map.
        Specified by:
        maybePruneDeletes in class Engine

      • updateMaxUnsafeAutoIdTimestamp

        public void updateMaxUnsafeAutoIdTimestamp​(long newTimestamp)
        Description copied from class: Engine
        Forces this engine to advance its max_unsafe_auto_id_timestamp marker to at least the given timestamp. The engine will disable optimization for all append-only whose timestamp at most newTimestamp.
        Specified by:
        updateMaxUnsafeAutoIdTimestamp in class Engine

      • getMaxSeqNoOfUpdatesOrDeletes

        public long getMaxSeqNoOfUpdatesOrDeletes()
        Description copied from class: Engine
        Returns the maximum sequence number of either update or delete operations have been processed in this engine or the sequence number from Engine.advanceMaxSeqNoOfUpdatesOrDeletes(long). An index request is considered as an update operation if it overwrites the existing documents in Lucene index with the same document id.

        A note on the optimization using max_seq_no_of_updates_or_deletes: For each operation O, the key invariants are:

        1. I1: There is no operation on docID(O) with seqno that is > MSU(O) and < seqno(O)
        2. I2: If MSU(O) < seqno(O) then docID(O) did not exist when O was applied; more precisely, if there is any O' with seqno(O') < seqno(O) and docID(O') = docID(O) then the one with the greatest seqno is a delete.

        When a receiving shard (either a replica or a follower) receives an operation O, it must first ensure its own MSU at least MSU(O), and then compares its MSU to its local checkpoint (LCP). If LCP < MSU then there's a gap: there may be some operations that act on docID(O) about which we do not yet know, so we cannot perform an add. Note this also covers the case where a future operation O' with seqNo(O') > seqNo(O) and docId(O') = docID(O) is processed before O. In that case MSU(O') is at least seqno(O') and this means MSU >= seqNo(O') > seqNo(O) > LCP (because O wasn't processed yet).

        However, if MSU <= LCP then there is no gap: we have processed every operation <= LCP, and no operation O' with seqno(O') > LCP and seqno(O') < seqno(O) also has docID(O') = docID(O), because such an operation would have seqno(O') > LCP >= MSU >= MSU(O) which contradicts the first invariant. Furthermore in this case we immediately know that docID(O) has been deleted (or never existed) without needing to check Lucene for the following reason. If there's no earlier operation on docID(O) then this is clear, so suppose instead that the preceding operation on docID(O) is O': 1. The first invariant above tells us that seqno(O') <= MSU(O) <= LCP so we have already applied O' to Lucene. 2. Also MSU(O) <= MSU <= LCP < seqno(O) (we discard O if seqno(O) <= LCP) so the second invariant applies, meaning that the O' was a delete.

        Therefore, if MSU <= LCP < seqno(O) we know that O can safely be optimized with and added to lucene with addDocument. Moreover, operations that are optimized using the MSU optimization must not be processed twice as this will create duplicates in Lucene. To avoid this we check the local checkpoint tracker to see if an operation was already processed.

        Specified by:
        getMaxSeqNoOfUpdatesOrDeletes in class Engine
        See Also:
        Engine.advanceMaxSeqNoOfUpdatesOrDeletes(long)

      • advanceMaxSeqNoOfUpdatesOrDeletes

        public void advanceMaxSeqNoOfUpdatesOrDeletes​(long maxSeqNoOfUpdatesOnPrimary)
        Description copied from class: Engine
        A replica shard receives a new max_seq_no_of_updates from its primary shard, then calls this method to advance this marker to at least the given sequence number.
        Specified by:
        advanceMaxSeqNoOfUpdatesOrDeletes in class Engine