org.elasticsearch.index.store

Class Store

java.lang.Object

org.elasticsearch.index.shard.AbstractIndexShardComponent

org.elasticsearch.index.store.Store

All Implemented Interfaces:

Closeable, AutoCloseable, RefCounted, IndexShardComponent

public class Store
extends AbstractIndexShardComponent
implements Closeable, RefCounted

A Store provides plain access to files written by an elasticsearch index shard. Each shard has a dedicated store that is uses to access Lucene's Directory which represents the lowest level of file abstraction in Lucene used to read and write Lucene indices. This class also provides access to metadata information like checksums for committed files. A committed file is a file that belongs to a segment written by a Lucene commit. Files that have not been committed ie. created during a merge or a shard refresh / NRT reopen are not considered in the MetadataSnapshot.

Note: If you use a store it's reference count should be increased before using it by calling #incRef and a corresponding #decRef must be called in a try/finally block to release the store again ie.:

      store.incRef();
      try {
        // use the store...

      } finally {
          store.decRef();
      }
 

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	Store.LegacyChecksums
static class	Store.MetadataSnapshot Represents a snapshot of the current directory build from the latest Lucene commit.
static interface	Store.OnClose A listener that is executed once the store is closed and all references to it are released
static class	Store.RecoveryDiff A class representing the diff between a recovery source and recovery target

Field Summary

Fields

Modifier and Type	Field and Description
static String	CHECKSUMS_PREFIX
static String	INDEX_STORE_STATS_REFRESH_INTERVAL

Fields inherited from class org.elasticsearch.index.shard.AbstractIndexShardComponent

deprecationLogger, indexSettings, logger, shardId

Constructor Summary

Constructors

Constructor and Description
Store(ShardId shardId, IndexSettingsService indexSettingsService, DirectoryService directoryService, ShardLock shardLock, Store.OnClose onClose)
Store(ShardId shardId, Settings indexSettings, DirectoryService directoryService, ShardLock shardLock)
Store(ShardId shardId, Settings indexSettings, DirectoryService directoryService, ShardLock shardLock, Store.OnClose onClose)

Method Summary

Methods

Modifier and Type	Method and Description
static boolean	canOpenIndex(ESLogger logger, Path indexLocation) Returns true iff the given location contains an index an the index can be successfully opened.
static void	checkIntegrity(StoreFileMetaData md, org.apache.lucene.store.Directory directory)
boolean	checkIntegrityNoException(StoreFileMetaData md)
static boolean	checkIntegrityNoException(StoreFileMetaData md, org.apache.lucene.store.Directory directory)
void	cleanupAndVerify(String reason, Store.MetadataSnapshot sourceMetaData) This method deletes every file in this store that is not contained in the given source meta data or is a legacy checksum file.
void	close()
org.apache.lucene.store.IndexOutput	createVerifyingOutput(String fileName, StoreFileMetaData metadata, org.apache.lucene.store.IOContext context) The returned IndexOutput might validate the files checksum if the file has been written with a newer lucene version and the metadata holds the necessary information to detect that it was been written by Lucene 4.8 or newer.
void	decRef() Decreases the refCount of this Store instance.If the refCount drops to 0, then this store is closed.
void	deleteQuiet(String... files)
static String	digestToString(long digest) Produces a string representation of the given digest value.
org.apache.lucene.store.Directory	directory()
void	failIfCorrupted()
Store.MetadataSnapshot	getMetadata() Returns a new MetadataSnapshot for the latest commit in this store.
Store.MetadataSnapshot	getMetadata(org.apache.lucene.index.IndexCommit commit) Returns a new MetadataSnapshot for the given commit.
Store.MetadataSnapshot	getMetadataOrEmpty() Returns a new MetadataSnapshot for the latest commit in this store or an empty snapshot if no index exists or can not be opened.
void	incRef() Increments the refCount of this Store instance.
static boolean	isAutogenerated(String name) Returns true if the file is auto-generated by the store and shouldn't be deleted during cleanup.
static boolean	isChecksum(String name)
boolean	isMarkedCorrupted()
void	markStoreCorrupted(IOException exception) Marks this store as corrupted.
org.apache.lucene.store.IndexInput	openVerifyingInput(String filename, org.apache.lucene.store.IOContext context, StoreFileMetaData metadata)
org.apache.lucene.index.SegmentInfos	readLastCommittedSegmentsInfo() Returns the last committed segments info for this store
static Store.MetadataSnapshot	readMetadataSnapshot(Path indexLocation, ESLogger logger) Reads a MetadataSnapshot from the given index locations or returns an empty snapshot if it can't be read.
int	refCount() Returns the current reference count.
void	removeCorruptionMarker() Deletes all corruption markers from this store.
void	renameFile(String from, String to)
void	renameTempFilesSafe(Map<String,String> tempFileMap) Renames all the given files form the key of the map to the value of the map.
StoreStats	stats()
boolean	tryIncRef() Tries to increment the refCount of this Store instance.
static void	tryOpenIndex(Path indexLocation) Tries to open an index for the given location.
static void	verify(org.apache.lucene.store.IndexInput input)
static void	verify(org.apache.lucene.store.IndexOutput output)

Methods inherited from class org.elasticsearch.index.shard.AbstractIndexShardComponent

indexSettings, nodeName, shardId

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

INDEX_STORE_STATS_REFRESH_INTERVAL

public static final String INDEX_STORE_STATS_REFRESH_INTERVAL

See Also:

Constant Field Values

CHECKSUMS_PREFIX

public static final String CHECKSUMS_PREFIX

See Also:

Constant Field Values

Constructor Detail

Store

public Store(ShardId shardId,
     Settings indexSettings,
     DirectoryService directoryService,
     ShardLock shardLock)
      throws IOException

Throws:

IOException

Store

@Inject
public Store(ShardId shardId,
            IndexSettingsService indexSettingsService,
            DirectoryService directoryService,
            ShardLock shardLock,
            Store.OnClose onClose)
      throws IOException

Throws:

IOException

Store

public Store(ShardId shardId,
     Settings indexSettings,
     DirectoryService directoryService,
     ShardLock shardLock,
     Store.OnClose onClose)
      throws IOException

Throws:

IOException

Method Detail

directory

public org.apache.lucene.store.Directory directory()

readLastCommittedSegmentsInfo

public org.apache.lucene.index.SegmentInfos readLastCommittedSegmentsInfo()
                                                                   throws IOException

Returns the last committed segments info for this store

Throws:

IOException - if the index is corrupted or the segments file is not present

getMetadataOrEmpty

public Store.MetadataSnapshot getMetadataOrEmpty()
                                          throws IOException

Returns a new MetadataSnapshot for the latest commit in this store or an empty snapshot if no index exists or can not be opened.

Throws:

org.apache.lucene.index.CorruptIndexException - if the lucene index is corrupted. This can be caused by a checksum mismatch or an unexpected exception when opening the index reading the segments file.

org.apache.lucene.index.IndexFormatTooOldException - if the lucene index is too old to be opened.

org.apache.lucene.index.IndexFormatTooNewException - if the lucene index is too new to be opened.

IOException

getMetadata

public Store.MetadataSnapshot getMetadata()
                                   throws IOException

Returns a new MetadataSnapshot for the latest commit in this store.

Throws:

org.apache.lucene.index.CorruptIndexException - if the lucene index is corrupted. This can be caused by a checksum mismatch or an unexpected exception when opening the index reading the segments file.

org.apache.lucene.index.IndexFormatTooOldException - if the lucene index is too old to be opened.

org.apache.lucene.index.IndexFormatTooNewException - if the lucene index is too new to be opened.

FileNotFoundException - if one or more files referenced by a commit are not present.

NoSuchFileException - if one or more files referenced by a commit are not present.

org.apache.lucene.index.IndexNotFoundException - if no index / valid commit-point can be found in this store

IOException

getMetadata

public Store.MetadataSnapshot getMetadata(org.apache.lucene.index.IndexCommit commit)
                                   throws IOException

Returns a new MetadataSnapshot for the given commit. If the given commit is null the latest commit point is used.

Throws:

org.apache.lucene.index.CorruptIndexException - if the lucene index is corrupted. This can be caused by a checksum mismatch or an unexpected exception when opening the index reading the segments file.

org.apache.lucene.index.IndexFormatTooOldException - if the lucene index is too old to be opened.

org.apache.lucene.index.IndexFormatTooNewException - if the lucene index is too new to be opened.

FileNotFoundException - if one or more files referenced by a commit are not present.

NoSuchFileException - if one or more files referenced by a commit are not present.

org.apache.lucene.index.IndexNotFoundException - if the commit point can't be found in this store

IOException

renameTempFilesSafe

public void renameTempFilesSafe(Map<String,String> tempFileMap)
                         throws IOException

Renames all the given files form the key of the map to the value of the map. All successfully renamed files are removed from the map in-place.

Throws:

IOException

stats

public StoreStats stats()
                 throws IOException

Throws:

IOException

renameFile

public void renameFile(String from,
              String to)
                throws IOException

Throws:

IOException

incRef

public final void incRef()

Increments the refCount of this Store instance. RefCounts are used to determine when a Store can be closed safely, i.e. as soon as there are no more references. Be sure to always call a corresponding decRef(), in a finally clause; otherwise the store may never be closed. Note that close() simply calls decRef(), which means that the Store will not really be closed until decRef() has been called for all outstanding references.

Note: Close can safely be called multiple times.

Specified by:

incRef in interface RefCounted

Throws:

org.apache.lucene.store.AlreadyClosedException - iff the reference counter can not be incremented.

See Also:

decRef(), tryIncRef()

tryIncRef

public final boolean tryIncRef()

Tries to increment the refCount of this Store instance. This method will return true iff the refCount was incremented successfully otherwise false. RefCounts are used to determine when a Store can be closed safely, i.e. as soon as there are no more references. Be sure to always call a corresponding decRef(), in a finally clause; otherwise the store may never be closed. Note that close() simply calls decRef(), which means that the Store will not really be closed until decRef() has been called for all outstanding references.

Note: Close can safely be called multiple times.

Specified by:

tryIncRef in interface RefCounted

See Also:

decRef(), incRef()

decRef

public final void decRef()

Decreases the refCount of this Store instance.If the refCount drops to 0, then this store is closed.

Specified by:

decRef in interface RefCounted

See Also:

incRef()

close

public void close()

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

readMetadataSnapshot

public static Store.MetadataSnapshot readMetadataSnapshot(Path indexLocation,
                                          ESLogger logger)
                                                   throws IOException

Reads a MetadataSnapshot from the given index locations or returns an empty snapshot if it can't be read.

Throws:

IOException - if the index we try to read is corrupted

canOpenIndex

public static boolean canOpenIndex(ESLogger logger,
                   Path indexLocation)
                            throws IOException

Returns true iff the given location contains an index an the index can be successfully opened. This includes reading the segment infos and possible corruption markers.

Throws:

IOException

tryOpenIndex

public static void tryOpenIndex(Path indexLocation)
                         throws IOException

Tries to open an index for the given location. This includes reading the segment infos and possible corruption markers. If the index can not be opened, an exception is thrown

Throws:

IOException

createVerifyingOutput

public org.apache.lucene.store.IndexOutput createVerifyingOutput(String fileName,
                                                        StoreFileMetaData metadata,
                                                        org.apache.lucene.store.IOContext context)
                                                          throws IOException

The returned IndexOutput might validate the files checksum if the file has been written with a newer lucene version and the metadata holds the necessary information to detect that it was been written by Lucene 4.8 or newer. If it has only a legacy checksum, returned IndexOutput will not verify the checksum.

Note: Checksums are calculated nevertheless since lucene does it by default sicne version 4.8.0. This method only adds the verification against the checksum in the given metadata and does not add any significant overhead.

Throws:

IOException

verify

public static void verify(org.apache.lucene.store.IndexOutput output)
                   throws IOException

Throws:

IOException

openVerifyingInput

public org.apache.lucene.store.IndexInput openVerifyingInput(String filename,
                                                    org.apache.lucene.store.IOContext context,
                                                    StoreFileMetaData metadata)
                                                      throws IOException

Throws:

IOException

verify

public static void verify(org.apache.lucene.store.IndexInput input)
                   throws IOException

Throws:

IOException

checkIntegrityNoException

public boolean checkIntegrityNoException(StoreFileMetaData md)

checkIntegrityNoException

public static boolean checkIntegrityNoException(StoreFileMetaData md,
                                org.apache.lucene.store.Directory directory)

checkIntegrity

public static void checkIntegrity(StoreFileMetaData md,
                  org.apache.lucene.store.Directory directory)
                           throws IOException

Throws:

IOException

isMarkedCorrupted

public boolean isMarkedCorrupted()
                          throws IOException

Throws:

IOException

removeCorruptionMarker

public void removeCorruptionMarker()
                            throws IOException

Deletes all corruption markers from this store.

Throws:

IOException

failIfCorrupted

public void failIfCorrupted()
                     throws IOException

Throws:

IOException

cleanupAndVerify

public void cleanupAndVerify(String reason,
                    Store.MetadataSnapshot sourceMetaData)
                      throws IOException

This method deletes every file in this store that is not contained in the given source meta data or is a legacy checksum file. After the delete it pulls the latest metadata snapshot from the store and compares it to the given snapshot. If the snapshots are inconsistent an illegal state exception is thrown

Parameters:

reason - the reason for this cleanup operation logged for each deleted file

sourceMetaData - the metadata used for cleanup. all files in this metadata should be kept around.

Throws:

IOException - if an IOException occurs

IllegalStateException - if the latest snapshot in this store differs from the given one after the cleanup.

refCount

public int refCount()

Returns the current reference count.

isChecksum

public static boolean isChecksum(String name)

isAutogenerated

public static boolean isAutogenerated(String name)

Returns true if the file is auto-generated by the store and shouldn't be deleted during cleanup. This includes write lock and checksum files

digestToString

public static String digestToString(long digest)

Produces a string representation of the given digest value.

deleteQuiet

public void deleteQuiet(String... files)

markStoreCorrupted

public void markStoreCorrupted(IOException exception)
                        throws IOException

Marks this store as corrupted. This method writes a corrupted_${uuid} file containing the given exception message. If a store contains a corrupted_${uuid} file isMarkedCorrupted() will return true.

Throws:

IOException