Module org.elasticsearch.server
Package org.elasticsearch.indices

Class SystemIndexDescriptor

java.lang.Object
org.elasticsearch.indices.SystemIndexDescriptor
All Implemented Interfaces:
Comparable<SystemIndexDescriptor>, IndexPatternMatcher
public class SystemIndexDescriptor extends Object implements IndexPatternMatcher, Comparable<SystemIndexDescriptor>
Uses a pattern string to define a protected space for indices belonging to a system feature, and, if needed, provides metadata for managing indices that match the pattern.

Any index name that matches a descriptor’s index pattern belongs to the descriptor. For example, if a descriptor had a pattern of ".example-index-*", then indices named ".example-index-1", ".example-index-reindex", and ".example-index-old" would all belong to the descriptor. If a node gains a new system index descriptor after an upgrade, then matching indices will automatically be marked as system indices (see SystemIndexMetadataUpgradeService).

SystemIndexDescriptor index patterns must begin with a "." character. Index patterns also have a "gotcha": the pattern definitions may look like the standard Elasticsearch multi-target syntax but the underlying implementation is different. Index patterns are actually defined in a mangled regex syntax where "." is always interpreted as a literal character and "*" is expanded to ".*". We don’t support date math or the "-" operator from the IndexNameExpressionResolver class. We intend for developers to use only the "*", "+", "~" (complement), "(", ")", and character class operators, but because of the implementation, other regex operators probably work.

Sample index patterns that we want to handle:

  1. .system-* - covers all index names beginning with ".system-".
  2. .system-[0-9]+ - covers all index names beginning with ".system-" and containing only one or more numerals after that
  3. .system-~(other-*) - covers all system indices beginning with ".system-", except for those beginning with ".system-other-"

The descriptor defines which, if any, Elasticsearch products are expected to read or modify it with calls to the REST API. Requests that do not include the correct product header should, in most cases, generate deprecation warnings. The exception is for "net new" system index descriptors, described below.

The descriptor also provides names for the thread pools that Elasticsearch should use to read, search, or modify the descriptor’s indices.

A SystemIndexDescriptor may be one of several types (see SystemIndexDescriptor.Type). The four types come from two different distinctions. The first is between "internal" and "external" system indices. The second is between "managed and unmanaged" system indices. The "internal/external" distinction is simple. Access to internal system indices via standard index APIs is deprecated, and system features that use internal system indices should provide any necessary APIs for operating on their state. An "external" system index, on the other hand, does not deprecate the use of standard index APIs.

The distinction between managed and unmanaged is simple in theory but not observed very well in our code. A "managed" system index is one whose settings, mappings, and aliases are defined by the SystemIndexDescriptor and managed by Elasticsearch. Many of the fields in this class, when added, were meant to be used only by managed system indices, and use of them should always be conditional on whether the system index is managed or not. However, we have not consistently enforced this, so our code may have inconsistent expectations about what fields will be defined for an unmanaged index. (In the future, we should refactor so that it is clear which fields are ignored by unmanaged system indices.)

A managed system index defines a "primary index" which is intended to be the main write index for the descriptor. The current behavior when creating a non-primary index is a little strange. A request to create a non-primary index with the Create Index API will fail. (See PR #86707) However, auto-creating the index by writing a document to it will succeed. (See PR #77045)

The mappings for managed system indices are automatically upgraded when all nodes in the cluster are compatible with the descriptor's mappings. See SystemIndexMappingUpdateService for details.

We hope to remove the currently deprecated forms of access to system indices in a future release. A newly added system index with no backwards-compatibility requirements may opt into our desired behavior by setting isNetNew to true. A "net new system index" strictly enforces its allowed product origins, and cannot be accessed by any REST API request that lacks a correct product header. A system index that is fully internal to Elasticsearch will not allow any product origins; such an index is fully "locked down," and in general can only be changed by restoring feature states from snapshots.

  • Field Details

    • DEFAULT_SETTINGS

      public static final Settings DEFAULT_SETTINGS

    • VERSION_META_KEY

      public static final String VERSION_META_KEY
      The version meta key for the integer system index mapping version
      See Also:

  • Constructor Details

    • SystemIndexDescriptor

      protected SystemIndexDescriptor(String indexPattern, String primaryIndex, String description, String mappings, Settings settings, String aliasName, int indexFormat, String mappingsNodeVersionMetaKey, String origin, @Deprecated Version minimumNodeVersion, SystemIndexDescriptor.Type type, List<String> allowedElasticProductOrigins, List<SystemIndexDescriptor> priorSystemIndexDescriptors, ExecutorNames executorNames, boolean isNetNew, boolean allowsTemplates)
      Creates a descriptor for system indices matching the supplied pattern. These indices will be managed by Elasticsearch internally if mappings or settings are provided.
      Parameters:
      indexPattern - The pattern of index names that this descriptor will be used for. Must start with a '.' character, must not overlap with any other descriptor patterns, and must allow a suffix (see note on indexPattern for details).
      primaryIndex - The primary index name of this descriptor. Used when creating the system index for the first time.
      description - The name of the plugin responsible for this system index.
      mappings - The mappings to apply to this index when auto-creating, if appropriate
      settings - The settings to apply to this index when auto-creating, if appropriate
      aliasName - An alias for the index, or null
      indexFormat - A value for the `index.format` setting. Pass 0 or higher.
      mappingsNodeVersionMetaKey - a mapping key under _meta where a version can be found, which indicates the Elasticsearch version when the index was created.
      origin - the client origin to use when creating this index. Internal system indices must not provide an origin, while external system indices must do so.
      minimumNodeVersion - the minimum cluster node version required for this descriptor
      type - The SystemIndexDescriptor.Type of system index
      allowedElasticProductOrigins - A list of allowed origin values that should be allowed access in the case of external system indices
      priorSystemIndexDescriptors - A list of system index descriptors that describe the same index in a way that is compatible with older versions of Elasticsearch
      allowsTemplates - if this system index descriptor allows templates to affect its settings (e.g. .kibana_ indices)

  • Method Details

    • getIndexPattern

      public String getIndexPattern()
      Specified by:
      getIndexPattern in interface IndexPatternMatcher
      Returns:
      The pattern of index names that this descriptor will be used for. Must start with a '.' character, must not overlap with any other descriptor patterns, and must allow a suffix (see note on indexPattern for details).

    • getPrimaryIndex

      public String getPrimaryIndex()
      Returns:
      The concrete name of an index being managed internally to Elasticsearch. Will be null for indices managed externally to Elasticsearch.

    • matchesIndexPattern

      public boolean matchesIndexPattern(String index)
      Checks whether an index name matches the system index name pattern for this descriptor.
      Parameters:
      index - The index name to be checked against the index pattern given at construction time.
      Returns:
      True if the name matches the pattern, false otherwise.

    • getMatchingIndices

      public List<String> getMatchingIndices(Metadata metadata)
      Retrieves a list of all indices which match this descriptor's pattern. This cannot be done via IndexNameExpressionResolver because that class can only handle simple wildcard expressions, but system index name patterns may use full Lucene regular expression syntax,
      Specified by:
      getMatchingIndices in interface IndexPatternMatcher
      Parameters:
      metadata - The current metadata to get the list of matching indices from
      Returns:
      A list of index names that match this descriptor

    • getDescription

      public String getDescription()
      Returns:
      A short description of the purpose of this system index.

    • toString

      public String toString()
      Overrides:
      toString in class Object

    • getMappings

      public String getMappings()

    • getSettings

      public Settings getSettings()

    • getAliasName

      public String getAliasName()

    • getIndexFormat

      public int getIndexFormat()

    • getMappingsNodeVersionMetaKey

      public String getMappingsNodeVersionMetaKey()

    • getMinimumNodeVersion

      public Version getMinimumNodeVersion()

    • isAutomaticallyManaged

      public boolean isAutomaticallyManaged()

    • getOrigin

      public String getOrigin()
      Get an origin string suitable for use in an OriginSettingClient. See SystemIndexDescriptor.Builder.setOrigin(String) for more information.
      Returns:
      an origin string to use for sub-requests

    • hasDynamicMappings

      public boolean hasDynamicMappings()

    • isExternal

      public boolean isExternal()

    • isInternal

      public boolean isInternal()

    • getAllowedElasticProductOrigins

      public List<String> getAllowedElasticProductOrigins()
      Requests from these products, if made with the proper security credentials, are allowed non-deprecated access to this descriptor's indices. (Product names may be specified in requests with the Task.X_ELASTIC_PRODUCT_ORIGIN_HTTP_HEADER).
      Returns:
      A list of product names.

    • isNetNew

      public boolean isNetNew()

    • allowsTemplates

      public boolean allowsTemplates()

    • getMappingsNodeVersion

      @Deprecated public Version getMappingsNodeVersion()
      Deprecated.
      Use of the mappings Version should be replaced with the value returned from getMappingsVersion()
      Returns:
      Elasticsearch version associated with this descriptor's mappings.

    • getMappingsVersion

      public SystemIndexDescriptor.MappingsVersion getMappingsVersion()

    • getMinimumMappingsVersionMessage

      public String getMinimumMappingsVersionMessage(String cause)
      Gets a standardized message when the node contains a data or master node whose version is less than that of the minimum supported version of this descriptor and its prior descriptors.
      Parameters:
      cause - the action being attempted that triggered the check. Used in the error message.
      Returns:
      the standardized error message

    • getMinimumNodeVersionMessage

      @Deprecated public String getMinimumNodeVersionMessage(String cause)
      Deprecated.
      Gets a standardized message when the node contains a data or master node whose version is less than that of the minimum supported version of this descriptor and its prior descriptors.
      Parameters:
      cause - the action being attempted that triggered the check. Used in the error message.
      Returns:
      the standardized error message

    • getDescriptorCompatibleWith

      @Deprecated public SystemIndexDescriptor getDescriptorCompatibleWith(Version version)
      Deprecated.
      Finds the descriptor that can be used within this cluster, by comparing the supplied minimum node version to this descriptor's minimum version and the prior descriptors minimum version.
      Parameters:
      version - the lower node version in the cluster
      Returns:
      null if the lowest node version is lower than the minimum version in this descriptor, or the appropriate descriptor if the supplied version is acceptable.

    • getDescriptorCompatibleWith

      public SystemIndexDescriptor getDescriptorCompatibleWith(SystemIndexDescriptor.MappingsVersion version)
      Finds the descriptor that can be used within this cluster, by comparing the supplied minimum node version to this descriptor's minimum version and the prior descriptors minimum version.
      Parameters:
      version - the lower node version in the cluster
      Returns:
      null if the lowest node version is lower than the minimum version in this descriptor, or the appropriate descriptor if the supplied version is acceptable.

    • getThreadPoolNames

      public ExecutorNames getThreadPoolNames()
      Returns:
      The names of thread pools that should be used for operations on this system index.

    • builder

      public static SystemIndexDescriptor.Builder builder()

    • compareTo

      public int compareTo(SystemIndexDescriptor other)
      Specified by:
      compareTo in interface Comparable<SystemIndexDescriptor>