Module org.elasticsearch.server
Package org.elasticsearch.indices

Class SystemIndexDescriptor

java.lang.Object
org.elasticsearch.indices.SystemIndexDescriptor
All Implemented Interfaces:
Comparable<SystemIndexDescriptor>, IndexMatcher, IndexPatternMatcher, SystemResourceDescriptor
public class SystemIndexDescriptor extends Object implements IndexPatternMatcher, SystemResourceDescriptor, Comparable<SystemIndexDescriptor>
Uses a pattern string to define a protected space for indices belonging to a system feature, and, if needed, provides metadata for managing indices that match the pattern.

Any index name that matches a descriptor’s index pattern belongs to the descriptor. For example, if a descriptor had a pattern of ".example-index-*", then indices named ".example-index-1", ".example-index-reindex", and ".example-index-old" would all belong to the descriptor. If a node gains a new system index descriptor after an upgrade, then matching indices will automatically be marked as system indices (see SystemIndexMetadataUpgradeService).

SystemIndexDescriptor index patterns must begin with a "." character. Index patterns also have a "gotcha": the pattern definitions may look like the standard Elasticsearch multi-target syntax but the underlying implementation is different. Index patterns are actually defined in a mangled regex syntax where "." is always interpreted as a literal character and "*" is expanded to ".*". We don’t support date math or the "-" operator from the IndexNameExpressionResolver class. We intend for developers to use only the "*", "+", "~" (complement), "(", ")", and character class operators, but because of the implementation, other regex operators probably work.

Sample index patterns that we want to handle:

  1. .system-* - covers all index names beginning with ".system-".
  2. .system-[0-9]+ - covers all index names beginning with ".system-" and containing only one or more numerals after that
  3. .system-~(other-*) - covers all system indices beginning with ".system-", except for those beginning with ".system-other-"

The descriptor defines which, if any, Elasticsearch products are expected to read or modify it with calls to the REST API. Requests that do not include the correct product header should, in most cases, generate deprecation warnings. The exception is for "net new" system index descriptors, described below.

The descriptor also provides names for the thread pools that Elasticsearch should use to read, search, or modify the descriptor’s indices.

A SystemIndexDescriptor may be one of several types (see SystemIndexDescriptor.Type). The four types come from two different distinctions. The first is between "internal" and "external" system indices. The second is between "managed and unmanaged" system indices. The "internal/external" distinction is simple. Access to internal system indices via standard index APIs is deprecated, and system features that use internal system indices should provide any necessary APIs for operating on their state. An "external" system index, on the other hand, does not deprecate the use of standard index APIs.

The distinction between managed and unmanaged is simple in theory but not observed very well in our code. A "managed" system index is one whose settings, mappings, and aliases are defined by the SystemIndexDescriptor and managed by Elasticsearch. Many of the fields in this class, when added, were meant to be used only by managed system indices, and use of them should always be conditional on whether the system index is managed or not. However, we have not consistently enforced this, so our code may have inconsistent expectations about what fields will be defined for an unmanaged index. (In the future, we should refactor so that it is clear which fields are ignored by unmanaged system indices.)

A managed system index defines a "primary index" which is intended to be the main write index for the descriptor. The current behavior when creating a non-primary index is a little strange. A request to create a non-primary index with the Create Index API will fail. (See PR #86707) However, auto-creating the index by writing a document to it will succeed. (See PR #77045)

The mappings for managed system indices are automatically upgraded when all nodes in the cluster are compatible with the descriptor's mappings. See SystemIndexMappingUpdateService for details. When the mappings change add the previous index descriptors with SystemIndexDescriptor.Builder.setPriorSystemIndexDescriptors(List). In a mixed cluster setting this enables auto creation of the index with compatible mappings.

We hope to remove the currently deprecated forms of access to system indices in a future release. A newly added system index with no backwards-compatibility requirements may opt into our desired behavior by setting isNetNew to true. A "net new system index" strictly enforces its allowed product origins, and cannot be accessed by any REST API request that lacks a correct product header. A system index that is fully internal to Elasticsearch will not allow any product origins; such an index is fully "locked down," and in general can only be changed by restoring feature states from snapshots.

  • Field Details

    • DEFAULT_SETTINGS

      public static final Settings DEFAULT_SETTINGS

    • VERSION_META_KEY

      public static final String VERSION_META_KEY
      The version meta key for the integer system index mapping version
      See Also:

  • Constructor Details

    • SystemIndexDescriptor

      protected SystemIndexDescriptor(String indexPattern, String primaryIndex, String description, String mappings, Settings settings, String aliasName, int indexFormat, String origin, String migrationScript, SystemIndexDescriptor.Type type, List<String> allowedElasticProductOrigins, List<SystemIndexDescriptor> priorSystemIndexDescriptors, ExecutorNames executorNames, boolean isNetNew, boolean allowsTemplates)
      Creates a descriptor for system indices matching the supplied pattern. These indices will be managed by Elasticsearch internally if mappings or settings are provided.
      Parameters:
      indexPattern - The pattern of index names that this descriptor will be used for. Must start with a '.' character, must not overlap with any other descriptor patterns, and must allow a suffix (see note on indexPattern for details).
      primaryIndex - The primary index name of this descriptor. Used when creating the system index for the first time.
      description - The name of the plugin responsible for this system index.
      mappings - The mappings to apply to this index when auto-creating, if appropriate
      settings - The settings to apply to this index when auto-creating, if appropriate
      aliasName - An alias for the index, or null
      indexFormat - A value for the `index.format` setting. Pass 0 or higher.
      origin - the client origin to use when creating this index. Internal system indices must not provide an origin, while external system indices must do so.
      migrationScript - The script to apply when migrating this system index, or null
      type - The SystemIndexDescriptor.Type of system index
      allowedElasticProductOrigins - A list of allowed origin values that should be allowed access in the case of external system indices
      priorSystemIndexDescriptors - A list of system index descriptors that describe the same index in a way that is compatible with older versions of Elasticsearch
      allowsTemplates - if this system index descriptor allows templates to affect its settings (e.g. .kibana_ indices)

  • Method Details

    • getIndexPattern

      public String getIndexPattern()
      Specified by:
      getIndexPattern in interface IndexPatternMatcher
      Returns:
      The pattern of index names that this descriptor will be used for. Must start with a '.' character, must not overlap with any other descriptor patterns, and must allow a suffix (see note on indexPattern for details).

    • getPrimaryIndex

      public String getPrimaryIndex()
      Returns:
      The concrete name of an index being managed internally to Elasticsearch. Will be null for indices managed externally to Elasticsearch.

    • matchesIndexPattern

      public boolean matchesIndexPattern(String index)
      Checks whether an index name matches the system index name pattern for this descriptor.
      Parameters:
      index - The index name to be checked against the index pattern given at construction time.
      Returns:
      True if the name matches the pattern, false otherwise.

    • getMatchingIndices

      public List<String> getMatchingIndices(Metadata metadata)
      Retrieves a list of all indices which match this descriptor's pattern. This cannot be done via IndexNameExpressionResolver because that class can only handle simple wildcard expressions, but system index name patterns may use full Lucene regular expression syntax,
      Specified by:
      getMatchingIndices in interface IndexMatcher
      Parameters:
      metadata - The current metadata to get the list of matching indices from
      Returns:
      A list of index names that match this descriptor

    • getDescription

      public String getDescription()
      Specified by:
      getDescription in interface SystemResourceDescriptor
      Returns:
      A short description of the purpose of this system resource.

    • toString

      public String toString()
      Overrides:
      toString in class Object

    • getMappings

      public String getMappings()

    • getSettings

      public Settings getSettings()

    • getAliasName

      public String getAliasName()

    • getIndexFormat

      public int getIndexFormat()

    • isAutomaticallyManaged

      public boolean isAutomaticallyManaged()
      Specified by:
      isAutomaticallyManaged in interface SystemResourceDescriptor

    • getOrigin

      public String getOrigin()
      Description copied from interface: SystemResourceDescriptor
      Get an origin string suitable for use in an OriginSettingClient. See SystemIndexDescriptor.Builder.setOrigin(String) for more information.
      Specified by:
      getOrigin in interface SystemResourceDescriptor
      Returns:
      an origin string to use for sub-requests

    • hasDynamicMappings

      public boolean hasDynamicMappings()

    • isExternal

      public boolean isExternal()
      Specified by:
      isExternal in interface SystemResourceDescriptor

    • getAllowedElasticProductOrigins

      public List<String> getAllowedElasticProductOrigins()
      Description copied from interface: SystemResourceDescriptor
      Requests from these products, if made with the proper security credentials, are allowed non-deprecated access to this descriptor's indices. (Product names may be specified in requests with the Task.X_ELASTIC_PRODUCT_ORIGIN_HTTP_HEADER).
      Specified by:
      getAllowedElasticProductOrigins in interface SystemResourceDescriptor
      Returns:
      A list of product names.

    • isNetNew

      public boolean isNetNew()

    • allowsTemplates

      public boolean allowsTemplates()

    • getMappingsVersion

      public SystemIndexDescriptor.MappingsVersion getMappingsVersion()

    • getMinimumMappingsVersionMessage

      public String getMinimumMappingsVersionMessage(String cause, SystemIndexDescriptor.MappingsVersion requiredMinimumMappingVersion)
      Gets a standardized message when the node contains a data or master node whose mappings version is less than that of the minimum supported version of this descriptor and its prior descriptors.
      Parameters:
      cause - the action being attempted that triggered the check. Used in the error message.
      Returns:
      the standardized error message

    • getDescriptorCompatibleWith

      public SystemIndexDescriptor getDescriptorCompatibleWith(SystemIndexDescriptor.MappingsVersion version)
      Finds the descriptor that can be used within this cluster, by comparing the supplied minimum mappings version to this descriptor's minimum version and the prior descriptors minimum version.
      Parameters:
      version - the lower mappings version in the cluster
      Returns:
      null if the lowest mappings version is lower than the minimum version in this descriptor, or the appropriate descriptor if the supplied version is acceptable.

    • getThreadPoolNames

      public ExecutorNames getThreadPoolNames()
      Specified by:
      getThreadPoolNames in interface SystemResourceDescriptor
      Returns:
      The names of thread pools that should be used for operations on this system index.

    • getMigrationScript

      public String getMigrationScript()

    • builder

      public static SystemIndexDescriptor.Builder builder()

    • compareTo

      public int compareTo(SystemIndexDescriptor other)
      Specified by:
      compareTo in interface Comparable<SystemIndexDescriptor>