Module org.elasticsearch.server
Package org.elasticsearch.cluster.metadata

Class DataStream

java.lang.Object
org.elasticsearch.cluster.metadata.DataStream
All Implemented Interfaces:
Diffable<DataStream>, IndexAbstraction, SimpleDiffable<DataStream>, Writeable, ToXContent, ToXContentObject
public final class DataStream extends Object implements SimpleDiffable<DataStream>, ToXContentObject, IndexAbstraction

  • Field Details

    • FAILURE_STORE_FEATURE_FLAG

      public static final FeatureFlag FAILURE_STORE_FEATURE_FLAG

    • ADDED_FAILURE_STORE_TRANSPORT_VERSION

      public static final TransportVersion ADDED_FAILURE_STORE_TRANSPORT_VERSION

    • ADDED_AUTO_SHARDING_EVENT_VERSION

      public static final TransportVersion ADDED_AUTO_SHARDING_EVENT_VERSION

    • ADD_DATA_STREAM_OPTIONS_VERSION

      public static final TransportVersion ADD_DATA_STREAM_OPTIONS_VERSION

    • BACKING_INDEX_PREFIX

      public static final String BACKING_INDEX_PREFIX
      See Also:

    • FAILURE_STORE_PREFIX

      public static final String FAILURE_STORE_PREFIX
      See Also:

    • DATE_FORMATTER

      public static final DateFormatter DATE_FORMATTER

    • TIMESTAMP_FIELD_NAME

      public static final String TIMESTAMP_FIELD_NAME
      See Also:

    • TIMESERIES_LEAF_READERS_SORTER

      public static final Comparator<org.apache.lucene.index.LeafReader> TIMESERIES_LEAF_READERS_SORTER

    • NAME_FIELD

      public static final ParseField NAME_FIELD

    • TIMESTAMP_FIELD_FIELD

      public static final ParseField TIMESTAMP_FIELD_FIELD

    • INDICES_FIELD

      public static final ParseField INDICES_FIELD

    • GENERATION_FIELD

      public static final ParseField GENERATION_FIELD

    • METADATA_FIELD

      public static final ParseField METADATA_FIELD

    • HIDDEN_FIELD

      public static final ParseField HIDDEN_FIELD

    • REPLICATED_FIELD

      public static final ParseField REPLICATED_FIELD

    • SYSTEM_FIELD

      public static final ParseField SYSTEM_FIELD

    • ALLOW_CUSTOM_ROUTING

      public static final ParseField ALLOW_CUSTOM_ROUTING

    • INDEX_MODE

      public static final ParseField INDEX_MODE

    • LIFECYCLE

      public static final ParseField LIFECYCLE

    • FAILURE_STORE_FIELD

      public static final ParseField FAILURE_STORE_FIELD

    • FAILURE_INDICES_FIELD

      public static final ParseField FAILURE_INDICES_FIELD

    • ROLLOVER_ON_WRITE_FIELD

      public static final ParseField ROLLOVER_ON_WRITE_FIELD

    • AUTO_SHARDING_FIELD

      public static final ParseField AUTO_SHARDING_FIELD

    • FAILURE_ROLLOVER_ON_WRITE_FIELD

      public static final ParseField FAILURE_ROLLOVER_ON_WRITE_FIELD

    • FAILURE_AUTO_SHARDING_FIELD

      public static final ParseField FAILURE_AUTO_SHARDING_FIELD

    • DATA_STREAM_OPTIONS_FIELD

      public static final ParseField DATA_STREAM_OPTIONS_FIELD

    • TS_EXTRACT_CONFIG

      public static final XContentParserConfiguration TS_EXTRACT_CONFIG

  • Constructor Details

  • Method Details

    • isFailureStoreFeatureFlagEnabled

      public static boolean isFailureStoreFeatureFlagEnabled()

    • read

      public static DataStream read(StreamInput in) throws IOException
      Throws:
      IOException

    • getType

      public IndexAbstraction.Type getType()
      Specified by:
      getType in interface IndexAbstraction
      Returns:
      the type of the index abstraction

    • getName

      public String getName()
      Specified by:
      getName in interface IndexAbstraction
      Returns:
      the name of the index abstraction

    • isDataStreamRelated

      public boolean isDataStreamRelated()
      Specified by:
      isDataStreamRelated in interface IndexAbstraction
      Returns:
      whether this index abstraction is related to data streams

    • getIndices

      public List<Index> getIndices()
      Specified by:
      getIndices in interface IndexAbstraction
      Returns:
      All Index of all concrete indices this index abstraction is referring to.

    • getFailureIndices

      public List<Index> getFailureIndices(Metadata ignored)
      Description copied from interface: IndexAbstraction
      It retrieves the failure indices of an index abstraction given it supports the failure store.
      Specified by:
      getFailureIndices in interface IndexAbstraction
      Parameters:
      ignored - certain abstractions require the matadata to lazily retrieve the failure indices.
      Returns:
      All concrete failure indices this index abstraction is referring to. If the failure store is not supported, it returns an empty list.

    • getFailureIndices

      public List<Index> getFailureIndices()

    • getGeneration

      public long getGeneration()

    • getWriteIndex

      public Index getWriteIndex()
      Description copied from interface: IndexAbstraction
      A write index is a dedicated concrete index, that accepts all the new documents that belong to an index abstraction.

      A write index may also be a regular concrete index of a index abstraction and may therefore also be returned by IndexAbstraction.getIndices(). An index abstraction may also not have a dedicated write index.

      Specified by:
      getWriteIndex in interface IndexAbstraction
      Returns:
      the write index of this index abstraction or null if this index abstraction doesn't have a write index.

    • getWriteFailureIndex

      public Index getWriteFailureIndex(Metadata metadata)
      Description copied from interface: IndexAbstraction
      A write failure index is a dedicated concrete index, that accepts all the new documents that belong to the failure store of an index abstraction. Only an index abstraction with true IndexAbstraction.isDataStreamRelated() supports a failure store.
      Specified by:
      getWriteFailureIndex in interface IndexAbstraction
      Parameters:
      metadata - is not necessary for data streams
      Returns:
      the write failure index if the failure store is enabled and there is already at least one failure, null otherwise

    • getWriteFailureIndex

      @Nullable public Index getWriteFailureIndex()
      Returns:
      the write failure index if the failure store is enabled and there is already at least one failure, null otherwise

    • isFailureStoreIndex

      public boolean isFailureStoreIndex(String indexName)
      Returns true if the index name provided belongs to a failure store index.

    • containsIndex

      public boolean containsIndex(String indexName)
      Returns true if the index name provided belongs to this data stream.

    • getDataStreamOptions

      public DataStreamOptions getDataStreamOptions()

    • rolloverOnWrite

      public boolean rolloverOnWrite()

    • isInternal

      public boolean isInternal()
      We define that a data stream is considered internal either if it is a system index or if its name starts with a dot. Note: Dot-prefixed internal data streams is a naming convention for internal data streams, but it's not yet enforced.
      Returns:
      true if it's a system index or has a dot-prefixed name.

    • selectTimeSeriesWriteIndex

      public Index selectTimeSeriesWriteIndex(Instant timestamp, Metadata metadata)
      Parameters:
      timestamp - The timestamp used to select a backing index based on its start and end time.
      metadata - The metadata that is used to fetch the start and end times for backing indices of this data stream.
      Returns:
      a backing index with a start time that is greater or equal to the provided timestamp and an end time that is less than the provided timestamp. Otherwise null is returned.

    • validate

      public void validate(Function<String,IndexMetadata> imSupplier)
      Validates this data stream. If this is a time series data stream then this method validates that temporal range of backing indices (defined by index.time_series.start_time and index.time_series.end_time) do not overlap with each other.
      Parameters:
      imSupplier - Function that supplies IndexMetadata instances based on the provided index name

    • getMetadata

      @Nullable public Map<String,Object> getMetadata()

    • isHidden

      public boolean isHidden()
      Specified by:
      isHidden in interface IndexAbstraction
      Returns:
      whether this index abstraction is hidden or not

    • isReplicated

      public boolean isReplicated()
      Determines whether this data stream is replicated from elsewhere, for example a remote cluster
      Returns:
      Whether this data stream is replicated.

    • isSystem

      public boolean isSystem()
      Specified by:
      isSystem in interface IndexAbstraction
      Returns:
      whether this index abstraction should be treated as a system index or not

    • isAllowCustomRouting

      public boolean isAllowCustomRouting()

    • isFailureStoreExplicitlyEnabled

      public boolean isFailureStoreExplicitlyEnabled()
      Determines whether this data stream has its failure store enabled explicitly in its metadata.

    • isFailureStoreEffectivelyEnabled

      public boolean isFailureStoreEffectivelyEnabled(DataStreamFailureStoreSettings dataStreamFailureStoreSettings)
      Returns whether this data stream has its failure store enabled, either explicitly in its metadata or implicitly via settings.

      If the failure store is either explicitly enabled or explicitly disabled in its options metadata, that value is returned. If not, it checks whether its name matches one of the patterns in the settings, and that the data stream is not internal (i.e. neither a dot-prefixed nor a system data stream).

      Parameters:
      dataStreamFailureStoreSettings - The settings to use to determine whether the failure store should be implicitly enabled

    • isFailureStoreEffectivelyEnabled

      public static boolean isFailureStoreEffectivelyEnabled(@Nullable DataStreamOptions options, DataStreamFailureStoreSettings dataStreamFailureStoreSettings, String name, SystemIndices systemIndices)
      Returns whether a data stream has its failure store enabled, either explicitly in its metadata or implicitly via settings, based on the given parameters. The logic is equivalent to that in isFailureStoreEffectivelyEnabled(DataStreamFailureStoreSettings).
      Parameters:
      options - The DataStreamOptions for the data stream (which may be null)
      dataStreamFailureStoreSettings - The settings to use to determine whether the failure store should be implicitly enabled
      name - The name of the data stream
      systemIndices - The SystemIndices instance to use to determine whether this is a system data stream

    • getIndexMode

      @Nullable public IndexMode getIndexMode()

    • getLifecycle

      @Nullable public DataStreamLifecycle getLifecycle()

    • getAutoShardingEvent

      public DataStreamAutoShardingEvent getAutoShardingEvent()
      Returns the latest auto sharding event that happened for this data stream

    • getDataComponent

      public DataStream.DataStreamIndices getDataComponent()

    • getFailureComponent

      public DataStream.DataStreamIndices getFailureComponent()

    • getDataStreamIndices

      public DataStream.DataStreamIndices getDataStreamIndices(boolean failureStore)

    • rollover

      public DataStream rollover(Index writeIndex, long generation, IndexMode indexModeFromTemplate, @Nullable DataStreamAutoShardingEvent autoShardingEvent)
      Performs a rollover on a DataStream instance and returns a new instance containing the updated list of backing indices and incremented generation.
      Parameters:
      writeIndex - new write index
      generation - new generation
      indexModeFromTemplate - the index mode that originates from the template that created this data stream
      autoShardingEvent - the auto sharding event this rollover operation is applying
      Returns:
      new DataStream instance with the rollover operation applied

    • unsafeRollover

      public DataStream unsafeRollover(Index writeIndex, long generation, IndexMode indexModeFromTemplate, DataStreamAutoShardingEvent autoShardingEvent)
      Like rollover(Index, long, IndexMode, DataStreamAutoShardingEvent), but does no validation, use with care only.

    • rolloverFailureStore

      public DataStream rolloverFailureStore(Index writeIndex, long generation)
      Performs a rollover on the failure store of a DataStream instance and returns a new instance containing the updated list of failure store indices and incremented generation.
      Parameters:
      writeIndex - new failure store write index
      generation - new generation
      Returns:
      new DataStream instance with the rollover operation applied

    • unsafeRolloverFailureStore

      public DataStream unsafeRolloverFailureStore(Index writeIndex, long generation)
      Like rolloverFailureStore(Index, long), but does no validation, use with care only.

    • nextWriteIndexAndGeneration

      public Tuple<String,Long> nextWriteIndexAndGeneration(Metadata clusterMetadata, DataStream.DataStreamIndices dataStreamIndices)
      Generates the next write index name and generation to be used for rolling over this data stream.
      Parameters:
      clusterMetadata - Cluster metadata
      dataStreamIndices - The data stream indices that we're generating the next write index name and generation for
      Returns:
      tuple of the next write index name and next generation.

    • unsafeNextWriteIndexAndGeneration

      public Tuple<String,Long> unsafeNextWriteIndexAndGeneration(Metadata clusterMetadata, DataStream.DataStreamIndices dataStreamIndices)
      Like nextWriteIndexAndGeneration(Metadata, DataStreamIndices), but does no validation, use with care only.

    • removeBackingIndex

      public DataStream removeBackingIndex(Index index)
      Removes the specified backing index and returns a new DataStream instance with the remaining backing indices.
      Parameters:
      index - the backing index to remove
      Returns:
      new DataStream instance with the remaining backing indices
      Throws:
      IllegalArgumentException - if index is not a backing index or is the current write index of the data stream

    • removeFailureStoreIndex

      public DataStream removeFailureStoreIndex(Index index)
      Removes the specified failure store index and returns a new DataStream instance with the remaining failure store indices.
      Parameters:
      index - the failure store index to remove
      Returns:
      new DataStream instance with the remaining failure store indices
      Throws:
      IllegalArgumentException - if index is not a failure store index or is the current failure store write index of the data stream

    • replaceBackingIndex

      public DataStream replaceBackingIndex(Index existingBackingIndex, Index newBackingIndex)
      Replaces the specified backing index with a new index and returns a new DataStream instance with the modified backing indices. An IllegalArgumentException is thrown if the index to be replaced is not a backing index for this data stream or if it is the DataStream's write index.
      Parameters:
      existingBackingIndex - the backing index to be replaced
      newBackingIndex - the new index that will be part of the DataStream
      Returns:
      new DataStream instance with backing indices that contain replacement index instead of the specified existing index.

    • replaceFailureStoreIndex

      public DataStream replaceFailureStoreIndex(Index existingFailureIndex, Index newFailureIndex)
      Replaces the specified failure store index with a new index and returns a new DataStream instance with the modified backing indices. An IllegalArgumentException is thrown if the index to be replaced is not a failure store index for this data stream or if it is the DataStream's failure store write index.
      Parameters:
      existingFailureIndex - the failure store index to be replaced
      newFailureIndex - the new index that will be part of the DataStream
      Returns:
      new DataStream instance with failure store indices that contain replacement index instead of the specified existing index.

    • addBackingIndex

      public DataStream addBackingIndex(Metadata clusterMetadata, Index index)
      Adds the specified index as a backing index and returns a new DataStream instance with the new combination of backing indices.
      Parameters:
      index - index to add to the data stream
      Returns:
      new DataStream instance with the added backing index
      Throws:
      IllegalArgumentException - if index is ineligible to be a backing index for the data stream

    • addFailureStoreIndex

      public DataStream addFailureStoreIndex(Metadata clusterMetadata, Index index)
      Adds the specified index as a failure store index and returns a new DataStream instance with the new combination of failure store indices.
      Parameters:
      index - index to add to the data stream's failure store
      Returns:
      new DataStream instance with the added failure store index
      Throws:
      IllegalArgumentException - if index is ineligible to be a failure store index for the data stream

    • promoteDataStream

      public DataStream promoteDataStream()

    • snapshot

      @Nullable public DataStream snapshot(Set<String> indicesInSnapshot, Metadata.Builder snapshotMetadataBuilder)
      Reconciles this data stream with a list of indices available in a snapshot. Allows snapshots to store accurate data stream definitions that do not reference backing indices and failure indices not contained in the snapshot.
      Parameters:
      indicesInSnapshot - List of indices in the snapshot
      snapshotMetadataBuilder - a metadata builder with the current view of the snapshot metadata
      Returns:
      Reconciled DataStream instance or null if no reconciled version of this data stream could be built from the given indices

    • getIndicesPastRetention

      public List<Index> getIndicesPastRetention(Function<String,IndexMetadata> indexMetadataSupplier, LongSupplier nowSupplier, DataStreamGlobalRetention globalRetention)
      Iterate over the backing indices and return the ones that are managed by the data stream lifecycle and past the configured retention in their lifecycle. NOTE that this specifically does not return the write index of the data stream as usually retention is treated differently for the write index (i.e. they first need to be rolled over)

    • getDownsamplingRoundsFor

      public List<DataStreamLifecycle.Downsampling.Round> getDownsamplingRoundsFor(Index index, Function<String,IndexMetadata> indexMetadataSupplier, LongSupplier nowSupplier)
      Returns a list of downsampling rounds this index is eligible for (based on the rounds `after` configuration) or an empty list if this data streams' lifecycle doesn't have downsampling configured or the index's generation age doesn't yet match any `after` downsampling configuration. An empty list is returned for indices that are not time series.

    • getNonWriteIndicesOlderThan

      public List<Index> getNonWriteIndicesOlderThan(TimeValue retentionPeriod, Function<String,IndexMetadata> indexMetadataSupplier, @Nullable Predicate<IndexMetadata> indicesPredicate, LongSupplier nowSupplier)
      Returns the non-write backing indices and failure store indices that are older than the provided age, excluding the write indices. The index age is calculated from the rollover or index creation date (or the origination date if present). If an indices predicate is provided the returned list of indices will be filtered according to the predicate definition. This is useful for things like "return only the backing indices that are managed by the data stream lifecycle".

    • isIndexManagedByDataStreamLifecycle

      public boolean isIndexManagedByDataStreamLifecycle(Index index, Function<String,IndexMetadata> indexMetadataSupplier)
      Checks if the provided backing index is managed by the data stream lifecycle as part of this data stream. If the index is not a backing index or a failure store index of this data stream, or we cannot supply its metadata we return false.

    • getGenerationLifecycleDate

      @Nullable public TimeValue getGenerationLifecycleDate(IndexMetadata indexMetadata)
      Returns the generation date of the index whose metadata is passed. The generation date of the index represents the time at which the index started progressing towards the user configurable / business specific parts of the lifecycle (e.g. retention). The generation date is the origination date if it exists, or the rollover date if it exists and the origination date does not, or the creation date if neither the origination date nor the rollover date exist. If the index is the write index the generation date will be null because it is not eligible for retention or other parts of the lifecycle.
      Parameters:
      indexMetadata - The metadata of the index whose generation date is returned
      Returns:
      The generation date of the index, or null if this is the write index

    • getDefaultBackingIndexName

      public static String getDefaultBackingIndexName(String dataStreamName, long generation)
      Generates the name of the index that conforms to the default naming convention for backing indices on data streams given the specified data stream name and generation and the current system time.
      Parameters:
      dataStreamName - name of the data stream
      generation - generation of the data stream
      Returns:
      backing index name

    • getDefaultBackingIndexName

      public static String getDefaultBackingIndexName(String dataStreamName, long generation, long epochMillis)
      Generates the name of the index that conforms to the default naming convention for backing indices on data streams given the specified data stream name, generation, and time.
      Parameters:
      dataStreamName - name of the data stream
      generation - generation of the data stream
      epochMillis - creation time for the backing index
      Returns:
      backing index name

    • getDefaultFailureStoreName

      public static String getDefaultFailureStoreName(String dataStreamName, long generation, long epochMillis)
      Generates the name of the index that conforms to the default naming convention for backing indices on data streams given the specified data stream name, generation, and time.
      Parameters:
      dataStreamName - name of the data stream
      generation - generation of the data stream
      epochMillis - creation time for the backing index
      Returns:
      backing index name

    • readDiffFrom

      public static Diff<DataStream> readDiffFrom(StreamInput in) throws IOException
      Throws:
      IOException

    • writeTo

      public void writeTo(StreamOutput out) throws IOException
      Description copied from interface: Writeable
      Write this into the StreamOutput.
      Specified by:
      writeTo in interface Writeable
      Throws:
      IOException

    • fromXContent

      public static DataStream fromXContent(XContentParser parser) throws IOException
      Throws:
      IOException

    • toXContent

      public XContentBuilder toXContent(XContentBuilder builder, ToXContent.Params params) throws IOException
      Specified by:
      toXContent in interface ToXContent
      Throws:
      IOException

    • toXContent

      public XContentBuilder toXContent(XContentBuilder builder, ToXContent.Params params, @Nullable RolloverConfiguration rolloverConfiguration, @Nullable DataStreamGlobalRetention globalRetention) throws IOException
      Converts the data stream to XContent and passes the RolloverConditions, when provided, to the lifecycle.
      Throws:
      IOException

    • equals

      public boolean equals(Object o)
      Overrides:
      equals in class Object

    • hashCode

      public int hashCode()
      Overrides:
      hashCode in class Object

    • getWriteIndex

      public Index getWriteIndex(IndexRequest request, Metadata metadata)
      Specified by:
      getWriteIndex in interface IndexAbstraction

    • getParentDataStream

      public DataStream getParentDataStream()
      Specified by:
      getParentDataStream in interface IndexAbstraction
      Returns:
      the data stream to which this index belongs or null if this is not a concrete index or if it is a concrete index that does not belong to a data stream.

    • getIndicesWithinMaxAgeRange

      public static List<Index> getIndicesWithinMaxAgeRange(DataStream dataStream, Function<Index,IndexMetadata> indexProvider, TimeValue maxIndexAge, LongSupplier nowSupplier)
      Returns the indices created within the interval. Note that this strives to cover the entire interval so one backing index created before the specified age will also be return.

    • resolveDataStream

      public static DataStream resolveDataStream(IndexAbstraction indexAbstraction, Metadata metadata)
      Resolve the index abstraction to a data stream. This handles alias resolution as well as data stream resolution. This does NOT resolve a data stream by providing a concrete backing index.

    • getCanonicalTimestampBound

      public static Instant getCanonicalTimestampBound(Instant time)
      Modifies the passed Instant object to be used as a bound for a timestamp field in TimeSeries. It needs to be called in both backing index construction (rollover) and index selection for doc insertion. Failure to do so may lead to errors due to document timestamps exceeding the end time of the selected backing index for insertion.
      Parameters:
      time - The initial Instant object that's used to generate the canonical time
      Returns:
      A canonical Instant object to be used as a timestamp bound

    • builder

      public static DataStream.Builder builder(String name, List<Index> indices)

    • builder

      public static DataStream.Builder builder(String name, DataStream.DataStreamIndices backingIndices)

    • copy

      public DataStream.Builder copy()