Module org.elasticsearch.server
Package org.elasticsearch.cluster.metadata

Class DataStream

java.lang.Object
org.elasticsearch.cluster.metadata.DataStream
All Implemented Interfaces:
Diffable<DataStream>, IndexAbstraction, SimpleDiffable<DataStream>, Writeable, ToXContent, ToXContentObject
public final class DataStream extends Object implements SimpleDiffable<DataStream>, ToXContentObject, IndexAbstraction

  • Field Details

    • FAILURE_STORE_FEATURE_FLAG

      public static final FeatureFlag FAILURE_STORE_FEATURE_FLAG

    • ADDED_FAILURE_STORE_TRANSPORT_VERSION

      public static final TransportVersion ADDED_FAILURE_STORE_TRANSPORT_VERSION

    • BACKING_INDEX_PREFIX

      public static final String BACKING_INDEX_PREFIX
      See Also:

    • FAILURE_STORE_PREFIX

      public static final String FAILURE_STORE_PREFIX
      See Also:

    • DATE_FORMATTER

      public static final DateFormatter DATE_FORMATTER

    • TIMESTAMP_FIELD_NAME

      public static final String TIMESTAMP_FIELD_NAME
      See Also:

    • TIMESERIES_LEAF_READERS_SORTER

      public static Comparator<org.apache.lucene.index.LeafReader> TIMESERIES_LEAF_READERS_SORTER

    • NAME_FIELD

      public static final ParseField NAME_FIELD

    • TIMESTAMP_FIELD_FIELD

      public static final ParseField TIMESTAMP_FIELD_FIELD

    • INDICES_FIELD

      public static final ParseField INDICES_FIELD

    • GENERATION_FIELD

      public static final ParseField GENERATION_FIELD

    • METADATA_FIELD

      public static final ParseField METADATA_FIELD

    • HIDDEN_FIELD

      public static final ParseField HIDDEN_FIELD

    • REPLICATED_FIELD

      public static final ParseField REPLICATED_FIELD

    • SYSTEM_FIELD

      public static final ParseField SYSTEM_FIELD

    • ALLOW_CUSTOM_ROUTING

      public static final ParseField ALLOW_CUSTOM_ROUTING

    • INDEX_MODE

      public static final ParseField INDEX_MODE

    • LIFECYCLE

      public static final ParseField LIFECYCLE

    • FAILURE_STORE_FIELD

      public static final ParseField FAILURE_STORE_FIELD

    • FAILURE_INDICES_FIELD

      public static final ParseField FAILURE_INDICES_FIELD

    • ROLLOVER_ON_WRITE_FIELD

      public static final ParseField ROLLOVER_ON_WRITE_FIELD

    • TS_EXTRACT_CONFIG

      public static final XContentParserConfiguration TS_EXTRACT_CONFIG

  • Constructor Details

  • Method Details

    • isFailureStoreEnabled

      public static boolean isFailureStoreEnabled()

    • getType

      public IndexAbstraction.Type getType()
      Specified by:
      getType in interface IndexAbstraction
      Returns:
      the type of the index abstraction

    • getName

      public String getName()
      Specified by:
      getName in interface IndexAbstraction
      Returns:
      the name of the index abstraction

    • isDataStreamRelated

      public boolean isDataStreamRelated()
      Specified by:
      isDataStreamRelated in interface IndexAbstraction
      Returns:
      whether this index abstraction is related to data streams

    • getIndices

      public List<Index> getIndices()
      Specified by:
      getIndices in interface IndexAbstraction
      Returns:
      All Index of all concrete indices this index abstraction is referring to.

    • getGeneration

      public long getGeneration()

    • getFailureIndices

      public List<Index> getFailureIndices()

    • getWriteIndex

      public Index getWriteIndex()
      Description copied from interface: IndexAbstraction
      A write index is a dedicated concrete index, that accepts all the new documents that belong to an index abstraction.

      A write index may also be a regular concrete index of a index abstraction and may therefore also be returned by IndexAbstraction.getIndices(). An index abstraction may also not have a dedicated write index.

      Specified by:
      getWriteIndex in interface IndexAbstraction
      Returns:
      the write index of this index abstraction or null if this index abstraction doesn't have a write index.

    • rolloverOnWrite

      public boolean rolloverOnWrite()

    • selectTimeSeriesWriteIndex

      public Index selectTimeSeriesWriteIndex(Instant timestamp, Metadata metadata)
      Parameters:
      timestamp - The timestamp used to select a backing index based on its start and end time.
      metadata - The metadata that is used to fetch the start and end times for backing indices of this data stream.
      Returns:
      a backing index with a start time that is greater or equal to the provided timestamp and an end time that is less than the provided timestamp. Otherwise null is returned.

    • validate

      public void validate(Function<String,IndexMetadata> imSupplier)
      Validates this data stream. If this is a time series data stream then this method validates that temporal range of backing indices (defined by index.time_series.start_time and index.time_series.end_time) do not overlap with each other.
      Parameters:
      imSupplier - Function that supplies IndexMetadata instances based on the provided index name

    • getMetadata

      @Nullable public Map<String,Object> getMetadata()

    • isHidden

      public boolean isHidden()
      Specified by:
      isHidden in interface IndexAbstraction
      Returns:
      whether this index abstraction is hidden or not

    • isReplicated

      public boolean isReplicated()
      Determines whether this data stream is replicated from elsewhere, for example a remote cluster
      Returns:
      Whether this data stream is replicated.

    • isSystem

      public boolean isSystem()
      Specified by:
      isSystem in interface IndexAbstraction
      Returns:
      whether this index abstraction should be treated as a system index or not

    • isAllowCustomRouting

      public boolean isAllowCustomRouting()

    • isFailureStore

      public boolean isFailureStore()
      Determines if this data stream should persist ingest pipeline and mapping failures from bulk requests to a locally configured failure store.
      Returns:
      Whether this data stream should store ingestion failures.

    • getIndexMode

      @Nullable public IndexMode getIndexMode()

    • getLifecycle

      @Nullable public DataStreamLifecycle getLifecycle()

    • rollover

      public DataStream rollover(Index writeIndex, long generation, boolean timeSeries)
      Performs a rollover on a DataStream instance and returns a new instance containing the updated list of backing indices and incremented generation.
      Parameters:
      writeIndex - new write index
      generation - new generation
      timeSeries - whether the template that created this data stream is in time series mode
      Returns:
      new DataStream instance with the rollover operation applied

    • unsafeRollover

      public DataStream unsafeRollover(Index writeIndex, long generation, boolean timeSeries)
      Like rollover(Index, long, boolean), but does no validation, use with care only.

    • nextWriteIndexAndGeneration

      public Tuple<String,Long> nextWriteIndexAndGeneration(Metadata clusterMetadata)
      Performs a dummy rollover on a DataStream instance and returns the tuple of the next write index name and next generation that this DataStream should roll over to using rollover(Index, long, boolean).
      Parameters:
      clusterMetadata - Cluster metadata
      Returns:
      new DataStream instance with the dummy rollover operation applied

    • unsafeNextWriteIndexAndGeneration

      public Tuple<String,Long> unsafeNextWriteIndexAndGeneration(Metadata clusterMetadata)
      Like nextWriteIndexAndGeneration(Metadata), but does no validation, use with care only.

    • removeBackingIndex

      public DataStream removeBackingIndex(Index index)
      Removes the specified backing index and returns a new DataStream instance with the remaining backing indices.
      Parameters:
      index - the backing index to remove
      Returns:
      new DataStream instance with the remaining backing indices
      Throws:
      IllegalArgumentException - if index is not a backing index or is the current write index of the data stream

    • replaceBackingIndex

      public DataStream replaceBackingIndex(Index existingBackingIndex, Index newBackingIndex)
      Replaces the specified backing index with a new index and returns a new DataStream instance with the modified backing indices. An IllegalArgumentException is thrown if the index to be replaced is not a backing index for this data stream or if it is the DataStream's write index.
      Parameters:
      existingBackingIndex - the backing index to be replaced
      newBackingIndex - the new index that will be part of the DataStream
      Returns:
      new DataStream instance with backing indices that contain replacement index instead of the specified existing index.

    • addBackingIndex

      public DataStream addBackingIndex(Metadata clusterMetadata, Index index)
      Adds the specified index as a backing index and returns a new DataStream instance with the new combination of backing indices.
      Parameters:
      index - index to add to the data stream
      Returns:
      new DataStream instance with the added backing index
      Throws:
      IllegalArgumentException - if index is ineligible to be a backing index for the data stream

    • promoteDataStream

      public DataStream promoteDataStream()

    • snapshot

      @Nullable public DataStream snapshot(Collection<String> indicesInSnapshot)
      Reconciles this data stream with a list of indices available in a snapshot. Allows snapshots to store accurate data stream definitions that do not reference backing indices not contained in the snapshot.
      Parameters:
      indicesInSnapshot - List of indices in the snapshot
      Returns:
      Reconciled DataStream instance or null if no reconciled version of this data stream could be built from the given indices

    • getIndicesPastRetention

      public List<Index> getIndicesPastRetention(Function<String,IndexMetadata> indexMetadataSupplier, LongSupplier nowSupplier)
      Iterate over the backing indices and return the ones that are managed by the data stream lifecycle and past the configured retention in their lifecycle. NOTE that this specifically does not return the write index of the data stream as usually retention is treated differently for the write index (i.e. they first need to be rolled over)

    • getDownsamplingRoundsFor

      public List<DataStreamLifecycle.Downsampling.Round> getDownsamplingRoundsFor(Index index, Function<String,IndexMetadata> indexMetadataSupplier, LongSupplier nowSupplier)
      Returns a list of downsampling rounds this index is eligible for (based on the rounds `after` configuration) or an empty list if this data streams' lifecycle doesn't have downsampling configured or the index's generation age doesn't yet match any `after` downsampling configuration. An empty list is returned for indices that are not time series.

    • getNonWriteIndicesOlderThan

      public List<Index> getNonWriteIndicesOlderThan(TimeValue age, Function<String,IndexMetadata> indexMetadataSupplier, @Nullable Predicate<IndexMetadata> indicesPredicate, LongSupplier nowSupplier)
      Returns the non-write backing indices that are older than the provided age, *excluding the write index*. The index age is calculated from the rollover or index creation date (or the origination date if present). If an indices predicate is provided the returned list of indices will be filtered according to the predicate definition. This is useful for things like "return only the backing indices that are managed by the data stream lifecycle".

    • isIndexManagedByDataStreamLifecycle

      public boolean isIndexManagedByDataStreamLifecycle(Index index, Function<String,IndexMetadata> indexMetadataSupplier)
      Checks if the provided backing index is managed by the data stream lifecycle as part of this data stream. If the index is not a backing index of this data stream, or we cannot supply its metadata we return false.

    • getGenerationLifecycleDate

      @Nullable public TimeValue getGenerationLifecycleDate(IndexMetadata indexMetadata)
      Returns the generation date of the index whose metadata is passed. The generation date of the index represents the time at which the index started progressing towards the user configurable / business specific parts of the lifecycle (e.g. retention). The generation date is the origination date if it exists, or the rollover date if it exists and the origination date does not, or the creation date if neither the origination date nor the rollover date exist. If the index is the write index the generation date will be null because it is not eligible for retention or other parts of the lifecycle.
      Parameters:
      indexMetadata - The metadata of the index whose generation date is returned
      Returns:
      The generation date of the index, or null if this is the write index

    • getDefaultBackingIndexName

      public static String getDefaultBackingIndexName(String dataStreamName, long generation)
      Generates the name of the index that conforms to the default naming convention for backing indices on data streams given the specified data stream name and generation and the current system time.
      Parameters:
      dataStreamName - name of the data stream
      generation - generation of the data stream
      Returns:
      backing index name

    • getDefaultBackingIndexName

      public static String getDefaultBackingIndexName(String dataStreamName, long generation, long epochMillis)
      Generates the name of the index that conforms to the default naming convention for backing indices on data streams given the specified data stream name, generation, and time.
      Parameters:
      dataStreamName - name of the data stream
      generation - generation of the data stream
      epochMillis - creation time for the backing index
      Returns:
      backing index name

    • getDefaultFailureStoreName

      public static String getDefaultFailureStoreName(String dataStreamName, long generation, long epochMillis)
      Generates the name of the index that conforms to the default naming convention for backing indices on data streams given the specified data stream name, generation, and time.
      Parameters:
      dataStreamName - name of the data stream
      generation - generation of the data stream
      epochMillis - creation time for the backing index
      Returns:
      backing index name

    • readDiffFrom

      public static Diff<DataStream> readDiffFrom(StreamInput in) throws IOException
      Throws:
      IOException

    • writeTo

      public void writeTo(StreamOutput out) throws IOException
      Description copied from interface: Writeable
      Write this into the StreamOutput.
      Specified by:
      writeTo in interface Writeable
      Throws:
      IOException

    • fromXContent

      public static DataStream fromXContent(XContentParser parser) throws IOException
      Throws:
      IOException

    • toXContent

      public XContentBuilder toXContent(XContentBuilder builder, ToXContent.Params params) throws IOException
      Specified by:
      toXContent in interface ToXContent
      Throws:
      IOException

    • toXContent

      public XContentBuilder toXContent(XContentBuilder builder, ToXContent.Params params, @Nullable RolloverConfiguration rolloverConfiguration) throws IOException
      Converts the data stream to XContent and passes the RolloverConditions, when provided, to the lifecycle.
      Throws:
      IOException

    • equals

      public boolean equals(Object o)
      Overrides:
      equals in class Object

    • hashCode

      public int hashCode()
      Overrides:
      hashCode in class Object

    • getWriteIndex

      public Index getWriteIndex(IndexRequest request, Metadata metadata)
      Specified by:
      getWriteIndex in interface IndexAbstraction

    • getParentDataStream

      public DataStream getParentDataStream()
      Specified by:
      getParentDataStream in interface IndexAbstraction
      Returns:
      the data stream to which this index belongs or null if this is not a concrete index or if it is a concrete index that does not belong to a data stream.

    • getCanonicalTimestampBound

      public static Instant getCanonicalTimestampBound(Instant time)
      Modifies the passed Instant object to be used as a bound for a timestamp field in TimeSeries. It needs to be called in both backing index construction (rollover) and index selection for doc insertion. Failure to do so may lead to errors due to document timestamps exceeding the end time of the selected backing index for insertion.
      Parameters:
      time - The initial Instant object that's used to generate the canonical time
      Returns:
      A canonical Instant object to be used as a timestamp bound