java.lang.Object
org.elasticsearch.cluster.metadata.DataStream
- All Implemented Interfaces:
Diffable<DataStream>
,IndexAbstraction
,SimpleDiffable<DataStream>
,Writeable
,ToXContent
,ToXContentObject
public final class DataStream
extends Object
implements SimpleDiffable<DataStream>, ToXContentObject, IndexAbstraction
-
Nested Class Summary
Nested classes/interfaces inherited from interface org.elasticsearch.cluster.metadata.IndexAbstraction
IndexAbstraction.Alias, IndexAbstraction.ConcreteIndex, IndexAbstraction.Type
Nested classes/interfaces inherited from interface org.elasticsearch.cluster.SimpleDiffable
SimpleDiffable.CompleteDiff<T extends Diffable<T>>
Nested classes/interfaces inherited from interface org.elasticsearch.xcontent.ToXContent
ToXContent.DelegatingMapParams, ToXContent.MapParams, ToXContent.Params
Nested classes/interfaces inherited from interface org.elasticsearch.common.io.stream.Writeable
Writeable.Reader<V>, Writeable.Writer<V>
-
Field Summary
FieldsModifier and TypeFieldDescriptionstatic final TransportVersion
static final ParseField
static final String
static final DateFormatter
static final ParseField
static final FeatureFlag
static final ParseField
static final String
static final ParseField
static final ParseField
static final ParseField
static final ParseField
static final ParseField
static final ParseField
static final ParseField
static final ParseField
static final ParseField
static final ParseField
static Comparator<org.apache.lucene.index.LeafReader>
static final ParseField
static final String
static final XContentParserConfiguration
Fields inherited from interface org.elasticsearch.cluster.SimpleDiffable
EMPTY
Fields inherited from interface org.elasticsearch.xcontent.ToXContent
EMPTY, EMPTY_PARAMS
-
Constructor Summary
ConstructorsConstructorDescriptionDataStream
(String name, List<Index> indices, long generation, Map<String, Object> metadata, boolean hidden, boolean replicated, boolean system, boolean allowCustomRouting, IndexMode indexMode) DataStream
(String name, List<Index> indices, long generation, Map<String, Object> metadata, boolean hidden, boolean replicated, boolean system, boolean allowCustomRouting, IndexMode indexMode, DataStreamLifecycle lifecycle, boolean failureStore, List<Index> failureIndices) DataStream
(String name, List<Index> indices, long generation, Map<String, Object> metadata, boolean hidden, boolean replicated, boolean system, boolean allowCustomRouting, IndexMode indexMode, DataStreamLifecycle lifecycle, boolean failureStore, List<Index> failureIndices, boolean rolloverOnWrite) -
Method Summary
Modifier and TypeMethodDescriptionaddBackingIndex
(Metadata clusterMetadata, Index index) Adds the specified index as a backing index and returns a newDataStream
instance with the new combination of backing indices.boolean
static DataStream
fromXContent
(XContentParser parser) static Instant
Modifies the passed Instant object to be used as a bound for a timestamp field in TimeSeries.static String
getDefaultBackingIndexName
(String dataStreamName, long generation) Generates the name of the index that conforms to the default naming convention for backing indices on data streams given the specified data stream name and generation and the current system time.static String
getDefaultBackingIndexName
(String dataStreamName, long generation, long epochMillis) Generates the name of the index that conforms to the default naming convention for backing indices on data streams given the specified data stream name, generation, and time.static String
getDefaultFailureStoreName
(String dataStreamName, long generation, long epochMillis) Generates the name of the index that conforms to the default naming convention for backing indices on data streams given the specified data stream name, generation, and time.getDownsamplingRoundsFor
(Index index, Function<String, IndexMetadata> indexMetadataSupplier, LongSupplier nowSupplier) Returns a list of downsampling rounds this index is eligible for (based on the rounds `after` configuration) or an empty list if this data streams' lifecycle doesn't have downsampling configured or the index's generation age doesn't yet match any `after` downsampling configuration.long
getGenerationLifecycleDate
(IndexMetadata indexMetadata) Returns the generation date of the index whose metadata is passed.getIndicesPastRetention
(Function<String, IndexMetadata> indexMetadataSupplier, LongSupplier nowSupplier) Iterate over the backing indices and return the ones that are managed by the data stream lifecycle and past the configured retention in their lifecycle.getName()
getNonWriteIndicesOlderThan
(TimeValue age, Function<String, IndexMetadata> indexMetadataSupplier, Predicate<IndexMetadata> indicesPredicate, LongSupplier nowSupplier) Returns the non-write backing indices that are older than the provided age, *excluding the write index*.getType()
A write index is a dedicated concrete index, that accepts all the new documents that belong to an index abstraction.getWriteIndex
(IndexRequest request, Metadata metadata) int
hashCode()
boolean
boolean
boolean
Determines if this data stream should persist ingest pipeline and mapping failures from bulk requests to a locally configured failure store.static boolean
boolean
isHidden()
boolean
isIndexManagedByDataStreamLifecycle
(Index index, Function<String, IndexMetadata> indexMetadataSupplier) Checks if the provided backing index is managed by the data stream lifecycle as part of this data stream.boolean
Determines whether this data stream is replicated from elsewhere, for example a remote clusterboolean
isSystem()
nextWriteIndexAndGeneration
(Metadata clusterMetadata) Performs a dummy rollover on aDataStream
instance and returns the tuple of the next write index name and next generation that thisDataStream
should roll over to usingrollover(Index, long, boolean)
.static Diff<DataStream>
removeBackingIndex
(Index index) Removes the specified backing index and returns a newDataStream
instance with the remaining backing indices.replaceBackingIndex
(Index existingBackingIndex, Index newBackingIndex) Replaces the specified backing index with a new index and returns a newDataStream
instance with the modified backing indices.Performs a rollover on aDataStream
instance and returns a new instance containing the updated list of backing indices and incremented generation.boolean
selectTimeSeriesWriteIndex
(Instant timestamp, Metadata metadata) snapshot
(Collection<String> indicesInSnapshot) Reconciles this data stream with a list of indices available in a snapshot.toXContent
(XContentBuilder builder, ToXContent.Params params) toXContent
(XContentBuilder builder, ToXContent.Params params, RolloverConfiguration rolloverConfiguration) Converts the data stream to XContent and passes the RolloverConditions, when provided, to the lifecycle.unsafeNextWriteIndexAndGeneration
(Metadata clusterMetadata) LikenextWriteIndexAndGeneration(Metadata)
, but does no validation, use with care only.unsafeRollover
(Index writeIndex, long generation, boolean timeSeries) Likerollover(Index, long, boolean)
, but does no validation, use with care only.void
validate
(Function<String, IndexMetadata> imSupplier) Validates this data stream.void
writeTo
(StreamOutput out) Write this into the StreamOutput.Methods inherited from class java.lang.Object
clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait
Methods inherited from interface org.elasticsearch.cluster.SimpleDiffable
diff
Methods inherited from interface org.elasticsearch.xcontent.ToXContentObject
isFragment
-
Field Details
-
FAILURE_STORE_FEATURE_FLAG
-
ADDED_FAILURE_STORE_TRANSPORT_VERSION
-
BACKING_INDEX_PREFIX
- See Also:
-
FAILURE_STORE_PREFIX
- See Also:
-
DATE_FORMATTER
-
TIMESTAMP_FIELD_NAME
- See Also:
-
TIMESERIES_LEAF_READERS_SORTER
-
NAME_FIELD
-
TIMESTAMP_FIELD_FIELD
-
INDICES_FIELD
-
GENERATION_FIELD
-
METADATA_FIELD
-
HIDDEN_FIELD
-
REPLICATED_FIELD
-
SYSTEM_FIELD
-
ALLOW_CUSTOM_ROUTING
-
INDEX_MODE
-
LIFECYCLE
-
FAILURE_STORE_FIELD
-
FAILURE_INDICES_FIELD
-
ROLLOVER_ON_WRITE_FIELD
-
TS_EXTRACT_CONFIG
-
-
Constructor Details
-
DataStream
-
DataStream
public DataStream(String name, List<Index> indices, long generation, Map<String, Object> metadata, boolean hidden, boolean replicated, boolean system, boolean allowCustomRouting, IndexMode indexMode, DataStreamLifecycle lifecycle, boolean failureStore, List<Index> failureIndices, boolean rolloverOnWrite) -
DataStream
-
DataStream
- Throws:
IOException
-
-
Method Details
-
isFailureStoreEnabled
public static boolean isFailureStoreEnabled() -
getType
- Specified by:
getType
in interfaceIndexAbstraction
- Returns:
- the type of the index abstraction
-
getName
- Specified by:
getName
in interfaceIndexAbstraction
- Returns:
- the name of the index abstraction
-
isDataStreamRelated
public boolean isDataStreamRelated()- Specified by:
isDataStreamRelated
in interfaceIndexAbstraction
- Returns:
- whether this index abstraction is related to data streams
-
getIndices
- Specified by:
getIndices
in interfaceIndexAbstraction
- Returns:
- All
Index
of all concrete indices this index abstraction is referring to.
-
getGeneration
public long getGeneration() -
getFailureIndices
-
getWriteIndex
Description copied from interface:IndexAbstraction
A write index is a dedicated concrete index, that accepts all the new documents that belong to an index abstraction.A write index may also be a regular concrete index of a index abstraction and may therefore also be returned by
IndexAbstraction.getIndices()
. An index abstraction may also not have a dedicated write index.- Specified by:
getWriteIndex
in interfaceIndexAbstraction
- Returns:
- the write index of this index abstraction or
null
if this index abstraction doesn't have a write index.
-
rolloverOnWrite
public boolean rolloverOnWrite() -
selectTimeSeriesWriteIndex
- Parameters:
timestamp
- The timestamp used to select a backing index based on its start and end time.metadata
- The metadata that is used to fetch the start and end times for backing indices of this data stream.- Returns:
- a backing index with a start time that is greater or equal to the provided timestamp and
an end time that is less than the provided timestamp. Otherwise
null
is returned.
-
validate
Validates this data stream. If this is a time series data stream then this method validates that temporal range of backing indices (defined by index.time_series.start_time and index.time_series.end_time) do not overlap with each other.- Parameters:
imSupplier
- Function that suppliesIndexMetadata
instances based on the provided index name
-
getMetadata
-
isHidden
public boolean isHidden()- Specified by:
isHidden
in interfaceIndexAbstraction
- Returns:
- whether this index abstraction is hidden or not
-
isReplicated
public boolean isReplicated()Determines whether this data stream is replicated from elsewhere, for example a remote cluster- Returns:
- Whether this data stream is replicated.
-
isSystem
public boolean isSystem()- Specified by:
isSystem
in interfaceIndexAbstraction
- Returns:
- whether this index abstraction should be treated as a system index or not
-
isAllowCustomRouting
public boolean isAllowCustomRouting() -
isFailureStore
public boolean isFailureStore()Determines if this data stream should persist ingest pipeline and mapping failures from bulk requests to a locally configured failure store.- Returns:
- Whether this data stream should store ingestion failures.
-
getIndexMode
-
getLifecycle
-
rollover
Performs a rollover on aDataStream
instance and returns a new instance containing the updated list of backing indices and incremented generation.- Parameters:
writeIndex
- new write indexgeneration
- new generationtimeSeries
- whether the template that created this data stream is in time series mode- Returns:
- new
DataStream
instance with the rollover operation applied
-
unsafeRollover
Likerollover(Index, long, boolean)
, but does no validation, use with care only. -
nextWriteIndexAndGeneration
Performs a dummy rollover on aDataStream
instance and returns the tuple of the next write index name and next generation that thisDataStream
should roll over to usingrollover(Index, long, boolean)
.- Parameters:
clusterMetadata
- Cluster metadata- Returns:
- new
DataStream
instance with the dummy rollover operation applied
-
unsafeNextWriteIndexAndGeneration
LikenextWriteIndexAndGeneration(Metadata)
, but does no validation, use with care only. -
removeBackingIndex
Removes the specified backing index and returns a newDataStream
instance with the remaining backing indices.- Parameters:
index
- the backing index to remove- Returns:
- new
DataStream
instance with the remaining backing indices - Throws:
IllegalArgumentException
- ifindex
is not a backing index or is the current write index of the data stream
-
replaceBackingIndex
Replaces the specified backing index with a new index and returns a newDataStream
instance with the modified backing indices. AnIllegalArgumentException
is thrown if the index to be replaced is not a backing index for this data stream or if it is theDataStream
's write index.- Parameters:
existingBackingIndex
- the backing index to be replacednewBackingIndex
- the new index that will be part of theDataStream
- Returns:
- new
DataStream
instance with backing indices that contain replacement index instead of the specified existing index.
-
addBackingIndex
Adds the specified index as a backing index and returns a newDataStream
instance with the new combination of backing indices.- Parameters:
index
- index to add to the data stream- Returns:
- new
DataStream
instance with the added backing index - Throws:
IllegalArgumentException
- ifindex
is ineligible to be a backing index for the data stream
-
promoteDataStream
-
snapshot
Reconciles this data stream with a list of indices available in a snapshot. Allows snapshots to store accurate data stream definitions that do not reference backing indices not contained in the snapshot.- Parameters:
indicesInSnapshot
- List of indices in the snapshot- Returns:
- Reconciled
DataStream
instance ornull
if no reconciled version of this data stream could be built from the given indices
-
getIndicesPastRetention
public List<Index> getIndicesPastRetention(Function<String, IndexMetadata> indexMetadataSupplier, LongSupplier nowSupplier) Iterate over the backing indices and return the ones that are managed by the data stream lifecycle and past the configured retention in their lifecycle. NOTE that this specifically does not return the write index of the data stream as usually retention is treated differently for the write index (i.e. they first need to be rolled over) -
getDownsamplingRoundsFor
public List<DataStreamLifecycle.Downsampling.Round> getDownsamplingRoundsFor(Index index, Function<String, IndexMetadata> indexMetadataSupplier, LongSupplier nowSupplier) Returns a list of downsampling rounds this index is eligible for (based on the rounds `after` configuration) or an empty list if this data streams' lifecycle doesn't have downsampling configured or the index's generation age doesn't yet match any `after` downsampling configuration. An empty list is returned for indices that are not time series. -
getNonWriteIndicesOlderThan
public List<Index> getNonWriteIndicesOlderThan(TimeValue age, Function<String, IndexMetadata> indexMetadataSupplier, @Nullable Predicate<IndexMetadata> indicesPredicate, LongSupplier nowSupplier) Returns the non-write backing indices that are older than the provided age, *excluding the write index*. The index age is calculated from the rollover or index creation date (or the origination date if present). If an indices predicate is provided the returned list of indices will be filtered according to the predicate definition. This is useful for things like "return only the backing indices that are managed by the data stream lifecycle". -
isIndexManagedByDataStreamLifecycle
public boolean isIndexManagedByDataStreamLifecycle(Index index, Function<String, IndexMetadata> indexMetadataSupplier) Checks if the provided backing index is managed by the data stream lifecycle as part of this data stream. If the index is not a backing index of this data stream, or we cannot supply its metadata we return false. -
getGenerationLifecycleDate
Returns the generation date of the index whose metadata is passed. The generation date of the index represents the time at which the index started progressing towards the user configurable / business specific parts of the lifecycle (e.g. retention). The generation date is the origination date if it exists, or the rollover date if it exists and the origination date does not, or the creation date if neither the origination date nor the rollover date exist. If the index is the write index the generation date will be null because it is not eligible for retention or other parts of the lifecycle.- Parameters:
indexMetadata
- The metadata of the index whose generation date is returned- Returns:
- The generation date of the index, or null if this is the write index
-
getDefaultBackingIndexName
Generates the name of the index that conforms to the default naming convention for backing indices on data streams given the specified data stream name and generation and the current system time.- Parameters:
dataStreamName
- name of the data streamgeneration
- generation of the data stream- Returns:
- backing index name
-
getDefaultBackingIndexName
public static String getDefaultBackingIndexName(String dataStreamName, long generation, long epochMillis) Generates the name of the index that conforms to the default naming convention for backing indices on data streams given the specified data stream name, generation, and time.- Parameters:
dataStreamName
- name of the data streamgeneration
- generation of the data streamepochMillis
- creation time for the backing index- Returns:
- backing index name
-
getDefaultFailureStoreName
public static String getDefaultFailureStoreName(String dataStreamName, long generation, long epochMillis) Generates the name of the index that conforms to the default naming convention for backing indices on data streams given the specified data stream name, generation, and time.- Parameters:
dataStreamName
- name of the data streamgeneration
- generation of the data streamepochMillis
- creation time for the backing index- Returns:
- backing index name
-
readDiffFrom
- Throws:
IOException
-
writeTo
Description copied from interface:Writeable
Write this into the StreamOutput.- Specified by:
writeTo
in interfaceWriteable
- Throws:
IOException
-
fromXContent
- Throws:
IOException
-
toXContent
public XContentBuilder toXContent(XContentBuilder builder, ToXContent.Params params) throws IOException - Specified by:
toXContent
in interfaceToXContent
- Throws:
IOException
-
toXContent
public XContentBuilder toXContent(XContentBuilder builder, ToXContent.Params params, @Nullable RolloverConfiguration rolloverConfiguration) throws IOException Converts the data stream to XContent and passes the RolloverConditions, when provided, to the lifecycle.- Throws:
IOException
-
equals
-
hashCode
public int hashCode() -
getWriteIndex
- Specified by:
getWriteIndex
in interfaceIndexAbstraction
-
getParentDataStream
- Specified by:
getParentDataStream
in interfaceIndexAbstraction
- Returns:
- the data stream to which this index belongs or
null
if this is not a concrete index or if it is a concrete index that does not belong to a data stream.
-
getCanonicalTimestampBound
Modifies the passed Instant object to be used as a bound for a timestamp field in TimeSeries. It needs to be called in both backing index construction (rollover) and index selection for doc insertion. Failure to do so may lead to errors due to document timestamps exceeding the end time of the selected backing index for insertion.- Parameters:
time
- The initial Instant object that's used to generate the canonical time- Returns:
- A canonical Instant object to be used as a timestamp bound
-