Module org.elasticsearch.server

Package org.elasticsearch.action.datastreams.autosharding

Class DataStreamAutoShardingService

java.lang.Object

org.elasticsearch.action.datastreams.autosharding.DataStreamAutoShardingService

public class DataStreamAutoShardingService
extends Object

Calculates the optimal number of shards the data stream write index should have based on the indexing load.

Field Summary

Fields

Modifier and Type	Field	Description
static final Setting<Integer>	CLUSTER_AUTO_SHARDING_MAX_WRITE_THREADS	Represents the maximum number of write threads we expect a node to have in the environments where auto sharding will be enabled.
static final Setting<Integer>	CLUSTER_AUTO_SHARDING_MIN_WRITE_THREADS	Represents the minimum number of write threads we expect a node to have in the environments where auto sharding will be enabled.
static final NodeFeature	DATA_STREAM_AUTO_SHARDING_FEATURE
static final Setting<TimeValue>	DATA_STREAMS_AUTO_SHARDING_DECREASE_SHARDS_COOLDOWN	Represents the minimum amount of time between two scaling events if the next event will reduce the number of shards.
static final String	DATA_STREAMS_AUTO_SHARDING_ENABLED
static final Setting<List<String>>	DATA_STREAMS_AUTO_SHARDING_EXCLUDES_SETTING
static final Setting<TimeValue>	DATA_STREAMS_AUTO_SHARDING_INCREASE_SHARDS_COOLDOWN	Represents the minimum amount of time between two scaling events if the next event will increase the number of shards.

Constructor Summary

Constructors

Constructor	Description
DataStreamAutoShardingService(Settings settings, ClusterService clusterService, FeatureService featureService, LongSupplier nowSupplier)

Method Summary

Modifier and Type	Method	Description
AutoShardingResult	calculate(ClusterState state, DataStream dataStream, Double writeIndexLoad)	Computes the optimal number of shards for the provided data stream according to the write index's indexing load (to check if we must increase the number of shards, whilst the heuristics for decreasing the number of shards _might_ use the provided write indexing load).
void	init()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

DATA_STREAMS_AUTO_SHARDING_ENABLED

public static final String DATA_STREAMS_AUTO_SHARDING_ENABLED

See Also:

• Constant Field Values

DATA_STREAM_AUTO_SHARDING_FEATURE

public static final NodeFeature DATA_STREAM_AUTO_SHARDING_FEATURE

DATA_STREAMS_AUTO_SHARDING_EXCLUDES_SETTING

public static final Setting<List<String>> DATA_STREAMS_AUTO_SHARDING_EXCLUDES_SETTING

DATA_STREAMS_AUTO_SHARDING_INCREASE_SHARDS_COOLDOWN

public static final Setting<TimeValue> DATA_STREAMS_AUTO_SHARDING_INCREASE_SHARDS_COOLDOWN

Represents the minimum amount of time between two scaling events if the next event will increase the number of shards. We've chosen a value of 4.5minutes by default, just lower than the data stream lifecycle poll interval so we can increase shards with every DSL run, but we don't want it to be lower/0 as data stream lifecycle might run more often than the poll interval in case of a master failover.

DATA_STREAMS_AUTO_SHARDING_DECREASE_SHARDS_COOLDOWN

public static final Setting<TimeValue> DATA_STREAMS_AUTO_SHARDING_DECREASE_SHARDS_COOLDOWN

Represents the minimum amount of time between two scaling events if the next event will reduce the number of shards.

CLUSTER_AUTO_SHARDING_MIN_WRITE_THREADS

public static final Setting<Integer> CLUSTER_AUTO_SHARDING_MIN_WRITE_THREADS

Represents the minimum number of write threads we expect a node to have in the environments where auto sharding will be enabled.

CLUSTER_AUTO_SHARDING_MAX_WRITE_THREADS

public static final Setting<Integer> CLUSTER_AUTO_SHARDING_MAX_WRITE_THREADS

Represents the maximum number of write threads we expect a node to have in the environments where auto sharding will be enabled.

Constructor Details

DataStreamAutoShardingService

public DataStreamAutoShardingService(Settings settings,
 ClusterService clusterService,
 FeatureService featureService,
 LongSupplier nowSupplier)

Method Details

init

public void init()

calculate

public AutoShardingResult calculate(ClusterState state,
 DataStream dataStream,
 @Nullable
 Double writeIndexLoad)

Computes the optimal number of shards for the provided data stream according to the write index's indexing load (to check if we must increase the number of shards, whilst the heuristics for decreasing the number of shards _might_ use the provided write indexing load). The result type will indicate the recommendation of the auto sharding service : - not applicable if the data stream is excluded from auto sharding as configured by DATA_STREAMS_AUTO_SHARDING_EXCLUDES_SETTING or if the auto sharding functionality is disabled according to DATA_STREAMS_AUTO_SHARDING_EXCLUDES_SETTING, or if the cluster doesn't have the feature available - increase number of shards if the optimal number of shards it deems necessary for the provided data stream is GT the current number of shards - decrease the number of shards if the optimal number of shards it deems necessary for the provided data stream is LT the current number of shards If the recommendation is to INCREASE/DECREASE shards the reported cooldown period will be TimeValue.ZERO. If the auto sharding service thinks the number of shards must be changed but it can't recommend a change due to the cooldown period not lapsing, the result will be of type AutoShardingType.COOLDOWN_PREVENTED_INCREASE or AutoShardingType.COOLDOWN_PREVENTED_INCREASE with the remaining cooldown configured and the number of shards that should be configured for the data stream once the remaining cooldown lapses as the target number of shards. The NOT_APPLICABLE type result will report a cooldown period of TimeValue.MAX_VALUE. The NO_CHANGE_REQUIRED type will potentially report the remaining cooldown always report a cool down period of TimeValue.ZERO (as there'll be no new auto sharding event)