Package org.apache.flink.runtime.state.heap

Class HeapKeyedStateBackend<K>

java.lang.Object

org.apache.flink.runtime.state.AbstractKeyedStateBackend<K>

org.apache.flink.runtime.state.heap.HeapKeyedStateBackend<K>

Type Parameters:

K - The key by which state is keyed.

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.api.common.state.InternalCheckpointListener, CheckpointableKeyedStateBackend<K>, InternalKeyContext<K>, KeyedStateBackend<K>, KeyedStateFactory, PriorityQueueSetFactory, Snapshotable<SnapshotResult<KeyedStateHandle>>, TestableKeyedStateBackend<K>, org.apache.flink.util.Disposable

public class HeapKeyedStateBackend<K>
extends AbstractKeyedStateBackend<K>

A AbstractKeyedStateBackend that keeps state on the Java Heap and will serialize state to streams provided by a CheckpointStreamFactory upon checkpointing.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.runtime.state.AbstractKeyedStateBackend

AbstractKeyedStateBackend.PartitionStateFactory

Nested classes/interfaces inherited from interface org.apache.flink.runtime.state.KeyedStateBackend

KeyedStateBackend.KeySelectionListener<K>

Field Summary

Fields inherited from class org.apache.flink.runtime.state.AbstractKeyedStateBackend

cancelStreamRegistry, keyContext, keyGroupCompressionDecorator, keyGroupRange, keySerializer, kvStateRegistry, latencyTrackingStateConfig, numberOfKeyGroups, ttlTimeProvider, userCodeClassLoader

Constructor Summary

Constructors

Constructor

Description

HeapKeyedStateBackend(TaskKvStateRegistry kvStateRegistry, org.apache.flink.api.common.typeutils.TypeSerializer<K> keySerializer, ClassLoader userCodeClassLoader, org.apache.flink.api.common.ExecutionConfig executionConfig, TtlTimeProvider ttlTimeProvider, LatencyTrackingStateConfig latencyTrackingStateConfig, org.apache.flink.core.fs.CloseableRegistry cancelStreamRegistry, StreamCompressionDecorator keyGroupCompressionDecorator, Map<String,StateTable<K,?,?>> registeredKVStates, Map<String,HeapPriorityQueueSnapshotRestoreWrapper<?>> registeredPQStates, LocalRecoveryConfig localRecoveryConfig, HeapPriorityQueueSetFactory priorityQueueSetFactory, org.apache.flink.runtime.state.heap.HeapSnapshotStrategy<K> checkpointStrategy, SnapshotExecutionType snapshotExecutionType, org.apache.flink.runtime.state.heap.StateTableFactory<K> stateTableFactory, InternalKeyContext<K> keyContext)

Method Summary

Modifier and Type	Method	Description
<N,S extends org.apache.flink.api.common.state.State,T> void	applyToAllKeys(N namespace, org.apache.flink.api.common.typeutils.TypeSerializer<N> namespaceSerializer, org.apache.flink.api.common.state.StateDescriptor<S,T> stateDescriptor, KeyedStateFunction<K,S> function, AbstractKeyedStateBackend.PartitionStateFactory partitionStateFactory)
<T extends HeapPriorityQueueElement & PriorityComparable<? super T> & Keyed<?>> KeyGroupedInternalPriorityQueue<T>	create(String stateName, org.apache.flink.api.common.typeutils.TypeSerializer<T> byteOrderedElementSerializer)	Creates a KeyGroupedInternalPriorityQueue.
<T extends HeapPriorityQueueElement & PriorityComparable<? super T> & Keyed<?>> KeyGroupedInternalPriorityQueue<T>	create(String stateName, org.apache.flink.api.common.typeutils.TypeSerializer<T> byteOrderedElementSerializer, boolean allowFutureMetadataUpdates)	Creates a KeyGroupedInternalPriorityQueue.
<N,SV,SEV,S extends org.apache.flink.api.common.state.State,IS extends S> IS	createOrUpdateInternalState(org.apache.flink.api.common.typeutils.TypeSerializer<N> namespaceSerializer, org.apache.flink.api.common.state.StateDescriptor<S,SV> stateDesc, StateSnapshotTransformer.StateSnapshotTransformFactory<SEV> snapshotTransformFactory)	Creates or updates internal state and returns a new InternalKvState.
<N,SV,SEV,S extends org.apache.flink.api.common.state.State,IS extends S> IS	createOrUpdateInternalState(org.apache.flink.api.common.typeutils.TypeSerializer<N> namespaceSerializer, org.apache.flink.api.common.state.StateDescriptor<S,SV> stateDesc, StateSnapshotTransformer.StateSnapshotTransformFactory<SEV> snapshotTransformFactory, boolean allowFutureMetadataUpdates)	Creates or updates internal state and returns a new InternalKvState.
<N> Stream<K>	getKeys(String state, N namespace)
<N> Stream<org.apache.flink.api.java.tuple.Tuple2<K,N>>	getKeysAndNamespaces(String state)
LocalRecoveryConfig	getLocalRecoveryConfig()
void	notifyCheckpointAborted(long checkpointId)
void	notifyCheckpointComplete(long checkpointId)
int	numKeyValueStateEntries()	Returns the total number of state entries across all keys/namespaces.
int	numKeyValueStateEntries(Object namespace)	Returns the total number of state entries across all keys for the given namespace.
SavepointResources<K>	savepoint()	Returns a SavepointResources that can be used by SavepointSnapshotStrategy to write out a savepoint in the common/unified format.
RunnableFuture<SnapshotResult<KeyedStateHandle>>	snapshot(long checkpointId, long timestamp, CheckpointStreamFactory streamFactory, CheckpointOptions checkpointOptions)	Operation that writes a snapshot into a stream that is provided by the given CheckpointStreamFactory and returns a @RunnableFuture that gives a state handle to the snapshot.
String	toString()

Methods inherited from class org.apache.flink.runtime.state.AbstractKeyedStateBackend

applyToAllKeys, close, deregisterKeySelectionListener, dispose, getCurrentKey, getCurrentKeyGroupIndex, getKeyContext, getKeyGroupCompressionDecorator, getKeyGroupRange, getKeySerializer, getLatencyTrackingStateConfig, getNumberOfKeyGroups, getOrCreateKeyedState, getPartitionedState, notifyCheckpointSubsumed, numKeyValueStatesByName, publishQueryableStateIfEnabled, registerKeySelectionListener, requiresLegacySynchronousTimerSnapshots, setCurrentKey, setCurrentKeyAndKeyGroup, setCurrentKeyGroupIndex

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.flink.runtime.state.KeyedStateBackend

isSafeToReuseKVState

Methods inherited from interface org.apache.flink.runtime.state.KeyedStateFactory

createOrUpdateInternalState

Methods inherited from interface org.apache.flink.runtime.state.TestableKeyedStateBackend

getDelegatedKeyedStateBackend

Constructor Detail

HeapKeyedStateBackend

public HeapKeyedStateBackend(TaskKvStateRegistry kvStateRegistry,
                             org.apache.flink.api.common.typeutils.TypeSerializer<K> keySerializer,
                             ClassLoader userCodeClassLoader,
                             org.apache.flink.api.common.ExecutionConfig executionConfig,
                             TtlTimeProvider ttlTimeProvider,
                             LatencyTrackingStateConfig latencyTrackingStateConfig,
                             org.apache.flink.core.fs.CloseableRegistry cancelStreamRegistry,
                             StreamCompressionDecorator keyGroupCompressionDecorator,
                             Map<String,StateTable<K,?,?>> registeredKVStates,
                             Map<String,HeapPriorityQueueSnapshotRestoreWrapper<?>> registeredPQStates,
                             LocalRecoveryConfig localRecoveryConfig,
                             HeapPriorityQueueSetFactory priorityQueueSetFactory,
                             org.apache.flink.runtime.state.heap.HeapSnapshotStrategy<K> checkpointStrategy,
                             SnapshotExecutionType snapshotExecutionType,
                             org.apache.flink.runtime.state.heap.StateTableFactory<K> stateTableFactory,
                             InternalKeyContext<K> keyContext)

Method Detail

create

@Nonnull
public <T extends HeapPriorityQueueElement & PriorityComparable<? super T> & Keyed<?>> KeyGroupedInternalPriorityQueue<T> create(@Nonnull
                                                                                                                                 String stateName,
                                                                                                                                 @Nonnull
                                                                                                                                 org.apache.flink.api.common.typeutils.TypeSerializer<T> byteOrderedElementSerializer)

Description copied from interface: PriorityQueueSetFactory

Creates a KeyGroupedInternalPriorityQueue.

Type Parameters:

T - type of the stored elements.

Parameters:

stateName - unique name for associated with this queue.

byteOrderedElementSerializer - a serializer that with a format that is lexicographically ordered in alignment with elementPriorityComparator.

Returns:

the queue with the specified unique name.

create

public <T extends HeapPriorityQueueElement & PriorityComparable<? super T> & Keyed<?>> KeyGroupedInternalPriorityQueue<T> create(@Nonnull
                                                                                                                                 String stateName,
                                                                                                                                 @Nonnull
                                                                                                                                 org.apache.flink.api.common.typeutils.TypeSerializer<T> byteOrderedElementSerializer,
                                                                                                                                 boolean allowFutureMetadataUpdates)

Description copied from interface: PriorityQueueSetFactory

Creates a KeyGroupedInternalPriorityQueue.

Type Parameters:

T - type of the stored elements.

Parameters:

stateName - unique name for associated with this queue.

byteOrderedElementSerializer - a serializer that with a format that is lexicographically ordered in alignment with elementPriorityComparator.

allowFutureMetadataUpdates - whether allow metadata to update in the future or not.

Returns:

the queue with the specified unique name.

getKeys

public <N> Stream<K> getKeys(String state,
                             N namespace)

Parameters:

state - State variable for which existing keys will be returned.

namespace - Namespace for which existing keys will be returned.

Returns:

A stream of all keys for the given state and namespace. Modifications to the state during iterating over it keys are not supported.

getKeysAndNamespaces

public <N> Stream<org.apache.flink.api.java.tuple.Tuple2<K,N>> getKeysAndNamespaces(String state)

Parameters:

state - State variable for which existing keys will be returned.

Returns:

A stream of all keys for the given state and namespace. Modifications to the state during iterating over it keys are not supported. Implementations go not make any ordering guarantees about the returned tupes. Two records with the same key or namespace may not be returned near each other in the stream.

createOrUpdateInternalState

@Nonnull
public <N,SV,SEV,S extends org.apache.flink.api.common.state.State,IS extends S> IS createOrUpdateInternalState(@Nonnull
                                                                                                                                        org.apache.flink.api.common.typeutils.TypeSerializer<N> namespaceSerializer,
                                                                                                                                        @Nonnull
                                                                                                                                        org.apache.flink.api.common.state.StateDescriptor<S,SV> stateDesc,
                                                                                                                                        @Nonnull
                                                                                                                                        StateSnapshotTransformer.StateSnapshotTransformFactory<SEV> snapshotTransformFactory)
                                                                                                                                 throws Exception

Description copied from interface: KeyedStateFactory

Creates or updates internal state and returns a new InternalKvState.

Type Parameters:

N - The type of the namespace.

SV - The type of the stored state value.

SEV - The type of the stored state value or entry for collection types (list or map).

S - The type of the public API state.

IS - The type of internal state.

Parameters:

namespaceSerializer - TypeSerializer for the state namespace.

stateDesc - The StateDescriptor that contains the name of the state.

snapshotTransformFactory - factory of state snapshot transformer.

Throws:

Exception

createOrUpdateInternalState

@Nonnull
public <N,SV,SEV,S extends org.apache.flink.api.common.state.State,IS extends S> IS createOrUpdateInternalState(@Nonnull
                                                                                                                                        org.apache.flink.api.common.typeutils.TypeSerializer<N> namespaceSerializer,
                                                                                                                                        @Nonnull
                                                                                                                                        org.apache.flink.api.common.state.StateDescriptor<S,SV> stateDesc,
                                                                                                                                        @Nonnull
                                                                                                                                        StateSnapshotTransformer.StateSnapshotTransformFactory<SEV> snapshotTransformFactory,
                                                                                                                                        boolean allowFutureMetadataUpdates)
                                                                                                                                 throws Exception

Description copied from interface: KeyedStateFactory

Creates or updates internal state and returns a new InternalKvState.

Type Parameters:

N - The type of the namespace.

SV - The type of the stored state value.

SEV - The type of the stored state value or entry for collection types (list or map).

S - The type of the public API state.

IS - The type of internal state.

Parameters:

namespaceSerializer - TypeSerializer for the state namespace.

stateDesc - The StateDescriptor that contains the name of the state.

snapshotTransformFactory - factory of state snapshot transformer.

allowFutureMetadataUpdates - whether allow metadata to update in the future or not.

Throws:

Exception

snapshot

@Nonnull
public RunnableFuture<SnapshotResult<KeyedStateHandle>> snapshot(long checkpointId,
                                                                 long timestamp,
                                                                 @Nonnull
                                                                 CheckpointStreamFactory streamFactory,
                                                                 @Nonnull
                                                                 CheckpointOptions checkpointOptions)
                                                          throws Exception

Description copied from interface: Snapshotable

Operation that writes a snapshot into a stream that is provided by the given CheckpointStreamFactory and returns a @RunnableFuture that gives a state handle to the snapshot. It is up to the implementation if the operation is performed synchronous or asynchronous. In the later case, the returned Runnable must be executed first before obtaining the handle.

Parameters:

checkpointId - The ID of the checkpoint.

timestamp - The timestamp of the checkpoint.

streamFactory - The factory that we can use for writing our state to streams.

checkpointOptions - Options for how to perform this checkpoint.

Returns:

A runnable future that will yield a StateObject.

Throws:

Exception

savepoint

@Nonnull
public SavepointResources<K> savepoint()

Description copied from interface: CheckpointableKeyedStateBackend

Returns a SavepointResources that can be used by SavepointSnapshotStrategy to write out a savepoint in the common/unified format.

notifyCheckpointComplete

public void notifyCheckpointComplete(long checkpointId)

notifyCheckpointAborted

public void notifyCheckpointAborted(long checkpointId)

applyToAllKeys

public <N,S extends org.apache.flink.api.common.state.State,T> void applyToAllKeys(N namespace,
                                                                                               org.apache.flink.api.common.typeutils.TypeSerializer<N> namespaceSerializer,
                                                                                               org.apache.flink.api.common.state.StateDescriptor<S,T> stateDescriptor,
                                                                                               KeyedStateFunction<K,S> function,
                                                                                               AbstractKeyedStateBackend.PartitionStateFactory partitionStateFactory)
                                                                                        throws Exception

Overrides:

applyToAllKeys in class AbstractKeyedStateBackend<K>

Throws:

Exception

toString

public String toString()

Overrides:

toString in class Object

numKeyValueStateEntries

@VisibleForTesting
public int numKeyValueStateEntries()

Returns the total number of state entries across all keys/namespaces.

numKeyValueStateEntries

@VisibleForTesting
public int numKeyValueStateEntries(Object namespace)

Returns the total number of state entries across all keys for the given namespace.

getLocalRecoveryConfig

@VisibleForTesting
public LocalRecoveryConfig getLocalRecoveryConfig()