org.apache.flink.streaming.api.functions.source

Class MultipleIdsMessageAcknowledgingSourceBase<Type,UId,SessionId>

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.streaming.api.functions.source.RichSourceFunction<Type>

org.apache.flink.streaming.api.functions.source.MessageAcknowledgingSourceBase<Type,UId>

org.apache.flink.streaming.api.functions.source.MultipleIdsMessageAcknowledgingSourceBase<Type,UId,SessionId>

Type Parameters:

Type - The type of the messages created by the source.

UId - The type of the unique IDs which are consistent across sessions.

SessionId - The type of the IDs that are used for acknowledging elements (ids valid during session).

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, org.apache.flink.api.common.state.CheckpointListener, CheckpointedFunction, SourceFunction<Type>

@PublicEvolving
public abstract class MultipleIdsMessageAcknowledgingSourceBase<Type,UId,SessionId>
extends MessageAcknowledgingSourceBase<Type,UId>

Abstract base class for data sources that receive elements from a message queue and acknowledge them back by IDs. In contrast to MessageAcknowledgingSourceBase, this source handles two types of ids:

1. Session ids
2. Unique message ids

Session ids are used to acknowledge messages in a session. When a checkpoint is restored, unacknowledged messages are redelivered. Duplicates are detected using the unique message ids which are checkpointed.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.streaming.api.functions.source.SourceFunction

SourceFunction.SourceContext<T>

Field Summary

Fields

Modifier and Type	Field and Description
protected List<SessionId>	sessionIds
protected Deque<org.apache.flink.api.java.tuple.Tuple2<Long,List<SessionId>>>	sessionIdsPerSnapshot

Fields inherited from class org.apache.flink.streaming.api.functions.source.MessageAcknowledgingSourceBase

pendingCheckpoints

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	MultipleIdsMessageAcknowledgingSourceBase(Class<UId> idClass) Creates a new MessageAcknowledgingSourceBase for IDs of the given type.
protected	MultipleIdsMessageAcknowledgingSourceBase(org.apache.flink.api.common.typeinfo.TypeInformation<UId> idTypeInfo) Creates a new MessageAcknowledgingSourceBase for IDs of the given type.

Method Summary

Modifier and Type	Method and Description
protected void	acknowledgeIDs(long checkpointId, Set<UId> uniqueIds) Acknowledges the session ids.
protected abstract void	acknowledgeSessionIDs(List<SessionId> sessionIds) Acknowledges the session ids.
void	close()
void	open(org.apache.flink.configuration.Configuration parameters)
void	snapshotState(org.apache.flink.runtime.state.FunctionSnapshotContext context) This method is called when a snapshot for a checkpoint is requested.

Methods inherited from class org.apache.flink.streaming.api.functions.source.MessageAcknowledgingSourceBase

addId, initializeState, notifyCheckpointAborted, notifyCheckpointComplete

Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.streaming.api.functions.source.SourceFunction

cancel, run

Field Detail

sessionIdsPerSnapshot

protected transient Deque<org.apache.flink.api.java.tuple.Tuple2<Long,List<SessionId>>> sessionIdsPerSnapshot

sessionIds

protected transient List<SessionId> sessionIds

Constructor Detail

MultipleIdsMessageAcknowledgingSourceBase

protected MultipleIdsMessageAcknowledgingSourceBase(Class<UId> idClass)

Creates a new MessageAcknowledgingSourceBase for IDs of the given type.

Parameters:

idClass - The class of the message ID type, used to create a serializer for the message IDs.

MultipleIdsMessageAcknowledgingSourceBase

protected MultipleIdsMessageAcknowledgingSourceBase(org.apache.flink.api.common.typeinfo.TypeInformation<UId> idTypeInfo)

Creates a new MessageAcknowledgingSourceBase for IDs of the given type.

Parameters:

idTypeInfo - The type information of the message ID type, used to create a serializer for the message IDs.

Method Detail

open

public void open(org.apache.flink.configuration.Configuration parameters)
          throws Exception

Specified by:

open in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

open in class org.apache.flink.api.common.functions.AbstractRichFunction

Throws:

Exception

close

public void close()
           throws Exception

Specified by:

close in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

close in class MessageAcknowledgingSourceBase<Type,UId>

Throws:

Exception

acknowledgeIDs

protected final void acknowledgeIDs(long checkpointId,
                                    Set<UId> uniqueIds)

Acknowledges the session ids.

Specified by:

acknowledgeIDs in class MessageAcknowledgingSourceBase<Type,UId>

Parameters:

checkpointId - The id of the current checkout to acknowledge ids for.

uniqueIds - The checkpointed unique ids which are ignored here. They only serve as a means of de-duplicating messages when the acknowledgment after a checkpoint fails.

acknowledgeSessionIDs

protected abstract void acknowledgeSessionIDs(List<SessionId> sessionIds)

Acknowledges the session ids.

Parameters:

sessionIds - The message ids for this session.

snapshotState

public void snapshotState(org.apache.flink.runtime.state.FunctionSnapshotContext context)
                   throws Exception

Description copied from interface: CheckpointedFunction

This method is called when a snapshot for a checkpoint is requested. This acts as a hook to the function to ensure that all state is exposed by means previously offered through FunctionInitializationContext when the Function was initialized, or offered now by FunctionSnapshotContext itself.

Specified by:

snapshotState in interface CheckpointedFunction

Overrides:

snapshotState in class MessageAcknowledgingSourceBase<Type,UId>

Parameters:

context - the context for drawing a snapshot of the operator

Throws:

Exception - Thrown, if state could not be created ot restored.