Package org.apache.flink.streaming.runtime.operators

Class GenericWriteAheadSink<IN>

java.lang.Object

org.apache.flink.streaming.api.operators.AbstractStreamOperator<IN>

org.apache.flink.streaming.runtime.operators.GenericWriteAheadSink<IN>

Type Parameters:

IN - Type of the elements emitted by this sink

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.streaming.api.operators.Input<IN>, org.apache.flink.streaming.api.operators.KeyContext, org.apache.flink.streaming.api.operators.KeyContextHandler, org.apache.flink.streaming.api.operators.OneInputStreamOperator<IN,IN>, org.apache.flink.streaming.api.operators.StreamOperator<IN>, org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator, org.apache.flink.streaming.api.operators.YieldingOperator<IN>

public abstract class GenericWriteAheadSink<IN>
extends org.apache.flink.streaming.api.operators.AbstractStreamOperator<IN>
implements org.apache.flink.streaming.api.operators.OneInputStreamOperator<IN,IN>

Generic Sink that emits its input elements into an arbitrary backend. This sink is integrated with Flink's checkpointing mechanism and can provide exactly-once guarantees; depending on the storage backend and sink/committer implementation.

Incoming records are stored within a AbstractStateBackend, and only committed if a checkpoint is completed.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field	Description
protected static org.slf4j.Logger	LOG
protected org.apache.flink.api.common.typeutils.TypeSerializer<IN>	serializer

Fields inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

combinedWatermark, config, lastRecordAttributes1, lastRecordAttributes2, latencyStats, metrics, output, processingTimeService, stateHandler, stateKeySelector1, stateKeySelector2, timeServiceManager

Constructor Summary

Constructors

Constructor	Description
GenericWriteAheadSink(CheckpointCommitter committer, org.apache.flink.api.common.typeutils.TypeSerializer<IN> serializer, String jobID)

Method Summary

Modifier and Type	Method	Description
void	close()
void	initializeState(org.apache.flink.runtime.state.StateInitializationContext context)
void	notifyCheckpointComplete(long checkpointId)
void	open()
void	processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<IN> element)
protected abstract boolean	sendValues(Iterable<IN> values, long checkpointId, long timestamp)	Write the given element into the backend.
void	snapshotState(org.apache.flink.runtime.state.StateSnapshotContext context)

Methods inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

beforeInitializeStateHandler, finish, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getStateKeySelector1, getStateKeySelector2, getTimeServiceManager, getUserCodeClassloader, hasKeyContext1, hasKeyContext2, initializeState, isAsyncStateProcessingEnabled, isUsingCustomRawKeyedState, notifyCheckpointAborted, prepareSnapshotPreBarrier, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processRecordAttributes, processRecordAttributes1, processRecordAttributes2, processWatermark, processWatermark, processWatermark1, processWatermark1, processWatermark2, processWatermark2, processWatermarkStatus, processWatermarkStatus, processWatermarkStatus1, processWatermarkStatus2, reportOrForwardLatencyMarker, setCurrentKey, setKeyContextElement1, setKeyContextElement2, setMailboxExecutor, setProcessingTimeService, setup, snapshotState, useSplittableTimers

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

notifyCheckpointAborted

Methods inherited from interface org.apache.flink.streaming.api.operators.Input

processLatencyMarker, processRecordAttributes, processWatermark, processWatermark, processWatermarkStatus

Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContext

getCurrentKey, setCurrentKey

Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContextHandler

hasKeyContext

Methods inherited from interface org.apache.flink.streaming.api.operators.OneInputStreamOperator

setKeyContextElement

Methods inherited from interface org.apache.flink.streaming.api.operators.StreamOperator

finish, getMetricGroup, getOperatorAttributes, getOperatorID, initializeState, prepareSnapshotPreBarrier, setKeyContextElement1, setKeyContextElement2, snapshotState

Field Detail

LOG

protected static final org.slf4j.Logger LOG

serializer

protected final org.apache.flink.api.common.typeutils.TypeSerializer<IN> serializer

Constructor Detail

GenericWriteAheadSink

public GenericWriteAheadSink(CheckpointCommitter committer,
                             org.apache.flink.api.common.typeutils.TypeSerializer<IN> serializer,
                             String jobID)
                      throws Exception

Throws:

Exception

Method Detail

initializeState

public void initializeState(org.apache.flink.runtime.state.StateInitializationContext context)
                     throws Exception

Specified by:

initializeState in interface org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator

Overrides:

initializeState in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<IN>

Throws:

Exception

open

public void open()
          throws Exception

Specified by:

open in interface org.apache.flink.streaming.api.operators.StreamOperator<IN>

Overrides:

open in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<IN>

Throws:

Exception

close

public void close()
           throws Exception

Specified by:

close in interface org.apache.flink.streaming.api.operators.StreamOperator<IN>

Overrides:

close in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<IN>

Throws:

Exception

snapshotState

public void snapshotState(org.apache.flink.runtime.state.StateSnapshotContext context)
                   throws Exception

Specified by:

snapshotState in interface org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator

Overrides:

snapshotState in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<IN>

Throws:

Exception

notifyCheckpointComplete

public void notifyCheckpointComplete(long checkpointId)
                              throws Exception

Specified by:

notifyCheckpointComplete in interface org.apache.flink.api.common.state.CheckpointListener

Overrides:

notifyCheckpointComplete in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<IN>

Throws:

Exception

sendValues

protected abstract boolean sendValues(Iterable<IN> values,
                                      long checkpointId,
                                      long timestamp)
                               throws Exception

Write the given element into the backend.

Parameters:

values - The values to be written

checkpointId - The checkpoint ID of the checkpoint to be written

timestamp - The wall-clock timestamp of the checkpoint

Returns:

true, if the sending was successful, false otherwise

Throws:

Exception

processElement

public void processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<IN> element)
                    throws Exception

Specified by:

processElement in interface org.apache.flink.streaming.api.operators.Input<IN>

Throws:

Exception