Package io.pravega.connectors.flink.sink

Class PravegaEventWriter<T>

java.lang.Object

io.pravega.connectors.flink.sink.PravegaEventWriter<T>

Type Parameters:

T - The type of the event to be written.

All Implemented Interfaces:

java.lang.AutoCloseable, org.apache.flink.api.connector.sink2.SinkWriter<T>

public class PravegaEventWriter<T>
extends java.lang.Object
implements org.apache.flink.api.connector.sink2.SinkWriter<T>

A Pravega SinkWriter implementation that is suitable for PravegaWriterMode.BEST_EFFORT and PravegaWriterMode.ATLEAST_ONCE.

Note that the difference between these two modes is that flushAndVerify() is called for each checkpoint in the PravegaWriterMode.ATLEAST_ONCE mode.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.api.connector.sink2.SinkWriter

org.apache.flink.api.connector.sink2.SinkWriter.Context

Field Summary

Fields

Modifier and Type	Field	Description
protected io.pravega.client.EventStreamClientFactory	clientFactory
protected java.util.concurrent.ExecutorService	executorService
protected java.util.concurrent.atomic.AtomicLong	pendingWritesCount
protected java.util.concurrent.atomic.AtomicReference<java.lang.Throwable>	writeError

Constructor Summary

Constructors

Constructor	Description
PravegaEventWriter(org.apache.flink.api.connector.sink2.Sink.InitContext context, io.pravega.client.ClientConfig clientConfig, io.pravega.client.stream.Stream stream, PravegaWriterMode writerMode, org.apache.flink.api.common.serialization.SerializationSchema<T> serializationSchema, PravegaEventRouter<T> eventRouter)	A Pravega non-transactional writer that handles PravegaWriterMode.BEST_EFFORT and PravegaWriterMode.ATLEAST_ONCE writer mode.

Method Summary

Modifier and Type	Method	Description
void	close()
void	flush(boolean endOfInput)
void	flushAndVerify()
protected PravegaEventRouter<T>	getEventRouter()
protected io.pravega.client.stream.EventStreamWriter<T>	getInternalWriter()
protected PravegaWriterMode	getWriterMode()
protected io.pravega.client.stream.EventStreamWriter<T>	initializeInternalWriter()
void	write(T element, org.apache.flink.api.connector.sink2.SinkWriter.Context context)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.api.connector.sink2.SinkWriter

writeWatermark

Field Detail

writeError

@VisibleForTesting
protected volatile java.util.concurrent.atomic.AtomicReference<java.lang.Throwable> writeError

pendingWritesCount

@VisibleForTesting
protected java.util.concurrent.atomic.AtomicLong pendingWritesCount

executorService

@VisibleForTesting
protected transient java.util.concurrent.ExecutorService executorService

clientFactory

@VisibleForTesting
protected transient io.pravega.client.EventStreamClientFactory clientFactory

Constructor Detail

PravegaEventWriter

public PravegaEventWriter(org.apache.flink.api.connector.sink2.Sink.InitContext context,
                          io.pravega.client.ClientConfig clientConfig,
                          io.pravega.client.stream.Stream stream,
                          PravegaWriterMode writerMode,
                          org.apache.flink.api.common.serialization.SerializationSchema<T> serializationSchema,
                          PravegaEventRouter<T> eventRouter)

A Pravega non-transactional writer that handles PravegaWriterMode.BEST_EFFORT and PravegaWriterMode.ATLEAST_ONCE writer mode.

Parameters:

context - Some runtime info from sink.

clientConfig - The Pravega client configuration.

stream - The destination stream.

writerMode - The Pravega writer mode.

serializationSchema - The implementation for serializing every event into pravega's storage format.

eventRouter - The implementation to extract the partition key from the event.

Method Detail

initializeInternalWriter

@VisibleForTesting
protected io.pravega.client.stream.EventStreamWriter<T> initializeInternalWriter()

write

public void write(T element,
                  org.apache.flink.api.connector.sink2.SinkWriter.Context context)
           throws java.io.IOException,
                  java.lang.InterruptedException

Specified by:

write in interface org.apache.flink.api.connector.sink2.SinkWriter<T>

Throws:

java.io.IOException

java.lang.InterruptedException

flush

public void flush(boolean endOfInput)
           throws java.io.IOException,
                  java.lang.InterruptedException

Specified by:

flush in interface org.apache.flink.api.connector.sink2.SinkWriter<T>

Throws:

java.io.IOException

java.lang.InterruptedException

flushAndVerify

@VisibleForTesting
public void flushAndVerify()
                    throws java.io.IOException,
                           java.lang.InterruptedException

Throws:

java.io.IOException

java.lang.InterruptedException

close

public void close()
           throws java.lang.Exception

Specified by:

close in interface java.lang.AutoCloseable

Throws:

java.lang.Exception

getWriterMode

@VisibleForTesting
protected PravegaWriterMode getWriterMode()

getEventRouter

@VisibleForTesting
@Nullable
protected PravegaEventRouter<T> getEventRouter()

getInternalWriter

@VisibleForTesting
protected io.pravega.client.stream.EventStreamWriter<T> getInternalWriter()