Package io.pravega.connectors.flink

Class FlinkPravegaWriter<T>

  • Type Parameters:
    T - The type of the event to be written.
    All Implemented Interfaces:
    java.io.Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, org.apache.flink.runtime.state.CheckpointListener, org.apache.flink.streaming.api.checkpoint.ListCheckpointed<io.pravega.connectors.flink.FlinkPravegaWriter.PendingTransaction>, org.apache.flink.streaming.api.functions.sink.SinkFunction<T>
    public class FlinkPravegaWriter<T>
extends org.apache.flink.streaming.api.functions.sink.RichSinkFunction<T>
implements org.apache.flink.streaming.api.checkpoint.ListCheckpointed<io.pravega.connectors.flink.FlinkPravegaWriter.PendingTransaction>, org.apache.flink.runtime.state.CheckpointListener
    Flink sink implementation for writing into pravega storage.
    See Also:
    Serialized Form

    • Nested Class Summary

      Nested Classes 
      Modifier and Type Class Description
      static class  FlinkPravegaWriter.Builder<T>
      A builder for FlinkPravegaWriter.

      • Nested classes/interfaces inherited from interface org.apache.flink.streaming.api.functions.sink.SinkFunction

        org.apache.flink.streaming.api.functions.sink.SinkFunction.Context<T extends java.lang.Object>

    • Constructor Summary

      Constructors 
      Modifier Constructor Description
      protected FlinkPravegaWriter​(io.pravega.client.ClientConfig clientConfig, io.pravega.client.stream.Stream stream, org.apache.flink.api.common.serialization.SerializationSchema<T> serializationSchema, PravegaEventRouter<T> eventRouter, PravegaWriterMode writerMode, long txnLeaseRenewalPeriod, boolean enableWatermark, boolean enableMetrics)
      The flink pravega writer instance which can be added as a sink to a Flink job.

    • Method Summary

      All Methods Static Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      static <T> FlinkPravegaWriter.Builder<T> builder()  
      void close()  
      protected io.pravega.client.EventStreamClientFactory createClientFactory​(java.lang.String scopeName, io.pravega.client.ClientConfig clientConfig)  
      protected java.util.concurrent.ExecutorService createExecutorService()  
      protected io.pravega.connectors.flink.FlinkPravegaWriter.AbstractInternalWriter createInternalWriter()  
      boolean getEnableWatermark()
      Gets this enable watermark flag.
      PravegaEventRouter<T> getEventRouter()
      Gets the associated event router.
      PravegaWriterMode getPravegaWriterMode()
      Gets this writer's operating mode.
      void invoke​(T event, org.apache.flink.streaming.api.functions.sink.SinkFunction.Context context)  
      void notifyCheckpointComplete​(long checkpointId)
      Notifies the writer that a checkpoint is complete.
      void open​(org.apache.flink.configuration.Configuration parameters)  
      void restoreState​(java.util.List<io.pravega.connectors.flink.FlinkPravegaWriter.PendingTransaction> pendingTransactionList)
      Restores the state, which here means the IDs of transaction for which we have to ensure that they are really committed.
      java.util.List<io.pravega.connectors.flink.FlinkPravegaWriter.PendingTransaction> snapshotState​(long checkpointId, long checkpointTime)  
      protected java.lang.String writerId()  

      • Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

        getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

      • Methods inherited from interface org.apache.flink.streaming.api.functions.sink.SinkFunction

        invoke

    • Constructor Detail

      • FlinkPravegaWriter

        protected FlinkPravegaWriter​(io.pravega.client.ClientConfig clientConfig,
                             io.pravega.client.stream.Stream stream,
                             org.apache.flink.api.common.serialization.SerializationSchema<T> serializationSchema,
                             PravegaEventRouter<T> eventRouter,
                             PravegaWriterMode writerMode,
                             long txnLeaseRenewalPeriod,
                             boolean enableWatermark,
                             boolean enableMetrics)
        The flink pravega writer instance which can be added as a sink to a Flink job.
        Parameters:
        clientConfig - The Pravega client configuration.
        stream - The destination stream.
        serializationSchema - The implementation for serializing every event into pravega's storage format.
        eventRouter - The implementation to extract the partition key from the event.
        writerMode - The Pravega writer mode.
        txnLeaseRenewalPeriod - Transaction lease renewal period in milliseconds.
        enableWatermark - Flag to indicate whether Pravega watermark needs to be enabled or not.
        enableMetrics - Flag to indicate whether metrics needs to be enabled or not.

    • Method Detail

      • getEventRouter

        public PravegaEventRouter<T> getEventRouter()
        Gets the associated event router.

      • getPravegaWriterMode

        public PravegaWriterMode getPravegaWriterMode()
        Gets this writer's operating mode.

      • getEnableWatermark

        public boolean getEnableWatermark()
        Gets this enable watermark flag.

      • open

        public void open​(org.apache.flink.configuration.Configuration parameters)
          throws java.lang.Exception
        Specified by:
        open in interface org.apache.flink.api.common.functions.RichFunction
        Overrides:
        open in class org.apache.flink.api.common.functions.AbstractRichFunction
        Throws:
        java.lang.Exception

      • invoke

        public void invoke​(T event,
                   org.apache.flink.streaming.api.functions.sink.SinkFunction.Context context)
            throws java.lang.Exception
        Specified by:
        invoke in interface org.apache.flink.streaming.api.functions.sink.SinkFunction<T>
        Throws:
        java.lang.Exception

      • close

        public void close()
           throws java.lang.Exception
        Specified by:
        close in interface org.apache.flink.api.common.functions.RichFunction
        Overrides:
        close in class org.apache.flink.api.common.functions.AbstractRichFunction
        Throws:
        java.lang.Exception

      • snapshotState

        public java.util.List<io.pravega.connectors.flink.FlinkPravegaWriter.PendingTransaction> snapshotState​(long checkpointId,
                                                                                                       long checkpointTime)
                                                                                                throws java.lang.Exception
        Specified by:
        snapshotState in interface org.apache.flink.streaming.api.checkpoint.ListCheckpointed<T>
        Throws:
        java.lang.Exception

      • restoreState

        public void restoreState​(java.util.List<io.pravega.connectors.flink.FlinkPravegaWriter.PendingTransaction> pendingTransactionList)
                  throws java.lang.Exception
        Restores the state, which here means the IDs of transaction for which we have to ensure that they are really committed. Note: restoreState is called before open.
        Specified by:
        restoreState in interface org.apache.flink.streaming.api.checkpoint.ListCheckpointed<T>
        Throws:
        java.lang.Exception

      • notifyCheckpointComplete

        public void notifyCheckpointComplete​(long checkpointId)
                              throws java.lang.Exception
        Notifies the writer that a checkpoint is complete.

        This call happens when the checkpoint has been fully committed (= second part of a two phase commit).

        This method is called under a mutually exclusive lock from the invoke() and trigger/restore methods, so there is no need for additional synchronization.

        Specified by:
        notifyCheckpointComplete in interface org.apache.flink.runtime.state.CheckpointListener
        Throws:
        java.lang.Exception

      • createClientFactory

        @VisibleForTesting
protected io.pravega.client.EventStreamClientFactory createClientFactory​(java.lang.String scopeName,
                                                                         io.pravega.client.ClientConfig clientConfig)

      • createInternalWriter

        @VisibleForTesting
protected io.pravega.connectors.flink.FlinkPravegaWriter.AbstractInternalWriter createInternalWriter()

      • createExecutorService

        @VisibleForTesting
protected java.util.concurrent.ExecutorService createExecutorService()

      • writerId

        protected java.lang.String writerId()