Package io.pravega.connectors.flink

Class FlinkPravegaReader<T>

  • Type Parameters:
    T - The type of the event to be written.
    All Implemented Interfaces:
    java.io.Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, org.apache.flink.api.common.functions.StoppableFunction, org.apache.flink.api.java.typeutils.ResultTypeQueryable<T>, org.apache.flink.streaming.api.checkpoint.ExternallyInducedSource<T,​io.pravega.client.stream.Checkpoint>, org.apache.flink.streaming.api.checkpoint.WithMasterCheckpointHook<io.pravega.client.stream.Checkpoint>, org.apache.flink.streaming.api.functions.source.ParallelSourceFunction<T>, org.apache.flink.streaming.api.functions.source.SourceFunction<T>
    public class FlinkPravegaReader<T>
extends org.apache.flink.streaming.api.functions.source.RichParallelSourceFunction<T>
implements org.apache.flink.api.java.typeutils.ResultTypeQueryable<T>, org.apache.flink.api.common.functions.StoppableFunction, org.apache.flink.streaming.api.checkpoint.ExternallyInducedSource<T,​io.pravega.client.stream.Checkpoint>
    Flink source implementation for reading from pravega storage.
    See Also:
    Serialized Form

    • Nested Class Summary

      Nested Classes 
      Modifier and Type Class Description
      static class  FlinkPravegaReader.Builder<T>
      A builder for FlinkPravegaReader.

      • Nested classes/interfaces inherited from interface org.apache.flink.streaming.api.checkpoint.ExternallyInducedSource

        org.apache.flink.streaming.api.checkpoint.ExternallyInducedSource.CheckpointTrigger

      • Nested classes/interfaces inherited from interface org.apache.flink.streaming.api.functions.source.SourceFunction

        org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T extends java.lang.Object>

    • Constructor Summary

      Constructors 
      Modifier Constructor Description
      protected FlinkPravegaReader​(java.lang.String hookUid, io.pravega.client.ClientConfig clientConfig, io.pravega.client.stream.ReaderGroupConfig readerGroupConfig, java.lang.String readerGroupScope, java.lang.String readerGroupName, org.apache.flink.api.common.serialization.DeserializationSchema<T> deserializationSchema, org.apache.flink.util.SerializedValue<AssignerWithTimeWindows<T>> assignerWithTimeWindows, org.apache.flink.api.common.time.Time eventReadTimeout, org.apache.flink.api.common.time.Time checkpointInitiateTimeout, boolean enableMetrics)
      Creates a new Flink Pravega reader instance which can be added as a source to a Flink job.

    • Method Summary

      All Methods Static Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      static <T> FlinkPravegaReader.Builder<T> builder()
      Gets a builder for FlinkPravegaReader to read Pravega streams using the Flink streaming API.
      void cancel()  
      void close()  
      protected io.pravega.client.EventStreamClientFactory createEventStreamClientFactory()
      Create the EventStreamClientFactory for the current configuration.
      protected io.pravega.client.stream.EventStreamReader<T> createEventStreamReader​(java.lang.String readerId)
      Create the EventStreamReader for the current configuration.
      org.apache.flink.runtime.checkpoint.MasterTriggerRestoreHook<io.pravega.client.stream.Checkpoint> createMasterTriggerRestoreHook()  
      protected io.pravega.client.admin.ReaderGroupManager createReaderGroupManager()
      Create the ReaderGroupManager for the current configuration.
      org.apache.flink.api.common.typeinfo.TypeInformation<T> getProducedType()  
      void open​(org.apache.flink.configuration.Configuration parameters)  
      void run​(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T> ctx)  
      void setCheckpointTrigger​(org.apache.flink.streaming.api.checkpoint.ExternallyInducedSource.CheckpointTrigger checkpointTrigger)  
      void stop()  

      • Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

        getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Field Detail

      • PRAVEGA_READER_METRICS_GROUP

        protected static final java.lang.String PRAVEGA_READER_METRICS_GROUP
        See Also:
        Constant Field Values

      • READER_GROUP_METRICS_GROUP

        protected static final java.lang.String READER_GROUP_METRICS_GROUP
        See Also:
        Constant Field Values

      • STREAM_METRICS_GROUP

        protected static final java.lang.String STREAM_METRICS_GROUP
        See Also:
        Constant Field Values

      • UNREAD_BYTES_METRICS_GAUGE

        protected static final java.lang.String UNREAD_BYTES_METRICS_GAUGE
        See Also:
        Constant Field Values

      • READER_GROUP_NAME_METRICS_GAUGE

        protected static final java.lang.String READER_GROUP_NAME_METRICS_GAUGE
        See Also:
        Constant Field Values

      • SCOPE_NAME_METRICS_GAUGE

        protected static final java.lang.String SCOPE_NAME_METRICS_GAUGE
        See Also:
        Constant Field Values

      • ONLINE_READERS_METRICS_GAUGE

        protected static final java.lang.String ONLINE_READERS_METRICS_GAUGE
        See Also:
        Constant Field Values

      • STREAM_NAMES_METRICS_GAUGE

        protected static final java.lang.String STREAM_NAMES_METRICS_GAUGE
        See Also:
        Constant Field Values

      • SEGMENT_POSITIONS_METRICS_GAUGE

        protected static final java.lang.String SEGMENT_POSITIONS_METRICS_GAUGE
        See Also:
        Constant Field Values

      • eventStreamClientFactory

        protected transient io.pravega.client.EventStreamClientFactory eventStreamClientFactory

      • readerGroupManager

        protected transient io.pravega.client.admin.ReaderGroupManager readerGroupManager

    • Constructor Detail

      • FlinkPravegaReader

        protected FlinkPravegaReader​(java.lang.String hookUid,
                             io.pravega.client.ClientConfig clientConfig,
                             io.pravega.client.stream.ReaderGroupConfig readerGroupConfig,
                             java.lang.String readerGroupScope,
                             java.lang.String readerGroupName,
                             org.apache.flink.api.common.serialization.DeserializationSchema<T> deserializationSchema,
                             org.apache.flink.util.SerializedValue<AssignerWithTimeWindows<T>> assignerWithTimeWindows,
                             org.apache.flink.api.common.time.Time eventReadTimeout,
                             org.apache.flink.api.common.time.Time checkpointInitiateTimeout,
                             boolean enableMetrics)
        Creates a new Flink Pravega reader instance which can be added as a source to a Flink job.

        The reader will use the given readerName to store its state (its positions in the stream segments) in Flink's checkpoints/savepoints. This name is used in a similar way as the operator UIDs (SingleOutputStreamOperator.uid(String)) to identify state when matching it into another job that resumes from this job's checkpoints/savepoints.

        Without specifying a readerName, the job will correctly checkpoint and recover, but new instances of the job can typically not resume this reader's state (positions).

        Parameters:
        hookUid - The UID of the source hook in the job graph.
        clientConfig - The Pravega client configuration.
        readerGroupConfig - The Pravega reader group configuration.
        readerGroupScope - The reader group scope name.
        readerGroupName - The reader group name.
        deserializationSchema - The implementation to deserialize events from Pravega streams.
        assignerWithTimeWindows - The serialized value of the implementation to extract timestamp from deserialized events (only in event-time mode).
        eventReadTimeout - The event read timeout.
        checkpointInitiateTimeout - The checkpoint initiation timeout.
        enableMetrics - Flag to indicate whether metrics needs to be enabled or not.

    • Method Detail

      • run

        public void run​(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T> ctx)
         throws java.lang.Exception
        Specified by:
        run in interface org.apache.flink.streaming.api.functions.source.SourceFunction<T>
        Throws:
        java.lang.Exception

      • cancel

        public void cancel()
        Specified by:
        cancel in interface org.apache.flink.streaming.api.functions.source.SourceFunction<T>

      • stop

        public void stop()
        Specified by:
        stop in interface org.apache.flink.api.common.functions.StoppableFunction

      • getProducedType

        public org.apache.flink.api.common.typeinfo.TypeInformation<T> getProducedType()
        Specified by:
        getProducedType in interface org.apache.flink.api.java.typeutils.ResultTypeQueryable<T>

      • open

        public void open​(org.apache.flink.configuration.Configuration parameters)
          throws java.lang.Exception
        Specified by:
        open in interface org.apache.flink.api.common.functions.RichFunction
        Overrides:
        open in class org.apache.flink.api.common.functions.AbstractRichFunction
        Throws:
        java.lang.Exception

      • close

        public void close()
           throws java.lang.Exception
        Specified by:
        close in interface org.apache.flink.api.common.functions.RichFunction
        Overrides:
        close in class org.apache.flink.api.common.functions.AbstractRichFunction
        Throws:
        java.lang.Exception

      • createMasterTriggerRestoreHook

        public org.apache.flink.runtime.checkpoint.MasterTriggerRestoreHook<io.pravega.client.stream.Checkpoint> createMasterTriggerRestoreHook()
        Specified by:
        createMasterTriggerRestoreHook in interface org.apache.flink.streaming.api.checkpoint.WithMasterCheckpointHook<T>

      • setCheckpointTrigger

        public void setCheckpointTrigger​(org.apache.flink.streaming.api.checkpoint.ExternallyInducedSource.CheckpointTrigger checkpointTrigger)
        Specified by:
        setCheckpointTrigger in interface org.apache.flink.streaming.api.checkpoint.ExternallyInducedSource<T,​io.pravega.client.stream.Checkpoint>

      • createReaderGroupManager

        protected io.pravega.client.admin.ReaderGroupManager createReaderGroupManager()
        Create the ReaderGroupManager for the current configuration.

      • createEventStreamClientFactory

        protected io.pravega.client.EventStreamClientFactory createEventStreamClientFactory()
        Create the EventStreamClientFactory for the current configuration.

      • createEventStreamReader

        protected io.pravega.client.stream.EventStreamReader<T> createEventStreamReader​(java.lang.String readerId)
        Create the EventStreamReader for the current configuration.
        Parameters:
        readerId - the readerID to use.