Package io.pravega.connectors.flink.source.reader

Class PravegaSourceReader<T>

  • Type Parameters:
    T - The final element type to emit.
    All Implemented Interfaces:
    java.lang.AutoCloseable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.api.connector.source.ExternallyInducedSourceReader<T,​PravegaSplit>, org.apache.flink.api.connector.source.SourceReader<T,​PravegaSplit>
    @Internal
public class PravegaSourceReader<T>
extends org.apache.flink.connector.base.source.reader.SourceReaderBase<io.pravega.client.stream.EventRead<java.nio.ByteBuffer>,​T,​PravegaSplit,​PravegaSplit>
implements org.apache.flink.api.connector.source.ExternallyInducedSourceReader<T,​PravegaSplit>
    A Pravega implementation of SourceReader. The PravegaSourceReader has a default recommended Flink implementation SourceReaderBase. It constructs with three major components, PravegaSplitReader, PravegaFetcherManager and PravegaRecordEmitter. Each reader will have a single threaded fetcher which will supply the split reader and assign all the splits assigned by the enumerator to it.

    PravegaSourceReader will not trigger checkpoints when receiving a trigger message from the checkpoint coordinator, but when a Pravega checkpoint event is received indicates that a checkpoint should be triggered.

    Due to Pravega's design, we don't have a mechanism to recover from a single reader currently so the failover strategy for Source Reader is to perform a full failover on Split Enumerator.

    • Field Summary

      • Fields inherited from class org.apache.flink.connector.base.source.reader.SourceReaderBase

        config, context, options, recordEmitter, splitFetcherManager

    • Constructor Summary

      Constructors 
      Constructor Description
      PravegaSourceReader​(java.util.function.Supplier<PravegaSplitReader> splitReaderSupplier, org.apache.flink.connector.base.source.reader.RecordEmitter<io.pravega.client.stream.EventRead<java.nio.ByteBuffer>,​T,​PravegaSplit> recordEmitter, org.apache.flink.configuration.Configuration config, org.apache.flink.api.connector.source.SourceReaderContext context)
      Creates a new Pravega Source Reader instance.

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      protected PravegaSplit initializedState​(PravegaSplit split)  
      protected void onSplitFinished​(java.util.Map finishedSplitIds)  
      org.apache.flink.core.io.InputStatus pollNext​(org.apache.flink.api.connector.source.ReaderOutput output)  
      java.util.Optional<java.lang.Long> shouldTriggerCheckpoint()  
      java.util.List<PravegaSplit> snapshotState​(long checkpointId)  
      protected PravegaSplit toSplitType​(java.lang.String splitId, PravegaSplit splitState)  

      • Methods inherited from class org.apache.flink.connector.base.source.reader.SourceReaderBase

        addSplits, close, getNumberOfCurrentlyAssignedSplits, handleSourceEvents, isAvailable, notifyNoMoreSplits, start

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

      • Methods inherited from interface java.lang.AutoCloseable

        close

      • Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

        notifyCheckpointAborted

      • Methods inherited from interface org.apache.flink.api.connector.source.SourceReader

        addSplits, handleSourceEvents, isAvailable, notifyCheckpointComplete, notifyNoMoreSplits, start

    • Constructor Detail

      • PravegaSourceReader

        public PravegaSourceReader​(java.util.function.Supplier<PravegaSplitReader> splitReaderSupplier,
                           org.apache.flink.connector.base.source.reader.RecordEmitter<io.pravega.client.stream.EventRead<java.nio.ByteBuffer>,​T,​PravegaSplit> recordEmitter,
                           org.apache.flink.configuration.Configuration config,
                           org.apache.flink.api.connector.source.SourceReaderContext context)
        Creates a new Pravega Source Reader instance.
        Parameters:
        splitReaderSupplier - The Pravega split reader supplier.
        recordEmitter - The Pravega Source reader record emitter.
        config - The Flink configuration.
        context - The Pravega Source reader context.

    • Method Detail

      • pollNext

        public org.apache.flink.core.io.InputStatus pollNext​(org.apache.flink.api.connector.source.ReaderOutput output)
                                              throws java.lang.Exception
        Specified by:
        pollNext in interface org.apache.flink.api.connector.source.SourceReader<T,​PravegaSplit>
        Overrides:
        pollNext in class org.apache.flink.connector.base.source.reader.SourceReaderBase<io.pravega.client.stream.EventRead<java.nio.ByteBuffer>,​T,​PravegaSplit,​PravegaSplit>
        Throws:
        java.lang.Exception

      • snapshotState

        public java.util.List<PravegaSplit> snapshotState​(long checkpointId)
        Specified by:
        snapshotState in interface org.apache.flink.api.connector.source.SourceReader<T,​PravegaSplit>
        Overrides:
        snapshotState in class org.apache.flink.connector.base.source.reader.SourceReaderBase<io.pravega.client.stream.EventRead<java.nio.ByteBuffer>,​T,​PravegaSplit,​PravegaSplit>

      • shouldTriggerCheckpoint

        public java.util.Optional<java.lang.Long> shouldTriggerCheckpoint()
        Specified by:
        shouldTriggerCheckpoint in interface org.apache.flink.api.connector.source.ExternallyInducedSourceReader<T,​PravegaSplit>

      • onSplitFinished

        protected void onSplitFinished​(java.util.Map finishedSplitIds)
        Specified by:
        onSplitFinished in class org.apache.flink.connector.base.source.reader.SourceReaderBase<io.pravega.client.stream.EventRead<java.nio.ByteBuffer>,​T,​PravegaSplit,​PravegaSplit>

      • initializedState

        protected PravegaSplit initializedState​(PravegaSplit split)
        Specified by:
        initializedState in class org.apache.flink.connector.base.source.reader.SourceReaderBase<io.pravega.client.stream.EventRead<java.nio.ByteBuffer>,​T,​PravegaSplit,​PravegaSplit>

      • toSplitType

        protected PravegaSplit toSplitType​(java.lang.String splitId,
                                   PravegaSplit splitState)
        Specified by:
        toSplitType in class org.apache.flink.connector.base.source.reader.SourceReaderBase<io.pravega.client.stream.EventRead<java.nio.ByteBuffer>,​T,​PravegaSplit,​PravegaSplit>