Package io.pravega.connectors.flink.source.enumerator

Class PravegaSplitEnumerator

  • All Implemented Interfaces:
    java.lang.AutoCloseable, org.apache.flink.api.common.state.CheckpointListener, org.apache.flink.api.connector.source.SplitEnumerator<PravegaSplit,​io.pravega.client.stream.Checkpoint>
    @Internal
public class PravegaSplitEnumerator
extends java.lang.Object
implements org.apache.flink.api.connector.source.SplitEnumerator<PravegaSplit,​io.pravega.client.stream.Checkpoint>
    The enumerator class for Pravega source. It is a single instance on Flink jobmanager. It is the "brain" of the source to initialize the reader group when it starts, then discover and assign the subtasks.

    The PravegaSplitEnumerator will assign splits to source readers. Pravega source pushes splits eagerly so that the enumerator will create a source reader and assign one split(Pravega reader) to it. One Pravega Source reader is only mapped to one Pravega Split as design.

    We triggers and restores checkpoints on a Pravega ReaderGroup along with Flink checkpoints for dealing with failure. Due to Pravega's design, we don't have a mechanism to recover from a single reader currently. We will perform a full failover both when Split Enumerator fails or Source Reader fails. The Split Enumerator will be restored to to its last successful checkpoint.

    • Constructor Summary

      Constructors 
      Constructor Description
      PravegaSplitEnumerator​(org.apache.flink.api.connector.source.SplitEnumeratorContext<PravegaSplit> context, java.lang.String scope, java.lang.String readerGroupName, io.pravega.client.ClientConfig clientConfig, io.pravega.client.stream.ReaderGroupConfig readerGroupConfig, io.pravega.client.stream.Checkpoint checkpoint)
      Creates a new Pravega Split Enumerator instance which can connect to a Pravega reader group with the pravega stream.

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      void addReader​(int subtaskId)  
      void addSplitsBack​(java.util.List<PravegaSplit> splits, int subtaskId)  
      void close()  
      protected io.pravega.client.stream.ReaderGroup createReaderGroup()
      Create the ReaderGroup for the current configuration.
      protected io.pravega.client.admin.ReaderGroupManager createReaderGroupManager()
      Create the ReaderGroupManager for the current configuration.
      void handleSplitRequest​(int subtaskId, java.lang.String requesterHostname)  
      io.pravega.client.stream.Checkpoint snapshotState​(long chkPtID)  
      void start()  

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

      • Methods inherited from interface org.apache.flink.api.common.state.CheckpointListener

        notifyCheckpointAborted

      • Methods inherited from interface org.apache.flink.api.connector.source.SplitEnumerator

        handleSourceEvent, notifyCheckpointComplete

    • Constructor Detail

      • PravegaSplitEnumerator

        public PravegaSplitEnumerator​(org.apache.flink.api.connector.source.SplitEnumeratorContext<PravegaSplit> context,
                              java.lang.String scope,
                              java.lang.String readerGroupName,
                              io.pravega.client.ClientConfig clientConfig,
                              io.pravega.client.stream.ReaderGroupConfig readerGroupConfig,
                              io.pravega.client.stream.Checkpoint checkpoint)
        Creates a new Pravega Split Enumerator instance which can connect to a Pravega reader group with the pravega stream.
        Parameters:
        context - The Pravega Split Enumeratior context.
        scope - The reader group scope name.
        readerGroupName - The reader group name.
        clientConfig - The Pravega client configuration.
        readerGroupConfig - The Pravega reader group configuration.
        checkpoint - The Pravega checkpoint.

    • Method Detail

      • start

        public void start()
        Specified by:
        start in interface org.apache.flink.api.connector.source.SplitEnumerator<PravegaSplit,​io.pravega.client.stream.Checkpoint>

      • handleSplitRequest

        public void handleSplitRequest​(int subtaskId,
                               @Nullable
                               java.lang.String requesterHostname)
        Specified by:
        handleSplitRequest in interface org.apache.flink.api.connector.source.SplitEnumerator<PravegaSplit,​io.pravega.client.stream.Checkpoint>

      • addReader

        public void addReader​(int subtaskId)
        Specified by:
        addReader in interface org.apache.flink.api.connector.source.SplitEnumerator<PravegaSplit,​io.pravega.client.stream.Checkpoint>

      • snapshotState

        public io.pravega.client.stream.Checkpoint snapshotState​(long chkPtID)
                                                  throws java.lang.Exception
        Specified by:
        snapshotState in interface org.apache.flink.api.connector.source.SplitEnumerator<PravegaSplit,​io.pravega.client.stream.Checkpoint>
        Throws:
        java.lang.Exception

      • close

        public void close()
           throws java.io.IOException
        Specified by:
        close in interface java.lang.AutoCloseable
        Specified by:
        close in interface org.apache.flink.api.connector.source.SplitEnumerator<PravegaSplit,​io.pravega.client.stream.Checkpoint>
        Throws:
        java.io.IOException

      • addSplitsBack

        public void addSplitsBack​(java.util.List<PravegaSplit> splits,
                          int subtaskId)
        Specified by:
        addSplitsBack in interface org.apache.flink.api.connector.source.SplitEnumerator<PravegaSplit,​io.pravega.client.stream.Checkpoint>

      • createReaderGroup

        protected io.pravega.client.stream.ReaderGroup createReaderGroup()
        Create the ReaderGroup for the current configuration.
        Returns:
        An instance of ReaderGroup

      • createReaderGroupManager

        protected io.pravega.client.admin.ReaderGroupManager createReaderGroupManager()
        Create the ReaderGroupManager for the current configuration.
        Returns:
        An instance of ReaderGroupManager