Package org.apache.flink.runtime.checkpoint

Interface MasterTriggerRestoreHook<T>

  • Type Parameters:
    T - The type of the data produced by the hook and stored as part of the checkpoint metadata. If the hook never stores any data, this can be typed to Void.
    public interface MasterTriggerRestoreHook<T>
    The interface for hooks that can be called by the checkpoint coordinator when triggering or restoring a checkpoint. Such a hook is useful for example when preparing external systems for taking or restoring checkpoints.

    The triggerCheckpoint(long, long, Executor) method (called when triggering a checkpoint) can return a result (via a future) that will be stored as part of the checkpoint metadata. When restoring a checkpoint, that stored result will be given to the restoreCheckpoint(long, Object) method. The hook's identifier is used to map data to hook in the presence of multiple hooks, and when resuming a savepoint that was potentially created by a different job. The identifier has a similar role as for example the operator UID in the streaming API.

    It is possible that a job fails (and is subsequently restarted) before any checkpoints were successful. In that situation, the checkpoint coordination calls reset() to give the hook an opportunity to, for example, reset an external system to initial conditions.

    The MasterTriggerRestoreHook is defined when creating the streaming dataflow graph. It is attached to the job graph, which gets sent to the cluster for execution. To avoid having to make the hook itself serializable, these hooks are attached to the job graph via a MasterTriggerRestoreHook.Factory.

    • Nested Class Summary

      Nested Classes 
      Modifier and Type Interface Description
      static interface  MasterTriggerRestoreHook.Factory
      A factory to instantiate a MasterTriggerRestoreHook.

    • Method Summary

      All Methods Instance Methods Abstract Methods Default Methods 
      Modifier and Type Method Description
      default void close()
      Tear-down method for the hook.
      org.apache.flink.core.io.SimpleVersionedSerializer<T> createCheckpointDataSerializer()
      Creates a the serializer to (de)serializes the data stored by this hook.
      String getIdentifier()
      Gets the identifier of this hook.
      default void reset()
      This method is called by the checkpoint coordinator to reset the hook when execution is restarted in the absence of any checkpoint state.
      void restoreCheckpoint​(long checkpointId, T checkpointData)
      This method is called by the checkpoint coordinator prior to restoring the state of a checkpoint.
      CompletableFuture<T> triggerCheckpoint​(long checkpointId, long timestamp, Executor executor)
      This method is called by the checkpoint coordinator prior when triggering a checkpoint, prior to sending the "trigger checkpoint" messages to the source tasks.

    • Method Detail

      • getIdentifier

        String getIdentifier()
        Gets the identifier of this hook. The identifier is used to identify a specific hook in the presence of multiple hooks and to give it the correct checkpointed data upon checkpoint restoration.

        The identifier should be unique between different hooks of a job, but deterministic/constant so that upon resuming a savepoint, the hook will get the correct data. For example, if the hook calls into another storage system and persists namespace/schema specific information, then the name of the storage system, together with the namespace/schema name could be an appropriate identifier.

        When multiple hooks of the same name are created and attached to a job graph, only the first one is actually used. This can be exploited to deduplicate hooks that would do the same thing.

        Returns:
        The identifier of the hook.

      • reset

        default void reset()
            throws Exception
        This method is called by the checkpoint coordinator to reset the hook when execution is restarted in the absence of any checkpoint state.
        Throws:
        Exception - Exceptions encountered when calling the hook will cause execution to fail.

      • close

        default void close()
            throws Exception
        Tear-down method for the hook.
        Throws:
        Exception - Exceptions encountered when calling close will be logged.

      • triggerCheckpoint

        @Nullable
CompletableFuture<T> triggerCheckpoint​(long checkpointId,
                                       long timestamp,
                                       Executor executor)
                                throws Exception
        This method is called by the checkpoint coordinator prior when triggering a checkpoint, prior to sending the "trigger checkpoint" messages to the source tasks.

        If the hook implementation wants to store data as part of the checkpoint, it may return that data via a future, otherwise it should return null. The data is stored as part of the checkpoint metadata under the hooks identifier (see getIdentifier()).

        If the action by this hook needs to be executed synchronously, then this method should directly execute the action synchronously. The returned future (if any) would typically be a completed future.

        If the action should be executed asynchronously and only needs to complete before the checkpoint is considered completed, then the method may use the given executor to execute the actual action and would signal its completion by completing the future. For hooks that do not need to store data, the future would be completed with null.

        Please note that this method should be non-blocking. Any heavy operation like IO operation should be executed asynchronously with given executor.

        Parameters:
        checkpointId - The ID (logical timestamp, monotonously increasing) of the checkpoint
        timestamp - The wall clock timestamp when the checkpoint was triggered, for info/logging purposes.
        executor - The executor for asynchronous actions
        Returns:
        Optionally, a future that signals when the hook has completed and that contains data to be stored with the checkpoint.
        Throws:
        Exception - Exceptions encountered when calling the hook will cause the checkpoint to abort.

      • restoreCheckpoint

        void restoreCheckpoint​(long checkpointId,
                       @Nullable
                       T checkpointData)
                throws Exception
        This method is called by the checkpoint coordinator prior to restoring the state of a checkpoint. If the checkpoint did store data from this hook, that data will be passed to this method.
        Parameters:
        checkpointId - The ID (logical timestamp) of the restored checkpoint
        checkpointData - The data originally stored in the checkpoint by this hook, possibly null.
        Throws:
        Exception - Exceptions thrown while restoring the checkpoint will cause the restore operation to fail and to possibly fall back to another checkpoint.

      • createCheckpointDataSerializer

        @Nullable
org.apache.flink.core.io.SimpleVersionedSerializer<T> createCheckpointDataSerializer()
        Creates a the serializer to (de)serializes the data stored by this hook. The serializer serializes the result of the Future returned by the triggerCheckpoint(long, long, Executor) method, and deserializes the data stored in the checkpoint into the object passed to the restoreCheckpoint(long, Object) method.

        If the hook never returns any data to be stored, then this method may return null as the serializer.

        Returns:
        The serializer to (de)serializes the data stored by this hook