Package org.apache.flink.runtime.minicluster

Class MiniCluster

  • All Implemented Interfaces:
    AutoCloseable, org.apache.flink.util.AutoCloseableAsync
    public class MiniCluster
extends Object
implements org.apache.flink.util.AutoCloseableAsync
    MiniCluster to execute Flink jobs locally.

    • Constructor Detail

      • MiniCluster

        public MiniCluster​(MiniClusterConfiguration miniClusterConfiguration)
        Creates a new Flink mini cluster based on the given configuration.
        Parameters:
        miniClusterConfiguration - The configuration for the mini cluster

      • MiniCluster

        public MiniCluster​(MiniClusterConfiguration miniClusterConfiguration,
                   Supplier<org.apache.flink.util.Reference<org.apache.flink.runtime.rpc.RpcSystem>> rpcSystemSupplier)

    • Method Detail

      • getIOExecutor

        protected Executor getIOExecutor()

      • isRunning

        public boolean isRunning()
        Checks if the mini cluster was started and is running.

      • start

        public void start()
           throws Exception
        Starts the mini cluster, based on the configured properties.
        Throws:
        Exception - This method passes on any exception that occurs during the startup of the mini cluster.

      • closeAsync

        public CompletableFuture<Void> closeAsync()
        Shuts down the mini cluster, failing all currently executing jobs. The mini cluster can be started again by calling the start() method again.

        This method shuts down all started services and components, even if an exception occurs in the process of shutting down some component.

        Specified by:
        closeAsync in interface org.apache.flink.util.AutoCloseableAsync
        Returns:
        Future which is completed once the MiniCluster has been completely shut down

      • closeAsyncWithoutCleaningHighAvailabilityData

        public CompletableFuture<Void> closeAsyncWithoutCleaningHighAvailabilityData()

      • startTaskManager

        public void startTaskManager()
                      throws Exception
        Starts additional TaskManager process.

        When the MiniCluster starts up, it always starts MiniClusterConfiguration.getNumTaskManagers() TaskManagers. All TaskManagers are indexed from 0 to the number of TaskManagers, started so far, minus one. This method starts a TaskManager with the next index which is the number of TaskManagers, started so far. The index always increases with each new started TaskManager. The indices of terminated TaskManagers are not reused after terminateTaskManager(int).

        Throws:
        Exception

      • useLocalCommunication

        @VisibleForTesting
protected boolean useLocalCommunication()

      • getConfiguration

        @VisibleForTesting
public org.apache.flink.configuration.Configuration getConfiguration()

      • overrideRestoreModeForChangelogStateBackend

        @Internal
public void overrideRestoreModeForChangelogStateBackend()

      • terminateTaskManager

        public CompletableFuture<Void> terminateTaskManager​(int index)
        Terminates a TaskManager with the given index.

        See startTaskManager() to understand how TaskManagers are indexed. This method terminates a TaskManager with a given index but it does not clear the index. The index stays occupied for the lifetime of the MiniCluster and its TaskManager stays terminated. The index is not reused if more TaskManagers are started with startTaskManager().

        Parameters:
        index - index of the TaskManager to terminate
        Returns:
        CompletableFuture of the given TaskManager termination

      • getJobStatus

        public CompletableFuture<org.apache.flink.api.common.JobStatus> getJobStatus​(org.apache.flink.api.common.JobID jobId)

      • triggerSavepoint

        public CompletableFuture<String> triggerSavepoint​(org.apache.flink.api.common.JobID jobId,
                                                  String targetDirectory,
                                                  boolean cancelJob,
                                                  org.apache.flink.core.execution.SavepointFormatType formatType)

      • triggerDetachedSavepoint

        public CompletableFuture<String> triggerDetachedSavepoint​(org.apache.flink.api.common.JobID jobId,
                                                          String targetDirectory,
                                                          boolean cancelJob,
                                                          org.apache.flink.core.execution.SavepointFormatType formatType)

      • triggerCheckpoint

        public CompletableFuture<String> triggerCheckpoint​(org.apache.flink.api.common.JobID jobID)

      • triggerCheckpoint

        public CompletableFuture<Long> triggerCheckpoint​(org.apache.flink.api.common.JobID jobID,
                                                 org.apache.flink.core.execution.CheckpointType checkpointType)

      • stopWithSavepoint

        public CompletableFuture<String> stopWithSavepoint​(org.apache.flink.api.common.JobID jobId,
                                                   String targetDirectory,
                                                   boolean terminate,
                                                   org.apache.flink.core.execution.SavepointFormatType formatType)

      • stopWithDetachedSavepoint

        public CompletableFuture<String> stopWithDetachedSavepoint​(org.apache.flink.api.common.JobID jobId,
                                                           String targetDirectory,
                                                           boolean terminate,
                                                           org.apache.flink.core.execution.SavepointFormatType formatType)

      • executeJobBlocking

        public org.apache.flink.api.common.JobExecutionResult executeJobBlocking​(JobGraph job)
                                                                  throws JobExecutionException,
                                                                         InterruptedException
        This method runs a job in blocking mode. The method returns only after the job completed successfully, or after it failed terminally.
        Parameters:
        job - The Flink job to execute
        Returns:
        The result of the job execution
        Throws:
        JobExecutionException - Thrown if anything went amiss during initial job launch, or if the job terminally failed.
        InterruptedException

      • createMetricRegistry

        protected MetricRegistryImpl createMetricRegistry​(org.apache.flink.configuration.Configuration config,
                                                  long maximumMessageSizeInBytes)
        Factory method to create the metric registry for the mini cluster.
        Parameters:
        config - The configuration of the mini cluster
        maximumMessageSizeInBytes - the maximum message size

      • createRemoteRpcService

        protected org.apache.flink.runtime.rpc.RpcService createRemoteRpcService​(org.apache.flink.configuration.Configuration configuration,
                                                                         String bindAddress,
                                                                         int bindPort,
                                                                         org.apache.flink.runtime.rpc.RpcSystem rpcSystem)
                                                                  throws Exception
        Factory method to instantiate the remote RPC service.
        Parameters:
        configuration - Flink configuration.
        bindAddress - The address to bind the RPC service to.
        bindPort - The port range to bind the RPC service to.
        rpcSystem -
        Returns:
        The instantiated RPC service
        Throws:
        Exception

      • createRemoteRpcService

        protected org.apache.flink.runtime.rpc.RpcService createRemoteRpcService​(org.apache.flink.configuration.Configuration configuration,
                                                                         String externalAddress,
                                                                         String externalPortRange,
                                                                         String bindAddress,
                                                                         org.apache.flink.runtime.rpc.RpcSystem rpcSystem)
                                                                  throws Exception
        Factory method to instantiate the remote RPC service.
        Parameters:
        configuration - Flink configuration.
        externalAddress - The external address to access the RPC service.
        externalPortRange - The external port range to access the RPC service.
        bindAddress - The address to bind the RPC service to.
        rpcSystem -
        Returns:
        The instantiated RPC service
        Throws:
        Exception

      • createLocalRpcService

        protected org.apache.flink.runtime.rpc.RpcService createLocalRpcService​(org.apache.flink.configuration.Configuration configuration,
                                                                        org.apache.flink.runtime.rpc.RpcSystem rpcSystem)
                                                                 throws Exception
        Factory method to instantiate the local RPC service.
        Parameters:
        configuration - Flink configuration.
        rpcSystem -
        Returns:
        The instantiated RPC service
        Throws:
        Exception

      • invalidateClusterDataset

        public CompletableFuture<Void> invalidateClusterDataset​(org.apache.flink.util.AbstractID clusterDatasetId)

      • listCompletedClusterDatasetIds

        public CompletableFuture<Set<org.apache.flink.util.AbstractID>> listCompletedClusterDatasetIds()

      • reportHeartbeat

        public CompletableFuture<Void> reportHeartbeat​(org.apache.flink.api.common.JobID jobId,
                                               long expiredTimestamp)