Package org.apache.flink.runtime.scheduler.adaptivebatch

Class AdaptiveBatchScheduler

java.lang.Object

org.apache.flink.runtime.scheduler.SchedulerBase

org.apache.flink.runtime.scheduler.DefaultScheduler

org.apache.flink.runtime.scheduler.adaptivebatch.AdaptiveBatchScheduler

All Implemented Interfaces:

AutoCloseable, CheckpointScheduling, JobGraphUpdateListener, GlobalFailureHandler, SchedulerNG, SchedulerOperations, org.apache.flink.util.AutoCloseableAsync

public class AdaptiveBatchScheduler
extends DefaultScheduler
implements JobGraphUpdateListener

This scheduler decides the parallelism of JobVertex according to the data volume it consumes. A dynamically built up ExecutionGraph is used for this purpose.

Field Summary

Fields inherited from class org.apache.flink.runtime.scheduler.DefaultScheduler

executionDeployer, executionSlotAllocator, failoverStrategy, log, schedulingStrategy, shuffleMaster

Fields inherited from class org.apache.flink.runtime.scheduler.SchedulerBase

executionVertexVersioner, inputsLocationsRetriever, jobInfo, jobManagerJobMetricGroup, operatorCoordinatorHandler, stateLocationRetriever

Constructor Summary

Constructors

Constructor

Description

AdaptiveBatchScheduler(org.slf4j.Logger log, AdaptiveExecutionHandler adaptiveExecutionHandler, Executor ioExecutor, org.apache.flink.configuration.Configuration jobMasterConfiguration, Consumer<org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor> startUpAction, org.apache.flink.util.concurrent.ScheduledExecutor delayExecutor, ClassLoader userCodeLoader, CheckpointsCleaner checkpointsCleaner, CheckpointRecoveryFactory checkpointRecoveryFactory, JobManagerJobMetricGroup jobManagerJobMetricGroup, SchedulingStrategyFactory schedulingStrategyFactory, FailoverStrategy.Factory failoverStrategyFactory, RestartBackoffTimeStrategy restartBackoffTimeStrategy, ExecutionOperations executionOperations, ExecutionVertexVersioner executionVertexVersioner, ExecutionSlotAllocatorFactory executionSlotAllocatorFactory, long initializationTimestamp, org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor mainThreadExecutor, JobStatusListener jobStatusListener, Collection<org.apache.flink.core.failure.FailureEnricher> failureEnrichers, ExecutionGraphFactory executionGraphFactory, ShuffleMaster<?> shuffleMaster, Duration rpcTimeout, VertexParallelismAndInputInfosDecider vertexParallelismAndInputInfosDecider, int defaultMaxParallelism, BlocklistOperations blocklistOperations, org.apache.flink.configuration.JobManagerOptions.HybridPartitionDataConsumeConstraint hybridPartitionDataConsumeConstraint, BatchJobRecoveryHandler jobRecoveryHandler, ExecutionPlanSchedulingContext executionPlanSchedulingContext)

Method Summary

Modifier and Type	Method	Description
void	allocateSlotsAndDeploy(List<ExecutionVertexID> verticesToDeploy)	Allocate slots and deploy the vertex when slots are returned.
CompletableFuture<Void>	closeAsync()
List<CompletableFuture<Integer>>	computeDynamicSourceParallelism()
static int	computeMaxParallelism(int parallelism, int defaultMaxParallelism)
static VertexParallelismStore	computeVertexParallelismStoreForDynamicGraph(Iterable<JobVertex> vertices, int defaultMaxParallelism)	Compute the VertexParallelismStore for all given vertices in a dynamic graph, which will set defaults and ensure that the returned store contains valid parallelisms, with the configured default max parallelism.
protected MarkPartitionFinishedStrategy	getMarkPartitionFinishedStrategy()
protected void	handleTaskFailure(Execution failedExecution, Throwable error)
void	initializeVerticesIfPossible()
protected void	maybeRestartTasks(FailureHandlingResult failureHandlingResult)	Modifies the vertices which need to be restarted.
void	onNewJobVerticesAdded(List<JobVertex> newVertices, int pendingOperatorsCount)	Invoked when new JobVertex instances are added to the JobGraph of a specific job.
protected void	onTaskFailed(Execution execution)
protected void	onTaskFinished(Execution execution, IOMetrics ioMetrics)
protected void	resetForNewExecution(ExecutionVertexID executionVertexId)
protected void	resetForNewExecutions(Collection<ExecutionVertexID> vertices)
protected void	startSchedulingInternal()

Methods inherited from class org.apache.flink.runtime.scheduler.DefaultScheduler

cancelAllPendingSlotRequestsForVertex, cancelAllPendingSlotRequestsInternal, cancelExecution, createFailureHandlingResultSnapshot, getNumberOfRescales, getNumberOfRestarts, getUserCodeLoader, handleGlobalFailure, notifyCoordinatorsAboutTaskFailure, recordTaskFailure

Methods inherited from class org.apache.flink.runtime.scheduler.SchedulerBase

acknowledgeCheckpoint, archiveFromFailureHandlingResult, archiveGlobalFailure, cancel, computeVertexParallelismStore, computeVertexParallelismStore, computeVertexParallelismStore, computeVertexParallelismStore, declineCheckpoint, deliverCoordinationRequestToCoordinator, deliverOperatorEventToCoordinator, failJob, getDefaultMaxParallelism, getDefaultMaxParallelism, getExceptionHistory, getExecutionGraph, getExecutionJobVertex, getExecutionVertex, getJobGraph, getJobId, getJobTerminationFuture, getMainThreadExecutor, getResultPartitionAvailabilityChecker, getSchedulingTopology, notifyEndOfData, notifyKvStateRegistered, notifyKvStateUnregistered, registerJobMetrics, reportCheckpointMetrics, reportInitializationMetrics, requestCheckpointStats, requestJob, requestJobStatus, requestKvStateLocation, requestNextInputSplit, requestPartitionState, restoreState, setGlobalFailureCause, startCheckpointScheduler, startScheduling, stopCheckpointScheduler, stopWithSavepoint, transitionExecutionGraphState, transitionToRunning, transitionToScheduled, triggerCheckpoint, triggerSavepoint, updateAccumulators, updateTaskExecutionState

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.util.AutoCloseableAsync

close

Methods inherited from interface org.apache.flink.runtime.scheduler.SchedulerNG

requestJobResourceRequirements, updateJobResourceRequirements, updateTaskExecutionState

Constructor Detail

AdaptiveBatchScheduler

public AdaptiveBatchScheduler(org.slf4j.Logger log,
                              AdaptiveExecutionHandler adaptiveExecutionHandler,
                              Executor ioExecutor,
                              org.apache.flink.configuration.Configuration jobMasterConfiguration,
                              Consumer<org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor> startUpAction,
                              org.apache.flink.util.concurrent.ScheduledExecutor delayExecutor,
                              ClassLoader userCodeLoader,
                              CheckpointsCleaner checkpointsCleaner,
                              CheckpointRecoveryFactory checkpointRecoveryFactory,
                              JobManagerJobMetricGroup jobManagerJobMetricGroup,
                              SchedulingStrategyFactory schedulingStrategyFactory,
                              FailoverStrategy.Factory failoverStrategyFactory,
                              RestartBackoffTimeStrategy restartBackoffTimeStrategy,
                              ExecutionOperations executionOperations,
                              ExecutionVertexVersioner executionVertexVersioner,
                              ExecutionSlotAllocatorFactory executionSlotAllocatorFactory,
                              long initializationTimestamp,
                              org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor mainThreadExecutor,
                              JobStatusListener jobStatusListener,
                              Collection<org.apache.flink.core.failure.FailureEnricher> failureEnrichers,
                              ExecutionGraphFactory executionGraphFactory,
                              ShuffleMaster<?> shuffleMaster,
                              Duration rpcTimeout,
                              VertexParallelismAndInputInfosDecider vertexParallelismAndInputInfosDecider,
                              int defaultMaxParallelism,
                              BlocklistOperations blocklistOperations,
                              org.apache.flink.configuration.JobManagerOptions.HybridPartitionDataConsumeConstraint hybridPartitionDataConsumeConstraint,
                              BatchJobRecoveryHandler jobRecoveryHandler,
                              ExecutionPlanSchedulingContext executionPlanSchedulingContext)
                       throws Exception

Throws:

Exception

Method Detail

onNewJobVerticesAdded

public void onNewJobVerticesAdded(List<JobVertex> newVertices,
                                  int pendingOperatorsCount)
                           throws Exception

Description copied from interface: JobGraphUpdateListener

Invoked when new JobVertex instances are added to the JobGraph of a specific job. This allows interested components to react to the addition of new vertices to the job topology.

Specified by:

onNewJobVerticesAdded in interface JobGraphUpdateListener

Parameters:

newVertices - A list of newly added JobVertex instances.

pendingOperatorsCount - The number of pending operators.

Throws:

Exception

startSchedulingInternal

protected void startSchedulingInternal()

Overrides:

startSchedulingInternal in class DefaultScheduler

maybeRestartTasks

protected void maybeRestartTasks(FailureHandlingResult failureHandlingResult)

Modifies the vertices which need to be restarted. If any task needing restarting belongs to job vertices with unrecovered operator coordinators, all tasks within those job vertices need to be restarted once.

Overrides:

maybeRestartTasks in class DefaultScheduler

resetForNewExecutions

protected void resetForNewExecutions(Collection<ExecutionVertexID> vertices)

Overrides:

resetForNewExecutions in class SchedulerBase

closeAsync

public CompletableFuture<Void> closeAsync()

Specified by:

closeAsync in interface org.apache.flink.util.AutoCloseableAsync

Overrides:

closeAsync in class SchedulerBase

onTaskFinished

protected void onTaskFinished(Execution execution,
                              IOMetrics ioMetrics)

Overrides:

onTaskFinished in class DefaultScheduler

onTaskFailed

protected void onTaskFailed(Execution execution)

Overrides:

onTaskFailed in class DefaultScheduler

handleTaskFailure

protected void handleTaskFailure(Execution failedExecution,
                                 @Nullable
                                 Throwable error)

Overrides:

handleTaskFailure in class DefaultScheduler

allocateSlotsAndDeploy

public void allocateSlotsAndDeploy(List<ExecutionVertexID> verticesToDeploy)

Description copied from interface: SchedulerOperations

Allocate slots and deploy the vertex when slots are returned. Vertices will be deployed only after all of them have been assigned slots. The given order will be respected, i.e. tasks with smaller indices will be deployed earlier. Only vertices in CREATED state will be accepted. Errors will happen if scheduling Non-CREATED vertices.

Specified by:

allocateSlotsAndDeploy in interface SchedulerOperations

Overrides:

allocateSlotsAndDeploy in class DefaultScheduler

Parameters:

verticesToDeploy - The execution vertices to deploy

resetForNewExecution

protected void resetForNewExecution(ExecutionVertexID executionVertexId)

Overrides:

resetForNewExecution in class SchedulerBase

getMarkPartitionFinishedStrategy

protected MarkPartitionFinishedStrategy getMarkPartitionFinishedStrategy()

Overrides:

getMarkPartitionFinishedStrategy in class SchedulerBase

computeDynamicSourceParallelism

public List<CompletableFuture<Integer>> computeDynamicSourceParallelism()

initializeVerticesIfPossible

@VisibleForTesting
public void initializeVerticesIfPossible()

computeVertexParallelismStoreForDynamicGraph

@VisibleForTesting
public static VertexParallelismStore computeVertexParallelismStoreForDynamicGraph(Iterable<JobVertex> vertices,
                                                                                  int defaultMaxParallelism)

Compute the VertexParallelismStore for all given vertices in a dynamic graph, which will set defaults and ensure that the returned store contains valid parallelisms, with the configured default max parallelism.

Parameters:

vertices - the vertices to compute parallelism for

defaultMaxParallelism - the global default max parallelism

Returns:

the computed parallelism store

computeMaxParallelism

public static int computeMaxParallelism(int parallelism,
                                        int defaultMaxParallelism)