org.apache.flink.runtime.resourcemanager

Class ResourceManager<WorkerType extends Serializable>

java.lang.Object

org.apache.flink.runtime.rpc.RpcEndpoint<ResourceManagerGateway>

org.apache.flink.runtime.resourcemanager.ResourceManager<WorkerType>

All Implemented Interfaces:

LeaderContender

Direct Known Subclasses:

StandaloneResourceManager

public abstract class ResourceManager<WorkerType extends Serializable>
extends RpcEndpoint<ResourceManagerGateway>
implements LeaderContender

ResourceManager implementation. The resource manager is responsible for resource de-/allocation and bookkeeping. It offers the following methods as part of its rpc interface to interact with him remotely:

• #registerJobManager(UUID, UUID, String, JobID, ResourceID) registers a JobMaster at the resource manager
• requestSlot(UUID, UUID, SlotRequest) requests a slot from the resource manager

Field Summary

Fields

Modifier and Type	Field and Description
static String	RESOURCE_MANAGER_NAME

Fields inherited from class org.apache.flink.runtime.rpc.RpcEndpoint

log

Constructor Summary

Constructors

Constructor and Description
ResourceManager(RpcService rpcService, String resourceManagerEndpointId, ResourceID resourceId, ResourceManagerConfiguration resourceManagerConfiguration, HighAvailabilityServices highAvailabilityServices, HeartbeatServices heartbeatServices, SlotManager slotManager, MetricRegistry metricRegistry, JobLeaderIdService jobLeaderIdService, FatalErrorHandler fatalErrorHandler)

Method Summary

Modifier and Type	Method and Description
protected void	closeJobManagerConnection(org.apache.flink.api.common.JobID jobId, Exception cause) This method should be called by the framework once it detects that a currently registered job manager has failed.
protected void	closeTaskManagerConnection(ResourceID resourceID, Exception cause) This method should be called by the framework once it detects that a currently registered task executor has failed.
void	disconnectJobManager(org.apache.flink.api.common.JobID jobId, Exception cause)
void	disconnectTaskManager(ResourceID resourceId, Exception cause)
Integer	getNumberOfRegisteredTaskManagers(UUID requestLeaderSessionId)
void	grantLeadership(UUID newLeaderSessionID) Callback method when current resourceManager is granted leadership
void	handleError(Exception exception) Handles error occurring in the leader election service
void	heartbeatFromJobManager(ResourceID resourceID)
void	heartbeatFromTaskManager(ResourceID resourceID)
protected abstract void	initialize() Initializes the framework specific components.
protected boolean	isValid(UUID resourceManagerLeaderId) Checks whether the given resource manager leader id is matching the current leader id and not null.
protected void	jobLeaderLostLeadership(org.apache.flink.api.common.JobID jobId, UUID oldJobLeaderId)
void	notifySlotAvailable(UUID resourceManagerLeaderId, InstanceID instanceID, SlotID slotId, AllocationID allocationId) Notification from a TaskExecutor that a slot has become available
protected void	onFatalErrorAsync(Throwable t) Notifies the ResourceManager that a fatal error has occurred and it cannot proceed.
void	registerInfoMessageListener(String address) Registers an info message listener
Future<RegistrationResponse>	registerJobManager(UUID resourceManagerLeaderId, UUID jobManagerLeaderId, ResourceID jobManagerResourceId, String jobManagerAddress, org.apache.flink.api.common.JobID jobId)
Future<RegistrationResponse>	registerTaskExecutor(UUID resourceManagerLeaderId, String taskExecutorAddress, ResourceID taskExecutorResourceId, SlotReport slotReport) Register a TaskExecutor at the resource manager
protected void	removeJob(org.apache.flink.api.common.JobID jobId)
Acknowledge	requestSlot(UUID jobMasterLeaderID, UUID resourceManagerLeaderID, SlotRequest slotRequest) Requests a slot from the resource manager.
void	revokeLeadership() Callback method when current resourceManager loses leadership.
void	sendInfoMessage(String message)
void	shutDown() Shuts down the underlying RPC endpoint via the RPC service.
protected abstract void	shutDownApplication(ApplicationStatus finalStatus, String optionalDiagnostics) The framework specific code for shutting down the application.
void	shutDownCluster(ApplicationStatus finalStatus, String optionalDiagnostics) Cleanup application and shut down cluster
void	start() Starts the rpc endpoint.
abstract void	startNewWorker(ResourceProfile resourceProfile) Allocates a resource using the resource profile.
abstract void	stopWorker(InstanceID instanceId)
void	unRegisterInfoMessageListener(String address) Unregisters an info message listener
protected abstract WorkerType	workerStarted(ResourceID resourceID) Callback when a worker was started.

Methods inherited from class org.apache.flink.runtime.rpc.RpcEndpoint

callAsync, getAddress, getEndpointId, getMainThreadExecutor, getRpcService, getSelf, getSelfGatewayType, getTerminationFuture, runAsync, scheduleRunAsync, scheduleRunAsync, validateRunsInMainThread

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.runtime.leaderelection.LeaderContender

getAddress

Field Detail

RESOURCE_MANAGER_NAME

public static final String RESOURCE_MANAGER_NAME

See Also:

Constant Field Values

Constructor Detail

ResourceManager

public ResourceManager(RpcService rpcService,
                       String resourceManagerEndpointId,
                       ResourceID resourceId,
                       ResourceManagerConfiguration resourceManagerConfiguration,
                       HighAvailabilityServices highAvailabilityServices,
                       HeartbeatServices heartbeatServices,
                       SlotManager slotManager,
                       MetricRegistry metricRegistry,
                       JobLeaderIdService jobLeaderIdService,
                       FatalErrorHandler fatalErrorHandler)

Method Detail

start

public void start()
           throws Exception

Description copied from class: RpcEndpoint

Starts the rpc endpoint. This tells the underlying rpc server that the rpc endpoint is ready to process remote procedure calls. IMPORTANT: Whenever you override this method, call the parent implementation to enable rpc processing. It is advised to make the parent call last.

Overrides:

start in class RpcEndpoint<ResourceManagerGateway>

Throws:

Exception - indicating that something went wrong while starting the RPC endpoint

shutDown

public void shutDown()
              throws Exception

Description copied from class: RpcEndpoint

Shuts down the underlying RPC endpoint via the RPC service. After this method was called, the RPC endpoint will no longer be reachable, neither remotely, not via its self gateway. It will also not accepts executions in main thread any more (via RpcEndpoint.callAsync(Callable, Time) and RpcEndpoint.runAsync(Runnable)).

This method can be overridden to add RPC endpoint specific shut down code. The overridden method should always call the parent shut down method.

Overrides:

shutDown in class RpcEndpoint<ResourceManagerGateway>

Throws:

Exception - indicating that the something went wrong while shutting the RPC endpoint down

registerJobManager

public Future<RegistrationResponse> registerJobManager(UUID resourceManagerLeaderId,
                                                       UUID jobManagerLeaderId,
                                                       ResourceID jobManagerResourceId,
                                                       String jobManagerAddress,
                                                       org.apache.flink.api.common.JobID jobId)

registerTaskExecutor

public Future<RegistrationResponse> registerTaskExecutor(UUID resourceManagerLeaderId,
                                                         String taskExecutorAddress,
                                                         ResourceID taskExecutorResourceId,
                                                         SlotReport slotReport)

Register a TaskExecutor at the resource manager

Parameters:

resourceManagerLeaderId - The fencing token for the ResourceManager leader

taskExecutorAddress - The address of the TaskExecutor that registers

taskExecutorResourceId - The resource ID of the TaskExecutor that registers

Returns:

The response by the ResourceManager.

heartbeatFromTaskManager

public void heartbeatFromTaskManager(ResourceID resourceID)

heartbeatFromJobManager

public void heartbeatFromJobManager(ResourceID resourceID)

disconnectTaskManager

public void disconnectTaskManager(ResourceID resourceId,
                                  Exception cause)

disconnectJobManager

public void disconnectJobManager(org.apache.flink.api.common.JobID jobId,
                                 Exception cause)

requestSlot

public Acknowledge requestSlot(UUID jobMasterLeaderID,
                               UUID resourceManagerLeaderID,
                               SlotRequest slotRequest)
                        throws ResourceManagerException,
                               LeaderSessionIDException

Requests a slot from the resource manager.

Parameters:

slotRequest - Slot request

Returns:

Slot assignment

Throws:

ResourceManagerException

LeaderSessionIDException

notifySlotAvailable

public void notifySlotAvailable(UUID resourceManagerLeaderId,
                                InstanceID instanceID,
                                SlotID slotId,
                                AllocationID allocationId)

Notification from a TaskExecutor that a slot has become available

Parameters:

resourceManagerLeaderId - TaskExecutor's resource manager leader id

instanceID - TaskExecutor's instance id

slotId - The slot id of the available slot

registerInfoMessageListener

public void registerInfoMessageListener(String address)

Registers an info message listener

Parameters:

address - address of infoMessage listener to register to this resource manager

unRegisterInfoMessageListener

public void unRegisterInfoMessageListener(String address)

Unregisters an info message listener

Parameters:

address - of the info message listener to unregister from this resource manager

shutDownCluster

public void shutDownCluster(ApplicationStatus finalStatus,
                            String optionalDiagnostics)

Cleanup application and shut down cluster

Parameters:

finalStatus - of the Flink application

optionalDiagnostics - for the Flink application

getNumberOfRegisteredTaskManagers

public Integer getNumberOfRegisteredTaskManagers(UUID requestLeaderSessionId)
                                          throws LeaderIdMismatchException

Throws:

LeaderIdMismatchException

closeJobManagerConnection

protected void closeJobManagerConnection(org.apache.flink.api.common.JobID jobId,
                                         Exception cause)

This method should be called by the framework once it detects that a currently registered job manager has failed.

Parameters:

jobId - identifying the job whose leader shall be disconnected.

cause - The exception which cause the JobManager failed.

closeTaskManagerConnection

protected void closeTaskManagerConnection(ResourceID resourceID,
                                          Exception cause)

This method should be called by the framework once it detects that a currently registered task executor has failed.

Parameters:

resourceID - Id of the TaskManager that has failed.

cause - The exception which cause the TaskManager failed.

isValid

protected boolean isValid(UUID resourceManagerLeaderId)

Checks whether the given resource manager leader id is matching the current leader id and not null.

Parameters:

resourceManagerLeaderId - to check

Returns:

True if the given leader id matches the actual leader id and is not null; otherwise false

removeJob

protected void removeJob(org.apache.flink.api.common.JobID jobId)

jobLeaderLostLeadership

protected void jobLeaderLostLeadership(org.apache.flink.api.common.JobID jobId,
                                       UUID oldJobLeaderId)

sendInfoMessage

public void sendInfoMessage(String message)

onFatalErrorAsync

protected void onFatalErrorAsync(Throwable t)

Notifies the ResourceManager that a fatal error has occurred and it cannot proceed. This method should be used when asynchronous threads want to notify the ResourceManager of a fatal error.

Parameters:

t - The exception describing the fatal error

grantLeadership

public void grantLeadership(UUID newLeaderSessionID)

Callback method when current resourceManager is granted leadership

Specified by:

grantLeadership in interface LeaderContender

Parameters:

newLeaderSessionID - unique leadershipID

revokeLeadership

public void revokeLeadership()

Callback method when current resourceManager loses leadership.

Specified by:

revokeLeadership in interface LeaderContender

handleError

public void handleError(Exception exception)

Handles error occurring in the leader election service

Specified by:

handleError in interface LeaderContender

Parameters:

exception - Exception being thrown in the leader election service

initialize

protected abstract void initialize()
                            throws ResourceManagerException

Initializes the framework specific components.

Throws:

ResourceManagerException - which occurs during initialization and causes the resource manager to fail.

shutDownApplication

protected abstract void shutDownApplication(ApplicationStatus finalStatus,
                                            String optionalDiagnostics)

The framework specific code for shutting down the application. This should report the application's final status and shut down the resource manager cleanly. This method also needs to make sure all pending containers that are not registered yet are returned.

Parameters:

finalStatus - The application status to report.

optionalDiagnostics - An optional diagnostics message.

startNewWorker

@VisibleForTesting
public abstract void startNewWorker(ResourceProfile resourceProfile)

Allocates a resource using the resource profile.

Parameters:

resourceProfile - The resource description

stopWorker

public abstract void stopWorker(InstanceID instanceId)

workerStarted

protected abstract WorkerType workerStarted(ResourceID resourceID)

Callback when a worker was started.

Parameters:

resourceID - The worker resource id