org.apache.flink.runtime.jobgraph

Class JobGraph

java.lang.Object

org.apache.flink.runtime.jobgraph.JobGraph

All Implemented Interfaces:

Serializable

public class JobGraph
extends Object
implements Serializable

The JobGraph represents a Flink dataflow program, at the low level that the JobManager accepts. All programs from higher level APIs are transformed into JobGraphs.

The JobGraph is a graph of vertices and intermediate results that are connected together to form a DAG. Note that iterations (feedback edges) are currently not encoded inside the JobGraph but inside certain special vertices that establish the feedback channel amongst themselves.

The JobGraph defines the job-wide configuration settings, while each vertex and intermediate result define the characteristics of the concrete operation and intermediate data.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
JobGraph(JobID jobId, String jobName) Constructs a new job graph with the given job ID (or a random ID, if null is passed), the given name and the given execution configuration (see ExecutionConfig).
JobGraph(JobID jobId, String jobName, JobVertex... vertices) Constructs a new job graph with the given name, the given ExecutionConfig, the given jobId or a random one if null supplied, and the given job vertices.
JobGraph(JobVertex... vertices) Constructs a new job graph with no name, a random job ID, the given ExecutionConfig, and the given job vertices.
JobGraph(String jobName) Constructs a new job graph with the given name, the given ExecutionConfig, and a random job ID.
JobGraph(String jobName, JobVertex... vertices) Constructs a new job graph with the given name, the given ExecutionConfig, a random job ID, and the given job vertices.

Method Summary

Methods

Modifier and Type	Method and Description
void	addBlob(BlobKey key) Adds the BLOB referenced by the key to the JobGraph's dependencies.
void	addJar(Path jar) Adds the path of a JAR file required to run the job on a task manager.
void	addVertex(JobVertex vertex) Adds a new task vertex to the job graph if it is not already included.
JobVertex	findVertexByID(JobVertexID id) Searches for a vertex with a matching ID and returns it.
boolean	getAllowQueuedScheduling()
JobCheckpointingSettings	getCheckpointingSettings() Gets the settings for asynchronous snapshots.
List<URL>	getClasspaths()
Configuration	getJobConfiguration() Returns the configuration object for this job.
JobID	getJobID() Returns the ID of the job.
int	getMaximumParallelism() Gets the maximum parallelism of all operations in this job graph.
String	getName() Returns the name assigned to the job graph.
int	getNumberOfVertices() Returns the number of all vertices.
SavepointRestoreSettings	getSavepointRestoreSettings() Returns the configured savepoint restore setting.
ScheduleMode	getScheduleMode()
SerializedValue<ExecutionConfig>	getSerializedExecutionConfig() Returns the ExecutionConfig
long	getSessionTimeout() Gets the timeout after which the corresponding ExecutionGraph is removed at the job manager after it has been executed.
List<BlobKey>	getUserJarBlobKeys() Returns a set of BLOB keys referring to the JAR files required to run this job.
List<Path>	getUserJars() Gets the list of assigned user jar paths.
Iterable<JobVertex>	getVertices() Returns an Iterable to iterate all vertices registered with the job graph.
JobVertex[]	getVerticesAsArray() Returns an array of all job vertices that are registered with the job graph.
List<JobVertex>	getVerticesSortedTopologicallyFromSources()
boolean	hasUsercodeJarFiles() Checks whether the JobGraph has user code JAR files attached.
void	setAllowQueuedScheduling(boolean allowQueuedScheduling)
void	setClasspaths(List<URL> paths) Sets the classpaths required to run the job on a task manager.
void	setExecutionConfig(ExecutionConfig executionConfig) Sets the execution config.
void	setSavepointRestoreSettings(SavepointRestoreSettings settings) Sets the savepoint restore settings.
void	setScheduleMode(ScheduleMode scheduleMode)
void	setSessionTimeout(long sessionTimeout) Sets the timeout of the session in seconds.
void	setSnapshotSettings(JobCheckpointingSettings settings) Sets the settings for asynchronous snapshots.
String	toString()
void	uploadRequiredJarFiles(InetSocketAddress serverAddress, Configuration blobClientConfig) Uploads the previously added user jar file to the job manager through the job manager's BLOB server.
void	uploadUserJars(ActorGateway jobManager, scala.concurrent.duration.FiniteDuration askTimeout, Configuration blobClientConfig) Uploads the previously added user JAR files to the job manager through the job manager's BLOB server.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

JobGraph

public JobGraph(String jobName)

Constructs a new job graph with the given name, the given ExecutionConfig, and a random job ID. The ExecutionConfig will be serialized and can't be modified afterwards.

Parameters:

jobName - The name of the job.

JobGraph

public JobGraph(JobID jobId,
        String jobName)

Constructs a new job graph with the given job ID (or a random ID, if null is passed), the given name and the given execution configuration (see ExecutionConfig). The ExecutionConfig will be serialized and can't be modified afterwards.

Parameters:

jobId - The id of the job. A random ID is generated, if null is passed.

jobName - The name of the job.

JobGraph

public JobGraph(JobVertex... vertices)

Constructs a new job graph with no name, a random job ID, the given ExecutionConfig, and the given job vertices. The ExecutionConfig will be serialized and can't be modified afterwards.

Parameters:

vertices - The vertices to add to the graph.

JobGraph

public JobGraph(String jobName,
        JobVertex... vertices)

Constructs a new job graph with the given name, the given ExecutionConfig, a random job ID, and the given job vertices. The ExecutionConfig will be serialized and can't be modified afterwards.

Parameters:

jobName - The name of the job.

vertices - The vertices to add to the graph.

JobGraph

public JobGraph(JobID jobId,
        String jobName,
        JobVertex... vertices)

Constructs a new job graph with the given name, the given ExecutionConfig, the given jobId or a random one if null supplied, and the given job vertices. The ExecutionConfig will be serialized and can't be modified afterwards.

Parameters:

jobId - The id of the job. A random ID is generated, if null is passed.

jobName - The name of the job.

vertices - The vertices to add to the graph.

Method Detail

getJobID

public JobID getJobID()

Returns the ID of the job.

Returns:

the ID of the job

getName

public String getName()

Returns the name assigned to the job graph.

Returns:

the name assigned to the job graph

getJobConfiguration

public Configuration getJobConfiguration()

Returns the configuration object for this job. Job-wide parameters should be set into that configuration object.

Returns:

The configuration object for this job.

getSerializedExecutionConfig

public SerializedValue<ExecutionConfig> getSerializedExecutionConfig()

Returns the ExecutionConfig

Returns:

ExecutionConfig

getSessionTimeout

public long getSessionTimeout()

Gets the timeout after which the corresponding ExecutionGraph is removed at the job manager after it has been executed.

Returns:

a timeout as a long in seconds.

setSessionTimeout

public void setSessionTimeout(long sessionTimeout)

Sets the timeout of the session in seconds. The timeout specifies how long a job will be kept in the job manager after it finishes.

Parameters:

sessionTimeout - The timeout in seconds

setAllowQueuedScheduling

public void setAllowQueuedScheduling(boolean allowQueuedScheduling)

getAllowQueuedScheduling

public boolean getAllowQueuedScheduling()

setScheduleMode

public void setScheduleMode(ScheduleMode scheduleMode)

getScheduleMode

public ScheduleMode getScheduleMode()

setSavepointRestoreSettings

public void setSavepointRestoreSettings(SavepointRestoreSettings settings)

Sets the savepoint restore settings.

Parameters:

settings - The savepoint restore settings.

getSavepointRestoreSettings

public SavepointRestoreSettings getSavepointRestoreSettings()

Returns the configured savepoint restore setting.

Returns:

The configured savepoint restore settings.

setExecutionConfig

public void setExecutionConfig(ExecutionConfig executionConfig)
                        throws IOException

Sets the execution config. This method eagerly serialized the ExecutionConfig for future RPC transport. Further modification of the referenced ExecutionConfig object will not affect this serialized copy.

Parameters:

executionConfig - The ExecutionConfig to be serialized.

Throws:

IOException - Thrown if the serialization of the ExecutionConfig fails

addVertex

public void addVertex(JobVertex vertex)

Adds a new task vertex to the job graph if it is not already included.

Parameters:

vertex - the new task vertex to be added

getVertices

public Iterable<JobVertex> getVertices()

Returns an Iterable to iterate all vertices registered with the job graph.

Returns:

an Iterable to iterate all vertices registered with the job graph

getVerticesAsArray

public JobVertex[] getVerticesAsArray()

Returns an array of all job vertices that are registered with the job graph. The order in which the vertices appear in the list is not defined.

Returns:

an array of all job vertices that are registered with the job graph

getNumberOfVertices

public int getNumberOfVertices()

Returns the number of all vertices.

Returns:

The number of all vertices.

setSnapshotSettings

public void setSnapshotSettings(JobCheckpointingSettings settings)

Sets the settings for asynchronous snapshots. A value of null means that snapshotting is not enabled.

Parameters:

settings - The snapshot settings, or null, to disable snapshotting.

getCheckpointingSettings

public JobCheckpointingSettings getCheckpointingSettings()

Gets the settings for asynchronous snapshots. This method returns null, when checkpointing is not enabled.

Returns:

The snapshot settings, or null, if checkpointing is not enabled.

findVertexByID

public JobVertex findVertexByID(JobVertexID id)

Searches for a vertex with a matching ID and returns it.

Parameters:

id - the ID of the vertex to search for

Returns:

the vertex with the matching ID or null if no vertex with such ID could be found

setClasspaths

public void setClasspaths(List<URL> paths)

Sets the classpaths required to run the job on a task manager.

Parameters:

paths - paths of the directories/JAR files required to run the job on a task manager

getClasspaths

public List<URL> getClasspaths()

getMaximumParallelism

public int getMaximumParallelism()

Gets the maximum parallelism of all operations in this job graph.

Returns:

The maximum parallelism of this job graph

getVerticesSortedTopologicallyFromSources

public List<JobVertex> getVerticesSortedTopologicallyFromSources()
                                                          throws InvalidProgramException

Throws:

InvalidProgramException

addJar

public void addJar(Path jar)

Adds the path of a JAR file required to run the job on a task manager.

Parameters:

jar - path of the JAR file required to run the job on a task manager

getUserJars

public List<Path> getUserJars()

Gets the list of assigned user jar paths.

Returns:

The list of assigned user jar paths

addBlob

public void addBlob(BlobKey key)

Adds the BLOB referenced by the key to the JobGraph's dependencies.

Parameters:

key - path of the JAR file required to run the job on a task manager

hasUsercodeJarFiles

public boolean hasUsercodeJarFiles()

Checks whether the JobGraph has user code JAR files attached.

Returns:

True, if the JobGraph has user code JAR files attached, false otherwise.

getUserJarBlobKeys

public List<BlobKey> getUserJarBlobKeys()

Returns a set of BLOB keys referring to the JAR files required to run this job.

Returns:

set of BLOB keys referring to the JAR files required to run this job

uploadRequiredJarFiles

public void uploadRequiredJarFiles(InetSocketAddress serverAddress,
                          Configuration blobClientConfig)
                            throws IOException

Uploads the previously added user jar file to the job manager through the job manager's BLOB server.

Parameters:

serverAddress - the network address of the BLOB server

blobClientConfig - the blob client configuration

Throws:

IOException - thrown if an I/O error occurs during the upload

uploadUserJars

public void uploadUserJars(ActorGateway jobManager,
                  scala.concurrent.duration.FiniteDuration askTimeout,
                  Configuration blobClientConfig)
                    throws IOException

Uploads the previously added user JAR files to the job manager through the job manager's BLOB server. The respective port is retrieved from the JobManager. This function issues a blocking call.

Parameters:

jobManager - JobManager actor gateway

askTimeout - Ask timeout

blobClientConfig - the blob client configuration

Throws:

IOException - Thrown, if the file upload to the JobManager failed.

toString

public String toString()

Overrides:

toString in class Object