org.apache.flink.api.java

Class RemoteEnvironment

java.lang.Object

org.apache.flink.api.java.ExecutionEnvironment

org.apache.flink.api.java.RemoteEnvironment

public class RemoteEnvironment
extends ExecutionEnvironment

An ExecutionEnvironment that sends programs to a cluster for execution. The environment needs to be created with the address and port of the JobManager of the Flink cluster that should execute the programs.

Many programs executed via the remote environment depend on additional classes. Such classes may be the classes of functions (transformation, aggregation, ...) or libraries. Those classes must be attached to the remote environment as JAR files, to allow the environment to ship the classes into the cluster for the distributed execution.

Field Summary

Fields

Modifier and Type	Field and Description
protected String	host The hostname of the JobManager
protected int	port The port of the JobManager main actor system

Fields inherited from class org.apache.flink.api.java.ExecutionEnvironment

jobID, lastJobExecutionResult, LOG, sessionTimeout

Constructor Summary

Constructors

Constructor and Description
RemoteEnvironment(String host, int port, Configuration clientConfig, String[] jarFiles) Creates a new RemoteEnvironment that points to the master (JobManager) described by the given host name and port.
RemoteEnvironment(String host, int port, Configuration clientConfig, String[] jarFiles, URL[] globalClasspaths) Creates a new RemoteEnvironment that points to the master (JobManager) described by the given host name and port.
RemoteEnvironment(String host, int port, String... jarFiles) Creates a new RemoteEnvironment that points to the master (JobManager) described by the given host name and port.

Method Summary

Methods

Modifier and Type	Method and Description
protected void	dispose()
JobExecutionResult	execute(String jobName) Triggers the program execution.
String	getExecutionPlan() Creates the plan with which the system will execute the program, and returns it as a String using a JSON representation of the execution data flow graph.
void	startNewSession() Starts a new session, discarding the previous data flow and all of its intermediate results.
String	toString()

Methods inherited from class org.apache.flink.api.java.ExecutionEnvironment

addDefaultKryoSerializer, addDefaultKryoSerializer, areExplicitEnvironmentsAllowed, createCollectionsEnvironment, createHadoopInput, createHadoopInput, createInput, createInput, createLocalEnvironment, createLocalEnvironment, createLocalEnvironment, createProgramPlan, createProgramPlan, createProgramPlan, createRemoteEnvironment, createRemoteEnvironment, createRemoteEnvironment, execute, fromCollection, fromCollection, fromCollection, fromCollection, fromElements, fromParallelCollection, fromParallelCollection, generateSequence, getConfig, getExecutionEnvironment, getId, getIdString, getLastJobExecutionResult, getNumberOfExecutionRetries, getParallelism, getSessionTimeout, initializeContextEnvironment, readCsvFile, readFile, readFileOfPrimitives, readFileOfPrimitives, readHadoopFile, readHadoopFile, readHadoopFile, readHadoopFile, readSequenceFile, readTextFile, readTextFile, readTextFileWithValue, readTextFileWithValue, registerCachedFile, registerCachedFile, registerCachedFilesWithPlan, registerType, registerTypeWithKryoSerializer, registerTypeWithKryoSerializer, resetContextEnvironment, setDefaultLocalParallelism, setNumberOfExecutionRetries, setParallelism, setSessionTimeout

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

host

protected final String host

The hostname of the JobManager

port

protected final int port

The port of the JobManager main actor system

Constructor Detail

RemoteEnvironment

public RemoteEnvironment(String host,
                 int port,
                 String... jarFiles)

Creates a new RemoteEnvironment that points to the master (JobManager) described by the given host name and port.

Each program execution will have all the given JAR files in its classpath.

Parameters:

host - The host name or address of the master (JobManager), where the program should be executed.

port - The port of the master (JobManager), where the program should be executed.

jarFiles - The JAR files with code that needs to be shipped to the cluster. If the program uses user-defined functions, user-defined input formats, or any libraries, those must be provided in the JAR files.

RemoteEnvironment

public RemoteEnvironment(String host,
                 int port,
                 Configuration clientConfig,
                 String[] jarFiles)

Creates a new RemoteEnvironment that points to the master (JobManager) described by the given host name and port.

Each program execution will have all the given JAR files in its classpath.

Parameters:

host - The host name or address of the master (JobManager), where the program should be executed.

port - The port of the master (JobManager), where the program should be executed.

clientConfig - The configuration used by the client that connects to the cluster.

jarFiles - The JAR files with code that needs to be shipped to the cluster. If the program uses user-defined functions, user-defined input formats, or any libraries, those must be provided in the JAR files.

RemoteEnvironment

public RemoteEnvironment(String host,
                 int port,
                 Configuration clientConfig,
                 String[] jarFiles,
                 URL[] globalClasspaths)

Creates a new RemoteEnvironment that points to the master (JobManager) described by the given host name and port.

Each program execution will have all the given JAR files in its classpath.

Parameters:

host - The host name or address of the master (JobManager), where the program should be executed.

port - The port of the master (JobManager), where the program should be executed.

clientConfig - The configuration used by the client that connects to the cluster.

jarFiles - The JAR files with code that needs to be shipped to the cluster. If the program uses user-defined functions, user-defined input formats, or any libraries, those must be provided in the JAR files.

globalClasspaths - The paths of directories and JAR files that are added to each user code classloader on all nodes in the cluster. Note that the paths must specify a protocol (e.g. file://) and be accessible on all nodes (e.g. by means of a NFS share). The protocol must be supported by the URLClassLoader.

Method Detail

execute

public JobExecutionResult execute(String jobName)
                           throws Exception

Description copied from class: ExecutionEnvironment

Triggers the program execution. The environment will execute all parts of the program that have resulted in a "sink" operation. Sink operations are for example printing results (DataSet.print(), writing results (e.g. DataSet.writeAsText(String), DataSet.write(org.apache.flink.api.common.io.FileOutputFormat, String), or other generic data sinks created with DataSet.output(org.apache.flink.api.common.io.OutputFormat).

The program execution will be logged and displayed with the given job name.

Specified by:

execute in class ExecutionEnvironment

Returns:

The result of the job execution, containing elapsed time and accumulators.

Throws:

Exception - Thrown, if the program executions fails.

getExecutionPlan

public String getExecutionPlan()
                        throws Exception

Description copied from class: ExecutionEnvironment

Creates the plan with which the system will execute the program, and returns it as a String using a JSON representation of the execution data flow graph. Note that this needs to be called, before the plan is executed.

Specified by:

getExecutionPlan in class ExecutionEnvironment

Returns:

The execution plan of the program, as a JSON String.

Throws:

Exception - Thrown, if the compiler could not be instantiated, or the master could not be contacted to retrieve information relevant to the execution planning.

startNewSession

public void startNewSession()
                     throws Exception

Description copied from class: ExecutionEnvironment

Starts a new session, discarding the previous data flow and all of its intermediate results.

Specified by:

startNewSession in class ExecutionEnvironment

Throws:

Exception

dispose

protected void dispose()

toString

public String toString()

Overrides:

toString in class Object