io.cdap.cdap.api.spark

Interface SparkClientContext

All Superinterfaces:

ClientLocalizationContext, DatasetContext, FeatureFlagsProvider, LineageRecorder, MessagingContext, MetadataReader, MetadataWriter, PluginContext, RuntimeContext, SchedulableProgramContext, SecureStore, ServiceDiscoverer, Transactional, WorkflowInfoProvider

@Beta
public interface SparkClientContext
extends SchedulableProgramContext, RuntimeContext, DatasetContext, ClientLocalizationContext, Transactional, ServiceDiscoverer, PluginContext, WorkflowInfoProvider, SecureStore, MessagingContext, LineageRecorder, MetadataReader, MetadataWriter

A context for a Spark program to interact with CDAP. This context object will be provided to Spark program in the ProgramLifecycle.initialize(T) call.

Method Summary

Modifier and Type	Method and Description
long	getLogicalStartTime() Returns the logical start time of this Spark job.
Metrics	getMetrics() Returns a Metrics which can be used to emit custom metrics.
SparkSpecification	getSpecification()
ProgramState	getState() Return the state of the Spark program.
void	setDriverResources(Resources resources) Sets the resources requirement for the Spark driver process.
void	setExecutorResources(Resources resources) Sets the resources, such as memory and virtual cores, to use for each executor process for the Spark program.
void	setPySparkScript(String script, Iterable<URI> additionalPythonFiles) Sets a python script to run using PySpark.
void	setPySparkScript(String script, URI... additionalPythonFiles) Sets a python script to run using PySpark.
void	setPySparkScript(URI scriptLocation, Iterable<URI> additionalPythonFiles) Sets a location that points to a python script to run using PySpark.
void	setPySparkScript(URI scriptLocation, URI... additionalPythonFiles) Sets a location that points to a python script to run using PySpark.
<T> void	setSparkConf(T sparkConf) Sets a SparkConf to be used for the Spark execution.

Methods inherited from interface io.cdap.cdap.api.SchedulableProgramContext

getTriggeringScheduleInfo

Methods inherited from interface io.cdap.cdap.api.RuntimeContext

getAdmin, getApplicationSpecification, getClusterName, getDataTracer, getNamespace, getRunId, getRuntimeArguments

Methods inherited from interface io.cdap.cdap.api.data.DatasetContext

discardDataset, getDataset, getDataset, getDataset, getDataset, releaseDataset

Methods inherited from interface io.cdap.cdap.api.ClientLocalizationContext

localize, localize

Methods inherited from interface io.cdap.cdap.api.Transactional

execute, execute

Methods inherited from interface io.cdap.cdap.api.ServiceDiscoverer

getServiceURL, getServiceURL, getServiceURL, openConnection

Methods inherited from interface io.cdap.cdap.api.plugin.PluginContext

getPluginProperties, getPluginProperties, loadPluginClass, newPluginInstance, newPluginInstance

Methods inherited from interface io.cdap.cdap.api.feature.FeatureFlagsProvider

isFeatureEnabled

Methods inherited from interface io.cdap.cdap.api.workflow.WorkflowInfoProvider

getWorkflowInfo, getWorkflowToken

Methods inherited from interface io.cdap.cdap.api.security.store.SecureStore

get, getData, getMetadata, list

Methods inherited from interface io.cdap.cdap.api.messaging.MessagingContext

getDirectMessagePublisher, getMessageFetcher, getMessagePublisher

Methods inherited from interface io.cdap.cdap.api.lineage.field.LineageRecorder

flushLineage, record

Methods inherited from interface io.cdap.cdap.api.metadata.MetadataReader

getMetadata, getMetadata

Methods inherited from interface io.cdap.cdap.api.metadata.MetadataWriter

addProperties, addTags, addTags, removeMetadata, removeProperties, removeProperties, removeTags, removeTags

Method Detail

getSpecification

SparkSpecification getSpecification()

Returns:

The specification used to configure this Spark job instance.

getLogicalStartTime

long getLogicalStartTime()

Returns the logical start time of this Spark job. Logical start time is the time when this Spark job is supposed to start if this job is started by the scheduler. Otherwise it would be the current time when the job runs.

Returns:

Time in milliseconds since epoch time (00:00:00 January 1, 1970 UTC).

getMetrics

Metrics getMetrics()

Returns a Metrics which can be used to emit custom metrics.

Returns:

a Metrics for the Spark program

setDriverResources

void setDriverResources(Resources resources)

Sets the resources requirement for the Spark driver process.

Parameters:

resources - Resources that the driver should use

setExecutorResources

void setExecutorResources(Resources resources)

Sets the resources, such as memory and virtual cores, to use for each executor process for the Spark program.

Parameters:

resources - Resources that each executor should use

setSparkConf

<T> void setSparkConf(T sparkConf)

Sets a SparkConf to be used for the Spark execution. Only configurations set inside the ProgramLifecycle.initialize(T) call will affect the Spark execution.

Type Parameters:

T - the SparkConf type

setPySparkScript

void setPySparkScript(String script,
                      URI... additionalPythonFiles)

Sets a python script to run using PySpark.

See Also:

setPySparkScript(String, Iterable)

setPySparkScript

void setPySparkScript(String script,
                      Iterable<URI> additionalPythonFiles)

Sets a python script to run using PySpark. If this is set, it overrides whatever being set via the SparkConfigurer.setMainClassName(String) and the python script is the one always get executed.

Parameters:

script - the python script to run using PySpark

additionalPythonFiles - a list of addition python files to be included in the PYTHONPATH. Each can be local by having file scheme or remote, for example, by having hdfs or http scheme. If the URI has no scheme, it will be default based on the execution environment, which would be local file in local sandbox, and remote if running in distributed mode. Note that in distributed mode, using file scheme means the file has to be present on every node on the cluster, since the spark program can be submitted from any node.

setPySparkScript

void setPySparkScript(URI scriptLocation,
                      URI... additionalPythonFiles)

Sets a location that points to a python script to run using PySpark.

See Also:

setPySparkScript(URI, Iterable)

setPySparkScript

void setPySparkScript(URI scriptLocation,
                      Iterable<URI> additionalPythonFiles)

Sets a location that points to a python script to run using PySpark. If this is set, it overrides whatever being set via the SparkConfigurer.setMainClassName(String) and the python script is the one always get executed.

Parameters:

scriptLocation - location to the python script. It can be local by having file scheme or remote, for example, by having hdfs or http scheme. If the URI has no scheme, it will be default based on the execution environment, which would be local file in local sandbox, and remote if running in distributed mode. Note that in distributed mode, using file scheme means the file has to be present on every node on the cluster, since the spark program can be submitted from any node.

additionalPythonFiles - a list of addition python files to be included in the PYTHONPATH. Each location can be local or remote, with the same definition as the scriptLocation parameter.

getState

ProgramState getState()

Return the state of the Spark program.