SparkContext
Related Doc:
package spark
Applies an action to the underlying SparkContext.
Add a file to be downloaded with this Spark job on every node.
Add a file to be downloaded with this Spark job on every node.
If a file is added during execution, it will not be available until the next TaskSet starts.
can be either a local file, a file in HDFS (or other
Hadoop-supported filesystems), or an HTTP, HTTPS or FTP URI. To
access the file in Spark jobs, use SparkFiles.get(fileName) to
find its download location.
if true, a directory can be given in path. Currently
directories are only supported for Hadoop-supported filesystems.
A path can be added only once. Subsequent additions of the same path are ignored.
Add a file to be downloaded with this Spark job on every node.
Add a file to be downloaded with this Spark job on every node.
If a file is added during execution, it will not be available until the next TaskSet starts.
can be either a local file, a file in HDFS (or other
Hadoop-supported filesystems), or an HTTP, HTTPS or FTP URI. To
access the file in Spark jobs, use SparkFiles.get(fileName) to
find its download location.
A path can be added only once. Subsequent additions of the same path are ignored.
Adds a JAR dependency for all tasks to be executed on this
SparkContext in the future.
Adds a JAR dependency for all tasks to be executed on this
SparkContext in the future.
If a jar is added during execution, it will not be available until the next TaskSet starts.
can be either a local file, a file in HDFS (or other Hadoop-supported filesystems), an HTTP, HTTPS or FTP URI, or local:/path for a file on every worker node.
A path can be added only once. Subsequent additions of the same path are ignored.
A unique identifier for the Spark application.
A unique identifier for the Spark application. Its format depends on the scheduler implementation. (i.e. in case of local spark app something like 'local-1433865536131' in case of YARN something like 'application_1433865536131_34483' in case of MESOS something like 'driver-20170926223339-0001' )
Get an RDD for a Hadoop-readable dataset as PortableDataStream for each file (useful for binary data)
Get an RDD for a Hadoop-readable dataset as PortableDataStream for each file (useful for binary data)
For example, if you have the following files:
hdfs://a-hdfs-path/part-00000 hdfs://a-hdfs-path/part-00001 ... hdfs://a-hdfs-path/part-nnnnn
Do val rdd = sparkContext.binaryFiles("hdfs://a-hdfs-path"),
then rdd contains
(a-hdfs-path/part-00000, its content) (a-hdfs-path/part-00001, its content) ... (a-hdfs-path/part-nnnnn, its content)
Directory to the input data files, the path can be comma separated paths as the list of inputs.
A suggestion value of the minimal splitting number for input data.
RDD representing tuples of file path and corresponding file content
Partitioning is determined by data locality. This may result in too few partitions by default.
,
On some filesystems, .../path/* can be a more efficient way
to read all files in a directory rather than .../path/ or
.../path
Small files are preferred; very large files may cause bad performance.
Load data from a flat binary file, assuming the length of each record is constant.
Load data from a flat binary file, assuming the length of each record is constant.
Directory to the input data files, the path can be comma separated paths as the list of inputs.
The length at which to split the records
Configuration for setting up the dataset.
An RDD of data with values, represented as byte arrays
We ensure that the byte array for each record in the resulting RDD has the provided record length.
Broadcast a read-only variable to the cluster, returning a org.apache.spark.broadcast.Broadcast object for reading it in distributed functions.
Broadcast a read-only variable to the cluster, returning a org.apache.spark.broadcast.Broadcast object for reading it in distributed functions. The variable will be sent to each cluster only once.
value to broadcast to the Spark nodes
Broadcast object, a read-only variable cached on each machine
Cancel all jobs that have been scheduled or are running.
Cancel a given job if it's scheduled or running.
Cancel a given job if it's scheduled or running.
the job ID to cancel
Throws InterruptedException if the cancel message cannot be
sent
Cancel a given job if it's scheduled or running.
Cancel a given job if it's scheduled or running.
the job ID to cancel
optional reason for cancellation
Throws InterruptedException if the cancel message cannot be
sent
Cancel active jobs for the specified group.
Cancel active jobs for the specified group. See
org.apache.spark.SparkContext.setJobGroup for more information.
Cancel a given stage and all jobs associated with it.
Cancel a given stage and all jobs associated with it.
the stage ID to cancel
Throws InterruptedException if the cancel message cannot be
sent
Cancel a given stage and all jobs associated with it.
Cancel a given stage and all jobs associated with it.
the stage ID to cancel
reason for cancellation
Throws InterruptedException if the cancel message cannot be
sent
Clear the thread-local property for overriding the call sites of actions and RDDs.
Clear the current thread's job group ID and its description.
Create and register a CollectionAccumulator, which starts with
empty list and accumulates inputs by adding them into the list.
Create and register a CollectionAccumulator, which starts with
empty list and accumulates inputs by adding them into the list.
Default min number of partitions for Hadoop RDDs when not given by user Notice that we use math.min so the "defaultMinPartitions" cannot be higher than 2.
Default min number of partitions for Hadoop RDDs when not given by user Notice that we use math.min so the "defaultMinPartitions" cannot be higher than 2. The reasons for this are discussed in https://github.com/mesos/spark/pull/718
Default level of parallelism to use when not given by user (e.g.
Default level of parallelism to use when not given by user (e.g. parallelize and makeRDD).
Create and register a double accumulator, which starts with 0 and
accumulates inputs by add.
Create and register a double accumulator, which starts with 0 and
accumulates inputs by add.
Get an RDD that has no partitions or elements.
Applies an action to the underlying SparkContext.
Return a copy of this SparkContext's configuration.
Return a copy of this SparkContext's configuration. The configuration cannot be changed at runtime.
Return a map from the slave to the max memory available for caching and the remaining memory available for caching.
Get a local property set in this thread, or null if it is missing.
Get a local property set in this thread, or null if it is missing.
See org.apache.spark.SparkContext.setLocalProperty.
Returns an immutable map of RDDs that have marked themselves as persistent via cache() call.
Returns an immutable map of RDDs that have marked themselves as persistent via cache() call.
This does not necessarily mean the caching or computation was successful.
Return current scheduling mode
A default Hadoop Configuration for the Hadoop code (e.g.
A default Hadoop Configuration for the Hadoop code (e.g. file systems) that we reuse.
As it will be reused in all Hadoop RDDs, it's better not to modify it unless you plan to set some global configurations for all Hadoop RDDs.
Smarter version of hadoopFile() that uses class tags to figure out the classes of keys, values and the InputFormat so that users don't need to pass them directly.
Smarter version of hadoopFile() that uses class tags to figure out the classes of keys, values and the InputFormat so that users don't need to pass them directly. Instead, callers can just write, for example,
val file = sparkContext.hadoopFile[LongWritable, Text, TextInputFormat](path)directory to the input data files, the path can be comma separated paths as a list of inputs
RDD of tuples of key and corresponding value
Because Hadoop's RecordReader class re-uses the same Writable
object for each record, directly caching the returned RDD or
directly passing it to an aggregation or shuffle operation will
create many references to the same object. If you plan to
directly cache, sort, or aggregate Hadoop writable objects, you
should first copy them using a map function.
Smarter version of hadoopFile() that uses class tags to figure out the classes of keys, values and the InputFormat so that users don't need to pass them directly.
Smarter version of hadoopFile() that uses class tags to figure out the classes of keys, values and the InputFormat so that users don't need to pass them directly. Instead, callers can just write, for example,
val file = sparkContext.hadoopFile[LongWritable, Text, TextInputFormat](path, minPartitions)directory to the input data files, the path can be comma separated paths as a list of inputs
suggested minimum number of partitions for the resulting RDD
RDD of tuples of key and corresponding value
Because Hadoop's RecordReader class re-uses the same Writable
object for each record, directly caching the returned RDD or
directly passing it to an aggregation or shuffle operation will
create many references to the same object. If you plan to
directly cache, sort, or aggregate Hadoop writable objects, you
should first copy them using a map function.
Get an RDD for a Hadoop file with an arbitrary InputFormat
Get an RDD for a Hadoop file with an arbitrary InputFormat
directory to the input data files, the path can be comma separated paths as a list of inputs
storage format of the data to be read
Class of the key associated with the inputFormatClass
parameter
Class of the value associated with the inputFormatClass
parameter
suggested minimum number of partitions for the resulting RDD
RDD of tuples of key and corresponding value
Because Hadoop's RecordReader class re-uses the same Writable
object for each record, directly caching the returned RDD or
directly passing it to an aggregation or shuffle operation will
create many references to the same object. If you plan to
directly cache, sort, or aggregate Hadoop writable objects, you
should first copy them using a map function.
Get an RDD for a Hadoop-readable dataset from a Hadoop JobConf given its InputFormat and other necessary info (e.g.
Get an RDD for a Hadoop-readable dataset from a Hadoop JobConf
given its InputFormat and other necessary info (e.g. file name for
a filesystem-based dataset, table name for HyperTable), using the
older MapReduce API (org.apache.hadoop.mapred).
JobConf for setting up the dataset. Note: This will be put into a Broadcast. Therefore if you plan to reuse this conf to create multiple RDDs, you need to make sure you won't modify the conf. A safe approach is always creating a new conf for a new RDD.
storage format of the data to be read
Class of the key associated with the inputFormatClass
parameter
Class of the value associated with the inputFormatClass
parameter
Minimum number of Hadoop Splits to generate.
RDD of tuples of key and corresponding value
Because Hadoop's RecordReader class re-uses the same Writable
object for each record, directly caching the returned RDD or
directly passing it to an aggregation or shuffle operation will
create many references to the same object. If you plan to
directly cache, sort, or aggregate Hadoop writable objects, you
should first copy them using a map function.
true if context is stopped or in the midst of stopping.
Kill and reschedule the given task attempt.
Kill and reschedule the given task attempt. Task ids can be obtained from the Spark UI or through SparkListener.onTaskStart.
the task ID to kill. This id uniquely identifies the task attempt.
whether to interrupt the thread running the task.
the reason for killing the task, which should be a short string. If a task is killed multiple times with different reasons, only one reason will be reported.
Whether the task was successfully killed.
Returns a list of file paths that are added to resources.
Returns a list of jar files that are added to resources.
Create and register a long accumulator, which starts with 0 and
accumulates inputs by add.
Create and register a long accumulator, which starts with 0 and
accumulates inputs by add.
Distribute a local Scala collection to form an RDD, with one or more location preferences (hostnames of Spark nodes) for each object.
Distribute a local Scala collection to form an RDD, with one or more location preferences (hostnames of Spark nodes) for each object. Create a new partition for each collection item.
list of tuples of data and location preferences (hostnames of Spark nodes)
RDD representing data partitioned according to location preferences
Distribute a local Scala collection to form an RDD.
Distribute a local Scala collection to form an RDD.
This method is identical to parallelize.
Scala collection to distribute
number of partitions to divide the collection into
RDD representing distributed collection
Get an RDD for a given Hadoop file with an arbitrary new API InputFormat and extra configuration options to pass to the input format.
Get an RDD for a given Hadoop file with an arbitrary new API InputFormat and extra configuration options to pass to the input format.
directory to the input data files, the path can be comma separated paths as a list of inputs
storage format of the data to be read
Class of the key associated with the fClass parameter
Class of the value associated with the fClass parameter
Hadoop configuration
RDD of tuples of key and corresponding value
Because Hadoop's RecordReader class re-uses the same Writable
object for each record, directly caching the returned RDD or
directly passing it to an aggregation or shuffle operation will
create many references to the same object. If you plan to
directly cache, sort, or aggregate Hadoop writable objects, you
should first copy them using a map function.
Smarter version of newApiHadoopFile that uses class tags to
figure out the classes of keys, values and the
org.apache.hadoop.mapreduce.InputFormat (new MapReduce API) so
that user don't need to pass them directly.
Smarter version of newApiHadoopFile that uses class tags to
figure out the classes of keys, values and the
org.apache.hadoop.mapreduce.InputFormat (new MapReduce API) so
that user don't need to pass them directly. Instead, callers can
just write, for example:
val file = sparkContext.hadoopFile[LongWritable, Text, TextInputFormat](path)
directory to the input data files, the path can be comma separated paths as a list of inputs
RDD of tuples of key and corresponding value
Because Hadoop's RecordReader class re-uses the same Writable
object for each record, directly caching the returned RDD or
directly passing it to an aggregation or shuffle operation will
create many references to the same object. If you plan to
directly cache, sort, or aggregate Hadoop writable objects, you
should first copy them using a map function.
Get an RDD for a given Hadoop file with an arbitrary new API InputFormat and extra configuration options to pass to the input format.
Get an RDD for a given Hadoop file with an arbitrary new API InputFormat and extra configuration options to pass to the input format.
Configuration for setting up the dataset. Note: This will be put into a Broadcast. Therefore if you plan to reuse this conf to create multiple RDDs, you need to make sure you won't modify the conf. A safe approach is always creating a new conf for a new RDD.
storage format of the data to be read
Class of the key associated with the fClass parameter
Class of the value associated with the fClass parameter
Because Hadoop's RecordReader class re-uses the same Writable
object for each record, directly caching the returned RDD or
directly passing it to an aggregation or shuffle operation will
create many references to the same object. If you plan to
directly cache, sort, or aggregate Hadoop writable objects, you
should first copy them using a map function.
Load an RDD saved as a SequenceFile containing serialized objects, with NullWritable keys and BytesWritable values that contain a serialized partition.
Load an RDD saved as a SequenceFile containing serialized objects, with NullWritable keys and BytesWritable values that contain a serialized partition. This is still an experimental storage format and may not be supported exactly as is in future Spark releases. It will also be pretty slow if you use the default serializer (Java serialization), though the nice thing about it is that there's very little effort required to save arbitrary objects.
directory to the input data files, the path can be comma separated paths as a list of inputs
suggested minimum number of partitions for the resulting RDD
RDD representing deserialized data from the file(s)
Distribute a local Scala collection to form an RDD.
Distribute a local Scala collection to form an RDD.
Scala collection to distribute
number of partitions to divide the collection into
RDD representing distributed collection
avoid using parallelize(Seq()) to create an empty RDD.
Consider emptyRDD for an RDD with no partitions, or
parallelize(Seq[T]()) for an RDD of T with empty partitions.
Parallelize acts lazily. If seq is a mutable collection and is
altered after the call to parallelize and before the first action
on the RDD, the resultant RDD will reflect the modified
collection. Pass a copy of the argument to avoid this.
Creates a new RDD[Long] containing elements from start to
end(exclusive), increased by step every element.
Creates a new RDD[Long] containing elements from start to
end(exclusive), increased by step every element.
the start value.
the end value.
the incremental step
number of partitions to divide the collection into
RDD representing distributed range
if we need to cache this RDD, we should make sure each partition does not exceed limit.
Register the given accumulator with given name.
Register the given accumulator with given name.
Accumulators must be registered before use, or it will throw exception.
Register the given accumulator.
Register the given accumulator.
Accumulators must be registered before use, or it will throw exception.
Run a job on all partitions in an RDD and pass the results to a handler function.
Run a job on all partitions in an RDD and pass the results to a handler function.
target RDD to run tasks on
a function to run on each partition of the RDD
callback to pass each result to
Run a job on all partitions in an RDD and pass the results to a handler function.
Run a job on all partitions in an RDD and pass the results to a
handler function. The function that is run against each partition
additionally takes TaskContext argument.
target RDD to run tasks on
a function to run on each partition of the RDD
callback to pass each result to
Run a job on all partitions in an RDD and return the results in an array.
Run a job on all partitions in an RDD and return the results in an array.
target RDD to run tasks on
a function to run on each partition of the RDD
in-memory collection with a result of the job (each collection element will contain a result from one partition)
Run a job on all partitions in an RDD and return the results in an array.
Run a job on all partitions in an RDD and return the results in an
array. The function that is run against each partition additionally
takes TaskContext argument.
target RDD to run tasks on
a function to run on each partition of the RDD
in-memory collection with a result of the job (each collection element will contain a result from one partition)
Run a function on a given set of partitions in an RDD and return the results as an array.
Run a function on a given set of partitions in an RDD and return the results as an array.
target RDD to run tasks on
a function to run on each partition of the RDD
set of partitions to run on; some jobs may not want to compute on
all partitions of the target RDD, e.g. for operations like
first()
in-memory collection with a result of the job (each collection element will contain a result from one partition)
Run a function on a given set of partitions in an RDD and return the results as an array.
Run a function on a given set of partitions in an RDD and return
the results as an array. The function that is run against each
partition additionally takes TaskContext argument.
target RDD to run tasks on
a function to run on each partition of the RDD
set of partitions to run on; some jobs may not want to compute on
all partitions of the target RDD, e.g. for operations like
first()
in-memory collection with a result of the job (each collection element will contain a result from one partition)
Run a function on a given set of partitions in an RDD and pass the results to the given handler function.
Run a function on a given set of partitions in an RDD and pass the results to the given handler function. This is the main entry point for all actions in Spark.
target RDD to run tasks on
a function to run on each partition of the RDD
set of partitions to run on; some jobs may not want to compute on
all partitions of the target RDD, e.g. for operations like
first()
callback to pass each result to
Set the thread-local property for overriding the call sites of actions and RDDs.
Set the directory under which RDDs are going to be checkpointed.
Set the directory under which RDDs are going to be checkpointed.
path to the directory where checkpoint files will be stored (must be HDFS path if running in cluster)
Set a human readable description of the current job.
Assigns a group ID to all the jobs started by this thread until the group ID is set to a different value or cleared.
Assigns a group ID to all the jobs started by this thread until the group ID is set to a different value or cleared.
Often, a unit of execution in an application consists of multiple Spark actions or jobs. Application programmers can use this method to group all those jobs together and give a group description. Once set, the Spark web UI will associate such jobs with this group.
The application can also use
org.apache.spark.SparkContext.cancelJobGroup to cancel all
running jobs in this group. For example,
// In the main thread: sc.setJobGroup("some_job_to_cancel", "some job description") sc.parallelize(1 to 10000, 2).map { i => Thread.sleep(10); i }.count() // In a separate thread: sc.cancelJobGroup("some_job_to_cancel")
If true, then job cancellation will result in
Thread.interrupt() being called on the job's executor threads.
This is useful to help ensure that the tasks are actually stopped
in a timely manner, but is off by default due to HDFS-1208, where
HDFS may respond to Thread.interrupt() by marking nodes as dead.
Set a local property that affects jobs submitted from this thread, such as the Spark fair scheduler pool.
Set a local property that affects jobs submitted from this thread, such as the Spark fair scheduler pool. User-defined properties may also be set here. These properties are propagated through to worker tasks and can be accessed there via org.apache.spark.TaskContext#getLocalProperty.
These properties are inherited by child threads spawned from this thread. This may have unexpected consequences when working with thread pools. The standard java implementation of thread pools have worker threads spawn other worker threads. As a result, local properties may propagate unpredictably.
Control our logLevel.
Control our logLevel. This overrides any user-defined log settings.
The desired log level as a string. Valid log levels include: ALL, DEBUG, ERROR, FATAL, INFO, OFF, TRACE, WARN
Shut down the SparkContext.
Submit a job for execution and return a FutureJob holding the result.
Submit a job for execution and return a FutureJob holding the result.
target RDD to run tasks on
a function to run on each partition of the RDD
set of partitions to run on; some jobs may not want to compute on
all partitions of the target RDD, e.g. for operations like
first()
callback to pass each result to
function to be executed when the result is ready
Read a text file from HDFS, a local file system (available on all nodes), or any Hadoop-supported file system URI, and return it as an RDD of Strings.
Read a text file from HDFS, a local file system (available on all nodes), or any Hadoop-supported file system URI, and return it as an RDD of Strings.
path to the text file on a supported file system
suggested minimum number of partitions for the resulting RDD
RDD of lines of the text file
Build the union of a list of RDDs passed as variable-length arguments.
Build the union of a list of RDDs.
The version of Spark on which this application is running.
Read a directory of text files from HDFS, a local file system (available on all nodes), or any Hadoop-supported file system URI.
Read a directory of text files from HDFS, a local file system (available on all nodes), or any Hadoop-supported file system URI. Each file is read as a single record and returned in a key-value pair, where the key is the path of each file, the value is the content of each file.
For example, if you have the following files:
hdfs://a-hdfs-path/part-00000 hdfs://a-hdfs-path/part-00001 ... hdfs://a-hdfs-path/part-nnnnn
Do val rdd = sparkContext.wholeTextFile("hdfs://a-hdfs-path"),
then rdd contains
(a-hdfs-path/part-00000, its content) (a-hdfs-path/part-00001, its content) ... (a-hdfs-path/part-nnnnn, its content)
Directory to the input data files, the path can be comma separated paths as the list of inputs.
A suggestion value of the minimal splitting number for input data.
RDD representing tuples of file path and the corresponding file content
Partitioning is determined by data locality. This may result in too few partitions by default.
,
On some filesystems, .../path/* can be a more efficient way
to read all files in a directory rather than .../path/ or
.../path
Small files are preferred, large file is also allowable, but may cause bad performance.
Create an org.apache.spark.Accumulable shared variable, with a name for display in the Spark UI.
Create an org.apache.spark.Accumulable shared variable, with a
name for display in the Spark UI. Tasks can add values to the
accumulable using the += operator. Only the driver can access the
accumulable's value.
accumulator result type
type that can be added to the accumulator
(Since version 2.0.0) use AccumulatorV2
Create an org.apache.spark.Accumulable shared variable, to
which tasks can add values with +=.
Create an org.apache.spark.Accumulable shared variable, to
which tasks can add values with +=. Only the driver can access
the accumulable's value.
accumulator result type
type that can be added to the accumulator
(Since version 2.0.0) use AccumulatorV2
Create an accumulator from a "mutable collection" type.
Create an accumulator from a "mutable collection" type.
Growable and TraversableOnce are the standard APIs that guarantee += and ++=, implemented by standard mutable collections. So you can use this with mutable Map, Set, etc.
(Since version 2.0.0) use AccumulatorV2
Create an org.apache.spark.Accumulator variable of a given type, with a name for display in the Spark UI.
Create an org.apache.spark.Accumulator variable of a given
type, with a name for display in the Spark UI. Tasks can "add"
values to the accumulator using the += method. Only the driver
can access the accumulator's value.
(Since version 2.0.0) use AccumulatorV2
Create an org.apache.spark.Accumulator variable of a given
type, which tasks can "add" values to using the += method.
Create an org.apache.spark.Accumulator variable of a given
type, which tasks can "add" values to using the += method. Only
the driver can access the accumulator's value.
(Since version 2.0.0) use AccumulatorV2