Main entry point for Spark Streaming functionality.
Main entry point for Spark Streaming functionality. It provides methods used to create
org.apache.spark.streaming.dstream.DStreams from various input sources. It can be either
created by providing a Spark master URL and an appName, or from a org.apache.spark.SparkConf
configuration (see core Spark documentation), or from an existing org.apache.spark.SparkContext.
The associated SparkContext can be accessed using context.sparkContext
. After
creating and transforming DStreams, the streaming computation can be started and stopped
using context.start()
and context.stop()
, respectively.
context.awaitTermination()
allows the current thread to wait for the termination
of the context by stop()
or by an exception.
This is a simple class that represents an absolute instant of time.
This is a simple class that represents an absolute instant of time. Internally, it represents time as the difference, measured in milliseconds, between the current time and midnight, January 1, 1970 UTC. This is the same format as what is returned by System.currentTimeMillis.
Helper object that creates instance of org.apache.spark.streaming.Duration representing a given number of milliseconds.
Helper object that creates instance of org.apache.spark.streaming.Duration representing a given number of minutes.
Helper object that creates instance of org.apache.spark.streaming.Duration representing a given number of seconds.
StreamingContext object contains a number of utility functions related to the StreamingContext class.
Various implementations of DStream's.
Various implementations of DStream's.
Spark Streaming functionality. org.apache.spark.streaming.StreamingContext serves as the main entry point to Spark Streaming, while org.apache.spark.streaming.dstream.DStream is the data type representing a continuous sequence of RDDs, representing a continuous stream of data.
In addition, org.apache.spark.streaming.dstream.PairDStreamFunctions contains operations available only on DStreams of key-value pairs, such as
groupByKey
andreduceByKey
. These operations are automatically available on any DStream of the right type (e.g. DStream[(Int, Int)] through implicit conversions when youimport org.apache.spark.streaming.StreamingContext._
.For the Java API of Spark Streaming, take a look at the org.apache.spark.streaming.api.java.JavaStreamingContext which serves as the entry point, and the org.apache.spark.streaming.api.java.JavaDStream and the org.apache.spark.streaming.api.java.JavaPairDStream which have the DStream functionality.