streaming

Spark Streaming functionality. org.apache.spark.streaming.StreamingContext serves as the main entry point to Spark Streaming, while org.apache.spark.streaming.dstream.DStream is the data type representing a continuous sequence of RDDs, representing a continuous stream of data.

In addition, org.apache.spark.streaming.dstream.PairDStreamFunctions contains operations available only on DStreams of key-value pairs, such as groupByKey and reduceByKey. These operations are automatically available on any DStream of the right type (e.g. DStream[(Int, Int)] through implicit conversions.

For the Java API of Spark Streaming, take a look at the org.apache.spark.streaming.api.java.JavaStreamingContext which serves as the entry point, and the org.apache.spark.streaming.api.java.JavaDStream and the org.apache.spark.streaming.api.java.JavaPairDStream which have the DStream functionality.

Linear Supertypes

AnyRef, Any

Type Members

case class Duration(millis: Long) extends Product with Serializable
class StreamingContext extends Logging

Main entry point for Spark Streaming functionality.
Main entry point for Spark Streaming functionality. It provides methods used to create org.apache.spark.streaming.dstream.DStreams from various input sources. It can be either created by providing a Spark master URL and an appName, or from a org.apache.spark.SparkConf configuration (see core Spark documentation), or from an existing org.apache.spark.SparkContext. The associated SparkContext can be accessed using context.sparkContext. After creating and transforming DStreams, the streaming computation can be started and stopped using context.start() and context.stop(), respectively. context.awaitTermination() allows the current thread to wait for the termination of the context by stop() or by an exception.
final class StreamingContextState extends Enum[StreamingContextState]
case class Time(millis: Long) extends Product with Serializable

This is a simple class that represents an absolute instant of time.
This is a simple class that represents an absolute instant of time. Internally, it represents time as the difference, measured in milliseconds, between the current time and midnight, January 1, 1970 UTC. This is the same format as what is returned by System.currentTimeMillis.

Value Members

object Durations
object Milliseconds

Helper object that creates instance of org.apache.spark.streaming.Duration representing a given number of milliseconds.
object Minutes

Helper object that creates instance of org.apache.spark.streaming.Duration representing a given number of minutes.
object Seconds

Helper object that creates instance of org.apache.spark.streaming.Duration representing a given number of seconds.
object StreamingContext extends Logging

StreamingContext object contains a number of utility functions related to the StreamingContext class.
object Time extends Serializable
package api
package dstream

Various implementations of DStream's.
Various implementations of DStream's.

See also
org.apache.spark.streaming.dstream.DStream
package receiver
package scheduler
package util

package streaming

Type Members

case class Duration(millis: Long) extends Product with Serializable

class StreamingContext extends Logging

final class StreamingContextState extends Enum[StreamingContextState]

case class Time(millis: Long) extends Product with Serializable

Value Members

object Durations

object Milliseconds

object Minutes

object Seconds

object StreamingContext extends Logging

object Time extends Serializable

package api

package dstream

package receiver

package scheduler

package util

Inherited from AnyRef

Inherited from Any

Ungrouped