Package

org.apache.spark.streaming

dstream

Permalink

package dstream

Various implementations of DStream's.

See also

org.apache.spark.streaming.dstream.DStream

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. dstream
  2. AnyRef
  3. Any
  1. Hide All
  2. Show all
Visibility
  1. Public
  2. All

Type Members

  1. class ConstantInputDStream[T] extends InputDStream[T]

    Permalink

    An input stream that always returns the same RDD on each timestep.

    An input stream that always returns the same RDD on each timestep. Useful for testing.

  2. abstract class DStream[T] extends Serializable with Logging

    Permalink

    A Discretized Stream (DStream), the basic abstraction in Spark Streaming, is a continuous sequence of RDDs (of the same type) representing a continuous stream of data (see org.apache.spark.rdd.RDD in the Spark core documentation for more details on RDDs).

    A Discretized Stream (DStream), the basic abstraction in Spark Streaming, is a continuous sequence of RDDs (of the same type) representing a continuous stream of data (see org.apache.spark.rdd.RDD in the Spark core documentation for more details on RDDs). DStreams can either be created from live data (such as, data from TCP sockets, Kafka, Flume, etc.) using a org.apache.spark.streaming.StreamingContext or it can be generated by transforming existing DStreams using operations such as map, window and reduceByKeyAndWindow. While a Spark Streaming program is running, each DStream periodically generates a RDD, either from live data or by transforming the RDD generated by a parent DStream.

    This class contains the basic operations available on all DStreams, such as map, filter and window. In addition, org.apache.spark.streaming.dstream.PairDStreamFunctions contains operations available only on DStreams of key-value pairs, such as groupByKeyAndWindow and join. These operations are automatically available on any DStream of pairs (e.g., DStream[(Int, Int)] through implicit conversions.

    DStreams internally is characterized by a few basic properties:

    • A list of other DStreams that the DStream depends on
    • A time interval at which the DStream generates an RDD
    • A function that is used to generate an RDD after each time interval
  3. abstract class InputDStream[T] extends DStream[T]

    Permalink

    This is the abstract base class for all input streams.

    This is the abstract base class for all input streams. This class provides methods start() and stop() which is called by Spark Streaming system to start and stop receiving data. Input streams that can generate RDDs from new data by running a service/thread only on the driver node (that is, without running a receiver on worker nodes), can be implemented by directly inheriting this InputDStream. For example, FileInputDStream, a subclass of InputDStream, monitors a HDFS directory from the driver for new files and generates RDDs with the new files. For implementing input streams that requires running a receiver on the worker nodes, use org.apache.spark.streaming.dstream.ReceiverInputDStream as the parent class.

  4. class PairDStreamFunctions[K, V] extends Serializable

    Permalink

    Extra functions available on DStream of (key, value) pairs through an implicit conversion.

  5. abstract class ReceiverInputDStream[T] extends InputDStream[T]

    Permalink

    Abstract class for defining any org.apache.spark.streaming.dstream.InputDStream that has to start a receiver on worker nodes to receive external data.

    Abstract class for defining any org.apache.spark.streaming.dstream.InputDStream that has to start a receiver on worker nodes to receive external data. Specific implementations of ReceiverInputDStream must define the getReceiver() function that gets the receiver object of type org.apache.spark.streaming.receiver.Receiver that will be sent to the workers to receive data.

    T

    Class type of the object of this stream

Value Members

  1. object DStream extends Serializable

    Permalink

Inherited from AnyRef

Inherited from Any

Ungrouped