receiver

Type Members

1. trait ActorHelper extends Logging

   

   :: DeveloperApi :: A receiver trait to be mixed in with your Actor to gain access to the API for pushing received data into Spark Streaming for being processed.

   :: DeveloperApi :: A receiver trait to be mixed in with your Actor to gain access to the API for pushing received data into Spark Streaming for being processed.

   Find more details at: http://spark.apache.org/docs/latest/streaming-custom-receivers.html

   Annotations

   @DeveloperApi()

   Example:

   1. class MyActor extends Actor with ActorHelper{
          def receive {
              case anything: String => store(anything)
          }
      }
      // Can be used with an actorStream as follows
      ssc.actorStream[String](Props(new MyActor),"MyActorReceiver")

   Note

   Since Actor may exist outside the spark framework, It is thus user's responsibility to ensure the type safety, i.e parametrized type of push block and InputDStream should be same.

2. abstract class Receiver[T] extends Serializable

   

   :: DeveloperApi :: Abstract class of a receiver that can be run on worker nodes to receive external data.

   :: DeveloperApi :: Abstract class of a receiver that can be run on worker nodes to receive external data. A custom receiver can be defined by defining the functions onStart() and onStop(). onStart() should define the setup steps necessary to start receiving data, and onStop() should define the cleanup steps necessary to stop receiving data. Exceptions while receiving can be handled either by restarting the receiver with restart(...) or stopped completely by stop(...) or

   A custom receiver in Scala would look like this.

   class MyReceiver(storageLevel: StorageLevel) extends NetworkReceiver[String](storageLevel) {
       def onStart() {
           // Setup stuff (start threads, open sockets, etc.) to start receiving data.
           // Must start new thread to receive data, as onStart() must be non-blocking.
   
           // Call store(...) in those threads to store received data into Spark's memory.
   
           // Call stop(...), restart(...) or reportError(...) on any thread based on how
           // different errors needs to be handled.
   
           // See corresponding method documentation for more details
       }
   
       def onStop() {
           // Cleanup stuff (stop threads, close sockets, etc.) to stop receiving data.
       }
   }

   A custom receiver in Java would look like this.

   class MyReceiver extends Receiver<String> {
       public MyReceiver(StorageLevel storageLevel) {
           super(storageLevel);
       }
   
       public void onStart() {
            // Setup stuff (start threads, open sockets, etc.) to start receiving data.
            // Must start new thread to receive data, as onStart() must be non-blocking.
   
            // Call store(...) in those threads to store received data into Spark's memory.
   
            // Call stop(...), restart(...) or reportError(...) on any thread based on how
            // different errors needs to be handled.
   
            // See corresponding method documentation for more details
       }
   
       public void onStop() {
            // Cleanup stuff (stop threads, close sockets, etc.) to stop receiving data.
       }
   }

   Annotations

   @DeveloperApi()

3. case class Statistics(numberOfMsgs: Int, numberOfWorkers: Int, numberOfHiccups: Int, otherInfo: String) extends Product with Serializable

   

   :: DeveloperApi :: Statistics for querying the supervisor about state of workers.

   :: DeveloperApi :: Statistics for querying the supervisor about state of workers. Used in conjunction with StreamingContext.actorStream and org.apache.spark.streaming.receiver.ActorHelper.

   Annotations

   @DeveloperApi()