org.apache.spark.sql.streaming

SnappyStreamingContext

class SnappyStreamingContext extends StreamingContext with Serializable

Main entry point for SnappyData extensions to Spark Streaming. A SnappyStreamingContext extends Spark's org.apache.spark.streaming.StreamingContext to provides an ability to manipulate SQL like query on org.apache.spark.streaming.dstream.DStream. You can apply schema and register continuous SQL queries(CQ) over the data streams. A single shared SnappyStreamingContext makes it possible to re-use Executors across client connections or applications.

Self Type
SnappyStreamingContext
Linear Supertypes
Serializable, Serializable, StreamingContext, Logging, AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. SnappyStreamingContext
  2. Serializable
  3. Serializable
  4. StreamingContext
  5. Logging
  6. AnyRef
  7. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new SnappyStreamingContext(snappyContext: SnappyContext, batchDur: Duration)

    Attributes
    protected[org.apache.spark]

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any
  3. final def ##(): Int

    Definition Classes
    AnyRef → Any
  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any
  6. def actorStream[T](props: Props, name: String, storageLevel: StorageLevel, supervisorStrategy: SupervisorStrategy)(implicit arg0: ClassTag[T]): ReceiverInputDStream[T]

    Definition Classes
    StreamingContext
  7. def addStreamingListener(streamingListener: StreamingListener): Unit

    Definition Classes
    StreamingContext
  8. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  9. def awaitTermination(): Unit

    Definition Classes
    StreamingContext
  10. def awaitTerminationOrTimeout(timeout: Long): Boolean

    Definition Classes
    StreamingContext
  11. val batchDur: Duration

  12. def binaryRecordsStream(directory: String, recordLength: Int): DStream[Array[Byte]]

    Definition Classes
    StreamingContext
    Annotations
    @Experimental()
  13. def checkpoint(directory: String): Unit

    Definition Classes
    StreamingContext
  14. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  15. def createSchemaDStream(rowStream: DStream[Row], schema: StructType): SchemaDStream

  16. def createSchemaDStream[A <: Product](stream: DStream[A])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[A]): SchemaDStream

    Creates a SchemaDStream from an DStream of Product (e.

    Creates a SchemaDStream from an DStream of Product (e.g. case classes).

  17. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  18. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  19. def fileStream[K, V, F <: InputFormat[K, V]](directory: String, filter: (Path) ⇒ Boolean, newFilesOnly: Boolean, conf: Configuration)(implicit arg0: ClassTag[K], arg1: ClassTag[V], arg2: ClassTag[F]): InputDStream[(K, V)]

    Definition Classes
    StreamingContext
  20. def fileStream[K, V, F <: InputFormat[K, V]](directory: String, filter: (Path) ⇒ Boolean, newFilesOnly: Boolean)(implicit arg0: ClassTag[K], arg1: ClassTag[V], arg2: ClassTag[F]): InputDStream[(K, V)]

    Definition Classes
    StreamingContext
  21. def fileStream[K, V, F <: InputFormat[K, V]](directory: String)(implicit arg0: ClassTag[K], arg1: ClassTag[V], arg2: ClassTag[F]): InputDStream[(K, V)]

    Definition Classes
    StreamingContext
  22. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  23. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any
  24. def getSchemaDStream(tableName: String): SchemaDStream

  25. def getState(): StreamingContextState

    Definition Classes
    StreamingContext
    Annotations
    @DeveloperApi()
  26. val graph: DStreamGraph

    Definition Classes
    StreamingContext
  27. def hashCode(): Int

    Definition Classes
    AnyRef → Any
  28. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  29. def isTraceEnabled(): Boolean

    Attributes
    protected
    Definition Classes
    Logging
  30. def log: Logger

    Attributes
    protected
    Definition Classes
    Logging
  31. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  32. def logDebug(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  33. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  34. def logError(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  35. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  36. def logInfo(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  37. def logName: String

    Attributes
    protected
    Definition Classes
    Logging
  38. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  39. def logTrace(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  40. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  41. def logWarning(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  42. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  43. final def notify(): Unit

    Definition Classes
    AnyRef
  44. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  45. def queueStream[T](queue: Queue[RDD[T]], oneAtATime: Boolean, defaultRDD: RDD[T])(implicit arg0: ClassTag[T]): InputDStream[T]

    Definition Classes
    StreamingContext
  46. def queueStream[T](queue: Queue[RDD[T]], oneAtATime: Boolean)(implicit arg0: ClassTag[T]): InputDStream[T]

    Definition Classes
    StreamingContext
  47. def rawSocketStream[T](hostname: String, port: Int, storageLevel: StorageLevel)(implicit arg0: ClassTag[T]): ReceiverInputDStream[T]

    Definition Classes
    StreamingContext
  48. def receiverStream[T](receiver: Receiver[T])(implicit arg0: ClassTag[T]): ReceiverInputDStream[T]

    Definition Classes
    StreamingContext
  49. def registerCQ(queryStr: String): SchemaDStream

    Registers and executes given SQL query and returns SchemaDStream to consume the results

    Registers and executes given SQL query and returns SchemaDStream to consume the results

    queryStr
    returns

  50. def remember(duration: Duration): Unit

    Definition Classes
    StreamingContext
  51. val snappyContext: SnappyContext

  52. def socketStream[T](hostname: String, port: Int, converter: (InputStream) ⇒ Iterator[T], storageLevel: StorageLevel)(implicit arg0: ClassTag[T]): ReceiverInputDStream[T]

    Definition Classes
    StreamingContext
  53. def socketTextStream(hostname: String, port: Int, storageLevel: StorageLevel): ReceiverInputDStream[String]

    Definition Classes
    StreamingContext
  54. def sparkContext: SparkContext

    Definition Classes
    StreamingContext
  55. def sql(sqlText: String): DataFrame

  56. def start(): Unit

    Start the execution of the streams.

    Start the execution of the streams. Also registers population of AQP tables from stream tables if present.

    Definition Classes
    SnappyStreamingContext → StreamingContext
    Exceptions thrown
    IllegalStateException

    if the StreamingContext is already stopped

  57. def stop(stopSparkContext: Boolean, stopGracefully: Boolean): Unit

    Definition Classes
    SnappyStreamingContext → StreamingContext
  58. def stop(stopSparkContext: Boolean): Unit

    Definition Classes
    StreamingContext
  59. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  60. def textFileStream(directory: String): DStream[String]

    Definition Classes
    StreamingContext
  61. def toString(): String

    Definition Classes
    AnyRef → Any
  62. def transform[T](dstreams: Seq[DStream[_]], transformFunc: (Seq[RDD[_]], Time) ⇒ RDD[T])(implicit arg0: ClassTag[T]): DStream[T]

    Definition Classes
    StreamingContext
  63. def union[T](streams: Seq[DStream[T]])(implicit arg0: ClassTag[T]): DStream[T]

    Definition Classes
    StreamingContext
  64. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  65. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  66. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Deprecated Value Members

  1. def awaitTermination(timeout: Long): Unit

    Definition Classes
    StreamingContext
    Annotations
    @deprecated
    Deprecated

    (Since version 1.3.0) Use awaitTerminationOrTimeout(Long) instead

  2. def networkStream[T](receiver: Receiver[T])(implicit arg0: ClassTag[T]): ReceiverInputDStream[T]

    Definition Classes
    StreamingContext
    Annotations
    @deprecated
    Deprecated

    (Since version 1.0.0) Use receiverStream

Inherited from Serializable

Inherited from Serializable

Inherited from StreamingContext

Inherited from Logging

Inherited from AnyRef

Inherited from Any

Ungrouped