org.apache.spark.sql.streaming

SchemaDStream

final class SchemaDStream extends DStream[Row]

A SQL based DStream with support for schema/Product This class offers the ability to manipulate SQL query on DStreams It is similar to SchemaRDD, which offers the similar functions Internally, RDD of each batch duration is treated as a small table and CQs are evaluated on those small tables Some of the abstraction and code is borrowed from the project: https://github.com/Intel-bigdata/spark-streamingsql

Linear Supertypes
DStream[Row], Logging, Serializable, Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. SchemaDStream
  2. DStream
  3. Logging
  4. Serializable
  5. Serializable
  6. AnyRef
  7. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new SchemaDStream(ssc: SnappyStreamingContext, logicalPlan: LogicalPlan)

  2. new SchemaDStream(snsc: SnappyStreamingContext, queryExecution: QueryExecution)

    snsc
    queryExecution

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any
  3. final def ##(): Int

    Definition Classes
    AnyRef → Any
  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any
  6. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  7. val baseScope: Option[String]

    Attributes
    protected[org.apache.spark.streaming]
    Definition Classes
    DStream
  8. def cache(): DStream[Row]

    Definition Classes
    DStream
  9. def checkpoint(interval: Duration): DStream[Row]

    Definition Classes
    DStream
  10. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  11. def columns: Array[String]

    Returns all column names as an array.

  12. def compute(validTime: Time): Option[RDD[Row]]

    Method that generates a RDD for the given time

    Method that generates a RDD for the given time

    Definition Classes
    SchemaDStream → DStream
  13. def context: StreamingContext

    Definition Classes
    DStream
  14. def count(): DStream[Long]

    Definition Classes
    DStream
  15. def countByValue(numPartitions: Int)(implicit ord: Ordering[Row]): DStream[(Row, Long)]

    Definition Classes
    DStream
  16. def countByValueAndWindow(windowDuration: Duration, slideDuration: Duration, numPartitions: Int)(implicit ord: Ordering[Row]): DStream[(Row, Long)]

    Definition Classes
    DStream
  17. def countByWindow(windowDuration: Duration, slideDuration: Duration): DStream[Long]

    Definition Classes
    DStream
  18. def createRDDWithLocalProperties[U](time: Time)(body: ⇒ U): U

    Attributes
    protected
    Definition Classes
    DStream
  19. def dependencies: List[DStream[InternalRow]]

    List of parent DStreams on which this DStream depends on

    List of parent DStreams on which this DStream depends on

    Definition Classes
    SchemaDStream → DStream
  20. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  21. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  22. def filter(filterFunc: (Row) ⇒ Boolean): DStream[Row]

    Return a new DStream containing only the elements that satisfy a predicate.

    Return a new DStream containing only the elements that satisfy a predicate.

    Definition Classes
    SchemaDStream → DStream
  23. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  24. def flatMap[U](flatMapFunc: (Row) ⇒ Traversable[U])(implicit arg0: ClassTag[U]): DStream[U]

    Definition Classes
    DStream
  25. def foreachDataFrame(foreachFunc: (DataFrame, Time) ⇒ Unit): Unit

    Apply a function to each DataFrame in this SchemaDStream.

    Apply a function to each DataFrame in this SchemaDStream. This is an output operator, so 'this' SchemaDStream will be registered as an output stream and therefore materialized.

  26. def foreachDataFrame(foreachFunc: (DataFrame) ⇒ Unit): Unit

    Apply a function to each DataFrame in this SchemaDStream.

    Apply a function to each DataFrame in this SchemaDStream. This is an output operator, so 'this' SchemaDStream will be registered as an output stream and therefore materialized.

  27. def foreachRDD(foreachFunc: (RDD[Row], Time) ⇒ Unit): Unit

    Definition Classes
    DStream
  28. def foreachRDD(foreachFunc: (RDD[Row]) ⇒ Unit): Unit

    Definition Classes
    DStream
  29. var generatedRDDs: Map[Time, RDD[Row]]

    Definition Classes
    DStream
  30. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any
  31. final def getOrCompute(time: Time): Option[RDD[Row]]

    Definition Classes
    DStream
  32. def glom(): DStream[Array[Row]]

    Definition Classes
    DStream
  33. var graph: DStreamGraph

    Definition Classes
    DStream
  34. def hashCode(): Int

    Definition Classes
    AnyRef → Any
  35. def initGeneratedRDDs(): Unit

    Definition Classes
    DStream
  36. def initializeAfterContextStart(time: Time): Unit

    Definition Classes
    DStream
  37. def isInitialized: Boolean

    Definition Classes
    DStream
  38. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  39. def isTraceEnabled(): Boolean

    Attributes
    protected
    Definition Classes
    Logging
  40. def log: Logger

    Attributes
    protected
    Definition Classes
    Logging
  41. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  42. def logDebug(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  43. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  44. def logError(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  45. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  46. def logInfo(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  47. def logName: String

    Attributes
    protected
    Definition Classes
    Logging
  48. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  49. def logTrace(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  50. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  51. def logWarning(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  52. val logicalPlan: LogicalPlan

  53. def map[U](mapFunc: (Row) ⇒ U)(implicit arg0: ClassTag[U]): DStream[U]

    Definition Classes
    DStream
  54. def mapPartitions[U](mapPartFunc: (Iterator[Row]) ⇒ Iterator[U], preservePartitioning: Boolean)(implicit arg0: ClassTag[U]): DStream[U]

    Definition Classes
    DStream
  55. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  56. final def notify(): Unit

    Definition Classes
    AnyRef
  57. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  58. def persist(): DStream[Row]

    Definition Classes
    DStream
  59. def persist(level: StorageLevel): DStream[Row]

    Definition Classes
    DStream
  60. def print(num: Int): Unit

    Definition Classes
    DStream
  61. def print(): Unit

    Definition Classes
    DStream
  62. def printSchema(): Unit

  63. val queryExecution: QueryExecution

  64. def reduce(reduceFunc: (Row, Row) ⇒ Row): DStream[Row]

    Definition Classes
    DStream
  65. def reduceByWindow(reduceFunc: (Row, Row) ⇒ Row, invReduceFunc: (Row, Row) ⇒ Row, windowDuration: Duration, slideDuration: Duration): DStream[Row]

    Definition Classes
    DStream
  66. def reduceByWindow(reduceFunc: (Row, Row) ⇒ Row, windowDuration: Duration, slideDuration: Duration): DStream[Row]

    Definition Classes
    DStream
  67. def register(): DStream[Row]

    Definition Classes
    DStream
  68. def registerAsTable(tableName: String): Unit

    Registers this SchemaDStream as a table in the catalog.

  69. def repartition(numPartitions: Int): DStream[Row]

    Definition Classes
    DStream
  70. def saveAsObjectFiles(prefix: String, suffix: String): Unit

    Definition Classes
    DStream
  71. def saveAsTextFiles(prefix: String, suffix: String): Unit

    Definition Classes
    DStream
  72. def schema: StructType

    Returns the schema of this SchemaDStream (represented by a StructType).

  73. def slice(fromTime: Time, toTime: Time): Seq[RDD[Row]]

    Definition Classes
    DStream
  74. def slice(interval: Interval): Seq[RDD[Row]]

    Definition Classes
    DStream
  75. def slideDuration: Duration

    Time interval after which the DStream generates a RDD

    Time interval after which the DStream generates a RDD

    Definition Classes
    SchemaDStream → DStream
  76. val snsc: SnappyStreamingContext

  77. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  78. def toString(): String

    Definition Classes
    AnyRef → Any
  79. def transform[U](transformFunc: (RDD[Row], Time) ⇒ RDD[U])(implicit arg0: ClassTag[U]): DStream[U]

    Definition Classes
    DStream
  80. def transform[U](transformFunc: (RDD[Row]) ⇒ RDD[U])(implicit arg0: ClassTag[U]): DStream[U]

    Definition Classes
    DStream
  81. def transformWith[U, V](other: DStream[U], transformFunc: (RDD[Row], RDD[U], Time) ⇒ RDD[V])(implicit arg0: ClassTag[U], arg1: ClassTag[V]): DStream[V]

    Definition Classes
    DStream
  82. def transformWith[U, V](other: DStream[U], transformFunc: (RDD[Row], RDD[U]) ⇒ RDD[V])(implicit arg0: ClassTag[U], arg1: ClassTag[V]): DStream[V]

    Definition Classes
    DStream
  83. def union(that: DStream[Row]): DStream[Row]

    Definition Classes
    DStream
  84. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  85. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  86. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  87. def window(windowDuration: Duration, slideDuration: Duration): DStream[Row]

    Definition Classes
    DStream
  88. def window(windowDuration: Duration): DStream[Row]

    Definition Classes
    DStream

Deprecated Value Members

  1. def foreach(foreachFunc: (RDD[Row], Time) ⇒ Unit): Unit

    Definition Classes
    DStream
    Annotations
    @deprecated
    Deprecated

    (Since version 0.9.0) use foreachRDD

  2. def foreach(foreachFunc: (RDD[Row]) ⇒ Unit): Unit

    Definition Classes
    DStream
    Annotations
    @deprecated
    Deprecated

    (Since version 0.9.0) use foreachRDD

Inherited from DStream[Row]

Inherited from Logging

Inherited from Serializable

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped