SchemaDStream

Instance Constructors

new SchemaDStream(ssc: SnappyStreamingContext, logicalPlan: LogicalPlan)
new SchemaDStream(snsc: SnappyStreamingContext, queryExecution: QueryExecution)

Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def asInstanceOf[T0]: T0

Definition Classes
Any
val baseScope: Option[String]

Attributes
protected[org.apache.spark.streaming]
Definition Classes
DStream
def cache(): SchemaDStream

Persist RDDs of this SchemaDStream with the default storage level (MEMORY_ONLY_SER)
Persist RDDs of this SchemaDStream with the default storage level (MEMORY_ONLY_SER)

Definition Classes
SchemaDStream → DStream
def checkpoint(interval: Duration): SchemaDStream

Enable periodic checkpointing of RDDs of this SchemaDStream
Enable periodic checkpointing of RDDs of this SchemaDStream
interval
Time interval after which generated RDD will be checkpointed

Definition Classes
SchemaDStream → DStream
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
def columns: Array[String]

Returns all column names as an array.
def compute(validTime: Time): Option[RDD[Row]]

Method that generates a RDD for the given time
Method that generates a RDD for the given time

Definition Classes
SchemaDStream → DStream
def context: StreamingContext

Definition Classes
DStream
def count(): DStream[Long]

Definition Classes
DStream
def countByValue(numPartitions: Int)(implicit ord: Ordering[Row]): DStream[(Row, Long)]

Definition Classes
DStream
def countByValueAndWindow(windowDuration: Duration, slideDuration: Duration, numPartitions: Int)(implicit ord: Ordering[Row]): DStream[(Row, Long)]

Definition Classes
DStream
def countByWindow(windowDuration: Duration, slideDuration: Duration): DStream[Long]

Definition Classes
DStream
def createRDDWithLocalProperties[U](time: Time, displayInnerRDDOps: Boolean)(body: ⇒ U): U

Attributes
protected[org.apache.spark.streaming]
Definition Classes
DStream
def dependencies: List[DStream[InternalRow]]

List of parent DStreams on which this SchemaDStream depends on
List of parent DStreams on which this SchemaDStream depends on

Definition Classes
SchemaDStream → DStream
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def equals(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def filter(filterFunc: (Row) ⇒ Boolean): SchemaDStream

Return a new SchemaDStream containing only the elements that satisfy a predicate.
Return a new SchemaDStream containing only the elements that satisfy a predicate.

Definition Classes
SchemaDStream → DStream
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
def flatMap[U](flatMapFunc: (Row) ⇒ TraversableOnce[U])(implicit arg0: ClassTag[U]): DStream[U]

Return a new DStream by applying a function to all elements of this SchemaDStream, and then flattening the results
Return a new DStream by applying a function to all elements of this SchemaDStream, and then flattening the results

Definition Classes
SchemaDStream → DStream
def foreachDataFrame(foreachFunc: (DataFrame, Time) ⇒ Unit, needsConversion: Boolean): Unit

Apply a function to each DataFrame in this SchemaDStream.
Apply a function to each DataFrame in this SchemaDStream. This is an output operator, so 'this' SchemaDStream will be registered as an output stream and therefore materialized.
def foreachDataFrame(foreachFunc: (DataFrame, Time) ⇒ Unit): Unit

Apply a function to each DataFrame in this SchemaDStream.
Apply a function to each DataFrame in this SchemaDStream. This is an output operator, so 'this' SchemaDStream will be registered as an output stream and therefore materialized.
def foreachDataFrame(foreachFunc: VoidFunction2[DataFrame, Time]): Unit

Apply a function to each DataFrame in this SchemaDStream.
Apply a function to each DataFrame in this SchemaDStream. This is an output operator, so 'this' SchemaDStream will be registered as an output stream and therefore materialized.
def foreachDataFrame(foreachFunc: VoidFunction[DataFrame], needsConversion: Boolean): Unit

Apply a function to each DataFrame in this SchemaDStream.
Apply a function to each DataFrame in this SchemaDStream. This is an output operator, so 'this' SchemaDStream will be registered as an output stream and therefore materialized.
def foreachDataFrame(foreachFunc: VoidFunction[DataFrame]): Unit

Apply a function to each DataFrame in this SchemaDStream.
Apply a function to each DataFrame in this SchemaDStream. This is an output operator, so 'this' SchemaDStream will be registered as an output stream and therefore materialized.
def foreachDataFrame(foreachFunc: (DataFrame) ⇒ Unit, needsConversion: Boolean): Unit

Apply a function to each DataFrame in this SchemaDStream.
Apply a function to each DataFrame in this SchemaDStream. This is an output operator, so 'this' SchemaDStream will be registered as an output stream and therefore materialized.
def foreachDataFrame(foreachFunc: (DataFrame) ⇒ Unit): Unit

Apply a function to each DataFrame in this SchemaDStream.
Apply a function to each DataFrame in this SchemaDStream. This is an output operator, so 'this' SchemaDStream will be registered as an output stream and therefore materialized.
def foreachRDD(foreachFunc: (RDD[Row], Time) ⇒ Unit): Unit

Definition Classes
DStream
def foreachRDD(foreachFunc: (RDD[Row]) ⇒ Unit): Unit

Definition Classes
DStream
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
def glom(): DStream[Array[Row]]

Return a new DStream in which each RDD is generated by applying glom() to each RDD of this SchemaDStream.
Return a new DStream in which each RDD is generated by applying glom() to each RDD of this SchemaDStream. Applying glom() to an RDD coalesces all elements within each partition into an array.

Definition Classes
SchemaDStream → DStream
def hashCode(): Int

Definition Classes
AnyRef → Any
def initializeLogIfNecessary(isInterpreter: Boolean): Unit

Attributes
protected
Definition Classes
Logging
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
def isTraceEnabled(): Boolean

Attributes
protected
Definition Classes
Logging
def log: Logger

Attributes
protected
Definition Classes
Logging
def logDebug(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logDebug(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logError(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logError(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logInfo(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logInfo(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logName: String

Attributes
protected
Definition Classes
Logging
def logTrace(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logTrace(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logWarning(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logWarning(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
val logicalPlan: LogicalPlan
def map[U](mapFunc: (Row) ⇒ U)(implicit arg0: ClassTag[U]): DStream[U]

Return a new DStream by applying a function to all elements of this SchemaDStream.
Return a new DStream by applying a function to all elements of this SchemaDStream.

Definition Classes
SchemaDStream → DStream
def mapPartitions[U](mapPartFunc: (Iterator[Row]) ⇒ Iterator[U], preservePartitioning: Boolean = false)(implicit arg0: ClassTag[U]): DStream[U]

Return a new DStream in which each RDD is generated by applying mapPartitions() to each RDDs of this SchemaDStream.
Return a new DStream in which each RDD is generated by applying mapPartitions() to each RDDs of this SchemaDStream. Applying mapPartitions() to an RDD applies a function to each partition of the RDD.

Definition Classes
SchemaDStream → DStream
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
def persist(): SchemaDStream

Persist RDDs of this SchemaDStream with the default storage level (MEMORY_ONLY_SER)
Persist RDDs of this SchemaDStream with the default storage level (MEMORY_ONLY_SER)

Definition Classes
SchemaDStream → DStream
def persist(level: StorageLevel): SchemaDStream

Persist the RDDs of this SchemaDStream with the given storage level
Persist the RDDs of this SchemaDStream with the given storage level

Definition Classes
SchemaDStream → DStream
def print(num: Int): Unit

Definition Classes
DStream
def print(): Unit

Definition Classes
DStream
def printSchema(): Unit
val queryExecution: QueryExecution
def reduce(reduceFunc: (Row, Row) ⇒ Row): DStream[Row]

Definition Classes
DStream
def reduceByWindow(reduceFunc: (Row, Row) ⇒ Row, invReduceFunc: (Row, Row) ⇒ Row, windowDuration: Duration, slideDuration: Duration): DStream[Row]

Definition Classes
DStream
def reduceByWindow(reduceFunc: (Row, Row) ⇒ Row, windowDuration: Duration, slideDuration: Duration): DStream[Row]

Definition Classes
DStream
def registerAsTable(tableName: String): Unit

Registers this SchemaDStream as a table in the catalog.
def repartition(numPartitions: Int): SchemaDStream

Return a new SchemaDStream with an increased or decreased level of parallelism.
Return a new SchemaDStream with an increased or decreased level of parallelism. Each RDD in the returned SchemaDStream has exactly numPartitions partitions.

Definition Classes
SchemaDStream → DStream
def saveAsObjectFiles(prefix: String, suffix: String): Unit

Definition Classes
DStream
def saveAsTextFiles(prefix: String, suffix: String): Unit

Definition Classes
DStream
def schema: StructType

Returns the schema of this SchemaDStream (represented by a StructType).
def slice(fromTime: Time, toTime: Time): Seq[RDD[Row]]

Definition Classes
DStream
def slice(interval: Interval): Seq[RDD[Row]]

Definition Classes
DStream
def slideDuration: Duration

Time interval after which the SchemaDStream generates a RDD
Time interval after which the SchemaDStream generates a RDD

Definition Classes
SchemaDStream → DStream
val snsc: SnappyStreamingContext
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
def toString(): String

Definition Classes
AnyRef → Any
def transform[U](transformFunc: (RDD[Row], Time) ⇒ RDD[U])(implicit arg0: ClassTag[U]): DStream[U]

Return a new DStream in which each RDD is generated by applying a function on each RDD of 'this' SchemaDStream.
Return a new DStream in which each RDD is generated by applying a function on each RDD of 'this' SchemaDStream.

Definition Classes
SchemaDStream → DStream
def transform[U](transformFunc: (RDD[Row]) ⇒ RDD[U])(implicit arg0: ClassTag[U]): DStream[U]

Return a new DStream in which each RDD is generated by applying a function on each RDD of 'this' SchemaDStream.
Return a new DStream in which each RDD is generated by applying a function on each RDD of 'this' SchemaDStream.

Definition Classes
SchemaDStream → DStream
def transformWith[U, V](other: DStream[U], transformFunc: (RDD[Row], RDD[U], Time) ⇒ RDD[V])(implicit arg0: ClassTag[U], arg1: ClassTag[V]): DStream[V]

Return a new DStream in which each RDD is generated by applying a function on each RDD of 'this' DStream and 'other' DStream.
Return a new DStream in which each RDD is generated by applying a function on each RDD of 'this' DStream and 'other' DStream.

Definition Classes
SchemaDStream → DStream
def transformWith[U, V](other: DStream[U], transformFunc: (RDD[Row], RDD[U]) ⇒ RDD[V])(implicit arg0: ClassTag[U], arg1: ClassTag[V]): DStream[V]

Return a new DStream in which each RDD is generated by applying a function on each RDD of 'this' SchemaDStream and 'other' SchemaDStream.
Return a new DStream in which each RDD is generated by applying a function on each RDD of 'this' SchemaDStream and 'other' SchemaDStream.

Definition Classes
SchemaDStream → DStream
def union(that: DStream[Row]): DStream[Row]

Definition Classes
DStream
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
def window(windowDuration: Duration, slideDuration: Duration): DStream[Row]

Definition Classes
DStream
def window(windowDuration: Duration): DStream[Row]

Definition Classes
DStream

Related Doc: package streaming

class SchemaDStream extends DStream[Row]

Instance Constructors

new SchemaDStream(ssc: SnappyStreamingContext, logicalPlan: LogicalPlan)

new SchemaDStream(snsc: SnappyStreamingContext, queryExecution: QueryExecution)

Value Members

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: Any): Boolean

final def asInstanceOf[T0]: T0

val baseScope: Option[String]

def cache(): SchemaDStream

def checkpoint(interval: Duration): SchemaDStream

def clone(): AnyRef

def columns: Array[String]

def compute(validTime: Time): Option[RDD[Row]]

def context: StreamingContext

def count(): DStream[Long]

def countByValue(numPartitions: Int)(implicit ord: Ordering[Row]): DStream[(Row, Long)]

def countByValueAndWindow(windowDuration: Duration, slideDuration: Duration, numPartitions: Int)(implicit ord: Ordering[Row]): DStream[(Row, Long)]

def countByWindow(windowDuration: Duration, slideDuration: Duration): DStream[Long]

def createRDDWithLocalProperties[U](time: Time, displayInnerRDDOps: Boolean)(body: ⇒ U): U

def dependencies: List[DStream[InternalRow]]

final def eq(arg0: AnyRef): Boolean

def equals(arg0: Any): Boolean

def filter(filterFunc: (Row) ⇒ Boolean): SchemaDStream

def finalize(): Unit

def flatMap[U](flatMapFunc: (Row) ⇒ TraversableOnce[U])(implicit arg0: ClassTag[U]): DStream[U]

def foreachDataFrame(foreachFunc: (DataFrame, Time) ⇒ Unit, needsConversion: Boolean): Unit

def foreachDataFrame(foreachFunc: (DataFrame, Time) ⇒ Unit): Unit

def foreachDataFrame(foreachFunc: VoidFunction2[DataFrame, Time]): Unit

def foreachDataFrame(foreachFunc: VoidFunction[DataFrame], needsConversion: Boolean): Unit

def foreachDataFrame(foreachFunc: VoidFunction[DataFrame]): Unit

def foreachDataFrame(foreachFunc: (DataFrame) ⇒ Unit, needsConversion: Boolean): Unit

def foreachDataFrame(foreachFunc: (DataFrame) ⇒ Unit): Unit

def foreachRDD(foreachFunc: (RDD[Row], Time) ⇒ Unit): Unit

def foreachRDD(foreachFunc: (RDD[Row]) ⇒ Unit): Unit

final def getClass(): Class[_]

def glom(): DStream[Array[Row]]

def hashCode(): Int

def initializeLogIfNecessary(isInterpreter: Boolean): Unit

final def isInstanceOf[T0]: Boolean

def isTraceEnabled(): Boolean

def log: Logger

def logDebug(msg: ⇒ String, throwable: Throwable): Unit

def logDebug(msg: ⇒ String): Unit

def logError(msg: ⇒ String, throwable: Throwable): Unit

def logError(msg: ⇒ String): Unit

def logInfo(msg: ⇒ String, throwable: Throwable): Unit

def logInfo(msg: ⇒ String): Unit

def logName: String

def logTrace(msg: ⇒ String, throwable: Throwable): Unit

def logTrace(msg: ⇒ String): Unit

def logWarning(msg: ⇒ String, throwable: Throwable): Unit

def logWarning(msg: ⇒ String): Unit

val logicalPlan: LogicalPlan

def map[U](mapFunc: (Row) ⇒ U)(implicit arg0: ClassTag[U]): DStream[U]

def mapPartitions[U](mapPartFunc: (Iterator[Row]) ⇒ Iterator[U], preservePartitioning: Boolean = false)(implicit arg0: ClassTag[U]): DStream[U]

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

def persist(): SchemaDStream

def persist(level: StorageLevel): SchemaDStream

def print(num: Int): Unit

def print(): Unit

def printSchema(): Unit

val queryExecution: QueryExecution

def reduce(reduceFunc: (Row, Row) ⇒ Row): DStream[Row]

def reduceByWindow(reduceFunc: (Row, Row) ⇒ Row, invReduceFunc: (Row, Row) ⇒ Row, windowDuration: Duration, slideDuration: Duration): DStream[Row]

def reduceByWindow(reduceFunc: (Row, Row) ⇒ Row, windowDuration: Duration, slideDuration: Duration): DStream[Row]

def registerAsTable(tableName: String): Unit

def repartition(numPartitions: Int): SchemaDStream

def saveAsObjectFiles(prefix: String, suffix: String): Unit

def saveAsTextFiles(prefix: String, suffix: String): Unit

def schema: StructType

def slice(fromTime: Time, toTime: Time): Seq[RDD[Row]]

def slice(interval: Interval): Seq[RDD[Row]]

def slideDuration: Duration

val snsc: SnappyStreamingContext