streaming

Type Members

1. case class CompositeOffset(offsets: Seq[Option[Offset]]) extends Offset with Product with Serializable

   An ordered collection of offsets, used to track the progress of processing data from one or more Sources that are present in a streaming query.

2. class ConsoleSink extends Sink with Logging

3. class ConsoleSinkProvider extends StreamSinkProvider with DataSourceRegister

4. class FileStreamSink extends Sink with Logging

   A sink that writes out results to parquet files.

5. class FileStreamSinkLog extends HDFSMetadataLog[Seq[SinkFileStatus]]

   A special log for FileStreamSink.

6. class FileStreamSinkWriter extends Serializable with Logging

   Writes data given to a FileStreamSink to the given basePath in the given fileFormat, partitioned by the given partitionColumnNames.

7. class FileStreamSource extends Source with Logging

   A very simple source that reads text files from the given directory as they appear.

8. class ForeachSink[T] extends Sink with Serializable

   A Sink that forwards all data into ForeachWriter according to the contract defined by ForeachWriter.

9. class HDFSMetadataLog[T] extends MetadataLog[T] with Logging

   A MetadataLog implementation based on HDFS.

10. class IncrementalExecution extends QueryExecution

    A variant of QueryExecution that allows the execution of the given LogicalPlan plan incrementally.

11. case class LongOffset(offset: Long) extends Offset with Product with Serializable

    A simple offset for sources that produce a single linear stream of data.

12. case class MemoryPlan(sink: MemorySink, output: Seq[Attribute]) extends LeafNode with Product with Serializable

    Used to query the data that has been written into a MemorySink.

13. class MemorySink extends Sink with Logging

    A sink that stores the results in memory.

14. case class MemoryStream[A](id: Int, sqlContext: SQLContext)(implicit evidence$2: Encoder[A]) extends Source with Logging with Product with Serializable

    A Source that produces value stored in memory as they are added by the user.

15. trait MetadataLog[T] extends AnyRef

    A general MetadataLog that supports the following features:

16. class MetadataLogFileCatalog extends PartitioningAwareFileCatalog

    A FileCatalog that generates the list of files to processing by reading them from the metadata log files generated by the FileStreamSink.

17. trait Offset extends Serializable

    An offset is a monotonically increasing metric used to track progress in the computation of a stream.

18. case class OperatorStateId(checkpointLocation: String, operatorId: Long, batchId: Long) extends Product with Serializable

    Used to identify the state store for a given operator.

19. case class ProcessingTimeExecutor(processingTime: ProcessingTime, clock: Clock = ...) extends TriggerExecutor with Logging with Product with Serializable

    A trigger executor that runs a batch every intervalMs milliseconds.

20. trait Sink extends AnyRef

    An interface for systems that can collect the results of a streaming query.

21. case class SinkFileStatus(path: String, size: Long, isDir: Boolean, modificationTime: Long, blockReplication: Int, blockSize: Long, action: String) extends Product with Serializable

    The status of a file outputted by FileStreamSink.

22. trait Source extends AnyRef

    A source of continually arriving data for a streaming query.

23. case class StateStoreRestoreExec(keyExpressions: Seq[Attribute], stateId: Option[OperatorStateId], child: SparkPlan) extends SparkPlan with UnaryExecNode with StatefulOperator with Product with Serializable

    For each input tuple, the key is calculated and the value from the StateStore is added to the stream (in addition to the input tuple) if present.

24. case class StateStoreSaveExec(keyExpressions: Seq[Attribute], stateId: Option[OperatorStateId], returnAllStates: Option[Boolean], child: SparkPlan) extends SparkPlan with UnaryExecNode with StatefulOperator with Product with Serializable

    For each input tuple, the key is calculated and the tuple is put into the StateStore.

25. trait StatefulOperator extends SparkPlan

    An operator that saves or restores state from the StateStore.

26. class StreamExecution extends StreamingQuery with Logging

    Manages the execution of a streaming Spark SQL query that is occurring in a separate thread.

27. class StreamProgress extends Map[Source, Offset]

    A helper class that looks like a Map[Source, Offset].

28. case class StreamingExecutionRelation(source: Source, output: Seq[Attribute]) extends LeafNode with Product with Serializable

    Used to link a streaming Source of data into a org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.

29. class StreamingQueryListenerBus extends SparkListener with ListenerBus[StreamingQueryListener, Event]

    A bus to forward events to StreamingQueryListeners.

30. case class StreamingRelation(dataSource: DataSource, sourceName: String, output: Seq[Attribute]) extends LeafNode with Product with Serializable

    Used to link a streaming DataSource into a org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.

31. case class StreamingRelationExec(sourceName: String, output: Seq[Attribute]) extends SparkPlan with LeafExecNode with Product with Serializable

    A dummy physical plan for StreamingRelation to support org.apache.spark.sql.Dataset.explain

32. class TextSocketSource extends Source with Logging

    A source that reads text lines through a TCP socket, designed only for tutorials and debugging.

33. class TextSocketSourceProvider extends StreamSourceProvider with DataSourceRegister with Logging

34. trait TriggerExecutor extends AnyRef