Packages

package streaming

Ordering
  1. Alphabetic
Visibility
  1. Public
  2. Protected

Package Members

  1. package checkpointing
  2. package continuous
  3. package operators
  4. package runtime
  5. package sinks
  6. package sources
  7. package state

Type Members

  1. case class ConsoleRelation(sqlContext: SQLContext, data: DataFrame) extends BaseRelation with Product with Serializable
  2. class ConsoleSinkProvider extends SimpleTableProvider with DataSourceRegister with CreatableRelationProvider
  3. class LowLatencyMemoryStream[A] extends MemoryStreamBaseClass[A] with SupportsRealTimeMode

    A low latency memory source from memory, only for unit test purpose.

    A low latency memory source from memory, only for unit test purpose. This class is very similar to ContinuousMemoryStream, except that it implements the interface of SupportsRealTimeMode, rather than ContinuousStream The overall strategy here is: * LowLatencyMemoryStream maintains a list of records for each partition. addData() will distribute records evenly-ish across partitions. * RecordEndpoint is set up as an endpoint for executor-side LowLatencyMemoryStreamInputPartitionReader instances to poll. It returns the record at the specified offset within the list, or null if that offset doesn't yet have a record. This differs from the existing memory source implementation as data is sent once to tasks as part of the Partition/Split metadata at the beginning of a batch.

  4. case class LowLatencyMemoryStreamInputPartition(driverEndpointName: String, driverEndpointAddress: RpcAddress, partition: Int, startOffset: Int, endOffset: Int) extends InputPartition with Product with Serializable

    An input partition for LowLatency memory stream.

  5. case class LowLatencyMemoryStreamOffset(partitionNums: Map[Int, Int]) extends Offset with Product with Serializable
  6. class LowLatencyMemoryStreamPartitionReader extends SupportsRealTimeRead[InternalRow]

    An input partition reader for LowLatency memory stream.

    An input partition reader for LowLatency memory stream.

    Polls the driver endpoint for new records.

  7. class LowLatencyMemoryStreamReaderFactory extends PartitionReaderFactory
  8. trait LowPriorityLowLatencyMemoryStreamImplicits extends AnyRef

    Provides lower-priority implicits for LowLatencyMemoryStream to prevent ambiguity when both SparkSession and SQLContext are in scope.

    Provides lower-priority implicits for LowLatencyMemoryStream to prevent ambiguity when both SparkSession and SQLContext are in scope. The implicits in the companion object, which use SparkSession, take higher precedence.

  9. class ManifestFileCommitProtocol extends FileCommitProtocol with Serializable with Logging

    A FileCommitProtocol that tracks the list of valid files in a manifest file, used in structured streaming.

  10. abstract class Offset extends connector.read.streaming.Offset

    This class is an alias of org.apache.spark.sql.connector.read.streaming.Offset.

    This class is an alias of org.apache.spark.sql.connector.read.streaming.Offset. It's internal and deprecated. New streaming data source implementations should use data source v2 API, which will be supported in the long term.

    This class will be removed in a future release.

  11. trait Sink extends Table

    An interface for systems that can collect the results of a streaming query.

    An interface for systems that can collect the results of a streaming query. In order to preserve exactly once semantics a sink must be idempotent in the face of multiple attempts to add the same batch.

    Note that, we extend Table here, to make the v1 streaming sink API be compatible with data source v2.

  12. trait Source extends SparkDataStream

    A source of continually arriving data for a streaming query.

    A source of continually arriving data for a streaming query. A Source must have a monotonically increasing notion of progress that can be represented as an Offset. Spark will regularly query each Source to see if any more data is available.

    Note that, we extends SparkDataStream here, to make the v1 streaming source API be compatible with data source v2.

Deprecated Type Members

  1. class HDFSMetadataLog[T <: AnyRef] extends streaming.checkpointing.HDFSMetadataLog[T]
    Annotations
    @deprecated
    Deprecated

    use org.apache.spark.sql.execution.streaming.checkpointing.HDFSMetadataLog

Value Members

  1. object ConsoleTable extends Table with SupportsWrite
  2. object LowLatencyMemoryStream extends LowPriorityLowLatencyMemoryStreamImplicits
  3. object StreamingErrors

    Object for grouping error messages from streaming query exceptions

  4. object StreamingQueryPlanTraverseHelper extends Logging

    This is an utility object placing methods to traverse the query plan for streaming query.

    This is an utility object placing methods to traverse the query plan for streaming query. This is used for patterns of traversal which are repeated in multiple places.

Deprecated Value Members

  1. object SerializedOffset
    Annotations
    @deprecated
    Deprecated

    use org.apache.spark.sql.execution.streaming.runtime.SerializedOffset

Ungrouped