Packages

trait Source extends SparkDataStream

A source of continually arriving data for a streaming query. A Source must have a monotonically increasing notion of progress that can be represented as an Offset. Spark will regularly query each Source to see if any more data is available.

Note that, we extends SparkDataStream here, to make the v1 streaming source API be compatible with data source v2.

Linear Supertypes
SparkDataStream, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. Source
  2. SparkDataStream
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Abstract Value Members

  1. abstract def getBatch(start: Option[Offset], end: Offset): DataFrame

    Returns the data that is between the offsets (start, end].

    Returns the data that is between the offsets (start, end]. When start is None, then the batch should begin with the first record. This method must always return the same data for a particular start and end pair; even after the Source has been restarted on a different node.

    Higher layers will always call this method with a value of start greater than or equal to the last value passed to commit and a value of end less than or equal to the last value returned by getOffset

    It is possible for the Offset type to be a SerializedOffset when it was obtained from the log. Moreover, StreamExecution only compares the Offset JSON representation to determine if the two objects are equal. This could have ramifications when upgrading Offset JSON formats i.e., two equivalent Offset objects could differ between version. Consequently, StreamExecution may call this method with two such equivalent Offset objects. In which case, the Source should return an empty DataFrame

  2. abstract def getOffset: Option[Offset]

    Returns the maximum available offset for this source.

    Returns the maximum available offset for this source. Returns None if this source has never received any data.

  3. abstract def schema: StructType

    Returns the schema of the data from this source

  4. abstract def stop(): Unit
    Definition Classes
    SparkDataStream

Concrete Value Members

  1. def commit(end: connector.read.streaming.Offset): Unit
    Definition Classes
    Source → SparkDataStream
  2. def commit(end: Offset): Unit

    Informs the source that Spark has completed processing all data for offsets less than or equal to end and will only request offsets greater than end in the future.

  3. def deserializeOffset(json: String): connector.read.streaming.Offset
    Definition Classes
    Source → SparkDataStream
  4. def initialOffset(): connector.read.streaming.Offset
    Definition Classes
    Source → SparkDataStream