Type Members

1.  final class DataStreamReader extends Logging

   Interface used to load a streaming Dataset from external storage systems (e.g.

   Interface used to load a streaming Dataset from external storage systems (e.g. file systems, key-value stores, etc). Use SparkSession.readStream to access this.

   Annotations

   @Evolving()

   Since

   2.0.0

2.  final class DataStreamWriter[T] extends AnyRef

   Interface used to write a streaming Dataset to external storage systems (e.g.

   Interface used to write a streaming Dataset to external storage systems (e.g. file systems, key-value stores, etc). Use Dataset.writeStream to access this.

   Annotations

   @Evolving()

   Since

   2.0.0

3.  class SinkProgress extends Serializable

   Information about progress made for a sink in the execution of a StreamingQuery during a trigger.

   Information about progress made for a sink in the execution of a StreamingQuery during a trigger. See StreamingQueryProgress for more information.

   Annotations

   @Evolving()

   Since

   2.1.0

4.  class SourceProgress extends Serializable

   Information about progress made for a source in the execution of a StreamingQuery during a trigger.

   Information about progress made for a source in the execution of a StreamingQuery during a trigger. See StreamingQueryProgress for more information.

   Annotations

   @Evolving()

   Since

   2.1.0

5.  class StateOperatorProgress extends Serializable

   Information about updates made to stateful operators in a StreamingQuery during a trigger.

   Information about updates made to stateful operators in a StreamingQuery during a trigger.

   Annotations

   @Evolving()

6.  trait StreamingQuery extends AnyRef

   A handle to a query that is executing continuously in the background as new data arrives.

   A handle to a query that is executing continuously in the background as new data arrives. All these methods are thread-safe.

   Annotations

   @Evolving()

   Since

   2.0.0

7.  class StreamingQueryException extends Exception with SparkThrowable

   Exception that stopped a StreamingQuery.

   Exception that stopped a StreamingQuery. Use cause get the actual exception that caused the failure.

   Annotations

   @Evolving()

   Since

   2.0.0

8.  abstract class StreamingQueryListener extends Serializable

   Interface for listening to events related to StreamingQueries.

   Interface for listening to events related to StreamingQueries.

   Annotations

   @Evolving()

   Since

   2.0.0

   Note

   The methods are not thread-safe as they may be called from different threads.

9.  class StreamingQueryManager extends Logging

   A class to manage all the StreamingQuery active in a SparkSession.

   A class to manage all the StreamingQuery active in a SparkSession.

   Annotations

   @Evolving()

   Since

   2.0.0

10.  class StreamingQueryProgress extends Serializable

    Information about progress made in the execution of a StreamingQuery during a trigger.

    Information about progress made in the execution of a StreamingQuery during a trigger. Each event relates to processing done for a single trigger of the streaming query. Events are emitted even when no new data is available to be processed.

    Annotations

    @Evolving()

    Since

    2.1.0

11.  class StreamingQueryStatus extends Serializable

    Reports information about the instantaneous status of a streaming query.

    Reports information about the instantaneous status of a streaming query.

    Annotations

    @Evolving()

    Since

    2.1.0

12.  trait TestGroupState[S] extends GroupState[S]

    :: Experimental ::

    :: Experimental ::

    The extended version of GroupState interface with extra getters of state machine fields to improve testability of the GroupState implementations which inherit from the extended interface.

    Scala example of using TestGroupState:

    // Please refer to ScalaDoc of `GroupState` for the Scala definition of `mappingFunction()`
    
    import org.apache.spark.api.java.Optional
    import org.apache.spark.sql.streaming.GroupStateTimeout
    import org.apache.spark.sql.streaming.TestGroupState
    // other imports
    
    // test class setups
    
    test("MapGroupsWithState state transition function") {
      // Creates the prevState input for the state transition function
      // with desired configs. The `create()` API would guarantee that
      // the generated instance has the same behavior as the one built by
      // engine with the same configs.
      val prevState = TestGroupState.create[Int](
        optionalState = Optional.empty[Int],
        timeoutConf = NoTimeout,
        batchProcessingTimeMs = 1L,
        eventTimeWatermarkMs = Optional.of(1L),
        hasTimedOut = false)
    
      val key: String = ...
      val values: Iterator[Int] = ...
    
      // Asserts the prevState is in init state without updates.
      assert(!prevState.isUpdated)
    
      // Calls the state transition function with the test previous state
      // with desired configs.
      mappingFunction(key, values, prevState)
    
      // Asserts the test GroupState object has been updated but not removed
      // after calling the state transition function
      assert(prevState.isUpdated)
      assert(!prevState.isRemoved)
    }

    Java example of using TestGroupSate:

    // Please refer to ScalaDoc of `GroupState` for the Java definition of `mappingFunction()`
    
    import org.apache.spark.api.java.Optional;
    import org.apache.spark.sql.streaming.GroupStateTimeout;
    import org.apache.spark.sql.streaming.TestGroupState;
    // other imports
    
    // test class setups
    
    // test `MapGroupsWithState` state transition function `mappingFunction()`
    public void testMappingFunctionWithTestGroupState() {
      // Creates the prevState input for the state transition function
      // with desired configs. The `create()` API would guarantee that
      // the generated instance has the same behavior as the one built by
      // engine with the same configs.
      TestGroupState<Int> prevState = TestGroupState.create(
        Optional.empty(),
        GroupStateTimeout.NoTimeout(),
        1L,
        Optional.of(1L),
        false);
    
      String key = ...;
      Integer[] values = ...;
    
      // Asserts the prevState is in init state without updates.
      Assert.assertFalse(prevState.isUpdated());
    
      // Calls the state transition function with the test previous state
      // with desired configs.
      mappingFunction.call(key, Arrays.asList(values).iterator(), prevState);
    
      // Asserts the test GroupState object has been updated but not removed
      // after calling the state transition function
      Assert.assertTrue(prevState.isUpdated());
      Assert.assertFalse(prevState.isRemoved());
    }

    S

    User-defined type of the state to be stored for each group. Must be encodable into Spark SQL types (see Encoder for more details).

    Annotations

    @Experimental() @Evolving()

    Since

    3.2.0