Type Members

1.  final class DataStreamReader extends Logging

   Interface used to load a streaming Dataset from external storage systems (e.g.

   Interface used to load a streaming Dataset from external storage systems (e.g. file systems, key-value stores, etc). Use SparkSession.readStream to access this.

   Annotations

   @Evolving()

   Since

   2.0.0

2.  final class DataStreamWriter[T] extends AnyRef

   Interface used to write a streaming Dataset to external storage systems (e.g.

   Interface used to write a streaming Dataset to external storage systems (e.g. file systems, key-value stores, etc). Use Dataset.writeStream to access this.

   Annotations

   @Evolving()

   Since

   2.0.0

3.  trait StreamingQuery extends api.StreamingQuery[Dataset]

   <invalid inheritdoc annotation>

4.  class StreamingQueryManager extends Logging

   A class to manage all the StreamingQuery active in a SparkSession.

   A class to manage all the StreamingQuery active in a SparkSession.

   Annotations

   @Evolving()

   Since

   2.0.0

5.  trait TestGroupState[S] extends GroupState[S]

   :: Experimental ::

   :: Experimental ::

   The extended version of GroupState interface with extra getters of state machine fields to improve testability of the GroupState implementations which inherit from the extended interface.

   Scala example of using TestGroupState:

   // Please refer to ScalaDoc of `GroupState` for the Scala definition of `mappingFunction()`
   
   import org.apache.spark.api.java.Optional
   import org.apache.spark.sql.streaming.GroupStateTimeout
   import org.apache.spark.sql.streaming.TestGroupState
   // other imports
   
   // test class setups
   
   test("MapGroupsWithState state transition function") {
     // Creates the prevState input for the state transition function
     // with desired configs. The `create()` API would guarantee that
     // the generated instance has the same behavior as the one built by
     // engine with the same configs.
     val prevState = TestGroupState.create[Int](
       optionalState = Optional.empty[Int],
       timeoutConf = NoTimeout,
       batchProcessingTimeMs = 1L,
       eventTimeWatermarkMs = Optional.of(1L),
       hasTimedOut = false)
   
     val key: String = ...
     val values: Iterator[Int] = ...
   
     // Asserts the prevState is in init state without updates.
     assert(!prevState.isUpdated)
   
     // Calls the state transition function with the test previous state
     // with desired configs.
     mappingFunction(key, values, prevState)
   
     // Asserts the test GroupState object has been updated but not removed
     // after calling the state transition function
     assert(prevState.isUpdated)
     assert(!prevState.isRemoved)
   }

   Java example of using TestGroupSate:

   // Please refer to ScalaDoc of `GroupState` for the Java definition of `mappingFunction()`
   
   import org.apache.spark.api.java.Optional;
   import org.apache.spark.sql.streaming.GroupStateTimeout;
   import org.apache.spark.sql.streaming.TestGroupState;
   // other imports
   
   // test class setups
   
   // test `MapGroupsWithState` state transition function `mappingFunction()`
   public void testMappingFunctionWithTestGroupState() {
     // Creates the prevState input for the state transition function
     // with desired configs. The `create()` API would guarantee that
     // the generated instance has the same behavior as the one built by
     // engine with the same configs.
     TestGroupState<Int> prevState = TestGroupState.create(
       Optional.empty(),
       GroupStateTimeout.NoTimeout(),
       1L,
       Optional.of(1L),
       false);
   
     String key = ...;
     Integer[] values = ...;
   
     // Asserts the prevState is in init state without updates.
     Assertions.assertFalse(prevState.isUpdated());
   
     // Calls the state transition function with the test previous state
     // with desired configs.
     mappingFunction.call(key, Arrays.asList(values).iterator(), prevState);
   
     // Asserts the test GroupState object has been updated but not removed
     // after calling the state transition function
     Assertions.assertTrue(prevState.isUpdated());
     Assertions.assertFalse(prevState.isRemoved());
   }

   S

   User-defined type of the state to be stored for each group. Must be encodable into Spark SQL types (see Encoder for more details).

   Annotations

   @Experimental() @Evolving()

   Since

   3.2.0