testing

Type Members

1. class BatchCountListener extends StreamingListener

   

2. class ColumnGenerator extends Serializable

   

3. trait DataFrameSuiteBase extends TestSuite with SharedSparkContext with DataFrameSuiteBaseLike

   

   :: Experimental :: Base class for testing Spark DataFrames.

4. trait DataFrameSuiteBaseLike extends SparkContextProvider with TestSuiteLike with Serializable

   

5. trait DatasetSuiteBase extends DataFrameSuiteBase with DatasetSuiteBaseLike

   

6. trait DatasetSuiteBaseLike extends DataFrameSuiteBaseLike

   

7. class HDFSCluster extends HDFSClusterLike

   

8. trait HDFSClusterLike extends AnyRef

   

9. class JavaDataFrameSuiteBase extends SharedJavaSparkContext with DataFrameSuiteBaseLike with JavaTestSuite

   

10. class JavaDatasetSuiteBase extends JavaDataFrameSuiteBase with DatasetSuiteBaseLike with Serializable

    

11. class JavaStreamingSuiteBase extends JavaSuiteBase with StreamingSuiteCommon

    

    This is the base trait for Spark Streaming testsuite.

    This is the base trait for Spark Streaming testsuite. This provides basic functionality to run user-defined set of input on user-defined stream operations, and verify the output. This implementation is designed to work with JUnit for java users.

    Note: this always uses the manual clock

12. class JavaSuiteBase extends SharedJavaSparkContext

    

13. trait JavaTestSuite extends TestSuiteLike

    

14. trait LocalSparkContext extends BeforeAndAfterEach with BeforeAndAfterAll

    

    Manages a local sc SparkContext variable, correctly stopping it after each test.

15. trait PerTestSparkContext extends LocalSparkContext with BeforeAndAfterEach

    

    Provides a local sc SparkContext variable, correctly stopping it after each test.

    Provides a local sc SparkContext variable, correctly stopping it after each test. The stopping logic is provided in LocalSparkContext

16. class PerfListener extends SparkListener

    

    This listener collects basic execution time information to be used in micro type performance tests.

    This listener collects basic execution time information to be used in micro type performance tests. Be careful imposing strict limits as there is a large amount of variability.

17. trait RDDComparisons extends RDDComparisonsLike with TestSuite

    

18. trait RDDComparisonsLike extends TestSuiteLike

    

19. class SharedJavaSparkContext extends SparkContextProvider

    

20. trait SharedMiniCluster extends BeforeAndAfterAll with HDFSClusterLike with YARNClusterLike

    

    Shares an HDFS MiniCluster based SparkContext between all tests in a suite and closes it at the end.

    Shares an HDFS MiniCluster based SparkContext between all tests in a suite and closes it at the end. This requires that the env variable SPARK_HOME is set. Further more if this is used, all Spark tests must run against the yarn mini cluster (see https://issues.apache.org/jira/browse/SPARK-10812 for details).

21. trait SharedSparkContext extends BeforeAndAfterAll with SparkContextProvider

    

    Shares a local SparkContext between all tests in a suite and closes it at the end.

22. trait SparkContextProvider extends AnyRef

    

23. trait StreamingActionBase extends StreamingSuiteBase

    

    Methods for testing Spark actions.

    Methods for testing Spark actions. Because actions don't return a DStream, you will need to verify the results of your test against mocks.

24. trait StreamingSuiteBase extends BeforeAndAfterAll with Logging with StreamingSuiteCommon with SharedSparkContext

    

    This is the base trait for Spark Streaming testsuites.

    This is the base trait for Spark Streaming testsuites. This provides basic functionality to run user-defined set of input on user-defined stream operations, and verify the output.

25. class TestInputStream[T] extends FriendlyInputDStream[T]

    

    This is a input stream just for the testsuites.

    This is a input stream just for the testsuites. This is equivalent to a checkpointable, replayable, reliable message queue like Kafka. It requires a sequence as input, and returns the i_th element at the i_th batch unde manual clock. Based on TestInputStream class from TestSuiteBase in the Apache Spark project.

26. class TestOutputStream[T] extends Serializable

    

    This is a output stream just for testing.

    This is a output stream just for testing.

    The buffer contains a sequence of RDD's, each containing a sequence of items

27. trait TestSuite extends TestSuiteLike

    

28. trait TestSuiteLike extends AnyRef

    

29. class YARNCluster extends YARNClusterLike

    

    Shares an HDFS MiniCluster based SparkContext between all tests in a suite and closes it at the end.

    Shares an HDFS MiniCluster based SparkContext between all tests in a suite and closes it at the end. This requires that the env variable SPARK_HOME is set. Further more if this is used, all Spark tests must run against the yarn mini cluster (see https://issues.apache.org/jira/browse/SPARK-10812 for details).

30. trait YARNClusterLike extends AnyRef