Execute unary DStream operation with a list of inputs and no expected output
Execute unary DStream operation with a list of inputs and no expected output
Sequence of input collections
Unary DStream operation to be applied to the input
Setup work to be called when creating a new SparkContext.
Setup work to be called when creating a new SparkContext. Default implementation currently sets a checkpoint directory.
This _should_ be called by the context provider automatically.
Test binary DStream operation with two lists of inputs, with number of batches to run same as the number of input values.
Test binary DStream operation with two lists of inputs, with number of batches to run same as the number of input values. The size of the two input lists should be the same.
Each input micro-batch is a list of values or as null to simulate empty batch.
First sequence of input collections
Second sequence of input collections
Binary DStream operation to be applied to the 2 inputs
Sequence of expected output collections
Compare output values with expected output values within the same output batch ordered or unOrdered. Comparing doubles may not work well in case of unordered.
Test unary DStream operation with a list of inputs, with number of batches to run same as the number of input values.
Test unary DStream operation with a list of inputs, with number of batches to run same as the number of input values.
Each input micro-batch is a list of values or as null to simulate empty batch.
Sequence of input collections
Binary DStream operation to be applied to the 2 inputs
Sequence of expected output collections
Compare output values with expected output values within the same output batch ordered or unordered. Comparing doubles may not work well in case of unordered.
Test binary DStream and RDD operation with two lists of inputs, with number of batches to run same as the number of input values corresponding to the DStream.
Test binary DStream and RDD operation with two lists of inputs, with number of batches to run same as the number of input values corresponding to the DStream.
Each input micro-batch is a list of values or as null to simulate empty batch.
Sequence of input collections corresponding to the DStream
Sequence of input values corresponding to the RDD
Binary DStream and RDD operation to be applied to the 2 inputs
Sequence of expected output collections
Compare output values with expected output values within the same output batch ordered or unOrdered. Comparing doubles may not work well in case of unordered.
Verify whether the output values after running a DStream operation is same as the expected output values, by comparing the output collections either as lists (order matters) or sets (order does not matter)
Verify whether the output values after running a DStream operation is same as the expected output values, by comparing the output collections either as lists (order matters) or sets (order does not matter)
Compare output values with expected output values within the same output batch ordered or unordered. Comparing doubles may not work well in case of unordered.
Methods for testing Spark actions. Because actions don't return a DStream, you will need to verify the results of your test against mocks.