SparkTest
Related Docs:
object SparkTest
| package sparktest
Converts $"col name" into a Column.
Converts $"col name" into a Column.
2.0.0
Create a dataframe using a json
Create a dataframe using a json
Example: val df = dataframe("{a:1, b:true}", "{a:2, b:false}") | a | b | +---+-------+ | 1 | true | | 2 | false |
json's agruments, each argument represent one line of our dataframe
a dataframe
Creates a Dataset from a local Seq.
Creates a Dataset from a local Seq.
1.6.0
1.6.1
1.6.0
1.6.1
2.0.0
2.0.0
2.0.0
2.0.0
2.0.0
2.0.0
2.0.0
1.6.1
1.6.0
1.6.1
1.6.1
1.6.0
1.6.1
1.6.1
1.6.0
1.6.1
1.6.1
1.6.0
1.6.1
1.6.1
1.6.0
1.6.1
1.6.1
1.6.0
1.6.1
1.6.1
1.6.0
1.6.1
1.6.1
1.6.0
1.6.1
Creates a Dataset from an RDD.
Creates a Dataset from an RDD.
1.6.0
An implicit conversion that turns a Scala Symbol into a Column.
An implicit conversion that turns a Scala Symbol into a Column.
1.3.0
You can wrap your Spark-Test's function using a configuration to customize the behaviour of the function Each functions who can be modified using the configuration have the tag @configuration
You can wrap your Spark-Test's function using a configuration to customize the behaviour of the function Each functions who can be modified using the configuration have the tag @configuration
Example: withConfiguration(failOnMissingExpectedCol = false, failOnMissingOriginalCol = false)({ df1.assertEquals(df2) }) result: ignore if there are any extra columns in df1 or df2
failOnMissingOriginalCol: if true, Return an exception if a column appear in the original DataFrame but not in the expected one failOnChangedDataTypeExpectedCol: if true, Return an exception if both columns don't have the same DataType failOnMissingExpectedCol: if true, Return an exception if a column appear in the expected DataFrame but not in the original one maxRowError: if > 0, print maxRowError's rows during the error handling else print every row's errors