Converts $"col name" into a Column.
Converts $"col name" into a Column.
2.0.0
Create a dataframe using a json
Create a dataframe using a json
Example: val df = dataframe("{a:1, b:true}", "{a:2, b:false}") | a | b | +---+-------+ | 1 | true | | 2 | false |
json's agruments, each argument represent one line of our dataframe
a dataframe
Creates a Dataset from a local Seq.
Creates a Dataset from a local Seq.
1.6.0
1.6.1
1.6.0
1.6.1
2.0.0
2.0.0
2.0.0
2.0.0
2.0.0
2.0.0
2.0.0
1.6.1
1.6.0
1.6.1
1.6.1
1.6.0
1.6.1
1.6.1
1.6.0
1.6.1
1.6.1
1.6.0
1.6.1
1.6.1
1.6.0
1.6.1
1.6.1
1.6.0
1.6.1
1.6.1
1.6.0
1.6.1
1.6.1
1.6.0
1.6.1
Creates a Dataset from an RDD.
Creates a Dataset from an RDD.
1.6.0
An implicit conversion that turns a Scala Symbol
into a Column.
An implicit conversion that turns a Scala Symbol
into a Column.
1.3.0
You can wrapped your Spark-Test's function using a configuration to customise the behaviour of the function Each functions who can be modified using the configuration have the tag @configuration
You can wrapped your Spark-Test's function using a configuration to customise the behaviour of the function Each functions who can be modified using the configuration have the tag @configuration
Example: withConfiguration(failOnMissingExpectedCol = false, failOnMissingOriginalCol = false)({ df1.assertEquals(df2) }) result: ignore if there are any extra columns in df1 or df2
failOnMissingOriginalCol: if true, Return an exception if a colonne appear in the original Dtaframe but not in the expected one failOnChangedDataTypeExpectedCol: if true, Return an exception if both colonne don't have the same DataType failOnMissingExpectedCol: if true, Return an exception if a colonne appear in the expected Dtaframe but not in the original one maxRowError: if > 0, print maxRowError's rows during the error handling else print every row's errors