Converts $"col name" into a Column.
Converts $"col name" into a Column.
2.0.0
Create a dataframe using a json
Create a dataframe using a json
Example: val df = dataframe("{a:1, b:true}", "{a:2, b:false}") | a | b | +---+-------+ | 1 | true | | 2 | false |
json's agruments, each argument represent one line of our dataframe
a dataframe
Create a dataframe using a json file
Create a dataframe using a json file
Example:
The path where the json is stored
The resulting DataFrame
Creates a Dataset from a local Seq.
Creates a Dataset from a local Seq.
1.6.0
1.6.1
1.6.0
1.6.1
2.0.0
2.0.0
2.0.0
2.0.0
2.0.0
2.0.0
2.0.0
1.6.1
1.6.0
1.6.1
1.6.1
1.6.0
1.6.1
1.6.1
1.6.0
1.6.1
1.6.1
1.6.0
1.6.1
1.6.1
1.6.0
1.6.1
1.6.1
1.6.0
1.6.1
1.6.1
1.6.0
1.6.1
1.6.1
1.6.0
1.6.1
Creates a Dataset from an RDD.
Creates a Dataset from an RDD.
1.6.0
An implicit conversion that turns a Scala Symbol
into a Column.
An implicit conversion that turns a Scala Symbol
into a Column.
1.3.0
You can wrap your Spark-Test's blocks using a different configuration to customize the behaviour of the functions.
You can wrap your Spark-Test's blocks using a different configuration to customize the behaviour of the functions.
Example: withConfiguration(failOnMissingExpectedCol = false, failOnMissingOriginalCol = false)({ df1.assertEquals(df2) }) result: ignore if there are any extra columns in df1 or df2
failOnMissingOriginalCol: if true, Return an exception if a column appear in the original DataFrame but not in the expected one failOnChangedDataTypeExpectedCol: if true, Return an exception if both columns don't have the same DataType failOnMissingExpectedCol: if true, Return an exception if a column appear in the expected DataFrame but not in the original one maxRowError: if > 0, print maxRowError's rows during the error handling else print every row's errors