SparkTestDfOps

Value Members

1. final def !=(arg0: Any): Boolean

   

   Definition Classes

   AnyRef → Any

2. final def ##(): Int

   

   Definition Classes

   AnyRef → Any

3. final def ==(arg0: Any): Boolean

   

   Definition Classes

   AnyRef → Any

4. final def asInstanceOf[T0]: T0

   

   Definition Classes

   Any

5. def assertApproxEquals(otherDf: DataFrame, approx: Double, ignoreNullableFlag: Boolean = false): Unit

   

   Approximate comparison between two DataFrames.

   Approximate comparison between two DataFrames. If the DataFrames match the instances of Double, Float, or Timestamp, 'approx' will be used to compare an approximate equality of the two DFs.

   otherDf

   DataFrame to compare to

   approx

   double for the approximate comparison. If the absolute value of the difference between two values is less than approx, then the two values are considered equal.

   ignoreNullableFlag

   flag to decide if nullable is ignored or not

6. def assertColumnEquality(rightLabel: String, leftLabel: String): Unit

   

7. def assertEquals[T](seq: Seq[T])(implicit arg0: Encoder[T]): Unit

   

   Verify if if the Dataframe and the Sequence are equals, can be custom using the Spark-Test configuration

   Verify if if the Dataframe and the Sequence are equals, can be custom using the Spark-Test configuration

   seq

   Sequence to compare to

   returns

   An exception if the Dataframe and the Sequence are not equals

8. def assertEquals(otherDf: DataFrame): Unit

   

   Verify if two Dataframes are equals, can be custom using the Spark-Test configuration

   Verify if two Dataframes are equals, can be custom using the Spark-Test configuration

   otherDf

   DataFrame to compare to

   returns

   An exception if both Dataframes are not equals

9. def clone(): AnyRef

   

   Attributes

   protected[java.lang]

   Definition Classes

   AnyRef

   Annotations

   @throws( ... )

10. def compareColumn(rightLabel: String, leftLabel: String): Boolean

    

11. final def eq(arg0: AnyRef): Boolean

    

    Definition Classes

    AnyRef

12. def equals(arg0: Any): Boolean

    

    Definition Classes

    AnyRef → Any

13. def finalize(): Unit

    

    Attributes

    protected[java.lang]

    Definition Classes

    AnyRef

    Annotations

    @throws( classOf[java.lang.Throwable] )

14. final def getClass(): Class[_]

    

    Definition Classes

    AnyRef → Any

15. def getRowsDifferences(otherDf: DataFrame): Seq[Seq[ObjectModification]]

    

    Gives each modifications for each row between this Dataframe and otherDf

    Gives each modifications for each row between this Dataframe and otherDf

    otherDf

    DataFrame to compare to

    returns

    Return the Sequence of modifications for each rows

16. def hashCode(): Int

    

    Definition Classes

    AnyRef → Any

17. final def isInstanceOf[T0]: Boolean

    

    Definition Classes

    Any

18. final def ne(arg0: AnyRef): Boolean

    

    Definition Classes

    AnyRef

19. final def notify(): Unit

    

    Definition Classes

    AnyRef

20. final def notifyAll(): Unit

    

    Definition Classes

    AnyRef

21. def reduceColumn(otherDf: DataFrame): Try[(DataFrame, DataFrame)]

    

    Depending on the configuration, try to modify the schema of the two DataFrames so that they can be compared.

    Depending on the configuration, try to modify the schema of the two DataFrames so that they can be compared. Throw a SchemaError otherwise.

    otherDf

    DataFrame to compare to

    returns

    Both reduced Dataframes

22. def reportErrorComparison(otherDf: DataFrame, modifications: Seq[Seq[ObjectModification]]): String

    

    Return a stringified version of the first modifications between this Dataframe and otherDf The number of showed modifications can be modified using the Spark-Test configuration's value: maxRowError

    Return a stringified version of the first modifications between this Dataframe and otherDf The number of showed modifications can be modified using the Spark-Test configuration's value: maxRowError

    otherDf

    DataFrame to compare to

    modifications

    Sequence of modifications for each rows

    returns

    Stringified version of modifications

23. def reportErrorComparisonLight(otherDf: DataFrame, modifications: Seq[Seq[ObjectModification]]): String

    

    Return a stringified version of the first modifications between this Dataframe and otherDf The number of showed modifications can be modified using the Spark-Test configuration's value: maxRowError

    Return a stringified version of the first modifications between this Dataframe and otherDf The number of showed modifications can be modified using the Spark-Test configuration's value: maxRowError

    otherDf

    DataFrame to compare to

    modifications

    Sequence of modifications for each rows

    returns

    Stringified version of modifications

24. def showCaseClass(className: String): Unit

    

    Display the schema of the DataFrame as a case class.

    Display the schema of the DataFrame as a case class. Example : val df = Seq(1, 2, 3).toDF("id") df.showCaseClass("Id")

    result : case class Id ( id:Int )

    className

    name of the case class

25. final def synchronized[T0](arg0: ⇒ T0): T0

    

    Definition Classes

    AnyRef

26. def toString(): String

    

    Definition Classes

    AnyRef → Any

27. final def wait(): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

28. final def wait(arg0: Long, arg1: Int): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

29. final def wait(arg0: Long): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )