datasetComparison

Type Members

1. final case class BadProvidedSchema(diffSchema: String, cause: Throwable = None.orNull) extends DatasetComparisonException with Product with Serializable

   

2. case class ComparisonResult(refRowCount: Long, newRowCount: Long, refDuplicateCount: Long, newDuplicateCount: Long, passedCount: Long, usedSchemaSelector: List[Column], resultDF: Option[DataFrame], diffCount: Long = 0, passedOptions: String = "", additionalInfo: Map[String, String] = Map.empty) extends Product with Serializable

   
   refRowCount

   Row Count of the reference data

   newRowCount

   Row Count of the new data

   usedSchemaSelector

   Selector used to align schemas created from reference data schema

   resultDF

   Result dataframe, if None, there were no differences between reference and new data

   diffCount

   Number of differences if there are any

   passedOptions

   Raw options passed to the job by user. Might be empty if comparison used as a library

3. class DatasetComparator extends AnyRef

   

   Class that is the brain of the DatasetComparison module.

   Class that is the brain of the DatasetComparison module. This class should be used in case of using DatasetComparison as a library. In case of running the DatasetComparison as SparkJob, please use the DatasetComparisonJob.

4. class DatasetComparisonException extends Exception

   

5. final case class DatasetsDifferException(refPath: String, stdPath: String, outPath: String, expectedCount: Long, actualCount: Long, cause: Throwable = None.orNull) extends DatasetComparisonException with Product with Serializable

   

6. final case class DuplicateRowsInDF(countRef: Long, countNew: Long) extends DatasetComparisonException with Product with Serializable

   

7. final case class MissingArgumentException(message: String, cause: Throwable = None.orNull) extends Exception with Product with Serializable

   

8. final case class SchemasDifferException(diffSchema: String, cause: Throwable = None.orNull) extends DatasetComparisonException with Product with Serializable