ingest

Type Members

1. case class AuditLog(jobid: String, paths: String, domain: String, schema: String, success: Boolean, count: Long, countAccepted: Long, countRejected: Long, timestamp: Timestamp, duration: Long, message: String) extends Product with Serializable

   

2. final case class CatCountFreq(category: String, countDiscrete: Long, frequency: Double) extends Product with Serializable

   

3. class DsvIngestionJob extends IngestionJob

   

   Main class to ingest delimiter separated values file

4. trait DsvValidator extends AnyRef

   

5. trait IngestionJob extends SparkJob

   

6. class JsonIngestionJob extends IngestionJob

   

   Main class to complex json delimiter separated values file If your json contains only one level simple attribute aka.

   Main class to complex json delimiter separated values file If your json contains only one level simple attribute aka. kind of dsv but in json format please use SIMPLE_JSON instead. It's way faster

7. case class LoadConfig(domain: String = "", schema: String = "", paths: List[Path] = Nil) extends Product with Serializable

   
   domain

   domain name of the dataset

   schema

   schema name of the dataset

   paths

   Absolute path of the file to ingest (present in the ingesting area of the domain)

8. case class MetricRecord(domain: String, schema: String, attribute: String, min: Option[Double], max: Option[Double], mean: Option[Double], missingValues: Option[Long], standardDev: Option[Double], variance: Option[Double], sum: Option[Double], skewness: Option[Double], kurtosis: Option[Double], percentile25: Option[Double], median: Option[Double], percentile75: Option[Double], countDistinct: Option[Long], catCountFreq: Option[Seq[CatCountFreq]], missingValuesDiscrete: Option[Long], count: Long, cometTime: Long, cometStage: String) extends Product with Serializable

   

9. class PositionIngestionJob extends DsvIngestionJob

   

   Main class to ingest delimiter separated values file

10. case class RejectedRecord(jobid: String, timestamp: Timestamp, domain: String, schema: String, error: String, path: String) extends Product with Serializable

    

11. class SimpleJsonIngestionJob extends DsvIngestionJob

    

    Parse a simple one level json file.

    Parse a simple one level json file. Complex types such as arrays & maps are not supported. Use JsonIngestionJob instead. This class is for simple json only that makes it way faster.