Settings

Type Members

1. final case class Airflow(endpoint: String, ingest: String) extends Product with Serializable

   
   endpoint

   : Airflow REST API endpoint, aka. http://127.0.0.1:8080/api/experimental

2. final case class Area(pending: String, unresolved: String, archive: String, ingesting: String, accepted: String, rejected: String, business: String) extends Product with Serializable

   

   datasets in the data pipeline go through several stages and are stored on disk at each of these stages.

   datasets in the data pipeline go through several stages and are stored on disk at each of these stages. This setting allow to customize the folder names of each of these stages.

   pending

   : Name of the pending area

   unresolved

   : Named of the unresolved area

   archive

   : Name of the archive area

   ingesting

   : Name of the ingesting area

   accepted

   : Name of the accepted area

   rejected

   : Name of the rejected area

   business

   : Name of the business area

3. final case class Atlas(uri: String, user: String, password: String, owner: String) extends Product with Serializable

   

4. final case class Audit(path: String, index: IndexSinkSettings, maxErrors: Int) extends Product with Serializable

   

5. final case class Comet(jobId: String, datasets: String, metadata: String, metrics: Metrics, audit: Audit, archive: Boolean, lock: Lock, writeFormat: String, launcher: String, chewerPrefix: String, analyze: Boolean, hive: Boolean, grouped: Boolean, mergeForceDistinct: Boolean, area: Area, airflow: Airflow, elasticsearch: Elasticsearch, hadoop: Map[String, String], jdbc: Map[String, Jdbc], jdbcEngines: Map[String, JdbcEngine], atlas: Atlas, privacy: Privacy, fileSystem: Option[String], disablePrivacy: Boolean) extends Serializable with Product

   
   datasets

   : Absolute path, datasets root folder beneath which each area is defined.

   metadata

   : Absolute path, location where all types / domains and auto jobs are defined

   metrics

   : Absolute path, location where all computed metrics are stored

   audit

   : Absolute path, location where all log are stored

   archive

   : Should we backup the ingested datasets ? true by default

   writeFormat

   : Choose between parquet, orc ... Default is parquet

   launcher

   : Cron Job Manager : simple (useful for testing) or airflow ? simple by default

   analyze

   : Should we create basics Hive statistics on the generated dataset ? true by default

   hive

   : Should we create a Hive Table ? true by default

   area

   : see Area above

   airflow

   : Airflow end point. Should be defined even if simple launccher is used instead of airflow.

   disablePrivacy

   : Disable privacy

6. final case class Elasticsearch(active: Boolean, options: Map[String, String]) extends Product with Serializable

   

7. sealed abstract class IndexSinkSettings extends AnyRef

   

   Configuration for IndexSink

   Configuration for IndexSink

   This is used to define an auxiliary output for Audit or Metrics data, in addition to the Parquets The default Index Sink is None, but additional types exists (such as BigQuery or Jdbc)

   Annotations

   @JsonTypeInfo()

8. final case class Jdbc(uri: String, user: String = "", password: String = "", engineOverride: Option[String] = None) extends Product with Serializable

   

   Describes a connection to a JDBC-accessible database engine

   Describes a connection to a JDBC-accessible database engine

   uri

   the URI of the database engine. It must start with "jdbc:"

   user

   the username under which to connect to the database engine

   password

   the password to use in order to connect to the database engine

   engineOverride

   the index into the Comet.jdbcEngines map of the underlying database engine, in case one cannot use the engine name from the uri

   Note

   the use case for engineOverride is when you need to have an alternate schema definition (e.g. non-standard table names) alongside with the regular schema definition, on the same underlying engine.

9. final case class JdbcEngine(driver: String, tables: Map[String, TableDdl]) extends Product with Serializable

   

   Describes how to use a specific type of JDBC-accessible database engine

   Describes how to use a specific type of JDBC-accessible database engine

   driver

   the qualified class name of the JDBC Driver to use for the specific engine

   tables

   for each of the Standard Table Names used by Comet, the specific SQL DDL statements as expected in the engine's own dialect.

10. final case class Lock(path: String, metricsTimeout: Long, ingestionTimeout: Long, pollTime: FiniteDuration = ..., refreshTime: FiniteDuration = ...) extends Product with Serializable

    

11. final case class Metrics(path: String, discreteMaxCardinality: Int, active: Boolean, index: IndexSinkSettings) extends Product with Serializable

    
    discreteMaxCardinality

    : Max number of unique values allowed in cardinality compute

12. final case class Privacy(options: Map[String, String]) extends Product with Serializable