Package

com.ebiznext.comet

config

Permalink

package config

Visibility

Public
All

Type Members

final case class Settings(comet: Comet, sparkConfig: Config) extends Product with Serializable

This class holds the current Comet settings and an assembly of reference instances for core, shared services
This class holds the current Comet settings and an assembly of reference instances for core, shared services
SMELL: this may be the start of a Dependency Injection root (but at 2-3 objects, is DI justified? probably not quite yet) — cchepelov
class SparkEnv extends StrictLogging

Any Spark Job will inherit from this class.
Any Spark Job will inherit from this class. All properties defined in application conf file and prefixed by the "spark" key will be loaded into the Spark Job
sealed abstract class StorageArea extends AnyRef

Annotations
@JsonSerialize() @JsonDeserialize()
final class StorageAreaDeserializer extends JsonDeserializer[StorageArea]
final class StorageAreaSerializer extends JsonSerializer[StorageArea]
trait UdfRegistration extends AnyRef

Value Members

object DatasetArea

Utilities methods to reference datasets paths Datasets paths are constructed as follows :
Utilities methods to reference datasets paths Datasets paths are constructed as follows :
- root path defined by the COMET_DATASETS env var or datasets applciation property
- followed by the area name
- followed by the the domain name
object Settings extends StrictLogging with Serializable
object StorageArea

After going through the data pipeline a dataset may be referenced through a Hive table in a Hive Database.
After going through the data pipeline a dataset may be referenced through a Hive table in a Hive Database. For each input domain, 3 Hive databases may be created :
- The rejected database : contains tables referencing rejected records for each schema in the domain
- The accepted database : contains tables referencing
- The business database : contains tables where autjob tables are created by default
- The ciustom database : contains tables where autojob tables are created when a specific area is defined

Ungrouped