config

Type Members

1. sealed trait DeletionConfig extends AnyRef

   

2. case class HBaseDeletionConfig(keysWithScan: RDD[(KeyWithCorrelation, Scan)], tableName: String, hbaseConfigModel: Option[HBaseConfigModel], keyValueMatchingStrategy: KeyValueMatchingStrategy, dryRun: Boolean) extends DeletionConfig with Product with Serializable

   

   Contains the configuration settings for an HBase Deletion Job

   Contains the configuration settings for an HBase Deletion Job

   keysWithScan

   list of distinct keys to delete (from config or input model), together with the HBase scan that will be used to match the HBase rowKeys to delete

   tableName

   namespace:table name of the HBase table to handle derived from the KeyValueModel

   hbaseConfigModel

   HBaseConfigModel that will be used to create an HBaseConnection inside the executors

   keyValueMatchingStrategy

   KeyValueMatchingStrategy defined in the BatchJobModel

3. case class HdfsDeletionConfig(keysToDeleteWithCorrelation: Seq[KeyWithCorrelation], rawModel: RawModel, rawMatchingStrategy: RawMatchingStrategy, rawMatchingCondition: Column, partitionPruningCondition: Column, stagingDirUri: String, backupDirUri: String, missingPathFailure: Boolean = false, dryRun: Boolean) extends DeletionConfig with Product with Serializable

   

   Contains the configuration settings for an Hdfs Deletion Job

   Contains the configuration settings for an Hdfs Deletion Job

   keysToDeleteWithCorrelation

   list of distinct keys to delete (from config or input model)

   rawModel

   RawModel to handle

   rawMatchingStrategy

   RawMatchingStrategy defined in the BatchJobModel

   rawMatchingCondition

   WHERE condition derived from the RawMatchingStrategy

   partitionPruningCondition

   WHERE condition derived from the PartitionPruningStrategy

   stagingDirUri

   staging directory path to use (from config or default = rawModel.uri + "/staging")

   backupDirUri

   backup directory parent path to use (from config or default = rawModel.uri.parent + "/staging")

   missingPathFailure

   if true a missing path inside rawModel results in deletion failure