Package

com.ebiznext.comet.database

extractor

Permalink

package extractor

Visibility
  1. Public
  2. All

Type Members

  1. case class ExtractScriptGenConfig(domain: String = "", scriptTemplateFile: File = File("."), scriptOutputDir: File = File(".")) extends Product with Serializable

    Permalink

  2. case class TemplateParams(tableToExport: String, columnsToExport: List[String], fullExport: Boolean, deltaColumn: Option[String], dsvDelimiter: String, exportOutputFileBase: String, scriptOutputFile: File) extends Product with Serializable

    Permalink

    Params for the script's mustache template

    Params for the script's mustache template

    tableToExport

    table to export

    columnsToExport

    cols to export

    fullExport

    if table is going to be fully or delta exported

    deltaColumn

    if delta exported, which is the col holding the date of last update

    dsvDelimiter

    export result dsv delimiter

    exportOutputFileBase

    export dsv file base name (will be completed by current datetime when executed)

    scriptOutputFile

    where the script is produced

Value Members

  1. object ExtractScriptGenConfig extends Serializable

    Permalink

  2. object Main extends App with StrictLogging

    Permalink

    Generate extraction scripts based on a given domain The schemas should at least, specify :

    Generate extraction scripts based on a given domain The schemas should at least, specify :

    • a table name (schemas.name)
    • a file pattern (schemas.pattern) which is used as the export file base name
    • a write mode (schemas.metadata.write): APPEND or OVERWRITE
    • a delta column (schemas.merge.timestamp) if in APPEND mode : the column which is used to determine new rows for each exports
    • the columns to extract (schemas.attributes.name*)

    You also have to provide a Mustache (http://mustache.github.io/mustache.5.html) template file.

    Here you'll write your extraction export process (sqlplus for Oracle, pgsql for PostgreSQL as an example). In that template you can use the following parameters:

    table_name -> the table to export delimiter -> the resulting dsv file delimiter columns -> the columns to export columns is a Mustache map, it gives you access, for each column, to:

    • name -> the column name
    • trailing_col_char -> the separator to append to the column (, if there are more columns to come, "" otherwise) Here is an example how to use it in a template: SELECT {{#columns}} TO_CHAR({{name}}){{trailing_col_char}} {{/columns}} FROM {{table_name}}; export_file -> the export file name full_export -> if the export is a full or delta export (the logic is to be implemented in your script)

    Usage: comet [script-gen] [options]

    Command: script-gen --domain <value> The domain for which to generate extract scripts --templateFile <value> Script template file --scriptsOutputDir <value> Scripts output folder

  3. object ScriptGen extends StrictLogging

    Permalink

  4. object TemplateParams extends Serializable

    Permalink

Ungrouped