ingestion

Type Members

1. case class IncorrectUserPKException(userPKs: Seq[String], dbPKs: Seq[String]) extends Exception with Product with Serializable

   

2. case class PostgresConnectionDetails(server: String, port: Int, databaseName: String, user: String, password: String, sslFactory: Option[String]) extends RDBMConnectionDetails with Product with Serializable

   

3. class PostgresExtractor extends RDBMExtractor

   

   Created by Vicky Avison on 27/04/18.

4. case class PostgresMetadata(schemaName: String, tableName: String, pkCols: String) extends Product with Serializable

   

5. trait RDBMConnectionDetails extends AnyRef

   

   Created by Vicky Avison on 30/04/18.

6. case class RDBMExtractionTableConfig(tableName: String, pkCols: Option[Seq[String]] = None, lastUpdatedColumn: Option[String] = None, maxRowsPerPartition: Option[Int] = None) extends Product with Serializable

   

   Table configuration used for RDBM extraction

   Table configuration used for RDBM extraction

   tableName

   The name of the table

   pkCols

   Optionally, the primary key columns for this table (don't need if the implementation of RDBMExtractor is capable of getting this information itself)

   lastUpdatedColumn

   Optionally, the last updated column for this table (don't need if the implementation of RDBMExtractor is capable of getting this information itself)

   maxRowsPerPartition

   Optionally, the maximum number of rows to be read per Dataset partition for this table This number will be used to generate predicates to be passed to org.apache.spark.sql.SparkSession.read.jdbc If this is not set, the DataFrame will only have one partition. This could result in memory issues when extracting large tables. Be careful not to create too many partitions in parallel on a large cluster; otherwise Spark might crash your external database systems. You can also control the maximum number of jdbc connections to open by limiting the number of executors for your application.

7. trait RDBMExtractor extends AnyRef

   

   Waimak RDBM connection mechanism

8. case class SQLServerConnectionDetails(server: String, port: Int, databaseName: String, user: String, password: String) extends RDBMConnectionDetails with Product with Serializable

   

   Connection details for a SQL Server instance

   Connection details for a SQL Server instance

   server

   the SQL Server instance

   port

   the port

   databaseName

   the database name

   user

   the user name

   password

   the password

9. abstract class SQLServerExtractor extends RDBMExtractor

   

   A mechanism for generating Waimak actions to extract data from a SQL Server instance

10. class SQLServerTemporalExtractor extends SQLServerExtractor with Logging

    

    A mechanism for generating Waimak actions to extract data from a SQL Server instance containing temporal tables Tables can be a mixture of temporal and non-temporal - both will be handled appropriately

11. case class SQLServerTemporalTableMetadata(schemaName: String, tableName: String, historyTableSchema: Option[String] = None, historyTableName: Option[String] = None, startColName: Option[String] = None, endColName: Option[String] = None, primaryKeys: String) extends Product with Serializable

    

12. class SQLServerViewExtractor extends SQLServerExtractor

    

    Created by Vicky Avison on 12/04/18.

13. case class TableExtractionMetadata(schemaName: String, tableName: String, primaryKeys: Seq[String], lastUpdatedColumn: Option[String] = None) extends Product with Serializable