metastore

Type Members

1. trait HiveDBConnector extends HadoopDBConnector

   

   Hive trait that implements the Hive-specific HadoopDBConnector functions

2. case class HiveDummyConnector(context: SparkFlowContext) extends HiveDBConnector with Product with Serializable

   

   A dummy Hive database connector that does not submit the DDLs but collects all that have been submitted in a List.

   A dummy Hive database connector that does not submit the DDLs but collects all that have been submitted in a List. This is useful for testing or using in flows where you wish to collect the DDLs and run them manually.

3. case class HiveSparkSQLConnector(context: SparkFlowContext, database: String, createDatabaseIfNotExists: Boolean = false) extends HiveDBConnector with Product with Serializable

   

   A Hive connector that uses the Hive Metastore configured in the SparkSession.

   A Hive connector that uses the Hive Metastore configured in the SparkSession. Hive support must be enabled on the SparkSession to use this connector. The connector uses the filesystem configured in the SparkFlowContext to discover partitions therefore table paths must exist on that filesystem.

   context

   The flow context object containing the SparkSession and FileSystem

   database

   Database to create tables in

   createDatabaseIfNotExists

   Whether to create the database if it does not exist (default false)