Class

org.apache.spark.sql.catalyst.catalog

ExternalCatalog

Related Doc: package catalog

Permalink

abstract class ExternalCatalog extends AnyRef

Interface for the system catalog (of functions, partitions, tables, and databases).

This is only used for non-temporary items, and implementations must be thread-safe as they can be accessed in multiple threads. This is an external catalog because it is expected to interact with external systems.

Implementations should throw NoSuchDatabaseException when databases don't exist.

Linear Supertypes
AnyRef, Any
Known Subclasses
InMemoryCatalog
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. ExternalCatalog
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new ExternalCatalog()

    Permalink

Abstract Value Members

  1. abstract def alterDatabase(dbDefinition: CatalogDatabase): Unit

    Permalink

    Alter a database whose name matches the one specified in dbDefinition, assuming the database exists.

    Alter a database whose name matches the one specified in dbDefinition, assuming the database exists.

    Note: If the underlying implementation does not support altering a certain field, this becomes a no-op.

  2. abstract def alterPartitions(db: String, table: String, parts: Seq[CatalogTablePartition]): Unit

    Permalink

    Alter one or many table partitions whose specs that match those specified in parts, assuming the partitions exist.

    Alter one or many table partitions whose specs that match those specified in parts, assuming the partitions exist.

    Note: If the underlying implementation does not support altering a certain field, this becomes a no-op.

  3. abstract def alterTable(tableDefinition: CatalogTable): Unit

    Permalink

    Alter a table whose database and name match the ones specified in tableDefinition, assuming the table exists.

    Alter a table whose database and name match the ones specified in tableDefinition, assuming the table exists. Note that, even though we can specify database in tableDefinition, it's used to identify the table, not to alter the table's database, which is not allowed.

    Note: If the underlying implementation does not support altering a certain field, this becomes a no-op.

  4. abstract def alterTableSchema(db: String, table: String, schema: StructType): Unit

    Permalink

    Alter the schema of a table identified by the provided database and table name.

    Alter the schema of a table identified by the provided database and table name. The new schema should still contain the existing bucket columns and partition columns used by the table. This method will also update any Spark SQL-related parameters stored as Hive table properties (such as the schema itself).

    db

    Database that table to alter schema for exists in

    table

    Name of table to alter schema for

    schema

    Updated schema to be used for the table (must contain existing partition and bucket columns)

  5. abstract def createDatabase(dbDefinition: CatalogDatabase, ignoreIfExists: Boolean): Unit

    Permalink

  6. abstract def createFunction(db: String, funcDefinition: CatalogFunction): Unit

    Permalink

  7. abstract def createPartitions(db: String, table: String, parts: Seq[CatalogTablePartition], ignoreIfExists: Boolean): Unit

    Permalink

  8. abstract def createTable(tableDefinition: CatalogTable, ignoreIfExists: Boolean): Unit

    Permalink

  9. abstract def databaseExists(db: String): Boolean

    Permalink

  10. abstract def dropDatabase(db: String, ignoreIfNotExists: Boolean, cascade: Boolean): Unit

    Permalink

  11. abstract def dropFunction(db: String, funcName: String): Unit

    Permalink

  12. abstract def dropPartitions(db: String, table: String, parts: Seq[TablePartitionSpec], ignoreIfNotExists: Boolean, purge: Boolean, retainData: Boolean): Unit

    Permalink

  13. abstract def dropTable(db: String, table: String, ignoreIfNotExists: Boolean, purge: Boolean): Unit

    Permalink

  14. abstract def functionExists(db: String, funcName: String): Boolean

    Permalink

  15. abstract def getDatabase(db: String): CatalogDatabase

    Permalink

  16. abstract def getFunction(db: String, funcName: String): CatalogFunction

    Permalink

  17. abstract def getPartition(db: String, table: String, spec: TablePartitionSpec): CatalogTablePartition

    Permalink

  18. abstract def getPartitionOption(db: String, table: String, spec: TablePartitionSpec): Option[CatalogTablePartition]

    Permalink

    Returns the specified partition or None if it does not exist.

  19. abstract def getTable(db: String, table: String): CatalogTable

    Permalink

  20. abstract def getTableOption(db: String, table: String): Option[CatalogTable]

    Permalink

  21. abstract def listDatabases(pattern: String): Seq[String]

    Permalink

  22. abstract def listDatabases(): Seq[String]

    Permalink

  23. abstract def listFunctions(db: String, pattern: String): Seq[String]

    Permalink

  24. abstract def listPartitionNames(db: String, table: String, partialSpec: Option[TablePartitionSpec] = None): Seq[String]

    Permalink

    List the names of all partitions that belong to the specified table, assuming it exists.

    List the names of all partitions that belong to the specified table, assuming it exists.

    For a table with partition columns p1, p2, p3, each partition name is formatted as p1=v1/p2=v2/p3=v3. Each partition column name and value is an escaped path name, and can be decoded with the ExternalCatalogUtils.unescapePathName method.

    The returned sequence is sorted as strings.

    A partial partition spec may optionally be provided to filter the partitions returned, as described in the listPartitions method.

    db

    database name

    table

    table name

    partialSpec

    partition spec

  25. abstract def listPartitions(db: String, table: String, partialSpec: Option[TablePartitionSpec] = None): Seq[CatalogTablePartition]

    Permalink

    List the metadata of all partitions that belong to the specified table, assuming it exists.

    List the metadata of all partitions that belong to the specified table, assuming it exists.

    A partial partition spec may optionally be provided to filter the partitions returned. For instance, if there exist partitions (a='1', b='2'), (a='1', b='3') and (a='2', b='4'), then a partial spec of (a='1') will return the first two only.

    db

    database name

    table

    table name

    partialSpec

    partition spec

  26. abstract def listPartitionsByFilter(db: String, table: String, predicates: Seq[Expression]): Seq[CatalogTablePartition]

    Permalink

    List the metadata of partitions that belong to the specified table, assuming it exists, that satisfy the given partition-pruning predicate expressions.

    List the metadata of partitions that belong to the specified table, assuming it exists, that satisfy the given partition-pruning predicate expressions.

    db

    database name

    table

    table name

    predicates

    partition-pruning predicates

  27. abstract def listTables(db: String, pattern: String): Seq[String]

    Permalink

  28. abstract def listTables(db: String): Seq[String]

    Permalink

  29. abstract def loadDynamicPartitions(db: String, table: String, loadPath: String, partition: TablePartitionSpec, replace: Boolean, numDP: Int, holdDDLTime: Boolean): Unit

    Permalink

  30. abstract def loadPartition(db: String, table: String, loadPath: String, partition: TablePartitionSpec, isOverwrite: Boolean, holdDDLTime: Boolean, inheritTableSpecs: Boolean): Unit

    Permalink

  31. abstract def loadTable(db: String, table: String, loadPath: String, isOverwrite: Boolean, holdDDLTime: Boolean): Unit

    Permalink

  32. abstract def renameFunction(db: String, oldName: String, newName: String): Unit

    Permalink

  33. abstract def renamePartitions(db: String, table: String, specs: Seq[TablePartitionSpec], newSpecs: Seq[TablePartitionSpec]): Unit

    Permalink

    Override the specs of one or many existing table partitions, assuming they exist.

    Override the specs of one or many existing table partitions, assuming they exist. This assumes index i of specs corresponds to index i of newSpecs.

  34. abstract def renameTable(db: String, oldName: String, newName: String): Unit

    Permalink

  35. abstract def setCurrentDatabase(db: String): Unit

    Permalink

  36. abstract def tableExists(db: String, table: String): Boolean

    Permalink

Concrete Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  4. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any

  5. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  6. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  7. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  8. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  9. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any

  10. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  11. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any

  12. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  13. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef

  14. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef

  15. def requireDbExists(db: String): Unit

    Permalink
    Attributes
    protected

  16. def requireFunctionExists(db: String, funcName: String): Unit

    Permalink
    Attributes
    protected

  17. def requireFunctionNotExists(db: String, funcName: String): Unit

    Permalink
    Attributes
    protected

  18. def requireTableExists(db: String, table: String): Unit

    Permalink
    Attributes
    protected

  19. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef

  20. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any

  21. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  22. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  23. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from AnyRef

Inherited from Any

Ungrouped