Class

org.apache.spark.sql.catalyst.catalog

SessionCatalog

Related Doc: package catalog

Permalink

class SessionCatalog extends Logging

An internal catalog that is used by a Spark Session. This internal catalog serves as a proxy to the underlying metastore (e.g. Hive Metastore) and it also manages temporary tables and functions of the Spark Session that it belongs to.

This class must be thread-safe.

Linear Supertypes
Logging, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. SessionCatalog
  2. Logging
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new SessionCatalog(externalCatalog: ExternalCatalog)

    Permalink
  2. new SessionCatalog(externalCatalog: ExternalCatalog, functionRegistry: FunctionRegistry, conf: CatalystConf)

    Permalink
  3. new SessionCatalog(externalCatalog: ExternalCatalog, functionResourceLoader: FunctionResourceLoader, functionRegistry: FunctionRegistry, conf: CatalystConf, hadoopConf: Configuration)

    Permalink

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  4. def alterDatabase(dbDefinition: CatalogDatabase): Unit

    Permalink
  5. def alterPartitions(tableName: TableIdentifier, parts: Seq[CatalogTablePartition]): Unit

    Permalink

    Alter one or many table partitions whose specs that match those specified in parts, assuming the partitions exist.

    Alter one or many table partitions whose specs that match those specified in parts, assuming the partitions exist.

    If no database is specified, assume the table is in the current database.

    Note: If the underlying implementation does not support altering a certain field, this becomes a no-op.

  6. def alterTable(tableDefinition: CatalogTable): Unit

    Permalink

    Alter the metadata of an existing metastore table identified by tableDefinition.

    Alter the metadata of an existing metastore table identified by tableDefinition.

    If no database is specified in tableDefinition, assume the table is in the current database.

    Note: If the underlying implementation does not support altering a certain field, this becomes a no-op.

  7. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  8. def clearTempTables(): Unit

    Permalink

    Drop all existing temporary tables.

    Drop all existing temporary tables. For testing only.

  9. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  10. def createDatabase(dbDefinition: CatalogDatabase, ignoreIfExists: Boolean): Unit

    Permalink
  11. def createFunction(funcDefinition: CatalogFunction, ignoreIfExists: Boolean): Unit

    Permalink

    Create a metastore function in the database specified in funcDefinition.

    Create a metastore function in the database specified in funcDefinition. If no such database is specified, create it in the current database.

  12. def createPartitions(tableName: TableIdentifier, parts: Seq[CatalogTablePartition], ignoreIfExists: Boolean): Unit

    Permalink

    Create partitions in an existing table, assuming it exists.

    Create partitions in an existing table, assuming it exists. If no database is specified, assume the table is in the current database.

  13. def createTable(tableDefinition: CatalogTable, ignoreIfExists: Boolean): Unit

    Permalink

    Create a metastore table in the database specified in tableDefinition.

    Create a metastore table in the database specified in tableDefinition. If no such database is specified, create it in the current database.

  14. def createTempFunction(name: String, info: ExpressionInfo, funcDefinition: FunctionBuilder, ignoreIfExists: Boolean): Unit

    Permalink

    Create a temporary function.

    Create a temporary function. This assumes no database is specified in funcDefinition.

  15. def createTempView(name: String, tableDefinition: LogicalPlan, overrideIfExists: Boolean): Unit

    Permalink

    Create a temporary table.

  16. var currentDb: String

    Permalink
    Attributes
    protected
  17. def databaseExists(db: String): Boolean

    Permalink
  18. def defaultTablePath(tableIdent: TableIdentifier): String

    Permalink
  19. def dropDatabase(db: String, ignoreIfNotExists: Boolean, cascade: Boolean): Unit

    Permalink
  20. def dropFunction(name: FunctionIdentifier, ignoreIfNotExists: Boolean): Unit

    Permalink

    Drop a metastore function.

    Drop a metastore function. If no database is specified, assume the function is in the current database.

  21. def dropPartitions(tableName: TableIdentifier, specs: Seq[TablePartitionSpec], ignoreIfNotExists: Boolean): Unit

    Permalink

    Drop partitions from a table, assuming they exist.

    Drop partitions from a table, assuming they exist. If no database is specified, assume the table is in the current database.

  22. def dropTable(name: TableIdentifier, ignoreIfNotExists: Boolean): Unit

    Permalink

    Drop a table.

    Drop a table.

    If a database is specified in name, this will drop the table from that database. If no database is specified, this will first attempt to drop a temporary table with the same name, then, if that does not exist, drop the table from the current database.

  23. def dropTempFunction(name: String, ignoreIfNotExists: Boolean): Unit

    Permalink

    Drop a temporary function.

  24. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  25. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  26. def failFunctionLookup(name: String): Nothing

    Permalink
    Attributes
    protected
  27. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  28. def formatDatabaseName(name: String): String

    Permalink

    Format database name, taking into account case sensitivity.

    Format database name, taking into account case sensitivity.

    Attributes
    protected[this]
  29. def formatTableName(name: String): String

    Permalink

    Format table name, taking into account case sensitivity.

    Format table name, taking into account case sensitivity.

    Attributes
    protected[this]
  30. def functionExists(name: FunctionIdentifier): Boolean

    Permalink

    Check if the specified function exists.

  31. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  32. def getCurrentDatabase: String

    Permalink
  33. def getDatabaseMetadata(db: String): CatalogDatabase

    Permalink
  34. def getDefaultDBPath(db: String): String

    Permalink

    Get the path for creating a non-default database when database location is not provided by users.

  35. def getFunctionMetadata(name: FunctionIdentifier): CatalogFunction

    Permalink

    Retrieve the metadata of a metastore function.

    Retrieve the metadata of a metastore function.

    If a database is specified in name, this will return the function in that database. If no database is specified, this will return the function in the current database.

  36. def getPartition(tableName: TableIdentifier, spec: TablePartitionSpec): CatalogTablePartition

    Permalink

    Retrieve the metadata of a table partition, assuming it exists.

    Retrieve the metadata of a table partition, assuming it exists. If no database is specified, assume the table is in the current database.

  37. def getTableMetadata(name: TableIdentifier): CatalogTable

    Permalink

    Retrieve the metadata of an existing metastore table.

    Retrieve the metadata of an existing metastore table. If no database is specified, assume the table is in the current database. If the specified table is not found in the database then an NoSuchTableException is thrown.

  38. def getTableMetadataOption(name: TableIdentifier): Option[CatalogTable]

    Permalink

    Retrieve the metadata of an existing metastore table.

    Retrieve the metadata of an existing metastore table. If no database is specified, assume the table is in the current database. If the specified table is not found in the database then return None if it doesn't exist.

  39. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  40. def initializeLogIfNecessary(isInterpreter: Boolean): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  41. def invalidateTable(name: TableIdentifier): Unit

    Permalink

    Invalidate the cache entry for a metastore table, if any.

  42. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  43. def isTemporaryTable(name: TableIdentifier): Boolean

    Permalink

    Return whether a table with the specified name is a temporary table.

    Return whether a table with the specified name is a temporary table.

    Note: The temporary table cache is checked only when database is not explicitly specified.

  44. def isTraceEnabled(): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  45. def listDatabases(pattern: String): Seq[String]

    Permalink
  46. def listDatabases(): Seq[String]

    Permalink
  47. def listFunctions(db: String, pattern: String): Seq[FunctionIdentifier]

    Permalink

    List all matching functions in the specified database, including temporary functions.

  48. def listFunctions(db: String): Seq[FunctionIdentifier]

    Permalink

    List all functions in the specified database, including temporary functions.

  49. def listPartitions(tableName: TableIdentifier, partialSpec: Option[TablePartitionSpec] = None): Seq[CatalogTablePartition]

    Permalink

    List the metadata of all partitions that belong to the specified table, assuming it exists.

    List the metadata of all partitions that belong to the specified table, assuming it exists.

    A partial partition spec may optionally be provided to filter the partitions returned. For instance, if there exist partitions (a='1', b='2'), (a='1', b='3') and (a='2', b='4'), then a partial spec of (a='1') will return the first two only.

  50. def listTables(db: String, pattern: String): Seq[TableIdentifier]

    Permalink

    List all matching tables in the specified database, including temporary tables.

  51. def listTables(db: String): Seq[TableIdentifier]

    Permalink

    List all tables in the specified database, including temporary tables.

  52. def loadFunctionResources(resources: Seq[FunctionResource]): Unit

    Permalink

    Loads resources such as JARs and Files for a function.

    Loads resources such as JARs and Files for a function. Every resource is represented by a tuple (resource type, resource uri).

  53. def loadPartition(name: TableIdentifier, loadPath: String, partition: TablePartitionSpec, isOverwrite: Boolean, holdDDLTime: Boolean, inheritTableSpecs: Boolean, isSkewedStoreAsSubdir: Boolean): Unit

    Permalink

    Load files stored in given path into the partition of an existing metastore table.

    Load files stored in given path into the partition of an existing metastore table. If no database is specified, assume the table is in the current database. If the specified table is not found in the database then an NoSuchTableException is thrown.

  54. def loadTable(name: TableIdentifier, loadPath: String, isOverwrite: Boolean, holdDDLTime: Boolean): Unit

    Permalink

    Load files stored in given path into an existing metastore table.

    Load files stored in given path into an existing metastore table. If no database is specified, assume the table is in the current database. If the specified table is not found in the database then an NoSuchTableException is thrown.

  55. def log: Logger

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  56. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  57. def logDebug(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  58. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  59. def logError(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  60. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  61. def logInfo(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  62. def logName: String

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  63. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  64. def logTrace(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  65. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  66. def logWarning(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  67. def lookupFunction(name: FunctionIdentifier, children: Seq[Expression]): Expression

    Permalink

    Return an Expression that represents the specified function, assuming it exists.

    Return an Expression that represents the specified function, assuming it exists.

    For a temporary function or a permanent function that has been loaded, this method will simply lookup the function through the FunctionRegistry and create an expression based on the builder.

    For a permanent function that has not been loaded, we will first fetch its metadata from the underlying external catalog. Then, we will load all resources associated with this function (i.e. jars and files). Finally, we create a function builder based on the function class and put the builder into the FunctionRegistry. The name of this function in the FunctionRegistry will be databaseName.functionName.

  68. def lookupRelation(name: TableIdentifier, alias: Option[String] = None): LogicalPlan

    Permalink

    Return a LogicalPlan that represents the given table.

    Return a LogicalPlan that represents the given table.

    If a database is specified in name, this will return the table from that database. If no database is specified, this will first attempt to return a temporary table with the same name, then, if that does not exist, return the table from the current database.

  69. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  70. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  71. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  72. def refreshTable(name: TableIdentifier): Unit

    Permalink

    Refresh the cache entry for a metastore table, if any.

  73. def renamePartitions(tableName: TableIdentifier, specs: Seq[TablePartitionSpec], newSpecs: Seq[TablePartitionSpec]): Unit

    Permalink

    Override the specs of one or many existing table partitions, assuming they exist.

    Override the specs of one or many existing table partitions, assuming they exist.

    This assumes index i of specs corresponds to index i of newSpecs. If no database is specified, assume the table is in the current database.

  74. def renameTable(oldName: TableIdentifier, newName: TableIdentifier): Unit

    Permalink

    Rename a table.

    Rename a table.

    If a database is specified in oldName, this will rename the table in that database. If no database is specified, this will first attempt to rename a temporary table with the same name, then, if that does not exist, rename the table in the current database.

    This assumes the database specified in oldName matches the one specified in newName.

  75. def requireDbExists(db: String): Unit

    Permalink
    Attributes
    protected[this]
  76. def requireTableExists(name: TableIdentifier): Unit

    Permalink
    Attributes
    protected[this]
  77. def setCurrentDatabase(db: String): Unit

    Permalink
  78. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  79. def tableExists(name: TableIdentifier): Boolean

    Permalink

    Return whether a table with the specified name exists.

    Return whether a table with the specified name exists.

    Note: If a database is explicitly specified, then this will return whether the table exists in that particular database instead. In that case, even if there is a temporary table with the same name, we will return false if the specified database does not contain the table.

  80. val tempTables: HashMap[String, LogicalPlan]

    Permalink

    List of temporary tables, mapping from table name to their logical plan.

    List of temporary tables, mapping from table name to their logical plan.

    Attributes
    protected
  81. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any
  82. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  83. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  84. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from Logging

Inherited from AnyRef

Inherited from Any

Ungrouped