Class/Object

org.apache.spark.sql.catalyst.catalog

SessionCatalog

Related Docs: object SessionCatalog | package catalog

Permalink

class SessionCatalog extends Logging

An internal catalog that is used by a Spark Session. This internal catalog serves as a proxy to the underlying metastore (e.g. Hive Metastore) and it also manages temporary tables and functions of the Spark Session that it belongs to.

This class must be thread-safe.

Linear Supertypes
Logging, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. SessionCatalog
  2. Logging
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new SessionCatalog(externalCatalog: ExternalCatalog)

    Permalink
  2. new SessionCatalog(externalCatalog: ExternalCatalog, functionRegistry: FunctionRegistry, conf: CatalystConf)

    Permalink
  3. new SessionCatalog(externalCatalog: ExternalCatalog, functionResourceLoader: FunctionResourceLoader, functionRegistry: FunctionRegistry, conf: CatalystConf, hadoopConf: Configuration)

    Permalink

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  4. def alterDatabase(dbDefinition: CatalogDatabase): Unit

    Permalink
  5. def alterPartitions(tableName: TableIdentifier, parts: Seq[CatalogTablePartition]): Unit

    Permalink

    Alter one or many table partitions whose specs that match those specified in parts, assuming the partitions exist.

    Alter one or many table partitions whose specs that match those specified in parts, assuming the partitions exist.

    If no database is specified, assume the table is in the current database.

    Note: If the underlying implementation does not support altering a certain field, this becomes a no-op.

  6. def alterTable(tableDefinition: CatalogTable): Unit

    Permalink

    Alter the metadata of an existing metastore table identified by tableDefinition.

    Alter the metadata of an existing metastore table identified by tableDefinition.

    If no database is specified in tableDefinition, assume the table is in the current database.

    Note: If the underlying implementation does not support altering a certain field, this becomes a no-op.

  7. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  8. def clearTempTables(): Unit

    Permalink

    Drop all existing temporary tables.

    Drop all existing temporary tables. For testing only.

  9. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  10. def createDatabase(dbDefinition: CatalogDatabase, ignoreIfExists: Boolean): Unit

    Permalink
  11. def createFunction(funcDefinition: CatalogFunction, ignoreIfExists: Boolean): Unit

    Permalink

    Create a metastore function in the database specified in funcDefinition.

    Create a metastore function in the database specified in funcDefinition. If no such database is specified, create it in the current database.

  12. def createPartitions(tableName: TableIdentifier, parts: Seq[CatalogTablePartition], ignoreIfExists: Boolean): Unit

    Permalink

    Create partitions in an existing table, assuming it exists.

    Create partitions in an existing table, assuming it exists. If no database is specified, assume the table is in the current database.

  13. def createTable(tableDefinition: CatalogTable, ignoreIfExists: Boolean): Unit

    Permalink

    Create a metastore table in the database specified in tableDefinition.

    Create a metastore table in the database specified in tableDefinition. If no such database is specified, create it in the current database.

  14. def createTempFunction(name: String, info: ExpressionInfo, funcDefinition: FunctionBuilder, ignoreIfExists: Boolean): Unit

    Permalink

    Create a temporary function.

    Create a temporary function. This assumes no database is specified in funcDefinition.

  15. def createTempView(name: String, tableDefinition: LogicalPlan, overrideIfExists: Boolean): Unit

    Permalink

    Create a temporary table.

  16. var currentDb: String

    Permalink
    Attributes
    protected
  17. def databaseExists(db: String): Boolean

    Permalink
  18. def defaultTablePath(tableIdent: TableIdentifier): String

    Permalink
  19. def dropDatabase(db: String, ignoreIfNotExists: Boolean, cascade: Boolean): Unit

    Permalink
  20. def dropFunction(name: FunctionIdentifier, ignoreIfNotExists: Boolean): Unit

    Permalink

    Drop a metastore function.

    Drop a metastore function. If no database is specified, assume the function is in the current database.

  21. def dropPartitions(tableName: TableIdentifier, specs: Seq[TablePartitionSpec], ignoreIfNotExists: Boolean): Unit

    Permalink

    Drop partitions from a table, assuming they exist.

    Drop partitions from a table, assuming they exist. If no database is specified, assume the table is in the current database.

  22. def dropTable(name: TableIdentifier, ignoreIfNotExists: Boolean): Unit

    Permalink

    Drop a table.

    Drop a table.

    If a database is specified in name, this will drop the table from that database. If no database is specified, this will first attempt to drop a temporary table with the same name, then, if that does not exist, drop the table from the current database.

  23. def dropTempFunction(name: String, ignoreIfNotExists: Boolean): Unit

    Permalink

    Drop a temporary function.

  24. def dropTempView(name: String): Unit

    Permalink

    Drop a temporary view.

  25. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  26. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  27. def failFunctionLookup(name: String): Nothing

    Permalink
    Attributes
    protected
  28. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  29. def formatDatabaseName(name: String): String

    Permalink

    Format database name, taking into account case sensitivity.

    Format database name, taking into account case sensitivity.

    Attributes
    protected[this]
  30. def formatTableName(name: String): String

    Permalink

    Format table name, taking into account case sensitivity.

    Format table name, taking into account case sensitivity.

    Attributes
    protected[this]
  31. def functionExists(name: FunctionIdentifier): Boolean

    Permalink

    Check if the specified function exists.

  32. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  33. def getCurrentDatabase: String

    Permalink
  34. def getDatabaseMetadata(db: String): CatalogDatabase

    Permalink
  35. def getDefaultDBPath(db: String): String

    Permalink

    Get the path for creating a non-default database when database location is not provided by users.

  36. def getFunctionMetadata(name: FunctionIdentifier): CatalogFunction

    Permalink

    Retrieve the metadata of a metastore function.

    Retrieve the metadata of a metastore function.

    If a database is specified in name, this will return the function in that database. If no database is specified, this will return the function in the current database.

  37. def getPartition(tableName: TableIdentifier, spec: TablePartitionSpec): CatalogTablePartition

    Permalink

    Retrieve the metadata of a table partition, assuming it exists.

    Retrieve the metadata of a table partition, assuming it exists. If no database is specified, assume the table is in the current database.

  38. def getTableMetadata(name: TableIdentifier): CatalogTable

    Permalink

    Retrieve the metadata of an existing permanent table/view.

    Retrieve the metadata of an existing permanent table/view. If no database is specified, assume the table/view is in the current database. If the specified table/view is not found in the database then a NoSuchTableException is thrown.

  39. def getTableMetadataOption(name: TableIdentifier): Option[CatalogTable]

    Permalink

    Retrieve the metadata of an existing metastore table.

    Retrieve the metadata of an existing metastore table. If no database is specified, assume the table is in the current database. If the specified table is not found in the database then return None if it doesn't exist.

  40. def getTempView(name: String): Option[LogicalPlan]

    Permalink

    Return a temporary view exactly as it was stored.

  41. def getTempViewOrPermanentTableMetadata(name: TableIdentifier): CatalogTable

    Permalink

    Retrieve the metadata of an existing temporary view or permanent table/view.

    Retrieve the metadata of an existing temporary view or permanent table/view.

    If a database is specified in name, this will return the metadata of table/view in that database. If no database is specified, this will first attempt to get the metadata of a temporary view with the same name, then, if that does not exist, return the metadata of table/view in the current database.

  42. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  43. def initializeLogIfNecessary(isInterpreter: Boolean): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  44. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  45. def isTemporaryTable(name: TableIdentifier): Boolean

    Permalink

    Return whether a table with the specified name is a temporary table.

    Return whether a table with the specified name is a temporary table.

    Note: The temporary table cache is checked only when database is not explicitly specified.

  46. def isTraceEnabled(): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  47. def listDatabases(pattern: String): Seq[String]

    Permalink
  48. def listDatabases(): Seq[String]

    Permalink
  49. def listFunctions(db: String, pattern: String): Seq[(FunctionIdentifier, String)]

    Permalink

    List all matching functions in the specified database, including temporary functions.

    List all matching functions in the specified database, including temporary functions. This returns the function identifier and the scope in which it was defined (system or user defined).

  50. def listFunctions(db: String): Seq[(FunctionIdentifier, String)]

    Permalink

    List all functions in the specified database, including temporary functions.

    List all functions in the specified database, including temporary functions. This returns the function identifier and the scope in which it was defined (system or user defined).

  51. def listPartitions(tableName: TableIdentifier, partialSpec: Option[TablePartitionSpec] = None): Seq[CatalogTablePartition]

    Permalink

    List the metadata of all partitions that belong to the specified table, assuming it exists.

    List the metadata of all partitions that belong to the specified table, assuming it exists.

    A partial partition spec may optionally be provided to filter the partitions returned. For instance, if there exist partitions (a='1', b='2'), (a='1', b='3') and (a='2', b='4'), then a partial spec of (a='1') will return the first two only.

  52. def listTables(db: String, pattern: String): Seq[TableIdentifier]

    Permalink

    List all matching tables in the specified database, including temporary tables.

  53. def listTables(db: String): Seq[TableIdentifier]

    Permalink

    List all tables in the specified database, including temporary tables.

  54. def loadFunctionResources(resources: Seq[FunctionResource]): Unit

    Permalink

    Loads resources such as JARs and Files for a function.

    Loads resources such as JARs and Files for a function. Every resource is represented by a tuple (resource type, resource uri).

  55. def loadPartition(name: TableIdentifier, loadPath: String, partition: TablePartitionSpec, isOverwrite: Boolean, holdDDLTime: Boolean, inheritTableSpecs: Boolean, isSkewedStoreAsSubdir: Boolean): Unit

    Permalink

    Load files stored in given path into the partition of an existing metastore table.

    Load files stored in given path into the partition of an existing metastore table. If no database is specified, assume the table is in the current database. If the specified table is not found in the database then a NoSuchTableException is thrown.

  56. def loadTable(name: TableIdentifier, loadPath: String, isOverwrite: Boolean, holdDDLTime: Boolean): Unit

    Permalink

    Load files stored in given path into an existing metastore table.

    Load files stored in given path into an existing metastore table. If no database is specified, assume the table is in the current database. If the specified table is not found in the database then a NoSuchTableException is thrown.

  57. def log: Logger

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  58. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  59. def logDebug(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  60. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  61. def logError(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  62. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  63. def logInfo(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  64. def logName: String

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  65. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  66. def logTrace(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  67. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  68. def logWarning(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  69. def lookupFunction(name: FunctionIdentifier, children: Seq[Expression]): Expression

    Permalink

    Return an Expression that represents the specified function, assuming it exists.

    Return an Expression that represents the specified function, assuming it exists.

    For a temporary function or a permanent function that has been loaded, this method will simply lookup the function through the FunctionRegistry and create an expression based on the builder.

    For a permanent function that has not been loaded, we will first fetch its metadata from the underlying external catalog. Then, we will load all resources associated with this function (i.e. jars and files). Finally, we create a function builder based on the function class and put the builder into the FunctionRegistry. The name of this function in the FunctionRegistry will be databaseName.functionName.

  70. def lookupFunctionInfo(name: FunctionIdentifier): ExpressionInfo

    Permalink

    Look up the ExpressionInfo associated with the specified function, assuming it exists.

  71. def lookupRelation(name: TableIdentifier, alias: Option[String] = None): LogicalPlan

    Permalink

    Return a LogicalPlan that represents the given table.

    Return a LogicalPlan that represents the given table.

    If a database is specified in name, this will return the table from that database. If no database is specified, this will first attempt to return a temporary table with the same name, then, if that does not exist, return the table from the current database.

  72. def makeFunctionBuilder(name: String, functionClassName: String): FunctionBuilder

    Permalink

    Construct a FunctionBuilder based on the provided class that represents a function.

    Construct a FunctionBuilder based on the provided class that represents a function.

    This performs reflection to decide what type of Expression to return in the builder.

  73. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  74. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  75. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  76. def refreshTable(name: TableIdentifier): Unit

    Permalink

    Refresh the cache entry for a metastore table, if any.

  77. def renamePartitions(tableName: TableIdentifier, specs: Seq[TablePartitionSpec], newSpecs: Seq[TablePartitionSpec]): Unit

    Permalink

    Override the specs of one or many existing table partitions, assuming they exist.

    Override the specs of one or many existing table partitions, assuming they exist.

    This assumes index i of specs corresponds to index i of newSpecs. If no database is specified, assume the table is in the current database.

  78. def renameTable(oldName: TableIdentifier, newName: TableIdentifier): Unit

    Permalink

    Rename a table.

    Rename a table.

    If a database is specified in oldName, this will rename the table in that database. If no database is specified, this will first attempt to rename a temporary table with the same name, then, if that does not exist, rename the table in the current database.

    This assumes the database specified in oldName matches the one specified in newName.

  79. def reset(): Unit

    Permalink

    Drop all existing databases (except "default"), tables, partitions and functions, and set the current database to "default".

    Drop all existing databases (except "default"), tables, partitions and functions, and set the current database to "default".

    This is mainly used for tests.

  80. def setCurrentDatabase(db: String): Unit

    Permalink
  81. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  82. def tableExists(name: TableIdentifier): Boolean

    Permalink

    Return whether a table/view with the specified name exists.

    Return whether a table/view with the specified name exists. If no database is specified, check with current database.

  83. val tempTables: HashMap[String, LogicalPlan]

    Permalink

    List of temporary tables, mapping from table name to their logical plan.

    List of temporary tables, mapping from table name to their logical plan.

    Attributes
    protected
  84. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any
  85. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  86. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  87. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from Logging

Inherited from AnyRef

Inherited from Any

Ungrouped