org.apache.spark.sql.catalyst.catalog

SessionCatalog

class SessionCatalog extends Logging

An internal catalog that is used by a Spark Session. This internal catalog serves as a proxy to the underlying metastore (e.g. Hive Metastore) and it also manages temporary tables and functions of the Spark Session that it belongs to.

This class must be thread-safe.

Linear Supertypes
Logging, AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. SessionCatalog
  2. Logging
  3. AnyRef
  4. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new SessionCatalog(externalCatalog: ExternalCatalog)

  2. new SessionCatalog(externalCatalog: ExternalCatalog, functionRegistry: FunctionRegistry, conf: CatalystConf)

  3. new SessionCatalog(externalCatalog: ExternalCatalog, functionResourceLoader: FunctionResourceLoader, functionRegistry: FunctionRegistry, conf: CatalystConf, hadoopConf: Configuration)

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any
  3. final def ##(): Int

    Definition Classes
    AnyRef → Any
  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any
  6. def alterDatabase(dbDefinition: CatalogDatabase): Unit

  7. def alterPartitions(tableName: TableIdentifier, parts: Seq[CatalogTablePartition]): Unit

    Alter one or many table partitions whose specs that match those specified in parts, assuming the partitions exist.

    Alter one or many table partitions whose specs that match those specified in parts, assuming the partitions exist.

    If no database is specified, assume the table is in the current database.

    Note: If the underlying implementation does not support altering a certain field, this becomes a no-op.

  8. def alterTable(tableDefinition: CatalogTable): Unit

    Alter the metadata of an existing metastore table identified by tableDefinition.

    Alter the metadata of an existing metastore table identified by tableDefinition.

    If no database is specified in tableDefinition, assume the table is in the current database.

    Note: If the underlying implementation does not support altering a certain field, this becomes a no-op.

  9. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  10. def clearTempTables(): Unit

    Drop all existing temporary tables.

    Drop all existing temporary tables. For testing only.

  11. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  12. def createDatabase(dbDefinition: CatalogDatabase, ignoreIfExists: Boolean): Unit

  13. def createFunction(funcDefinition: CatalogFunction, ignoreIfExists: Boolean): Unit

    Create a metastore function in the database specified in funcDefinition.

    Create a metastore function in the database specified in funcDefinition. If no such database is specified, create it in the current database.

  14. def createPartitions(tableName: TableIdentifier, parts: Seq[CatalogTablePartition], ignoreIfExists: Boolean): Unit

    Create partitions in an existing table, assuming it exists.

    Create partitions in an existing table, assuming it exists. If no database is specified, assume the table is in the current database.

  15. def createTable(tableDefinition: CatalogTable, ignoreIfExists: Boolean): Unit

    Create a metastore table in the database specified in tableDefinition.

    Create a metastore table in the database specified in tableDefinition. If no such database is specified, create it in the current database.

  16. def createTempFunction(name: String, info: ExpressionInfo, funcDefinition: (Seq[Expression]) ⇒ Expression, ignoreIfExists: Boolean): Unit

    Create a temporary function.

    Create a temporary function. This assumes no database is specified in funcDefinition.

  17. def createTempView(name: String, tableDefinition: LogicalPlan, overrideIfExists: Boolean): Unit

    Create a temporary table.

  18. var currentDb: String

    Attributes
    protected
  19. def databaseExists(db: String): Boolean

  20. def defaultTablePath(tableIdent: TableIdentifier): String

  21. def dropDatabase(db: String, ignoreIfNotExists: Boolean, cascade: Boolean): Unit

  22. def dropFunction(name: FunctionIdentifier, ignoreIfNotExists: Boolean): Unit

    Drop a metastore function.

    Drop a metastore function. If no database is specified, assume the function is in the current database.

  23. def dropPartitions(tableName: TableIdentifier, specs: Seq[TablePartitionSpec], ignoreIfNotExists: Boolean): Unit

    Drop partitions from a table, assuming they exist.

    Drop partitions from a table, assuming they exist. If no database is specified, assume the table is in the current database.

  24. def dropTable(name: TableIdentifier, ignoreIfNotExists: Boolean): Unit

    Drop a table.

    Drop a table.

    If a database is specified in name, this will drop the table from that database. If no database is specified, this will first attempt to drop a temporary table with the same name, then, if that does not exist, drop the table from the current database.

  25. def dropTempFunction(name: String, ignoreIfNotExists: Boolean): Unit

    Drop a temporary function.

  26. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  27. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  28. def failFunctionLookup(name: String): Nothing

    Attributes
    protected
  29. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  30. def formatDatabaseName(name: String): String

    Format database name, taking into account case sensitivity.

    Format database name, taking into account case sensitivity.

    Attributes
    protected[this]
  31. def formatTableName(name: String): String

    Format table name, taking into account case sensitivity.

    Format table name, taking into account case sensitivity.

    Attributes
    protected[this]
  32. def functionExists(name: FunctionIdentifier): Boolean

    Check if the specified function exists.

  33. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any
  34. def getCurrentDatabase: String

  35. def getDatabaseMetadata(db: String): CatalogDatabase

  36. def getDefaultDBPath(db: String): String

    Get the path for creating a non-default database when database location is not provided by users.

  37. def getFunctionMetadata(name: FunctionIdentifier): CatalogFunction

    Retrieve the metadata of a metastore function.

    Retrieve the metadata of a metastore function.

    If a database is specified in name, this will return the function in that database. If no database is specified, this will return the function in the current database.

  38. def getPartition(tableName: TableIdentifier, spec: TablePartitionSpec): CatalogTablePartition

    Retrieve the metadata of a table partition, assuming it exists.

    Retrieve the metadata of a table partition, assuming it exists. If no database is specified, assume the table is in the current database.

  39. def getTableMetadata(name: TableIdentifier): CatalogTable

    Retrieve the metadata of an existing metastore table.

    Retrieve the metadata of an existing metastore table. If no database is specified, assume the table is in the current database. If the specified table is not found in the database then a NoSuchTableException is thrown.

  40. def getTableMetadataOption(name: TableIdentifier): Option[CatalogTable]

    Retrieve the metadata of an existing metastore table.

    Retrieve the metadata of an existing metastore table. If no database is specified, assume the table is in the current database. If the specified table is not found in the database then return None if it doesn't exist.

  41. def hashCode(): Int

    Definition Classes
    AnyRef → Any
  42. def initializeLogIfNecessary(isInterpreter: Boolean): Unit

    Attributes
    protected
    Definition Classes
    Logging
  43. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  44. def isTemporaryTable(name: TableIdentifier): Boolean

    Return whether a table with the specified name is a temporary table.

    Return whether a table with the specified name is a temporary table.

    Note: The temporary table cache is checked only when database is not explicitly specified.

  45. def isTraceEnabled(): Boolean

    Attributes
    protected
    Definition Classes
    Logging
  46. def listDatabases(pattern: String): Seq[String]

  47. def listDatabases(): Seq[String]

  48. def listFunctions(db: String, pattern: String): Seq[(FunctionIdentifier, String)]

    List all matching functions in the specified database, including temporary functions.

    List all matching functions in the specified database, including temporary functions. This returns the function identifier and the scope in which it was defined (system or user defined).

  49. def listFunctions(db: String): Seq[(FunctionIdentifier, String)]

    List all functions in the specified database, including temporary functions.

    List all functions in the specified database, including temporary functions. This returns the function identifier and the scope in which it was defined (system or user defined).

  50. def listPartitions(tableName: TableIdentifier, partialSpec: Option[TablePartitionSpec] = None): Seq[CatalogTablePartition]

    List the metadata of all partitions that belong to the specified table, assuming it exists.

    List the metadata of all partitions that belong to the specified table, assuming it exists.

    A partial partition spec may optionally be provided to filter the partitions returned. For instance, if there exist partitions (a='1', b='2'), (a='1', b='3') and (a='2', b='4'), then a partial spec of (a='1') will return the first two only.

  51. def listTables(db: String, pattern: String): Seq[TableIdentifier]

    List all matching tables in the specified database, including temporary tables.

  52. def listTables(db: String): Seq[TableIdentifier]

    List all tables in the specified database, including temporary tables.

  53. def loadFunctionResources(resources: Seq[FunctionResource]): Unit

    Loads resources such as JARs and Files for a function.

    Loads resources such as JARs and Files for a function. Every resource is represented by a tuple (resource type, resource uri).

  54. def loadPartition(name: TableIdentifier, loadPath: String, partition: TablePartitionSpec, isOverwrite: Boolean, holdDDLTime: Boolean, inheritTableSpecs: Boolean, isSkewedStoreAsSubdir: Boolean): Unit

    Load files stored in given path into the partition of an existing metastore table.

    Load files stored in given path into the partition of an existing metastore table. If no database is specified, assume the table is in the current database. If the specified table is not found in the database then a NoSuchTableException is thrown.

  55. def loadTable(name: TableIdentifier, loadPath: String, isOverwrite: Boolean, holdDDLTime: Boolean): Unit

    Load files stored in given path into an existing metastore table.

    Load files stored in given path into an existing metastore table. If no database is specified, assume the table is in the current database. If the specified table is not found in the database then a NoSuchTableException is thrown.

  56. def log: Logger

    Attributes
    protected
    Definition Classes
    Logging
  57. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  58. def logDebug(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  59. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  60. def logError(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  61. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  62. def logInfo(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  63. def logName: String

    Attributes
    protected
    Definition Classes
    Logging
  64. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  65. def logTrace(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  66. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  67. def logWarning(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  68. def lookupFunction(name: FunctionIdentifier, children: Seq[Expression]): Expression

    Return an Expression that represents the specified function, assuming it exists.

    Return an Expression that represents the specified function, assuming it exists.

    For a temporary function or a permanent function that has been loaded, this method will simply lookup the function through the FunctionRegistry and create an expression based on the builder.

    For a permanent function that has not been loaded, we will first fetch its metadata from the underlying external catalog. Then, we will load all resources associated with this function (i.e. jars and files). Finally, we create a function builder based on the function class and put the builder into the FunctionRegistry. The name of this function in the FunctionRegistry will be databaseName.functionName.

  69. def lookupRelation(name: TableIdentifier, alias: Option[String] = None): LogicalPlan

    Return a LogicalPlan that represents the given table.

    Return a LogicalPlan that represents the given table.

    If a database is specified in name, this will return the table from that database. If no database is specified, this will first attempt to return a temporary table with the same name, then, if that does not exist, return the table from the current database.

  70. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  71. final def notify(): Unit

    Definition Classes
    AnyRef
  72. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  73. def refreshTable(name: TableIdentifier): Unit

    Refresh the cache entry for a metastore table, if any.

  74. def renamePartitions(tableName: TableIdentifier, specs: Seq[TablePartitionSpec], newSpecs: Seq[TablePartitionSpec]): Unit

    Override the specs of one or many existing table partitions, assuming they exist.

    Override the specs of one or many existing table partitions, assuming they exist.

    This assumes index i of specs corresponds to index i of newSpecs. If no database is specified, assume the table is in the current database.

  75. def renameTable(oldName: TableIdentifier, newName: TableIdentifier): Unit

    Rename a table.

    Rename a table.

    If a database is specified in oldName, this will rename the table in that database. If no database is specified, this will first attempt to rename a temporary table with the same name, then, if that does not exist, rename the table in the current database.

    This assumes the database specified in oldName matches the one specified in newName.

  76. def setCurrentDatabase(db: String): Unit

  77. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  78. def tableExists(name: TableIdentifier): Boolean

    Return whether a table with the specified name exists.

    Return whether a table with the specified name exists.

    Note: If a database is explicitly specified, then this will return whether the table exists in that particular database instead. In that case, even if there is a temporary table with the same name, we will return false if the specified database does not contain the table.

  79. val tempTables: HashMap[String, LogicalPlan]

    List of temporary tables, mapping from table name to their logical plan.

    List of temporary tables, mapping from table name to their logical plan.

    Attributes
    protected
  80. def toString(): String

    Definition Classes
    AnyRef → Any
  81. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  82. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  83. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from Logging

Inherited from AnyRef

Inherited from Any

Ungrouped