Package

org.apache.spark.sql.execution

command

Permalink

package command

Visibility
  1. Public
  2. All

Type Members

  1. case class AddFileCommand(path: String) extends LeafNode with RunnableCommand with Product with Serializable

    Permalink

    Adds a file to the current session so it can be used.

  2. case class AddJarCommand(path: String) extends LeafNode with RunnableCommand with Product with Serializable

    Permalink

    Adds a jar to the current session so it can be used (for UDFs or serdes).

  3. case class AlterDatabasePropertiesCommand(databaseName: String, props: Map[String, String]) extends LeafNode with RunnableCommand with Product with Serializable

    Permalink

    A command for users to add new (key, value) pairs into DBPROPERTIES If the database does not exist, an error message will be issued to indicate the database does not exist.

    A command for users to add new (key, value) pairs into DBPROPERTIES If the database does not exist, an error message will be issued to indicate the database does not exist. The syntax of using this command in SQL is:

    ALTER (DATABASE|SCHEMA) database_name SET DBPROPERTIES (property_name=property_value, ...)
  4. case class AlterTableAddPartitionCommand(tableName: TableIdentifier, partitionSpecsAndLocs: Seq[(TablePartitionSpec, Option[String])], ifNotExists: Boolean) extends LeafNode with RunnableCommand with Product with Serializable

    Permalink

    Add Partition in ALTER TABLE: add the table partitions.

    Add Partition in ALTER TABLE: add the table partitions.

    An error message will be issued if the partition exists, unless 'ifNotExists' is true.

    The syntax of this command is:

    ALTER TABLE table ADD [IF NOT EXISTS] PARTITION spec1 [LOCATION 'loc1']
                                          PARTITION spec2 [LOCATION 'loc2']
  5. case class AlterTableDropPartitionCommand(tableName: TableIdentifier, specs: Seq[TablePartitionSpec], ifExists: Boolean, purge: Boolean, retainData: Boolean) extends LeafNode with RunnableCommand with Product with Serializable

    Permalink

    Drop Partition in ALTER TABLE: to drop a particular partition for a table.

    Drop Partition in ALTER TABLE: to drop a particular partition for a table.

    This removes the data and metadata for this partition. The data is actually moved to the .Trash/Current directory if Trash is configured, unless 'purge' is true, but the metadata is completely lost. An error message will be issued if the partition does not exist, unless 'ifExists' is true. Note: purge is always false when the target is a view.

    The syntax of this command is:

    ALTER TABLE table DROP [IF EXISTS] PARTITION spec1[, PARTITION spec2, ...] [PURGE];
  6. case class AlterTableRecoverPartitionsCommand(tableName: TableIdentifier, cmd: String = "ALTER TABLE RECOVER PARTITIONS") extends LeafNode with RunnableCommand with Product with Serializable

    Permalink

    Recover Partitions in ALTER TABLE: recover all the partition in the directory of a table and update the catalog.

    Recover Partitions in ALTER TABLE: recover all the partition in the directory of a table and update the catalog.

    The syntax of this command is:

    ALTER TABLE table RECOVER PARTITIONS;
    MSCK REPAIR TABLE table;
  7. case class AlterTableRenameCommand(oldName: TableIdentifier, newName: TableIdentifier, isView: Boolean) extends LeafNode with RunnableCommand with Product with Serializable

    Permalink

    A command that renames a table/view.

    A command that renames a table/view.

    The syntax of this command is:

    ALTER TABLE table1 RENAME TO table2;
    ALTER VIEW view1 RENAME TO view2;
  8. case class AlterTableRenamePartitionCommand(tableName: TableIdentifier, oldPartition: TablePartitionSpec, newPartition: TablePartitionSpec) extends LeafNode with RunnableCommand with Product with Serializable

    Permalink

    Alter a table partition's spec.

    Alter a table partition's spec.

    The syntax of this command is:

    ALTER TABLE table PARTITION spec1 RENAME TO PARTITION spec2;
  9. case class AlterTableSerDePropertiesCommand(tableName: TableIdentifier, serdeClassName: Option[String], serdeProperties: Option[Map[String, String]], partSpec: Option[TablePartitionSpec]) extends LeafNode with RunnableCommand with Product with Serializable

    Permalink

    A command that sets the serde class and/or serde properties of a table/view.

    A command that sets the serde class and/or serde properties of a table/view.

    The syntax of this command is:

    ALTER TABLE table [PARTITION spec] SET SERDE serde_name [WITH SERDEPROPERTIES props];
    ALTER TABLE table [PARTITION spec] SET SERDEPROPERTIES serde_properties;
  10. case class AlterTableSetLocationCommand(tableName: TableIdentifier, partitionSpec: Option[TablePartitionSpec], location: String) extends LeafNode with RunnableCommand with Product with Serializable

    Permalink

    A command that sets the location of a table or a partition.

    A command that sets the location of a table or a partition.

    For normal tables, this just sets the location URI in the table/partition's storage format. For datasource tables, this sets a "path" parameter in the table/partition's serde properties.

    The syntax of this command is:

    ALTER TABLE table_name [PARTITION partition_spec] SET LOCATION "loc";
  11. case class AlterTableSetPropertiesCommand(tableName: TableIdentifier, properties: Map[String, String], isView: Boolean) extends LeafNode with RunnableCommand with Product with Serializable

    Permalink

    A command that sets table/view properties.

    A command that sets table/view properties.

    The syntax of this command is:

    ALTER TABLE table1 SET TBLPROPERTIES ('key1' = 'val1', 'key2' = 'val2', ...);
    ALTER VIEW view1 SET TBLPROPERTIES ('key1' = 'val1', 'key2' = 'val2', ...);
  12. case class AlterTableUnsetPropertiesCommand(tableName: TableIdentifier, propKeys: Seq[String], ifExists: Boolean, isView: Boolean) extends LeafNode with RunnableCommand with Product with Serializable

    Permalink

    A command that unsets table/view properties.

    A command that unsets table/view properties.

    The syntax of this command is:

    ALTER TABLE table1 UNSET TBLPROPERTIES [IF EXISTS] ('key1', 'key2', ...);
    ALTER VIEW view1 UNSET TBLPROPERTIES [IF EXISTS] ('key1', 'key2', ...);
  13. case class AlterViewAsCommand(name: TableIdentifier, originalText: String, query: LogicalPlan) extends LeafNode with RunnableCommand with Product with Serializable

    Permalink

    Alter a view with given query plan.

    Alter a view with given query plan. If the view name contains database prefix, this command will alter a permanent view matching the given name, or throw an exception if view not exist. Else, this command will try to alter a temporary view first, if view not exist, try permanent view next, if still not exist, throw an exception.

    name

    the name of this view.

    originalText

    the original SQL text of this view. Note that we can only alter a view by SQL API, which means we always have originalText.

    query

    the logical plan that represents the view; this is used to generate a canonicalized version of the SQL that can be saved in the catalog.

  14. case class AnalyzeColumnCommand(tableIdent: TableIdentifier, columnNames: Seq[String]) extends LeafNode with RunnableCommand with Product with Serializable

    Permalink

    Analyzes the given columns of the given table to generate statistics, which will be used in query optimizations.

  15. case class AnalyzeTableCommand(tableIdent: TableIdentifier, noscan: Boolean = true) extends LeafNode with RunnableCommand with Product with Serializable

    Permalink

    Analyzes the given table to generate statistics, which will be used in query optimizations.

  16. case class CacheTableCommand(tableIdent: TableIdentifier, plan: Option[LogicalPlan], isLazy: Boolean) extends LeafNode with RunnableCommand with Product with Serializable

    Permalink
  17. case class CreateDataSourceTableAsSelectCommand(table: CatalogTable, mode: SaveMode, query: LogicalPlan) extends LeafNode with RunnableCommand with Product with Serializable

    Permalink

    A command used to create a data source table using the result of a query.

    A command used to create a data source table using the result of a query.

    Note: This is different from CreateHiveTableAsSelectCommand. Please check the syntax for difference. This is not intended for temporary tables.

    The syntax of using this command in SQL is:

    CREATE TABLE [IF NOT EXISTS] [db_name.]table_name
    USING format OPTIONS ([option1_name "option1_value", option2_name "option2_value", ...])
    AS SELECT ...
  18. case class CreateDataSourceTableCommand(table: CatalogTable, ignoreIfExists: Boolean) extends LeafNode with RunnableCommand with Product with Serializable

    Permalink

    A command used to create a data source table.

    A command used to create a data source table.

    Note: This is different from CreateTableCommand. Please check the syntax for difference. This is not intended for temporary tables.

    The syntax of using this command in SQL is:

    CREATE TABLE [IF NOT EXISTS] [db_name.]table_name
    [(col1 data_type [COMMENT col_comment], ...)]
    USING format OPTIONS ([option1_name "option1_value", option2_name "option2_value", ...])
  19. case class CreateDatabaseCommand(databaseName: String, ifNotExists: Boolean, path: Option[String], comment: Option[String], props: Map[String, String]) extends LeafNode with RunnableCommand with Product with Serializable

    Permalink

    A command for users to create a new database.

    A command for users to create a new database.

    It will issue an error message when the database with the same name already exists, unless 'ifNotExists' is true. The syntax of using this command in SQL is:

    CREATE (DATABASE|SCHEMA) [IF NOT EXISTS] database_name
      [COMMENT database_comment]
      [LOCATION database_directory]
      [WITH DBPROPERTIES (property_name=property_value, ...)];
  20. case class CreateFunctionCommand(databaseName: Option[String], functionName: String, className: String, resources: Seq[FunctionResource], isTemp: Boolean) extends LeafNode with RunnableCommand with Product with Serializable

    Permalink

    The DDL command that creates a function.

    The DDL command that creates a function. To create a temporary function, the syntax of using this command in SQL is:

    CREATE TEMPORARY FUNCTION functionName
    AS className [USING JAR\FILE 'uri' [, JAR|FILE 'uri']]

    To create a permanent function, the syntax in SQL is:

    CREATE FUNCTION [databaseName.]functionName
    AS className [USING JAR\FILE 'uri' [, JAR|FILE 'uri']]
  21. case class CreateTableCommand(table: CatalogTable, ifNotExists: Boolean) extends LeafNode with RunnableCommand with Product with Serializable

    Permalink

    A command to create a table.

    A command to create a table.

    Note: This is currently used only for creating Hive tables. This is not intended for temporary tables.

    The syntax of using this command in SQL is:

    CREATE [EXTERNAL] TABLE [IF NOT EXISTS] [db_name.]table_name
    [(col1 data_type [COMMENT col_comment], ...)]
    [COMMENT table_comment]
    [PARTITIONED BY (col3 data_type [COMMENT col_comment], ...)]
    [CLUSTERED BY (col1, ...) [SORTED BY (col1 [ASC|DESC], ...)] INTO num_buckets BUCKETS]
    [SKEWED BY (col1, col2, ...) ON ((col_value, col_value, ...), ...)
    [STORED AS DIRECTORIES]
    [ROW FORMAT row_format]
    [STORED AS file_format | STORED BY storage_handler_class [WITH SERDEPROPERTIES (...)]]
    [LOCATION path]
    [TBLPROPERTIES (property_name=property_value, ...)]
    [AS select_statement];
  22. case class CreateTableLikeCommand(targetTable: TableIdentifier, sourceTable: TableIdentifier, ifNotExists: Boolean) extends LeafNode with RunnableCommand with Product with Serializable

    Permalink

    A command to create a MANAGED table with the same definition of the given existing table.

    A command to create a MANAGED table with the same definition of the given existing table. In the target table definition, the table comment is always empty but the column comments are identical to the ones defined in the source table.

    The CatalogTable attributes copied from the source table are storage(inputFormat, outputFormat, serde, compressed, properties), schema, provider, partitionColumnNames, bucketSpec.

    The syntax of using this command in SQL is:

    CREATE TABLE [IF NOT EXISTS] [db_name.]table_name
    LIKE [other_db_name.]existing_table_name
  23. case class CreateViewCommand(name: TableIdentifier, userSpecifiedColumns: Seq[(String, Option[String])], comment: Option[String], properties: Map[String, String], originalText: Option[String], child: LogicalPlan, allowExisting: Boolean, replace: Boolean, viewType: ViewType) extends LeafNode with RunnableCommand with Product with Serializable

    Permalink

    Create or replace a view with given query plan.

    Create or replace a view with given query plan. This command will convert the query plan to canonicalized SQL string, and store it as view text in metastore, if we need to create a permanent view.

    name

    the name of this view.

    userSpecifiedColumns

    the output column names and optional comments specified by users, can be Nil if not specified.

    comment

    the comment of this view.

    properties

    the properties of this view.

    originalText

    the original SQL text of this view, can be None if this view is created via Dataset API.

    child

    the logical plan that represents the view; this is used to generate a canonicalized version of the SQL that can be saved in the catalog.

    allowExisting

    if true, and if the view already exists, noop; if false, and if the view already exists, throws analysis exception.

    replace

    if true, and if the view already exists, updates it; if false, and if the view already exists, throws analysis exception.

    viewType

    the expected view type to be created with this command.

  24. case class DescribeDatabaseCommand(databaseName: String, extended: Boolean) extends LeafNode with RunnableCommand with Product with Serializable

    Permalink

    A command for users to show the name of the database, its comment (if one has been set), and its root location on the filesystem.

    A command for users to show the name of the database, its comment (if one has been set), and its root location on the filesystem. When extended is true, it also shows the database's properties If the database does not exist, an error message will be issued to indicate the database does not exist. The syntax of using this command in SQL is

    DESCRIBE DATABASE [EXTENDED] db_name
  25. case class DescribeFunctionCommand(functionName: FunctionIdentifier, isExtended: Boolean) extends LeafNode with RunnableCommand with Product with Serializable

    Permalink

    A command for users to get the usage of a registered function.

    A command for users to get the usage of a registered function. The syntax of using this command in SQL is

    DESCRIBE FUNCTION [EXTENDED] upper;
  26. case class DescribeTableCommand(table: TableIdentifier, partitionSpec: TablePartitionSpec, isExtended: Boolean, isFormatted: Boolean) extends LeafNode with RunnableCommand with Product with Serializable

    Permalink

    Command that looks like

    Command that looks like

    DESCRIBE [EXTENDED|FORMATTED] table_name partitionSpec?;
  27. case class DropDatabaseCommand(databaseName: String, ifExists: Boolean, cascade: Boolean) extends LeafNode with RunnableCommand with Product with Serializable

    Permalink

    A command for users to remove a database from the system.

    A command for users to remove a database from the system.

    'ifExists': - true, if database_name does't exist, no action - false (default), if database_name does't exist, a warning message will be issued 'cascade': - true, the dependent objects are automatically dropped before dropping database. - false (default), it is in the Restrict mode. The database cannot be dropped if it is not empty. The inclusive tables must be dropped at first.

    The syntax of using this command in SQL is:

    DROP DATABASE [IF EXISTS] database_name [RESTRICT|CASCADE];
  28. case class DropFunctionCommand(databaseName: Option[String], functionName: String, ifExists: Boolean, isTemp: Boolean) extends LeafNode with RunnableCommand with Product with Serializable

    Permalink

    The DDL command that drops a function.

    The DDL command that drops a function. ifExists: returns an error if the function doesn't exist, unless this is true. isTemp: indicates if it is a temporary function.

  29. case class DropTableCommand(tableName: TableIdentifier, ifExists: Boolean, isView: Boolean, purge: Boolean) extends LeafNode with RunnableCommand with Product with Serializable

    Permalink

    Drops a table/view from the metastore and removes it if it is cached.

    Drops a table/view from the metastore and removes it if it is cached.

    The syntax of this command is:

    DROP TABLE [IF EXISTS] table_name;
    DROP VIEW [IF EXISTS] [db_name.]view_name;
  30. case class ExecutedCommandExec(cmd: RunnableCommand) extends SparkPlan with Product with Serializable

    Permalink

    A physical operator that executes the run method of a RunnableCommand and saves the result to prevent multiple executions.

  31. case class ExplainCommand(logicalPlan: LogicalPlan, extended: Boolean = false, codegen: Boolean = false) extends LeafNode with RunnableCommand with Product with Serializable

    Permalink

    An explain command for users to see how a command will be executed.

    An explain command for users to see how a command will be executed.

    Note that this command takes in a logical plan, runs the optimizer on the logical plan (but do NOT actually execute it).

    EXPLAIN (EXTENDED | CODEGEN) SELECT * FROM ...
    logicalPlan

    plan to explain

    extended

    whether to do extended explain or not

    codegen

    whether to output generated code from whole-stage codegen or not

  32. case class ListFilesCommand(files: Seq[String] = Seq.empty[String]) extends LeafNode with RunnableCommand with Product with Serializable

    Permalink

    Returns a list of file paths that are added to resources.

    Returns a list of file paths that are added to resources. If file paths are provided, return the ones that are added to resources.

  33. case class ListJarsCommand(jars: Seq[String] = Seq.empty[String]) extends LeafNode with RunnableCommand with Product with Serializable

    Permalink

    Returns a list of jar files that are added to resources.

    Returns a list of jar files that are added to resources. If jar files are provided, return the ones that are added to resources.

  34. case class LoadDataCommand(table: TableIdentifier, path: String, isLocal: Boolean, isOverwrite: Boolean, partition: Option[TablePartitionSpec]) extends LeafNode with RunnableCommand with Product with Serializable

    Permalink

    A command that loads data into a Hive table.

    A command that loads data into a Hive table.

    The syntax of this command is:

    LOAD DATA [LOCAL] INPATH 'filepath' [OVERWRITE] INTO TABLE tablename
    [PARTITION (partcol1=val1, partcol2=val2 ...)]
  35. case class PartitionStatistics(numFiles: Int, totalSize: Long) extends Product with Serializable

    Permalink
  36. trait RunnableCommand extends LeafNode with Command

    Permalink

    A logical command that is executed for its side-effects.

    A logical command that is executed for its side-effects. RunnableCommands are wrapped in ExecutedCommand during execution.

  37. case class SetCommand(kv: Option[(String, Option[String])]) extends LeafNode with RunnableCommand with Logging with Product with Serializable

    Permalink

    Command that runs

    Command that runs

    set key = value;
    set -v;
    set;
  38. case class SetDatabaseCommand(databaseName: String) extends LeafNode with RunnableCommand with Product with Serializable

    Permalink

    Command for setting the current database.

    Command for setting the current database.

    USE database_name;
  39. case class ShowColumnsCommand(databaseName: Option[String], tableName: TableIdentifier) extends LeafNode with RunnableCommand with Product with Serializable

    Permalink

    A command to list the column names for a table.

    A command to list the column names for a table. This function creates a ShowColumnsCommand logical plan.

    The syntax of using this command in SQL is:

    SHOW COLUMNS (FROM | IN) table_identifier [(FROM | IN) database];
  40. case class ShowCreateTableCommand(table: TableIdentifier) extends LeafNode with RunnableCommand with Product with Serializable

    Permalink
  41. case class ShowDatabasesCommand(databasePattern: Option[String]) extends LeafNode with RunnableCommand with Product with Serializable

    Permalink

    A command for users to list the databases/schemas.

    A command for users to list the databases/schemas. If a databasePattern is supplied then the databases that only matches the pattern would be listed. The syntax of using this command in SQL is:

    SHOW (DATABASES|SCHEMAS) [LIKE 'identifier_with_wildcards'];
  42. case class ShowFunctionsCommand(db: Option[String], pattern: Option[String], showUserFunctions: Boolean, showSystemFunctions: Boolean) extends LeafNode with RunnableCommand with Product with Serializable

    Permalink

    A command for users to list all of the registered functions.

    A command for users to list all of the registered functions. The syntax of using this command in SQL is:

    SHOW FUNCTIONS [LIKE pattern]

    For the pattern, '*' matches any sequence of characters (including no characters) and '|' is for alternation. For example, "show functions like 'yea*|windo*'" will return "window" and "year".

  43. case class ShowPartitionsCommand(tableName: TableIdentifier, spec: Option[TablePartitionSpec]) extends LeafNode with RunnableCommand with Product with Serializable

    Permalink

    A command to list the partition names of a table.

    A command to list the partition names of a table. If the partition spec is specified, partitions that match the spec are returned. AnalysisException exception is thrown under the following conditions:

    1. If the command is called for a non partitioned table. 2. If the partition spec refers to the columns that are not defined as partitioning columns.

    This function creates a ShowPartitionsCommand logical plan

    The syntax of using this command in SQL is:

    SHOW PARTITIONS [db_name.]table_name [PARTITION(partition_spec)]
  44. case class ShowTablePropertiesCommand(table: TableIdentifier, propertyKey: Option[String]) extends LeafNode with RunnableCommand with Product with Serializable

    Permalink

    A command for users to list the properties for a table.

    A command for users to list the properties for a table. If propertyKey is specified, the value for the propertyKey is returned. If propertyKey is not specified, all the keys and their corresponding values are returned. The syntax of using this command in SQL is:

    SHOW TBLPROPERTIES table_name[('propertyKey')];
  45. case class ShowTablesCommand(databaseName: Option[String], tableIdentifierPattern: Option[String]) extends LeafNode with RunnableCommand with Product with Serializable

    Permalink

    A command for users to get tables in the given database.

    A command for users to get tables in the given database. If a databaseName is not given, the current database will be used. The syntax of using this command in SQL is:

    SHOW TABLES [(IN|FROM) database_name] [[LIKE] 'identifier_with_wildcards'];
  46. case class StreamingExplainCommand(queryExecution: IncrementalExecution, extended: Boolean) extends LeafNode with RunnableCommand with Product with Serializable

    Permalink

    An explain command for users to see how a streaming batch is executed.

  47. case class TruncateTableCommand(tableName: TableIdentifier, partitionSpec: Option[TablePartitionSpec]) extends LeafNode with RunnableCommand with Product with Serializable

    Permalink

    A command to truncate table.

    A command to truncate table.

    The syntax of this command is:

    TRUNCATE TABLE tablename [PARTITION (partcol1=val1, partcol2=val2 ...)]
  48. case class UncacheTableCommand(tableIdent: TableIdentifier, ifExists: Boolean) extends LeafNode with RunnableCommand with Product with Serializable

    Permalink
  49. sealed trait ViewType extends AnyRef

    Permalink

    ViewType is used to specify the expected view type when we want to create or replace a view in CreateViewCommand.

Value Members

  1. object AnalyzeColumnCommand extends Logging with Serializable

    Permalink
  2. object AnalyzeTableCommand extends Logging with Serializable

    Permalink
  3. object ClearCacheCommand extends LeafNode with RunnableCommand with Product with Serializable

    Permalink

    Clear all cached data from the in-memory cache.

  4. object DDLUtils

    Permalink
  5. object GlobalTempView extends ViewType

    Permalink

    GlobalTempView means cross-session global temporary views.

    GlobalTempView means cross-session global temporary views. Its lifetime is the lifetime of the Spark application, i.e. it will be automatically dropped when the application terminates. It's tied to a system preserved database _global_temp, and we must use the qualified name to refer a global temp view, e.g. SELECT * FROM _global_temp.view1.

  6. object LocalTempView extends ViewType

    Permalink

    LocalTempView means session-scoped local temporary views.

    LocalTempView means session-scoped local temporary views. Its lifetime is the lifetime of the session that created it, i.e. it will be automatically dropped when the session terminates. It's not tied to any databases, i.e. we can't use db1.view1 to reference a local temporary view.

  7. object PersistedView extends ViewType

    Permalink

    PersistedView means cross-session persisted views.

    PersistedView means cross-session persisted views. Persisted views stay until they are explicitly dropped by user command. It's always tied to a database, default to the current database if not specified.

    Note that, Existing persisted view with the same name are not visible to the current session while the local temporary view exists, unless the view name is qualified by database.

  8. object ResetCommand extends LeafNode with RunnableCommand with Logging with Product with Serializable

    Permalink

    This command is for resetting SQLConf to the default values.

    This command is for resetting SQLConf to the default values. Command that runs

    reset;
  9. object SetCommand extends Serializable

    Permalink

Ungrouped