Type Members

1.  case class AlterNamespaceSetPropertiesExec(catalog: SupportsNamespaces, namespace: Seq[String], props: Map[String, String]) extends V2CommandExec with Product with Serializable

   Physical plan node for setting properties of namespace.

2.  case class AlterTableAddPartitionExec(table: SupportsPartitionManagement, partSpecs: Seq[ResolvedPartitionSpec], ignoreIfExists: Boolean) extends V2CommandExec with Product with Serializable

   Physical plan node for adding partitions of table.

3.  case class AlterTableDropPartitionExec(table: SupportsPartitionManagement, partSpecs: Seq[ResolvedPartitionSpec], ignoreIfNotExists: Boolean) extends V2CommandExec with Product with Serializable

   Physical plan node for dropping partitions of table.

4.  case class AlterTableExec(catalog: TableCatalog, ident: Identifier, changes: Seq[TableChange]) extends V2CommandExec with Product with Serializable

   Physical plan node for altering a table.

5.  case class AppendDataExec(table: SupportsWrite, writeOptions: CaseInsensitiveStringMap, query: SparkPlan, refreshCache: () ⇒ Unit) extends V2CommandExec with V2TableWriteExec with BatchWriteHelper with Product with Serializable

   Physical plan node for append into a v2 table.

   Physical plan node for append into a v2 table.

   Rows in the output data set are appended.

6.  case class AppendDataExecV1(table: SupportsWrite, writeOptions: CaseInsensitiveStringMap, plan: LogicalPlan, refreshCache: () ⇒ Unit) extends V2CommandExec with V1FallbackWriters with Product with Serializable

   Physical plan node for append into a v2 table using V1 write interfaces.

   Physical plan node for append into a v2 table using V1 write interfaces.

   Rows in the output data set are appended.

7.  case class AtomicCreateTableAsSelectExec(catalog: StagingTableCatalog, ident: Identifier, partitioning: Seq[Transform], plan: LogicalPlan, query: SparkPlan, properties: Map[String, String], writeOptions: CaseInsensitiveStringMap, ifNotExists: Boolean) extends V2CommandExec with TableWriteExecHelper with Product with Serializable

   Physical plan node for v2 create table as select, when the catalog is determined to support staging table creation.

   Physical plan node for v2 create table as select, when the catalog is determined to support staging table creation.

   A new table will be created using the schema of the query, and rows from the query are appended. The CTAS operation is atomic. The creation of the table is staged and the commit of the write should bundle the commitment of the metadata and the table contents in a single unit. If the write fails, the table is instructed to roll back all staged changes.

8.  case class AtomicReplaceTableAsSelectExec(catalog: StagingTableCatalog, ident: Identifier, partitioning: Seq[Transform], plan: LogicalPlan, query: SparkPlan, properties: Map[String, String], writeOptions: CaseInsensitiveStringMap, orCreate: Boolean, invalidateCache: (TableCatalog, Table, Identifier) ⇒ Unit) extends V2CommandExec with TableWriteExecHelper with Product with Serializable

   Physical plan node for v2 replace table as select when the catalog supports staging table replacement.

   Physical plan node for v2 replace table as select when the catalog supports staging table replacement.

   A new table will be created using the schema of the query, and rows from the query are appended. If the table exists, its contents and schema should be replaced with the schema and the contents of the query. This implementation is atomic. The table replacement is staged, and the commit operation at the end should perform the replacement of the table's metadata and contents. If the write fails, the table is instructed to roll back staged changes and any previously written table is left untouched.

9.  case class AtomicReplaceTableExec(catalog: StagingTableCatalog, identifier: Identifier, tableSchema: StructType, partitioning: Seq[Transform], tableProperties: Map[String, String], orCreate: Boolean, invalidateCache: (TableCatalog, Table, Identifier) ⇒ Unit) extends V2CommandExec with Product with Serializable
10.  case class BatchScanExec(output: Seq[AttributeReference], scan: Scan) extends SparkPlan with DataSourceV2ScanExecBase with Product with Serializable

    Physical plan node for scanning a batch of data from a data source v2.

11.  trait BatchWriteHelper extends AnyRef

    Helper for physical plans that build batch writes.

12.  case class ContinuousScanExec(output: Seq[Attribute], scan: Scan, stream: ContinuousStream, start: Offset) extends SparkPlan with DataSourceV2ScanExecBase with Product with Serializable

    Physical plan node for scanning data from a streaming data source with continuous mode.

13.  case class CreateNamespaceExec(catalog: SupportsNamespaces, namespace: Seq[String], ifNotExists: Boolean, properties: Map[String, String]) extends V2CommandExec with Product with Serializable

    Physical plan node for creating a namespace.

14.  case class CreateTableAsSelectExec(catalog: TableCatalog, ident: Identifier, partitioning: Seq[Transform], plan: LogicalPlan, query: SparkPlan, properties: Map[String, String], writeOptions: CaseInsensitiveStringMap, ifNotExists: Boolean) extends V2CommandExec with TableWriteExecHelper with Product with Serializable

    Physical plan node for v2 create table as select when the catalog does not support staging the table creation.

    Physical plan node for v2 create table as select when the catalog does not support staging the table creation.

    A new table will be created using the schema of the query, and rows from the query are appended. If either table creation or the append fails, the table will be deleted. This implementation is not atomic; for an atomic variant for catalogs that support the appropriate features, see CreateTableAsSelectStagingExec.

15.  case class CreateTableExec(catalog: TableCatalog, identifier: Identifier, tableSchema: StructType, partitioning: Seq[Transform], tableProperties: Map[String, String], ignoreIfExists: Boolean) extends V2CommandExec with Product with Serializable
16.  class DataSourcePartitioning extends Partitioning

    An adapter from public data source partitioning to catalyst internal Partitioning.

17.  class DataSourceRDD extends RDD[InternalRow]
18.  class DataSourceRDDPartition extends Partition with Serializable
19.  trait DataSourceV2ScanExecBase extends SparkPlan with LeafExecNode
20.  class DataSourceV2Strategy extends Strategy with PredicateHelper
21.  case class DeleteFromTableExec(table: SupportsDelete, condition: Array[Filter], refreshCache: () ⇒ Unit) extends V2CommandExec with Product with Serializable
22.  case class DescribeNamespaceExec(output: Seq[Attribute], catalog: SupportsNamespaces, namespace: Seq[String], isExtended: Boolean) extends V2CommandExec with Product with Serializable

    Physical plan node for describing a namespace.

23.  case class DescribeTableExec(output: Seq[Attribute], table: Table, isExtended: Boolean) extends V2CommandExec with Product with Serializable
24.  case class DropNamespaceExec(catalog: CatalogPlugin, namespace: Seq[String], ifExists: Boolean, cascade: Boolean) extends V2CommandExec with Product with Serializable

    Physical plan node for dropping a namespace.

25.  case class DropTableExec(catalog: TableCatalog, ident: Identifier, ifExists: Boolean, purge: Boolean, invalidateCache: () ⇒ Unit) extends V2CommandExec with Product with Serializable

    Physical plan node for dropping a table.

26.  class EmptyPartitionReader[T] extends PartitionReader[T]

    A PartitionReader with empty output.

27.  class FileBatchWrite extends BatchWrite with Logging
28.  trait FileDataSourceV2 extends TableProvider with DataSourceRegister

    A base interface for data source v2 implementations of the built-in file-based data sources.

29.  class FilePartitionReader[T] extends PartitionReader[T] with Logging
30.  abstract class FilePartitionReaderFactory extends PartitionReaderFactory
31.  trait FileScan extends Scan with Batch with SupportsReportStatistics with SupportsMetadata with Logging
32.  abstract class FileScanBuilder extends ScanBuilder with SupportsPushDownRequiredColumns
33.  abstract class FileTable extends Table with SupportsRead with SupportsWrite
34.  abstract class FileWriteBuilder extends WriteBuilder
35.  case class FileWriterFactory(description: WriteJobDescription, committer: FileCommitProtocol) extends DataWriterFactory with Product with Serializable
36.  case class MicroBatchScanExec(output: Seq[Attribute], scan: Scan, stream: MicroBatchStream, start: Offset, end: Offset) extends SparkPlan with DataSourceV2ScanExecBase with Product with Serializable

    Physical plan node for scanning a micro-batch of data from a data source.

37.  case class OverwriteByExpressionExec(table: SupportsWrite, deleteWhere: Array[Filter], writeOptions: CaseInsensitiveStringMap, query: SparkPlan, refreshCache: () ⇒ Unit) extends V2CommandExec with V2TableWriteExec with BatchWriteHelper with Product with Serializable

    Physical plan node for overwrite into a v2 table.

    Physical plan node for overwrite into a v2 table.

    Overwrites data in a table matched by a set of filters. Rows matching all of the filters will be deleted and rows in the output data set are appended.

    This plan is used to implement SaveMode.Overwrite. The behavior of SaveMode.Overwrite is to truncate the table -- delete all rows -- and append the output data set. This uses the filter AlwaysTrue to delete all rows.

38.  case class OverwriteByExpressionExecV1(table: SupportsWrite, deleteWhere: Array[Filter], writeOptions: CaseInsensitiveStringMap, plan: LogicalPlan, refreshCache: () ⇒ Unit) extends V2CommandExec with V1FallbackWriters with Product with Serializable

    Physical plan node for overwrite into a v2 table with V1 write interfaces.

    Physical plan node for overwrite into a v2 table with V1 write interfaces. Note that when this interface is used, the atomicity of the operation depends solely on the target data source.

    Overwrites data in a table matched by a set of filters. Rows matching all of the filters will be deleted and rows in the output data set are appended.

    This plan is used to implement SaveMode.Overwrite. The behavior of SaveMode.Overwrite is to truncate the table -- delete all rows -- and append the output data set. This uses the filter AlwaysTrue to delete all rows.

39.  case class OverwritePartitionsDynamicExec(table: SupportsWrite, writeOptions: CaseInsensitiveStringMap, query: SparkPlan, refreshCache: () ⇒ Unit) extends V2CommandExec with V2TableWriteExec with BatchWriteHelper with Product with Serializable

    Physical plan node for dynamic partition overwrite into a v2 table.

    Physical plan node for dynamic partition overwrite into a v2 table.

    Dynamic partition overwrite is the behavior of Hive INSERT OVERWRITE ... PARTITION queries, and Spark INSERT OVERWRITE queries when spark.sql.sources.partitionOverwriteMode=dynamic. Each partition in the output data set replaces the corresponding existing partition in the table or creates a new partition. Existing partitions for which there is no data in the output data set are not modified.

40.  class PartitionReaderFromIterator[InternalRow] extends PartitionReader[InternalRow]
41.  class PartitionReaderWithPartitionValues extends PartitionReader[InternalRow]

    A wrapper reader that always appends partition values to InternalRows produced by the input reader fileReader.

42.  class PartitionRecordReader[T] extends PartitionReader[T]
43.  case class RefreshTableExec(catalog: TableCatalog, ident: Identifier, invalidateCache: () ⇒ Unit) extends V2CommandExec with Product with Serializable
44.  case class RenameTableExec(catalog: TableCatalog, oldIdent: Identifier, newIdent: Identifier, invalidateCache: () ⇒ Option[StorageLevel], cacheTable: (DataFrame, Option[String], StorageLevel) ⇒ Unit) extends V2CommandExec with Product with Serializable

    Physical plan node for renaming a table.

45.  case class ReplaceTableAsSelectExec(catalog: TableCatalog, ident: Identifier, partitioning: Seq[Transform], plan: LogicalPlan, query: SparkPlan, properties: Map[String, String], writeOptions: CaseInsensitiveStringMap, orCreate: Boolean, invalidateCache: (TableCatalog, Table, Identifier) ⇒ Unit) extends V2CommandExec with TableWriteExecHelper with Product with Serializable

    Physical plan node for v2 replace table as select when the catalog does not support staging table replacement.

    Physical plan node for v2 replace table as select when the catalog does not support staging table replacement.

    A new table will be created using the schema of the query, and rows from the query are appended. If the table exists, its contents and schema should be replaced with the schema and the contents of the query. This is a non-atomic implementation that drops the table and then runs non-atomic CTAS. For an atomic implementation for catalogs with the appropriate support, see ReplaceTableAsSelectStagingExec.

46.  case class ReplaceTableExec(catalog: TableCatalog, ident: Identifier, tableSchema: StructType, partitioning: Seq[Transform], tableProperties: Map[String, String], orCreate: Boolean, invalidateCache: (TableCatalog, Table, Identifier) ⇒ Unit) extends V2CommandExec with Product with Serializable
47.  case class SetCatalogAndNamespaceExec(catalogManager: CatalogManager, catalogName: Option[String], namespace: Option[Seq[String]]) extends V2CommandExec with Product with Serializable

    Physical plan node for setting the current catalog and/or namespace.

48.  case class ShowCurrentNamespaceExec(output: Seq[Attribute], catalogManager: CatalogManager) extends V2CommandExec with Product with Serializable

    Physical plan node for showing current catalog/namespace.

49.  case class ShowNamespacesExec(output: Seq[Attribute], catalog: SupportsNamespaces, namespace: Seq[String], pattern: Option[String]) extends V2CommandExec with LeafExecNode with Product with Serializable

    Physical plan node for showing namespaces.

50.  case class ShowPartitionsExec(output: Seq[Attribute], catalog: TableCatalog, table: SupportsPartitionManagement, partitionSpec: Option[ResolvedPartitionSpec]) extends V2CommandExec with LeafExecNode with Product with Serializable

    Physical plan node for showing partitions.

51.  case class ShowTablePropertiesExec(output: Seq[Attribute], catalogTable: Table, propertyKey: Option[String]) extends V2CommandExec with Product with Serializable

    Physical plan node for showing table properties.

52.  case class ShowTablesExec(output: Seq[Attribute], catalog: TableCatalog, namespace: Seq[String], pattern: Option[String]) extends V2CommandExec with LeafExecNode with Product with Serializable

    Physical plan node for showing tables.

53.  trait SupportsV1Write extends SparkPlan

    A trait that allows Tables that use V1 Writer interfaces to append data.

54.  abstract class TextBasedFileScan extends FileScan
55.  sealed trait V1FallbackWriters extends V2CommandExec with SupportsV1Write

    Some helper interfaces that use V2 write semantics through the V1 writer interface.

56.  case class V1ScanWrapper(v1Scan: V1Scan, translatedFilters: Seq[Filter], handledFilters: Seq[Filter]) extends Scan with Product with Serializable
57.  abstract class V2CommandExec extends SparkPlan

    A physical operator that executes run() and saves the result to prevent multiple executions.

    A physical operator that executes run() and saves the result to prevent multiple executions. Any V2 commands that do not require triggering a spark job should extend this class.

58.  class V2SessionCatalog extends TableCatalog with SupportsNamespaces with SQLConfHelper

    A TableCatalog that translates calls to the v1 SessionCatalog.

59.  trait V2TableWriteExec extends V2CommandExec with UnaryExecNode

    The base physical plan for writing data into data source v2.

60.  case class WriteToDataSourceV2Exec(batchWrite: BatchWrite, query: SparkPlan) extends V2CommandExec with V2TableWriteExec with Product with Serializable
61.  case class WriteToDataSourceV2(batchWrite: BatchWrite, query: LogicalPlan) extends LogicalPlan with Product with Serializable

    Deprecated logical plan for writing data into data source v2.

    Deprecated logical plan for writing data into data source v2. This is being replaced by more specific logical plans, like org.apache.spark.sql.catalyst.plans.logical.AppendData.

    Annotations

    @deprecated

    Deprecated

    (Since version 2.4.0) Use specific logical plans like AppendData instead