datasources

Type Members

1. case class AnalyzeCreateTableAsSelect(sparkSession: SparkSession) extends Rule[LogicalPlan] with Product with Serializable

   Analyze the query in CREATE TABLE AS SELECT (CTAS).

2. class CaseInsensitiveMap extends Map[String, String] with Serializable

   Builds a map in which keys are case insensitive

3. case class CreateTableUsing(tableIdent: TableIdentifier, userSpecifiedSchema: Option[StructType], provider: String, temporary: Boolean, options: Map[String, String], partitionColumns: Array[String], bucketSpec: Option[BucketSpec], allowExisting: Boolean, managedIfNoPath: Boolean) extends LeafNode with Command with Product with Serializable

   Used to represent the operation of create table using a data source.

4. case class CreateTableUsingAsSelect(tableIdent: TableIdentifier, provider: String, partitionColumns: Array[String], bucketSpec: Option[BucketSpec], mode: SaveMode, options: Map[String, String], query: LogicalPlan) extends LeafNode with Command with Product with Serializable

   A node used to support CTAS statements and saveAsTable for the data source API.

5. case class CreateTempViewUsing(tableIdent: TableIdentifier, userSpecifiedSchema: Option[StructType], replace: Boolean, provider: String, options: Map[String, String]) extends LeafNode with RunnableCommand with Product with Serializable

6. case class DataSource(sparkSession: SparkSession, className: String, paths: Seq[String] = immutable.this.Nil, userSpecifiedSchema: Option[StructType] = scala.None, partitionColumns: Seq[String] = collection.this.Seq.empty[Nothing], bucketSpec: Option[BucketSpec] = scala.None, options: Map[String, String] = ...) extends Logging with Product with Serializable

   The main class responsible for representing a pluggable Data Source in Spark SQL.

7. case class DataSourceAnalysis(conf: CatalystConf) extends Rule[LogicalPlan] with Product with Serializable

   Replaces generic operations with specific variants that are designed to work with Spark SQL Data Sources.

8. trait FileCatalog extends AnyRef

   An interface for objects capable of enumerating the files that comprise a relation as well as the partitioning characteristics of those files.

9. trait FileFormat extends AnyRef

   Used to read and write data stored in files to/from the InternalRow format.

10. case class FilePartition(index: Int, files: Seq[PartitionedFile]) extends spark.Partition with Product with Serializable

    A collection of files that should be read as a single task possibly from multiple partitioned directories.

11. class FileScanRDD extends RDD[InternalRow]

12. class FindDataSourceTable extends Rule[LogicalPlan]

    Replaces SimpleCatalogRelation with data source table if its table property contains data source information.

13. class HadoopFileLinesReader extends Iterator[Text] with Closeable

    An adaptor from a PartitionedFile to an Iterator of Text, which are all of the lines in that file.

14. case class HadoopFsRelation(location: FileCatalog, partitionSchema: StructType, dataSchema: StructType, bucketSpec: Option[BucketSpec], fileFormat: FileFormat, options: Map[String, String])(sparkSession: SparkSession) extends BaseRelation with FileRelation with Product with Serializable

    Acts as a container for all of the metadata required to read from a datasource.

15. case class InsertIntoDataSourceCommand(logicalRelation: LogicalRelation, query: LogicalPlan, overwrite: Boolean) extends LeafNode with RunnableCommand with Product with Serializable

    Inserts the results of query in to a relation that extends InsertableRelation.

16. case class InsertIntoHadoopFsRelationCommand(outputPath: Path, partitionColumns: Seq[Attribute], bucketSpec: Option[BucketSpec], fileFormat: FileFormat, refreshFunction: () ⇒ Unit, options: Map[String, String], query: LogicalPlan, mode: SaveMode) extends LeafNode with RunnableCommand with Product with Serializable

    A command for writing data to a HadoopFsRelation.

17. class ListingFileCatalog extends PartitioningAwareFileCatalog

    A FileCatalog that generates the list of files to process by recursively listing all the files present in paths.

18. case class LogicalRelation(relation: BaseRelation, expectedOutputAttributes: Option[Seq[Attribute]] = scala.None, metastoreTableIdentifier: Option[TableIdentifier] = scala.None) extends LeafNode with MultiInstanceRelation with Product with Serializable

    Used to link a BaseRelation in to a logical query plan.

19. abstract class OutputWriter extends AnyRef

    ::Experimental:: OutputWriter is used together with HadoopFsRelation for persisting rows to the underlying file system.

20. abstract class OutputWriterFactory extends Serializable

    ::Experimental:: A factory that produces OutputWriters.

21. case class Partition(values: InternalRow, files: Seq[FileStatus]) extends Product with Serializable

    A collection of data files from a partitioned relation, along with the partition values in the form of an InternalRow.

22. case class PartitionDirectory(values: InternalRow, path: Path) extends Product with Serializable

    Holds a directory in a partitioned collection of files as well as as the partition values in the form of a Row.

23. case class PartitionSpec(partitionColumns: StructType, partitions: Seq[PartitionDirectory]) extends Product with Serializable

24. case class PartitionedFile(partitionValues: InternalRow, filePath: String, start: Long, length: Long, locations: Array[String] = ...) extends Product with Serializable

    A single file that should be read, along with partition column values that need to be prepended to each row.

25. abstract class PartitioningAwareFileCatalog extends FileCatalog with Logging

    An abstract class that represents FileCatalogs that are aware of partitioned tables.

26. case class PreWriteCheck(conf: SQLConf, catalog: SessionCatalog) extends (LogicalPlan) ⇒ Unit with Product with Serializable

    A rule to do various checks before inserting into or writing to a data source table.

27. case class PreprocessTableInsertion(conf: SQLConf) extends Rule[LogicalPlan] with Product with Serializable

    Preprocess the InsertIntoTable plan.

28. class RecordReaderIterator[T] extends Iterator[T] with Closeable

    An adaptor from a Hadoop RecordReader to an Iterator over the values returned.

29. case class RefreshResource(path: String) extends LeafNode with RunnableCommand with Product with Serializable

30. case class RefreshTable(tableIdent: TableIdentifier) extends LeafNode with RunnableCommand with Product with Serializable

31. class ResolveDataSource extends Rule[LogicalPlan]

    Try to replaces UnresolvedRelations with ResolveDataSource.

32. abstract class TextBasedFileFormat extends FileFormat

    The base class file format that is based on text file.