datasources

Type Members

1. class CaseInsensitiveMap extends Map[String, String] with Serializable

   Builds a map in which keys are case insensitive

2. case class CreateTableUsing(tableIdent: TableIdentifier, userSpecifiedSchema: Option[StructType], provider: String, temporary: Boolean, options: Map[String, String], partitionColumns: Array[String], bucketSpec: Option[BucketSpec], allowExisting: Boolean, managedIfNoPath: Boolean) extends LogicalPlan with Command with Product with Serializable

   Used to represent the operation of create table using a data source.

3. case class CreateTableUsingAsSelect(tableIdent: TableIdentifier, provider: String, partitionColumns: Array[String], bucketSpec: Option[BucketSpec], mode: SaveMode, options: Map[String, String], child: LogicalPlan) extends UnaryNode with Product with Serializable

   A node used to support CTAS statements and saveAsTable for the data source API.

4. case class CreateTempViewUsing(tableIdent: TableIdentifier, userSpecifiedSchema: Option[StructType], replace: Boolean, provider: String, options: Map[String, String]) extends LogicalPlan with RunnableCommand with Product with Serializable

5. case class DataSource(sparkSession: SparkSession, className: String, paths: Seq[String] = immutable.this.Nil, userSpecifiedSchema: Option[StructType] = scala.None, partitionColumns: Seq[String] = collection.this.Seq.empty[Nothing], bucketSpec: Option[BucketSpec] = scala.None, options: Map[String, String] = ...) extends Logging with Product with Serializable

   The main class responsible for representing a pluggable Data Source in Spark SQL.

6. trait FileCatalog extends AnyRef

   An interface for objects capable of enumerating the files that comprise a relation as well as the partitioning characteristics of those files.

7. trait FileFormat extends AnyRef

   Used to read and write data stored in files to/from the InternalRow format.

8. case class FilePartition(index: Int, files: Seq[PartitionedFile]) extends spark.Partition with Product with Serializable

   A collection of files that should be read as a single task possibly from multiple partitioned directories.

9. class FileScanRDD extends RDD[InternalRow]

10. class HadoopFileLinesReader extends Iterator[Text]

    An adaptor from a PartitionedFile to an Iterator of Text, which are all of the lines in that file.

11. case class HadoopFsRelation(sparkSession: SparkSession, location: FileCatalog, partitionSchema: StructType, dataSchema: StructType, bucketSpec: Option[BucketSpec], fileFormat: FileFormat, options: Map[String, String]) extends BaseRelation with FileRelation with Product with Serializable

    Acts as a container for all of the metadata required to read from a datasource.

12. class ListingFileCatalog extends PartitioningAwareFileCatalog

    A FileCatalog that generates the list of files to process by recursively listing all the files present in paths.

13. case class LogicalRelation(relation: BaseRelation, expectedOutputAttributes: Option[Seq[Attribute]] = scala.None, metastoreTableIdentifier: Option[TableIdentifier] = scala.None) extends LeafNode with MultiInstanceRelation with Product with Serializable

    Used to link a BaseRelation in to a logical query plan.

14. abstract class OutputWriter extends AnyRef

    ::Experimental:: OutputWriter is used together with HadoopFsRelation for persisting rows to the underlying file system.

15. abstract class OutputWriterFactory extends Serializable

    ::Experimental:: A factory that produces OutputWriters.

16. case class Partition(values: InternalRow, files: Seq[FileStatus]) extends Product with Serializable

    A collection of data files from a partitioned relation, along with the partition values in the form of an InternalRow.

17. case class PartitionedFile(partitionValues: InternalRow, filePath: String, start: Long, length: Long, locations: Array[String] = ...) extends Product with Serializable

    A single file that should be read, along with partition column values that need to be prepended to each row.

18. abstract class PartitioningAwareFileCatalog extends FileCatalog with Logging

    An abstract class that represents FileCatalogs that are aware of partitioned tables.

19. class RecordReaderIterator[T] extends Iterator[T]

    An adaptor from a Hadoop RecordReader to an Iterator over the values returned.

20. case class RefreshResource(path: String) extends LogicalPlan with RunnableCommand with Product with Serializable

21. case class RefreshTable(tableIdent: TableIdentifier) extends LogicalPlan with RunnableCommand with Product with Serializable

22. abstract class TextBasedFileFormat extends FileFormat

    The base class file format that is based on text file.

23. case class WriteRelation(sparkSession: SparkSession, dataSchema: StructType, path: String, prepareJobForWrite: (Job) ⇒ OutputWriterFactory, bucketSpec: Option[BucketSpec]) extends Product with Serializable

    A container for all the details required when writing to a table.