datasources

Type Members

1. class CatalogFileIndex extends FileIndex

   A FileIndex for a metastore catalog table.

2. case class CreateTable(tableDesc: CatalogTable, mode: SaveMode, query: Option[LogicalPlan]) extends LogicalPlan with Product with Serializable

   Create a table and optionally insert some data into it.

3. case class CreateTempViewUsing(tableIdent: TableIdentifier, userSpecifiedSchema: Option[StructType], replace: Boolean, global: Boolean, provider: String, options: Map[String, String]) extends LeafNode with RunnableCommand with Product with Serializable

   Create or replace a local/global temporary view with given data source.

4. case class DataSource(sparkSession: SparkSession, className: String, paths: Seq[String] = immutable.this.Nil, userSpecifiedSchema: Option[StructType] = scala.None, partitionColumns: Seq[String] = collection.this.Seq.empty[Nothing], bucketSpec: Option[BucketSpec] = scala.None, options: Map[String, String] = ..., catalogTable: Option[CatalogTable] = scala.None) extends Logging with Product with Serializable

   The main class responsible for representing a pluggable Data Source in Spark SQL.

5. case class DataSourceAnalysis(conf: SQLConf) extends Rule[LogicalPlan] with CastSupport with Product with Serializable

   Replaces generic operations with specific variants that are designed to work with Spark SQL Data Sources.

6. case class DataSourceStrategy(conf: SQLConf) extends Strategy with Logging with CastSupport with Product with Serializable

   A Strategy for planning scans over data sources defined using the sources API.

7. class FailureSafeParser[IN] extends AnyRef

8. trait FileFormat extends AnyRef

   Used to read and write data stored in files to/from the InternalRow format.

9. trait FileIndex extends AnyRef

   An interface for objects capable of enumerating the root paths of a relation as well as the partitions of a relation subject to some pruning expressions.

10. case class FilePartition(index: Int, files: Seq[PartitionedFile]) extends Partition with Product with Serializable

    A collection of file blocks that should be read as a single task (possibly from multiple partitioned directories).

11. class FileScanRDD extends RDD[InternalRow]

    An RDD that scans a list of file partitions.

12. abstract class FileStatusCache extends AnyRef

    A cache of the leaf files of partition directories.

13. class FindDataSourceTable extends Rule[LogicalPlan]

    Replaces CatalogRelation with data source table if its table provider is not hive.

14. class HadoopFileLinesReader extends Iterator[Text] with Closeable

    An adaptor from a PartitionedFile to an Iterator of Text, which are all of the lines in that file.

15. case class HadoopFsRelation(location: FileIndex, partitionSchema: StructType, dataSchema: StructType, bucketSpec: Option[BucketSpec], fileFormat: FileFormat, options: Map[String, String])(sparkSession: SparkSession) extends BaseRelation with FileRelation with Product with Serializable

    Acts as a container for all of the metadata required to read from a datasource.

16. class InMemoryFileIndex extends PartitioningAwareFileIndex

    A FileIndex that generates the list of files to process by recursively listing all the files present in paths.

17. case class InsertIntoDataSourceCommand(logicalRelation: LogicalRelation, query: LogicalPlan, overwrite: Boolean) extends LeafNode with RunnableCommand with Product with Serializable

    Inserts the results of query in to a relation that extends InsertableRelation.

18. case class InsertIntoHadoopFsRelationCommand(outputPath: Path, staticPartitions: TablePartitionSpec, ifPartitionNotExists: Boolean, partitionColumns: Seq[String], bucketSpec: Option[BucketSpec], fileFormat: FileFormat, options: Map[String, String], query: LogicalPlan, mode: SaveMode, catalogTable: Option[CatalogTable], fileIndex: Option[FileIndex]) extends LeafNode with RunnableCommand with Product with Serializable

    A command for writing data to a HadoopFsRelation.

19. case class LogicalRelation(relation: BaseRelation, output: Seq[AttributeReference], catalogTable: Option[CatalogTable]) extends LeafNode with MultiInstanceRelation with Product with Serializable

    Used to link a BaseRelation in to a logical query plan.

20. abstract class OutputWriter extends AnyRef

    OutputWriter is used together with HadoopFsRelation for persisting rows to the underlying file system.

21. abstract class OutputWriterFactory extends Serializable

    A factory that produces OutputWriters.

22. case class PartitionDirectory(values: InternalRow, files: Seq[FileStatus]) extends Product with Serializable

    A collection of data files from a partitioned relation, along with the partition values in the form of an InternalRow.

23. case class PartitionPath(values: InternalRow, path: Path) extends Product with Serializable

    Holds a directory in a partitioned collection of files as well as the partition values in the form of a Row.

24. case class PartitionSpec(partitionColumns: StructType, partitions: Seq[PartitionPath]) extends Product with Serializable

25. case class PartitionedFile(partitionValues: InternalRow, filePath: String, start: Long, length: Long, locations: Array[String] = ...) extends Product with Serializable

    A part (i.

26. abstract class PartitioningAwareFileIndex extends FileIndex with Logging

    An abstract class that represents FileIndexs that are aware of partitioned tables.

27. case class PreprocessTableCreation(sparkSession: SparkSession) extends Rule[LogicalPlan] with Product with Serializable

    Preprocess CreateTable, to do some normalization and checking.

28. case class PreprocessTableInsertion(conf: SQLConf) extends Rule[LogicalPlan] with CastSupport with Product with Serializable

    Preprocess the InsertIntoTable plan.

29. class RecordReaderIterator[T] extends Iterator[T] with Closeable

    An adaptor from a Hadoop RecordReader to an Iterator over the values returned.

30. case class RefreshResource(path: String) extends LeafNode with RunnableCommand with Product with Serializable

31. case class RefreshTable(tableIdent: TableIdentifier) extends LeafNode with RunnableCommand with Product with Serializable

32. class ResolveSQLOnFile extends Rule[LogicalPlan]

    Try to replaces UnresolvedRelations if the plan is for direct query on files.

33. class SQLHadoopMapReduceCommitProtocol extends HadoopMapReduceCommitProtocol with Serializable with Logging

    A variant of HadoopMapReduceCommitProtocol that allows specifying the actual Hadoop output committer using an option specified in SQLConf.

34. case class SaveIntoDataSourceCommand(query: LogicalPlan, provider: String, partitionColumns: Seq[String], options: Map[String, String], mode: SaveMode) extends LeafNode with RunnableCommand with Product with Serializable

    Saves the results of query in to a data source.

35. abstract class TextBasedFileFormat extends FileFormat

    The base class file format that is based on text file.