Trait

org.apache.spark.sql.execution.datasources

FileCatalog

Related Doc: package datasources

Permalink

trait FileCatalog extends AnyRef

An interface for objects capable of enumerating the files that comprise a relation as well as the partitioning characteristics of those files.

Linear Supertypes
AnyRef, Any
Known Subclasses
ListingFileCatalog, MetadataLogFileCatalog, PartitioningAwareFileCatalog
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. FileCatalog
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Abstract Value Members

  1. abstract def allFiles(): Seq[FileStatus]

    Permalink

    Returns all the valid files.

  2. abstract def listFiles(filters: Seq[Expression]): Seq[Partition]

    Permalink

    Returns all valid files grouped into partitions when the data is partitioned.

    Returns all valid files grouped into partitions when the data is partitioned. If the data is unpartitioned, this will return a single partition with no partition values.

    filters

    The filters used to prune which partitions are returned. These filters must only refer to partition columns and this method will only return files where these predicates are guaranteed to evaluate to true. Thus, these filters will not need to be evaluated again on the returned data.

  3. abstract def partitionSpec(): PartitionSpec

    Permalink

    Returns the specification of the partitions inferred from the data.

  4. abstract def paths: Seq[Path]

    Permalink

    Returns the list of input paths from which the catalog will get files.

  5. abstract def refresh(): Unit

    Permalink

    Refresh the file listing

Concrete Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  4. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any

  5. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  6. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  7. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  8. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  9. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any

  10. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  11. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any

  12. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  13. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef

  14. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef

  15. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef

  16. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any

  17. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  18. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  19. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from AnyRef

Inherited from Any

Ungrouped