Class/Object

org.apache.spark.sql.catalyst.catalog

CatalogTable

Related Docs: object CatalogTable | package catalog

Permalink

case class CatalogTable(identifier: TableIdentifier, tableType: CatalogTableType, storage: CatalogStorageFormat, schema: StructType, provider: Option[String] = None, partitionColumnNames: Seq[String] = Seq.empty, bucketSpec: Option[BucketSpec] = None, owner: String = "", createTime: Long = System.currentTimeMillis, lastAccessTime: Long = 1, properties: Map[String, String] = Map.empty, stats: Option[CatalogStatistics] = None, viewText: Option[String] = None, comment: Option[String] = None, unsupportedFeatures: Seq[String] = Seq.empty, tracksPartitionsInCatalog: Boolean = false, schemaPreservesCase: Boolean = true) extends Product with Serializable

A table defined in the catalog.

Note that Hive's metastore also tracks skewed columns. We should consider adding that in the future once we have a better understanding of how we want to handle skewed columns.

provider

the name of the data source provider for this table, e.g. parquet, json, etc. Can be None if this table is a View, should be "hive" for hive serde tables.

unsupportedFeatures

is a list of string descriptions of features that are used by the underlying table but not supported by Spark SQL yet.

tracksPartitionsInCatalog

whether this table's partition metadata is stored in the catalog. If false, it is inferred automatically based on file structure.

schemaPreservesCase

Whether or not the schema resolved for this table is case-sensitive. When using a Hive Metastore, this flag is set to false if a case- sensitive schema was unable to be read from the table properties. Used to trigger case-sensitive schema inference at query time, when configured.

Linear Supertypes
Serializable, Serializable, Product, Equals, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. CatalogTable
  2. Serializable
  3. Serializable
  4. Product
  5. Equals
  6. AnyRef
  7. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new CatalogTable(identifier: TableIdentifier, tableType: CatalogTableType, storage: CatalogStorageFormat, schema: StructType, provider: Option[String] = None, partitionColumnNames: Seq[String] = Seq.empty, bucketSpec: Option[BucketSpec] = None, owner: String = "", createTime: Long = System.currentTimeMillis, lastAccessTime: Long = 1, properties: Map[String, String] = Map.empty, stats: Option[CatalogStatistics] = None, viewText: Option[String] = None, comment: Option[String] = None, unsupportedFeatures: Seq[String] = Seq.empty, tracksPartitionsInCatalog: Boolean = false, schemaPreservesCase: Boolean = true)

    Permalink

    provider

    the name of the data source provider for this table, e.g. parquet, json, etc. Can be None if this table is a View, should be "hive" for hive serde tables.

    unsupportedFeatures

    is a list of string descriptions of features that are used by the underlying table but not supported by Spark SQL yet.

    tracksPartitionsInCatalog

    whether this table's partition metadata is stored in the catalog. If false, it is inferred automatically based on file structure.

    schemaPreservesCase

    Whether or not the schema resolved for this table is case-sensitive. When using a Hive Metastore, this flag is set to false if a case- sensitive schema was unable to be read from the table properties. Used to trigger case-sensitive schema inference at query time, when configured.

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  4. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any

  5. val bucketSpec: Option[BucketSpec]

    Permalink

  6. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  7. val comment: Option[String]

    Permalink

  8. val createTime: Long

    Permalink

  9. def dataSchema: StructType

    Permalink

    schema of this table's data columns

  10. def database: String

    Permalink

    Return the database this table was specified to belong to, assuming it exists.

  11. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  12. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  13. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any

  14. val identifier: TableIdentifier

    Permalink

  15. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any

  16. val lastAccessTime: Long

    Permalink

  17. def location: URI

    Permalink

    Return the table location, assuming it is specified.

  18. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  19. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef

  20. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef

  21. val owner: String

    Permalink

  22. val partitionColumnNames: Seq[String]

    Permalink

  23. def partitionSchema: StructType

    Permalink

    schema of this table's partition columns

  24. val properties: Map[String, String]

    Permalink

  25. val provider: Option[String]

    Permalink

    the name of the data source provider for this table, e.g.

    the name of the data source provider for this table, e.g. parquet, json, etc. Can be None if this table is a View, should be "hive" for hive serde tables.

  26. def qualifiedName: String

    Permalink

    Return the fully qualified name of this table, assuming the database was specified.

  27. val schema: StructType

    Permalink

  28. val schemaPreservesCase: Boolean

    Permalink

    Whether or not the schema resolved for this table is case-sensitive.

    Whether or not the schema resolved for this table is case-sensitive. When using a Hive Metastore, this flag is set to false if a case- sensitive schema was unable to be read from the table properties. Used to trigger case-sensitive schema inference at query time, when configured.

  29. def simpleString: String

    Permalink

    Readable string representation for the CatalogTable.

  30. val stats: Option[CatalogStatistics]

    Permalink

  31. val storage: CatalogStorageFormat

    Permalink

  32. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef

  33. val tableType: CatalogTableType

    Permalink

  34. def toLinkedHashMap: LinkedHashMap[String, String]

    Permalink

  35. def toString(): String

    Permalink
    Definition Classes
    CatalogTable → AnyRef → Any

  36. val tracksPartitionsInCatalog: Boolean

    Permalink

    whether this table's partition metadata is stored in the catalog.

    whether this table's partition metadata is stored in the catalog. If false, it is inferred automatically based on file structure.

  37. val unsupportedFeatures: Seq[String]

    Permalink

    is a list of string descriptions of features that are used by the underlying table but not supported by Spark SQL yet.

  38. def viewDefaultDatabase: Option[String]

    Permalink

    Return the default database name we use to resolve a view, should be None if the CatalogTable is not a View or created by older versions of Spark(before 2.2.0).

  39. def viewQueryColumnNames: Seq[String]

    Permalink

    Return the output column names of the query that creates a view, the column names are used to resolve a view, should be empty if the CatalogTable is not a View or created by older versions of Spark(before 2.2.0).

  40. val viewText: Option[String]

    Permalink

  41. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  42. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  43. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  44. def withNewStorage(locationUri: Option[URI] = storage.locationUri, inputFormat: Option[String] = storage.inputFormat, outputFormat: Option[String] = storage.outputFormat, compressed: Boolean = false, serde: Option[String] = storage.serde, properties: Map[String, String] = storage.properties): CatalogTable

    Permalink

    Syntactic sugar to update a field in storage.

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from AnyRef

Inherited from Any

Ungrouped