AvroHiveDialect

Value Members

1. final def !=(arg0: Any): Boolean

   

   Definition Classes

   AnyRef → Any

2. final def ##(): Int

   

   Definition Classes

   AnyRef → Any

3. final def ==(arg0: Any): Boolean

   

   Definition Classes

   AnyRef → Any

4. final def asInstanceOf[T0]: T0

   

   Definition Classes

   Any

5. def clone(): AnyRef

   

   Attributes

   protected[java.lang]

   Definition Classes

   AnyRef

   Annotations

   @throws( ... )

6. final def eq(arg0: AnyRef): Boolean

   

   Definition Classes

   AnyRef

7. def equals(arg0: Any): Boolean

   

   Definition Classes

   AnyRef → Any

8. def finalize(): Unit

   

   Attributes

   protected[java.lang]

   Definition Classes

   AnyRef

   Annotations

   @throws( classOf[java.lang.Throwable] )

9. final def getClass(): Class[_]

   

   Definition Classes

   AnyRef → Any

10. def hashCode(): Int

    

    Definition Classes

    AnyRef → Any

11. def input(path: Path, metastoreSchema: StructType, projectionSchema: StructType, predicate: Option[Predicate])(implicit fs: FileSystem, conf: Configuration): Publisher[Seq[Row]]

    
    path

    where to load the data from

    metastoreSchema

    the schema as present in the metastore and used to match up with the raw data in dialects where the schema is not present. For example with a CSV format in Hive, the metastoreSchema is required in order to know what each column represents. We can't use the projection schema for this because the projection schema might be in a different order.

    projectionSchema

    the schema required to read. This might not be the full schema present in the data but is required here because some file schemas can read data more efficiently if they know they can omit some fields (eg Parquet).

    predicate

    used by some implementations to filter data at a file read level (eg Parquet) The dataSchema represents the schema that was written for the data files. This won't necessarily be the same as the hive metastore schema, because partition values are not written to the data files. We must include this here because some hive formats don't store schema information with the data, eg delimited files. The readerSchema is the schema required by the caller which may be the same as the written data, or it may be a subset if a projection pushdown is being used.

    Definition Classes

    AvroHiveDialect → HiveDialect

12. final def isInstanceOf[T0]: Boolean

    

    Definition Classes

    Any

13. val logger: Logger

    

    Attributes

    protected

    Definition Classes

    Logging

14. final def ne(arg0: AnyRef): Boolean

    

    Definition Classes

    AnyRef

15. final def notify(): Unit

    

    Definition Classes

    AnyRef

16. final def notifyAll(): Unit

    

    Definition Classes

    AnyRef

17. def output(schema: StructType, path: Path, permission: Option[FsPermission], roundingMode: RoundingMode, metadata: Map[String, String])(implicit fs: FileSystem, conf: Configuration): HiveOutputStream

    

    Creates a new writer ready to do the bidding of the hive sink.

    Creates a new writer ready to do the bidding of the hive sink.

    schema

    the schema that will be written to the underlying file. Since this is Hive, the caller should have stripped out any partition values

    path

    the location to write the file

    permission

    optional permission to set on the file once completed

    metadata

    any metadata we wish to include in the file, this might not be supported by all filetypes

    Definition Classes

    AvroHiveDialect → HiveDialect

18. final def synchronized[T0](arg0: ⇒ T0): T0

    

    Definition Classes

    AnyRef

19. def toString(): String

    

    Definition Classes

    AnyRef → Any

20. final def wait(): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

21. final def wait(arg0: Long, arg1: Int): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

22. final def wait(arg0: Long): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )