DataStreamReader

Value Members

1. final def !=(arg0: Any): Boolean

   

   Definition Classes

   AnyRef → Any

2. final def ##(): Int

   

   Definition Classes

   AnyRef → Any

3. final def ==(arg0: Any): Boolean

   

   Definition Classes

   AnyRef → Any

4. final def asInstanceOf[T0]: T0

   

   Definition Classes

   Any

5. def clone(): AnyRef

   

   Attributes

   protected[java.lang]

   Definition Classes

   AnyRef

   Annotations

   @throws( ... )

6. def csv(path: String)(implicit trace: Trace): SIO[DataFrame]

   

   Loads a dataframe from a folder containing a stream of CSV files.

   Loads a dataframe from a folder containing a stream of CSV files.

   See UnderlyingDataStreamReader.csv for more information.

7. final def eq(arg0: AnyRef): Boolean

   

   Definition Classes

   AnyRef

8. def finalize(): Unit

   

   Attributes

   protected[java.lang]

   Definition Classes

   AnyRef

   Annotations

   @throws( classOf[java.lang.Throwable] )

9. final def getClass(): Class[_]

   

   Definition Classes

   AnyRef → Any

10. final def isInstanceOf[T0]: Boolean

    

    Definition Classes

    Any

11. def json(path: String)(implicit trace: Trace): SIO[DataFrame]

    

    Loads a dataframe from a folder containing a stream of JSON files.

    Loads a dataframe from a folder containing a stream of JSON files.

    See UnderlyingDataStreamReader.json for more information.

12. final def ne(arg0: AnyRef): Boolean

    

    Definition Classes

    AnyRef

13. final def notify(): Unit

    

    Definition Classes

    AnyRef

14. final def notifyAll(): Unit

    

    Definition Classes

    AnyRef

15. def option(key: String, value: Double): DataStreamReader

    

    Adds an option to the DataFrameReader.

16. def option(key: String, value: Float): DataStreamReader

    

    Adds an option to the DataFrameReader.

17. def option(key: String, value: Int): DataStreamReader

    

    Adds an option to the DataFrameReader.

18. def option(key: String, value: Boolean): DataStreamReader

    

    Adds an option to the DataFrameReader.

19. def option(key: String, value: String): DataStreamReader

    

    Adds an option to the DataFrameReader.

20. def options(options: Map[String, String]): DataStreamReader

    

    Adds multiple options to the DataFrameReader.

21. val options: Map[String, String]

    

22. def orc(path: String)(implicit trace: Trace): SIO[DataFrame]

    

    Loads a dataframe from a folder containing a stream of ORC files.

    Loads a dataframe from a folder containing a stream of ORC files.

    See UnderlyingDataStreamReader.orc for more information.

23. def parquet(path: String)(implicit trace: Trace): SIO[DataFrame]

    

    Loads a dataframe from a folder containing a stream of PARQUET files.

    Loads a dataframe from a folder containing a stream of PARQUET files.

    See UnderlyingDataStreamReader.parquet for more information.

24. def schema[T](implicit T: ToStructSchema[T]): DataStreamReader

    

    ZIO-Spark specifics function to generate the schema from a case class.

    ZIO-Spark specifics function to generate the schema from a case class.

    E.g.:

    import zio.spark._
    
    case class Person(name: String, age: Int)
    val ds: Dataset[Person] = SparkSession.read.schema[Person].csv("./path.csv").as[Person].getOrThrow

25. def schema(schemaString: String): Either[ParseException, DataStreamReader]

    

    Replace the data schema of the DataFrameReader.

    Replace the data schema of the DataFrameReader.

    We advice you to always use a schema even if some data sources can infer the schema. It allows you to increase your job speed, it ensures that the schema is the expected one and it is useful as documentation.

    E.g.:

    schema("a INT, b STRING, c DOUBLE").csv("test.csv")

    See UnderlyingDataStreamReader.schema for more information.

26. def schema(schema: StructType): DataStreamReader

    

    Replace the data schema of the DataFrameReader.

    Replace the data schema of the DataFrameReader.

    We advice you to always use a schema even if some data sources can infer the schema. It allows you to increase your job speed, it ensures that the schema is the expected one and it is useful as documentation.

    See UnderlyingDataStreamReader.schema for more information.

27. def socket(host: String, port: Int)(implicit trace: Trace): SIO[DataFrame]

    

28. final def synchronized[T0](arg0: ⇒ T0): T0

    

    Definition Classes

    AnyRef

29. def text(path: String)(implicit trace: Trace): SIO[DataFrame]

    

    Loads a dataframe from a folder containing a stream of TXT files.

    Loads a dataframe from a folder containing a stream of TXT files.

    The underlying schema of the Dataset contains a single string column named "value". The text files must be encoded as UTF-8.

    See UnderlyingDataStreamReader.textFile for more information.

30. def textFile(path: String)(implicit trace: Trace): SIO[Dataset[String]]

    

    Loads a dataset[String] from a folder containing a stream of TXT files.

    Loads a dataset[String] from a folder containing a stream of TXT files.

    See UnderlyingDataStreamReader.textFile for more information.

31. val userSpecifiedSchema: Option[StructType]

    

32. final def wait(): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

33. final def wait(arg0: Long, arg1: Int): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

34. final def wait(arg0: Long): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )