Class

com.vbounyasit.bigdata.testing.formats.DataFrameIO

DataFrameLoader

Related Doc: package DataFrameIO

Permalink

abstract class DataFrameLoader extends AnyRef

Linear Supertypes
AnyRef, Any
Known Subclasses
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. DataFrameLoader
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new DataFrameLoader(fileExtension: String)

    Permalink

Abstract Value Members

  1. abstract def convertFileToDataFrame(filePath: String)(implicit spark: SparkSession): DataFrame

    Permalink

    Converts a given resource file to a DataFrame

    Converts a given resource file to a DataFrame

    filePath

    the path to the file

    spark

    an implicit spark session

    returns

    the resulting DataFrame

    Attributes
    protected
  2. abstract def loadDataFrames(jobTablesMetadata: Seq[JobTableMetadata], ioType: String)(implicit spark: SparkSession): List[Either[ReadDataFramesFromFilesError, (JobTableMetadata, DataFrame)]]

    Permalink

    Loads DataFrames from Resource files for a given job

    Loads DataFrames from Resource files for a given job

    jobTablesMetadata

    the list of sources

    returns

    A Map of sourceInfo/DataFrame values

Concrete Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  5. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  6. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  7. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  8. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  9. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  10. final def getFileNotFoundException(jobTableMetadata: JobTableMetadata): ResourcesDataFileNotFound

    Permalink

    Builds the exception object for a file that is not found

    Builds the exception object for a file that is not found

    jobTableMetadata

    the Job table metadata

    returns

    the built resource not found exception

  11. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  12. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  13. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  14. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  15. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  16. final def readDfFromFile(jobTableMetadata: JobTableMetadata, ioType: String)(implicit spark: SparkSession): Either[ReadDataFramesFromFilesError, (JobTableMetadata, Option[DataFrame])]

    Permalink

    Reads a json file into a DataFrame

    Reads a json file into a DataFrame

    jobTableMetadata

    the jobSource info

    spark

    the implicit spark session

    returns

    the converted json to DataFrame

    Attributes
    protected
  17. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  18. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any
  19. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  20. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  21. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from AnyRef

Inherited from Any

Ungrouped