object DatasetMapper extends App with Serializable
The DatasetMapper object is the entry point for mapping and processing datasets using Spark.
It leverages the DatasetMapper generic class to parse and transform text-based data into structured datasets.
This object is specifically tailored to process data of type Movie using the provided parser and configuration.
It sets up the necessary implicit Spark session and encoders, and executes the main logic for dataset transformation.
Functionality Overview:
- Initializes a Spark session configured for local execution.
- Configures the DatasetMapper to use the MovieDatabase parser for processing rows of movie data.
- Defines default handling for missing or malformed data using the Movie.missing value.
- Invokes the dataset processing logic with a specified input file containing raw text data, displaying the first 20 rows of the resulting dataset.
Note: This object is designed to run as a standalone Spark application to demonstrate dataset mapping functionality.
- Alphabetic
- By Inheritance
- DatasetMapper
- Serializable
- App
- DelayedInit
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def args: Array[String]
- Attributes
- protected
- Definition Classes
- App
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @HotSpotIntrinsicCandidate() @native()
- implicit val encoder: Encoder[Movie]
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- final val executionStart: Long
- Definition Classes
- App
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @HotSpotIntrinsicCandidate() @native()
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @HotSpotIntrinsicCandidate() @native()
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def main(args: Array[String]): Unit
- Definition Classes
- App
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @HotSpotIntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @HotSpotIntrinsicCandidate() @native()
- implicit val spark: SparkSession
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
Deprecated Value Members
- def delayedInit(body: => Unit): Unit
- Definition Classes
- App → DelayedInit
- Annotations
- @deprecated
- Deprecated
(Since version 2.11.0) the delayedInit mechanism will disappear
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)