SparkApp

During the development lifecycle of Spark applications, it is useful to create sandbox environments comprising paths and Hive databases etc. which are tied to specific logical environments (e.g. dev, test, prod) and feature development (i.e Git branches). e.g. when working on a feature called new_feature for a project called my_project, the application should write its data to paths under /data/dev/my_project/new_feature/ and create tables in a database called dev_my_project_new_feature (actual implementation of what these environments should look like can be defined by extending Env or one of its subclasses - the final implementation should be a case class whose values define the environment i.e env, branch etc.)

This is a generic Spark Application which uses an implementation of Env to generate application-specific configuration and subsequently parse this configuration into a case class to be used for the application logic.

E: the type of the Env implementation (must be a case class)

Linear Supertypes

AnyRef, Any

Known Subclasses

WaimakApp

Instance Constructors

new SparkApp()(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[E])

Abstract Value Members

abstract def confDefaults(env: E): Map[String, String]

Default Spark configuration values to use for the application
Default Spark configuration values to use for the application
env
the environment
returns
a map containing default Spark configuration
abstract def run(sparkSession: SparkSession, env: E): Unit

Run the application for given environment and configuration case classes
Run the application for given environment and configuration case classes
sparkSession
the SparkSession
env
the environment

Attributes
protected

Concrete Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def asInstanceOf[T0]: T0

Definition Classes
Any
def cleanupEnv(sparkSession: SparkSession, envPrefix: String): Unit

Cleans up the environment associated with this application
Cleans up the environment associated with this application
sparkSession
the SparkSession
envPrefix
the prefix for keys in the SparkConf needed by the Env implementation
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
def createEnv(sparkSession: SparkSession, envPrefix: String): Unit

Create the environment associated with this application
Create the environment associated with this application
sparkSession
the SparkSession
envPrefix
the prefix for keys in the SparkConf needed by the Env implementation
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def equals(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
def hashCode(): Int

Definition Classes
AnyRef → Any
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
def parseEnv(sparkSession: SparkSession, envPrefix: String): E

Parses configuration in the SparkSession into the environment case class (type E)
Parses configuration in the SparkSession into the environment case class (type E)
sparkSession
the SparkSession
envPrefix
the prefix for keys in the SparkConf needed by the Env implementation
returns
a parsed case class of type E
def runSparkApp(sparkSession: SparkSession, envPrefix: String): Unit

Runs the application
Runs the application
N.B does not create the environment - use createEnv
sparkSession
the SparkSession
envPrefix
the prefix for keys in the SparkConf needed by the Env implementation
def runWithEnv(env: E, sparkSession: SparkSession): Unit

Attributes
protected
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
def toString(): String

Definition Classes
AnyRef → Any
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )

Related Doc: package app

abstract class SparkApp[E <: Env] extends AnyRef

Instance Constructors

new SparkApp()(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[E])

Abstract Value Members

abstract def confDefaults(env: E): Map[String, String]

abstract def run(sparkSession: SparkSession, env: E): Unit

Concrete Value Members

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: Any): Boolean

final def asInstanceOf[T0]: T0

def cleanupEnv(sparkSession: SparkSession, envPrefix: String): Unit

def clone(): AnyRef

def createEnv(sparkSession: SparkSession, envPrefix: String): Unit

final def eq(arg0: AnyRef): Boolean

def equals(arg0: Any): Boolean

def finalize(): Unit

final def getClass(): Class[_]

def hashCode(): Int

final def isInstanceOf[T0]: Boolean

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

def parseEnv(sparkSession: SparkSession, envPrefix: String): E

def runSparkApp(sparkSession: SparkSession, envPrefix: String): Unit

def runWithEnv(env: E, sparkSession: SparkSession): Unit

final def synchronized[T0](arg0: ⇒ T0): T0

def toString(): String

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

Inherited from AnyRef

Inherited from Any

Ungrouped