Class/Object

io.github.setl

SparkSessionBuilder

Related Docs: object SparkSessionBuilder | package setl

Permalink

class SparkSessionBuilder extends Builder[SparkSession]

Configure and build new sparkSession according to given usages


Usage:

// Auto-configure
val spark: SparkSession = new SparkSessionBuilder("cassandra", "postgres").build().get()

// Build with your own SparkConf
val spark: SparkSession = new SparkSessionBuilder().configure(yourSparkConf).build().get()
Annotations
@Evolving()
Linear Supertypes
Builder[SparkSession], Logging, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. SparkSessionBuilder
  2. Builder
  3. Logging
  4. AnyRef
  5. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new SparkSessionBuilder(usages: String*)

    Permalink

    usages

    usages of the sparkSession, could be a list of the following elements:

    • cassandra

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  4. def appName: String

    Permalink

    Get Spark application name

  5. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any

  6. def build(): SparkSessionBuilder.this.type

    Permalink

    Automatically build a SparkSession

    Automatically build a SparkSession

    Definition Classes
    SparkSessionBuilder → Builder

  7. def cassandraHost: String

    Permalink

    Get cassandar host value

  8. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  9. def configure(conf: SparkConf): SparkSessionBuilder.this.type

    Permalink

    Wrapper of withSparkConf

    Wrapper of withSparkConf

    conf

    spark configuration

  10. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  11. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  12. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  13. def get(): SparkSession

    Permalink

    Build a spark session with the current configuration

    Build a spark session with the current configuration

    returns

    spark session

    Definition Classes
    SparkSessionBuilder → Builder

  14. def get(key: String): String

    Permalink

    Get a SparkConf value

    Get a SparkConf value

    key

    key of spark conf

    returns

    string if the key exists, null otherwise

  15. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any

  16. def getOrCreate(): SparkSession

    Permalink
    Definition Classes
    Builder

  17. def getShufflePartitions: String

    Permalink

    Get spark.sql.shuffle.partitions

  18. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  19. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any

  20. def log: Logger

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  21. def logDebug(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  22. def logError(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  23. def logInfo(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  24. def logName: String

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  25. def logTrace(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  26. def logWarning(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  27. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  28. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef

  29. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef

  30. def registerClass(cls: Class[_]): SparkSessionBuilder.this.type

    Permalink

  31. def registerClasses(cls: Array[Class[_]]): SparkSessionBuilder.this.type

    Permalink

  32. def set(options: Map[String, String]): SparkSessionBuilder.this.type

    Permalink

  33. def set(key: String, value: String): SparkSessionBuilder.this.type

    Permalink

    Set a SparkConf property

    Set a SparkConf property

    key

    key of spark conf

    value

    value of spark conf

  34. def setAppName(name: String): SparkSessionBuilder.this.type

    Permalink

    Set the name of spark application

    Set the name of spark application

    name

    name of app

  35. def setCassandraHost(host: String): SparkSessionBuilder.this.type

    Permalink

    Set the application envir

    Set the application envir

    host

    cassandra host

  36. def setEnv(env: String): SparkSessionBuilder.this.type

    Permalink

    Set application environment

    Set application environment

    env

    LOCAL, DEV, PREPROD, PROD, EMR

  37. def setKryoRegistrationRequired(boolean: Boolean): SparkSessionBuilder.this.type

    Permalink

  38. def setShufflePartitions(par: Int): SparkSessionBuilder.this.type

    Permalink

    Set spark.sql.shuffle.partitions

    Set spark.sql.shuffle.partitions

    par

    default number of partition

  39. def setSparkMaster(url: String): SparkSessionBuilder.this.type

    Permalink

    Set Master URL for Spark

    Set Master URL for Spark

    url

    url of master

  40. def sparkMasterUrl: String

    Permalink

    Get Spark Master URL

  41. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef

  42. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any

  43. def useKryo: Boolean

    Permalink

  44. def useKryo(boo: Boolean): SparkSessionBuilder.this.type

    Permalink

  45. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  46. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  47. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  48. def withSparkConf(conf: SparkConf): SparkSessionBuilder.this.type

    Permalink

    Override the existing configuration with an user defined configuration

    Override the existing configuration with an user defined configuration

    conf

    spark configuration

Inherited from Builder[SparkSession]

Inherited from Logging

Inherited from AnyRef

Inherited from Any

Ungrouped