Object

it.agilelab.bigdata.wasp.consumers.spark

SparkSingletons

Related Doc: package spark

Permalink

object SparkSingletons extends Logging

Singletons an initialization code related to Spark.

While Spark provides various getOrCreate methods, this allows WASP to initialize Spark in a cleaner way and in just a few places and access the various entry points in a much more straightforward way.

Linear Supertypes
Logging, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. SparkSingletons
  2. Logging
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  5. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  6. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  7. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  8. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  9. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  10. def getSQLContext: SQLContext

    Permalink

    Returns the SQLContext singleton, or throws an exception if Spark was not initialized.

    Returns the SQLContext singleton, or throws an exception if Spark was not initialized.

    Annotations
    @throws( ... )
    Exceptions thrown

    IllegalStateException if Spark was not already initialized

  11. def getSparkContext: SparkContext

    Permalink

    Returns the SparkContext singleton, or throws an exception if Spark was not initialized.

    Returns the SparkContext singleton, or throws an exception if Spark was not initialized.

    Annotations
    @throws( ... )
    Exceptions thrown

    IllegalStateException if Spark was not already initialized

  12. def getSparkSession: SparkSession

    Permalink

    Returns the SparkSession singleton, or throws an exception if Spark was not initialized.

    Returns the SparkSession singleton, or throws an exception if Spark was not initialized.

    Annotations
    @throws( ... )
    Exceptions thrown

    IllegalStateException if Spark was not already initialized

  13. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  14. def initializeSpark(sparkConfigModel: SparkConfigModel, telemetryConfig: TelemetryConfigModel, kafkaConfigModel: KafkaConfigModel): Boolean

    Permalink

    Try to initialize the SparkSession in the SparkSingleton with the provided configuration.

    Try to initialize the SparkSession in the SparkSingleton with the provided configuration.

    If it does not exist, it will be created using the settings from sparkConfigModel and true will be returned.

    If the SparkSession already exists, nothing will be done, and false will be returned.

    Annotations
    @throws( ... )
    Exceptions thrown

    IllegalStateException if Spark was already initialized but not by using this method

  15. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  16. val logger: WaspLogger

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  17. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  18. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  19. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  20. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  21. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any
  22. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  23. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  24. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from Logging

Inherited from AnyRef

Inherited from Any

Ungrouped