kotlin-spark-api_3.0.2_2.12/org.jetbrains.kotlinx.spark.api/KSparkStreamingSession

KSparkStreamingSession

class KSparkStreamingSession(val ssc: <Error class: unknown class>) : Serializable

This wrapper over SparkSession and JavaStreamingContext provides several additional methods to create org.apache.spark.sql.Dataset

Constructors

fun KSparkStreamingSession(ssc: <Error class: unknown class>)

Functions

getSpark

fun getSpark(rddForConf: <Error class: unknown class><out <Error class: unknown class>, out <Error class: unknown class>>): <Error class: unknown class>

Creates new spark session from context of given JavaRDD, rddForConf.

fun getSpark(sscForConf: <Error class: unknown class>): <Error class: unknown class>

Creates new spark session from context of given JavaStreamingContext, sscForConf

fun getSpark(sc: <Error class: unknown class>): <Error class: unknown class>

Creates new spark session from given sc.

invokeRunAfterStart

fun invokeRunAfterStart()

setRunAfterStart

fun setRunAfterStart(block: KSparkStreamingSession.() -> Unit)

block will be run after the streaming session has started from a new context (so not when loading from a checkpoint) and before it's terminated.

withSpark

fun <T> withSpark(rddForConf: <Error class: unknown class><out <Error class: unknown class>, out <Error class: unknown class>>, func: KSparkSession.() -> T): T

Helper function to enter Spark scope from a provided like when using the foreachRDD function.

fun <T> withSpark(sscForConf: <Error class: unknown class>, func: KSparkSession.() -> T): T

Helper function to enter Spark scope from sscForConf like

fun <T> withSpark(sc: <Error class: unknown class>, func: KSparkSession.() -> T): T

Helper function to enter Spark scope from sc like

Properties

ssc

val ssc: <Error class: unknown class>