SparkSessionAPI

abstract def broadcast[T](value: T)(implicit arg0: ClassTag[T]): BroadcastAPI[T]

Broadcast a read-only variable to the cluster, returning a broadcast object for reading it in distributed functions.

abstract def collectionAccumulator[T](name: String): CollectionAccumulatorAPI[T]

Create and register a CollectionAccumulator, which starts with empty list and accumulates inputs by adding them into the list.

abstract def collectionAccumulator[T]: CollectionAccumulatorAPI[T]

Create and register a CollectionAccumulator, which starts with empty list and accumulates inputs by adding them into the list.

abstract def createDataset[T](data: Seq[T])(implicit arg0: ClassTag[T], enc: Encoder[T]): DataSetAPI[T]

Create new DataSet based on Scala collection

abstract def createRDD[T](data: Seq[T])(implicit arg0: ClassTag[T]): RDDAPI[T]

Create new RDD based on Scala collection

abstract def doubleAccumulator(name: String): DoubleAccumulatorAPI

Create and register a double accumulator, which starts with 0 and accumulates inputs by add.

abstract def doubleAccumulator: DoubleAccumulatorAPI

Create and register a double accumulator, which starts with 0 and accumulates inputs by add.

abstract def longAccumulator(name: String): LongAccumulatorAPI

Create and register a long accumulator, which starts with 0 and accumulates inputs by add.

abstract def longAccumulator: LongAccumulatorAPI

Create and register a long accumulator, which starts with 0 and accumulates inputs by add.

abstract def read[T]: ReaderExecutor[T]

Returns a ReaderExecutor that can be used to read non-streaming data in as a DataSet

abstract def register(acc: AccumulatorV2API[_, _]): Unit

Register the given accumulator.

abstract def register(acc: AccumulatorV2API[_, _], name: String): Unit

Register the given accumulator with given name.

abstract def textFile(path: String, minPartitions: Int = 2): RDDAPI[String]

Read a text file from HDFS, a local file system (available on all nodes), or any Hadoop-supported file system URI, and return it as an RDD of Strings.

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: Any): Boolean

final def asInstanceOf[T0]: T0

def clone(): AnyRef

def createDataset[T](data: RDDAPI[T])(implicit arg0: ClassTag[T], enc: Encoder[T]): DataSetAPI[T]

Create new DataSet based on RDD

final def eq(arg0: AnyRef): Boolean

def equals(arg0: Any): Boolean

def finalize(): Unit

final def getClass(): Class[_]

def hashCode(): Int

final def isInstanceOf[T0]: Boolean

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

final def synchronized[T0](arg0: ⇒ T0): T0

def toString(): String

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit