Packages

class SparkSession extends sql.SparkSession with Logging with ColumnConversions

The entry point to programming Spark with the Dataset and DataFrame API.

In environments that this has been created upfront (e.g. REPL, notebooks), use the builder to get an existing session:

SparkSession.builder().getOrCreate()

The builder can also be used to create a new session:

SparkSession.builder
  .master("local")
  .appName("Word Count")
  .config("spark.some.config.option", "some-value")
  .getOrCreate()
Self Type
SparkSession
Annotations
@Stable()
Linear Supertypes
ColumnConversions, Logging, sql.SparkSession, Closeable, AutoCloseable, Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. SparkSession
  2. ColumnConversions
  3. Logging
  4. SparkSession
  5. Closeable
  6. AutoCloseable
  7. Serializable
  8. AnyRef
  9. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Type Members

  1. implicit class LogStringContext extends AnyRef
    Definition Classes
    Logging

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. def MDC(key: LogKey, value: Any): MDC
    Attributes
    protected
    Definition Classes
    Logging
  5. def addArtifact(source: String, target: String): Unit

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
    Annotations
    @Experimental()
  6. def addArtifact(bytes: Array[Byte], target: String): Unit

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
    Annotations
    @Experimental()
  7. def addArtifact(uri: URI): Unit

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
    Annotations
    @Experimental()
  8. def addArtifact(path: String): Unit

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
    Annotations
    @Experimental()
  9. def addArtifacts(uri: URI*): Unit

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
    Annotations
    @Experimental() @varargs()
  10. def addTag(tag: String): Unit

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  11. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  12. def baseRelationToDataFrame(baseRelation: BaseRelation): DataFrame

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  13. lazy val catalog: Catalog

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
    Annotations
    @transient()
  14. def clearTags(): Unit

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  15. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
  16. def close(): Unit

    Stop the underlying SparkContext.

    Stop the underlying SparkContext.

    Definition Classes
    SparkSession → Closeable → AutoCloseable
    Since

    2.1.0

  17. lazy val conf: RuntimeConfig

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
    Annotations
    @transient()
  18. val converter: ColumnNodeToExpressionConverter
    Attributes
    protected[sql]
    Definition Classes
    SparkSession → ColumnConversions
  19. def createDataFrame(data: List[_], beanClass: Class[_]): DataFrame

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  20. def createDataFrame(rdd: JavaRDD[_], beanClass: Class[_]): DataFrame

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  21. def createDataFrame(rdd: RDD[_], beanClass: Class[_]): DataFrame

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  22. def createDataFrame(rows: List[Row], schema: StructType): DataFrame

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
    Annotations
    @DeveloperApi()
  23. def createDataFrame(rowRDD: JavaRDD[Row], schema: StructType): DataFrame

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
    Annotations
    @DeveloperApi()
  24. def createDataFrame(rowRDD: RDD[Row], schema: StructType): DataFrame

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
    Annotations
    @DeveloperApi()
  25. def createDataFrame[A <: Product](data: Seq[A])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[A]): DataFrame

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  26. def createDataFrame[A <: Product](rdd: RDD[A])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[A]): DataFrame

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  27. def createDataset[T](data: List[T])(implicit arg0: Encoder[T]): Dataset[T]

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  28. def createDataset[T](data: RDD[T])(implicit arg0: Encoder[T]): Dataset[T]

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  29. def createDataset[T](data: Seq[T])(implicit arg0: Encoder[T]): Dataset[T]

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  30. lazy val emptyDataFrame: DataFrame

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
    Annotations
    @transient()
  31. def emptyDataset[T](implicit arg0: Encoder[T]): Dataset[T]

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  32. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  33. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  34. def executeCommand(runner: String, command: String, options: Map[String, String]): DataFrame

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
    Annotations
    @Unstable()
  35. def experimental: ExperimentalMethods

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
    Annotations
    @Experimental() @Unstable()
  36. def expression(column: Column): Expression

    Convert a Column into an Expression.

    Convert a Column into an Expression.

    Definition Classes
    ColumnConversions
    Annotations
    @DeveloperApi()
  37. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  38. def getTags(): Set[String]

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  39. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  40. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  41. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  42. def interruptAll(): Seq[String]

    Request to interrupt all currently running SQL operations of this session.

    Request to interrupt all currently running SQL operations of this session.

    returns

    Sequence of SQL execution IDs requested to be interrupted.

    Definition Classes
    SparkSession → SparkSession
    Since

    4.0.0

    Note

    Only DataFrame/SQL operations started by this session can be interrupted.

    ,

    This method will wait up to 60 seconds for the interruption request to be issued.

  43. def interruptOperation(operationId: String): Seq[String]

    Request to interrupt a SQL operation of this session, given its SQL execution ID.

    Request to interrupt a SQL operation of this session, given its SQL execution ID.

    returns

    The execution ID requested to be interrupted, as a single-element sequence, or an empty sequence if the operation is not started by this session.

    Definition Classes
    SparkSession → SparkSession
    Since

    4.0.0

    Note

    Only DataFrame/SQL operations started by this session can be interrupted.

    ,

    This method will wait up to 60 seconds for the interruption request to be issued.

  44. def interruptTag(tag: String): Seq[String]

    Request to interrupt all currently running SQL operations of this session with the given job tag.

    Request to interrupt all currently running SQL operations of this session with the given job tag.

    returns

    Sequence of SQL execution IDs requested to be interrupted.

    Definition Classes
    SparkSession → SparkSession
    Since

    4.0.0

    Note

    Only DataFrame/SQL operations started by this session can be interrupted.

    ,

    This method will wait up to 60 seconds for the interruption request to be issued.

  45. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  46. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  47. def listenerManager: ExecutionListenerManager

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  48. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  49. def logBasedOnLevel(level: Level)(f: => MessageWithContext): Unit
    Attributes
    protected
    Definition Classes
    Logging
  50. def logDebug(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  51. def logDebug(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  52. def logDebug(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  53. def logDebug(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  54. def logError(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  55. def logError(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  56. def logError(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  57. def logError(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  58. def logInfo(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  59. def logInfo(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  60. def logInfo(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  61. def logInfo(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  62. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  63. def logTrace(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  64. def logTrace(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  65. def logTrace(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  66. def logTrace(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  67. def logWarning(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  68. def logWarning(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  69. def logWarning(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  70. def logWarning(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  71. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  72. def newSession(): SparkSession

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  73. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  74. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  75. def parseDataType(dataTypeString: String): DataType

    Parses the data type in our internal string representation.

    Parses the data type in our internal string representation. The data type string should have the same format as the one generated by toString in scala. It is only used by PySpark.

    Attributes
    protected[sql]
  76. def range(start: Long, end: Long, step: Long, numPartitions: Int): Dataset[Long]

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  77. def range(start: Long, end: Long, step: Long): Dataset[Long]

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  78. def range(start: Long, end: Long): Dataset[Long]

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  79. def range(end: Long): Dataset[Long]

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  80. def read: DataFrameReader

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  81. def readStream: DataStreamReader

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  82. def removeTag(tag: String): Unit

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  83. lazy val sessionState: SessionState

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
    Annotations
    @Unstable() @transient()
  84. lazy val sharedState: SharedState

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
    Annotations
    @Unstable() @transient()
  85. val sparkContext: SparkContext
    Definition Classes
    SparkSession → SparkSession
  86. def sql(sqlText: String): DataFrame

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  87. def sql(sqlText: String, args: Map[String, Any]): DataFrame

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  88. def sql(sqlText: String, args: Map[String, Any]): DataFrame

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  89. def sql(sqlText: String, args: Array[_]): DataFrame

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  90. val sqlContext: SQLContext

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  91. def stop(): Unit
    Definition Classes
    SparkSession
  92. def streams: StreamingQueryManager

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
    Annotations
    @Unstable()
  93. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  94. def table(tableName: String): DataFrame

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  95. def time[T](f: => T): T
    Definition Classes
    SparkSession
  96. implicit def toRichColumn(column: Column): RichColumn

    Wrap a Column with a RichColumn to provide the expr and named methods.

    Wrap a Column with a RichColumn to provide the expr and named methods.

    Definition Classes
    ColumnConversions
    Annotations
    @DeveloperApi()
  97. def toString(): String
    Definition Classes
    AnyRef → Any
  98. def tvf: TableValuedFunction

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  99. def udf: UDFRegistration

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  100. def version: String

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  101. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  102. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  103. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  104. def withActive[T](block: => T): T
    Definition Classes
    SparkSession
    Annotations
    @DeveloperApi()
  105. def withLogContext(context: Map[String, String])(body: => Unit): Unit
    Attributes
    protected
    Definition Classes
    Logging
  106. object implicits extends SQLImplicits
    Definition Classes
    SparkSession → SparkSession

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable]) @Deprecated
    Deprecated

    (Since version 9)

Inherited from ColumnConversions

Inherited from Logging

Inherited from sql.SparkSession

Inherited from Closeable

Inherited from AutoCloseable

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped