Packages

class SparkSession extends sql.SparkSession with Logging with ColumnConversions

The entry point to programming Spark with the Dataset and DataFrame API.

In environments that this has been created upfront (e.g. REPL, notebooks), use the builder to get an existing session:

SparkSession.builder().getOrCreate()

The builder can also be used to create a new session:

SparkSession.builder
  .master("local")
  .appName("Word Count")
  .config("spark.some.config.option", "some-value")
  .getOrCreate()
Self Type
SparkSession
Annotations
@Stable()
Linear Supertypes
ColumnConversions, Logging, sql.SparkSession, Closeable, AutoCloseable, Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. SparkSession
  2. ColumnConversions
  3. Logging
  4. SparkSession
  5. Closeable
  6. AutoCloseable
  7. Serializable
  8. AnyRef
  9. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Type Members

  1. implicit class LogStringContext extends AnyRef
    Definition Classes
    Logging

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. def addArtifact(source: String, target: String): Unit

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
    Annotations
    @Experimental()
  5. def addArtifact(bytes: Array[Byte], target: String): Unit

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
    Annotations
    @Experimental()
  6. def addArtifact(uri: URI): Unit

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
    Annotations
    @Experimental()
  7. def addArtifact(path: String): Unit

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
    Annotations
    @Experimental()
  8. def addArtifacts(uri: URI*): Unit

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
    Annotations
    @Experimental() @varargs()
  9. def addTag(tag: String): Unit

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  10. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  11. def baseRelationToDataFrame(baseRelation: BaseRelation): DataFrame

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  12. lazy val catalog: Catalog

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
    Annotations
    @transient()
  13. def clearTags(): Unit

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  14. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
  15. def close(): Unit

    Stop the underlying SparkContext.

    Stop the underlying SparkContext.

    Definition Classes
    SparkSession → Closeable → AutoCloseable
    Since

    2.1.0

  16. lazy val conf: RuntimeConfig

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
    Annotations
    @transient()
  17. val converter: ColumnNodeToExpressionConverter
    Attributes
    protected[sql]
    Definition Classes
    SparkSession → ColumnConversions
  18. def createDataFrame(data: List[_], beanClass: Class[_]): DataFrame

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  19. def createDataFrame(rdd: JavaRDD[_], beanClass: Class[_]): DataFrame

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  20. def createDataFrame(rdd: RDD[_], beanClass: Class[_]): DataFrame

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  21. def createDataFrame(rows: List[Row], schema: StructType): DataFrame

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
    Annotations
    @DeveloperApi()
  22. def createDataFrame(rowRDD: JavaRDD[Row], schema: StructType): DataFrame

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
    Annotations
    @DeveloperApi()
  23. def createDataFrame(rowRDD: RDD[Row], schema: StructType): DataFrame

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
    Annotations
    @DeveloperApi()
  24. def createDataFrame[A <: Product](data: Seq[A])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[A]): DataFrame

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  25. def createDataFrame[A <: Product](rdd: RDD[A])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[A]): DataFrame

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  26. def createDataset[T](data: List[T])(implicit arg0: Encoder[T]): Dataset[T]

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  27. def createDataset[T](data: RDD[T])(implicit arg0: Encoder[T]): Dataset[T]

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  28. def createDataset[T](data: Seq[T])(implicit arg0: Encoder[T]): Dataset[T]

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  29. lazy val emptyDataFrame: DataFrame

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
    Annotations
    @transient()
  30. def emptyDataset[T](implicit arg0: Encoder[T]): Dataset[T]

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  31. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  32. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  33. def executeCommand(runner: String, command: String, options: Map[String, String]): DataFrame

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
    Annotations
    @Unstable()
  34. def experimental: ExperimentalMethods

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
    Annotations
    @Experimental() @Unstable()
  35. def expression(column: Column): Expression

    Convert a Column into an Expression.

    Convert a Column into an Expression.

    Definition Classes
    ColumnConversions
    Annotations
    @DeveloperApi()
  36. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  37. def getTags(): Set[String]

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  38. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  39. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  40. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  41. def interruptAll(): Seq[String]

    Request to interrupt all currently running SQL operations of this session.

    Request to interrupt all currently running SQL operations of this session.

    returns

    Sequence of SQL execution IDs requested to be interrupted.

    Definition Classes
    SparkSession → SparkSession
    Since

    4.0.0

    Note

    Only DataFrame/SQL operations started by this session can be interrupted.

    ,

    This method will wait up to 60 seconds for the interruption request to be issued.

  42. def interruptOperation(operationId: String): Seq[String]

    Request to interrupt a SQL operation of this session, given its SQL execution ID.

    Request to interrupt a SQL operation of this session, given its SQL execution ID.

    returns

    The execution ID requested to be interrupted, as a single-element sequence, or an empty sequence if the operation is not started by this session.

    Definition Classes
    SparkSession → SparkSession
    Since

    4.0.0

    Note

    Only DataFrame/SQL operations started by this session can be interrupted.

    ,

    This method will wait up to 60 seconds for the interruption request to be issued.

  43. def interruptTag(tag: String): Seq[String]

    Request to interrupt all currently running SQL operations of this session with the given job tag.

    Request to interrupt all currently running SQL operations of this session with the given job tag.

    returns

    Sequence of SQL execution IDs requested to be interrupted.

    Definition Classes
    SparkSession → SparkSession
    Since

    4.0.0

    Note

    Only DataFrame/SQL operations started by this session can be interrupted.

    ,

    This method will wait up to 60 seconds for the interruption request to be issued.

  44. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  45. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  46. def listenerManager: ExecutionListenerManager

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  47. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  48. def logDebug(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  49. def logDebug(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  50. def logDebug(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  51. def logDebug(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  52. def logError(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  53. def logError(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  54. def logError(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  55. def logError(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  56. def logInfo(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  57. def logInfo(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  58. def logInfo(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  59. def logInfo(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  60. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  61. def logTrace(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  62. def logTrace(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  63. def logTrace(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  64. def logTrace(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  65. def logWarning(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  66. def logWarning(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  67. def logWarning(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  68. def logWarning(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  69. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  70. def newSession(): SparkSession

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  71. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  72. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  73. def parseDataType(dataTypeString: String): DataType

    Parses the data type in our internal string representation.

    Parses the data type in our internal string representation. The data type string should have the same format as the one generated by toString in scala. It is only used by PySpark.

    Attributes
    protected[sql]
  74. def range(start: Long, end: Long, step: Long, numPartitions: Int): Dataset[Long]

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  75. def range(start: Long, end: Long, step: Long): Dataset[Long]

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  76. def range(start: Long, end: Long): Dataset[Long]

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  77. def range(end: Long): Dataset[Long]

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  78. def read: DataFrameReader

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  79. def readStream: DataStreamReader

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  80. def removeTag(tag: String): Unit

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  81. lazy val sessionState: SessionState

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
    Annotations
    @Unstable() @transient()
  82. lazy val sharedState: SharedState

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
    Annotations
    @Unstable() @transient()
  83. val sparkContext: SparkContext
    Definition Classes
    SparkSession → SparkSession
  84. def sql(sqlText: String): DataFrame

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  85. def sql(sqlText: String, args: Map[String, Any]): DataFrame

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  86. def sql(sqlText: String, args: Map[String, Any]): DataFrame

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  87. def sql(sqlText: String, args: Array[_]): DataFrame

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  88. val sqlContext: SQLContext

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  89. def stop(): Unit
    Definition Classes
    SparkSession
  90. def streams: StreamingQueryManager

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
    Annotations
    @Unstable()
  91. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  92. def table(tableName: String): DataFrame

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  93. def time[T](f: => T): T
    Definition Classes
    SparkSession
  94. implicit def toRichColumn(column: Column): RichColumn

    Wrap a Column with a RichColumn to provide the expr and named methods.

    Wrap a Column with a RichColumn to provide the expr and named methods.

    Definition Classes
    ColumnConversions
    Annotations
    @DeveloperApi()
  95. def toString(): String
    Definition Classes
    AnyRef → Any
  96. def tvf: TableValuedFunction

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  97. def udf: UDFRegistration

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  98. def version: String

    <invalid inheritdoc annotation>

    <invalid inheritdoc annotation>

    Definition Classes
    SparkSession → SparkSession
  99. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  100. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  101. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  102. def withActive[T](block: => T): T
    Definition Classes
    SparkSession
    Annotations
    @DeveloperApi()
  103. def withLogContext(context: Map[String, String])(body: => Unit): Unit
    Attributes
    protected
    Definition Classes
    Logging
  104. object implicits extends SQLImplicits
    Definition Classes
    SparkSession → SparkSession

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable]) @Deprecated
    Deprecated

    (Since version 9)

Inherited from ColumnConversions

Inherited from Logging

Inherited from sql.SparkSession

Inherited from Closeable

Inherited from AutoCloseable

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped