Type Members

1.  class AnalysisException extends Exception with SparkThrowable with Serializable with WithOrigin

   Thrown when a query fails to analyze, usually because the query itself is invalid.

   Thrown when a query fails to analyze, usually because the query itself is invalid.

   Annotations

   @Stable()

   Since

   1.3.0

2.  trait Encoder[T] extends Serializable

   Used to convert a JVM object of type T to and from the internal Spark SQL representation.

   Used to convert a JVM object of type T to and from the internal Spark SQL representation.

   Scala

   Encoders are generally created automatically through implicits from a SparkSession, or can be explicitly created by calling static methods on Encoders.

   import spark.implicits._
   
   val ds = Seq(1, 2, 3).toDS() // implicitly provided (spark.implicits.newIntEncoder)

   Java

   Encoders are specified by calling static methods on Encoders.

   List<String> data = Arrays.asList("abc", "abc", "xyz");
   Dataset<String> ds = context.createDataset(data, Encoders.STRING());

   Encoders can be composed into tuples:

   Encoder<Tuple2<Integer, String>> encoder2 = Encoders.tuple(Encoders.INT(), Encoders.STRING());
   List<Tuple2<Integer, String>> data2 = Arrays.asList(new scala.Tuple2(1, "a");
   Dataset<Tuple2<Integer, String>> ds2 = context.createDataset(data2, encoder2);

   Or constructed from Java Beans:

   Encoders.bean(MyClass.class);

   Implementation

   • Encoders should be thread-safe.

   Annotations

   @implicitNotFound( ... )

   Since

   1.6.0

3.  trait Row extends Serializable

   Represents one row of output from a relational operator.

   Represents one row of output from a relational operator. Allows both generic access by ordinal, which will incur boxing overhead for primitives, as well as native primitive access.

   It is invalid to use the native primitive interface to retrieve a value that is null, instead a user must check isNullAt before attempting to retrieve a value that might be null.

   To create a new Row, use RowFactory.create() in Java or Row.apply() in Scala.

   A Row object can be constructed by providing field values. Example:

   import org.apache.spark.sql._
   
   // Create a Row from values.
   Row(value1, value2, value3, ...)
   // Create a Row from a Seq of values.
   Row.fromSeq(Seq(value1, value2, ...))

   A value of a row can be accessed through both generic access by ordinal, which will incur boxing overhead for primitives, as well as native primitive access. An example of generic access by ordinal:

   import org.apache.spark.sql._
   
   val row = Row(1, true, "a string", null)
   // row: Row = [1,true,a string,null]
   val firstValue = row(0)
   // firstValue: Any = 1
   val fourthValue = row(3)
   // fourthValue: Any = null

   For native primitive access, it is invalid to use the native primitive interface to retrieve a value that is null, instead a user must check isNullAt before attempting to retrieve a value that might be null. An example of native primitive access:

   // using the row from the previous example.
   val firstValue = row.getInt(0)
   // firstValue: Int = 1
   val isNull = row.isNullAt(3)
   // isNull: Boolean = true

   In Scala, fields in a Row object can be extracted in a pattern match. Example:

   import org.apache.spark.sql._
   
   val pairs = sql("SELECT key, value FROM src").rdd.map {
     case Row(key: Int, value: String) =>
       key -> value
   }

   Annotations

   @Stable()

   Since

   1.3.0

4.  class RowFactory extends AnyRef

   A factory class used to construct Row objects.

   A factory class used to construct Row objects.

   Annotations

   @Stable()

   Since

   1.3.0

5.  sealed abstract final class SaveMode extends Enum[SaveMode]

   SaveMode is used to specify the expected behavior of saving a DataFrame to a data source.

   SaveMode is used to specify the expected behavior of saving a DataFrame to a data source.

   Annotations

   @Stable()

   Since

   1.3.0