sql

Type Members

1. class Column extends Logging

   

   :: Experimental :: A column in a DataFrame.

   :: Experimental :: A column in a DataFrame.

   Annotations

   @Experimental()

   Since

   1.3.0

2. class ColumnName extends Column

   

   :: Experimental :: A convenient class used for constructing schema.

   :: Experimental :: A convenient class used for constructing schema.

   Annotations

   @Experimental()

   Since

   1.3.0

3. class DataFrame extends Serializable

   

   :: Experimental :: A distributed collection of data organized into named columns.

   :: Experimental :: A distributed collection of data organized into named columns.

   A DataFrame is equivalent to a relational table in Spark SQL. The following example creates a DataFrame by pointing Spark SQL to a Parquet data set.

   val people = sqlContext.read.parquet("...")  // in Scala
   DataFrame people = sqlContext.read().parquet("...")  // in Java

   Once created, it can be manipulated using the various domain-specific-language (DSL) functions defined in: DataFrame (this class), Column, and functions.

   To select a column from the data frame, use apply method in Scala and col in Java.

   val ageCol = people("age")  // in Scala
   Column ageCol = people.col("age")  // in Java

   Note that the Column type can also be manipulated through its various functions.

   // The following creates a new column that increases everybody's age by 10.
   people("age") + 10  // in Scala
   people.col("age").plus(10);  // in Java

   A more concrete example in Scala:

   // To create DataFrame using SQLContext
   val people = sqlContext.read.parquet("...")
   val department = sqlContext.read.parquet("...")
   
   people.filter("age > 30")
     .join(department, people("deptId") === department("id"))
     .groupBy(department("name"), "gender")
     .agg(avg(people("salary")), max(people("age")))

   and in Java:

   // To create DataFrame using SQLContext
   DataFrame people = sqlContext.read().parquet("...");
   DataFrame department = sqlContext.read().parquet("...");
   
   people.filter("age".gt(30))
     .join(department, people.col("deptId").equalTo(department("id")))
     .groupBy(department.col("name"), "gender")
     .agg(avg(people.col("salary")), max(people.col("age")));

   Annotations

   @Experimental()

   Since

   1.3.0

4. final class DataFrameNaFunctions extends AnyRef

   

   :: Experimental :: Functionality for working with missing data in DataFrames.

   :: Experimental :: Functionality for working with missing data in DataFrames.

   Annotations

   @Experimental()

   Since

   1.3.1

5. class DataFrameReader extends Logging

   

   :: Experimental :: Interface used to load a DataFrame from external storage systems (e.g.

   :: Experimental :: Interface used to load a DataFrame from external storage systems (e.g. file systems, key-value stores, etc). Use SQLContext.read to access this.

   Annotations

   @Experimental()

   Since

   1.4.0

6. final class DataFrameStatFunctions extends AnyRef

   

   :: Experimental :: Statistic functions for DataFrames.

   :: Experimental :: Statistic functions for DataFrames.

   Annotations

   @Experimental()

   Since

   1.4.0

7. final class DataFrameWriter extends AnyRef

   

   :: Experimental :: Interface used to write a DataFrame to external storage systems (e.g.

   :: Experimental :: Interface used to write a DataFrame to external storage systems (e.g. file systems, key-value stores, etc). Use DataFrame.write to access this.

   Annotations

   @Experimental()

   Since

   1.4.0

8. class ExperimentalMethods extends AnyRef

   

   :: Experimental :: Holder for experimental methods for the bravest.

   :: Experimental :: Holder for experimental methods for the bravest. We make NO guarantee about the stability regarding binary compatibility and source compatibility of methods here.

   sqlContext.experimental.extraStrategies += ...

   Annotations

   @Experimental()

   Since

   1.3.0

9. class GroupedData extends AnyRef

   

   :: Experimental :: A set of methods for aggregations on a DataFrame, created by DataFrame.groupBy.

   :: Experimental :: A set of methods for aggregations on a DataFrame, created by DataFrame.groupBy.

   Annotations

   @Experimental()

   Since

   1.3.0

10. class SQLContext extends Logging with Serializable

    

    The entry point for working with structured data (rows and columns) in Spark.

    The entry point for working with structured data (rows and columns) in Spark. Allows the creation of DataFrame objects as well as the execution of SQL queries.

    Since

    1.0.0

11. final class SaveMode extends Enum[SaveMode]

    

12. type Strategy = GenericStrategy[SparkPlan]

    

    Converts a logical plan into zero or more SparkPlans.

    Converts a logical plan into zero or more SparkPlans. This API is exposed for experimenting with the query planner and is not designed to be stable across spark releases. Developers writing libraries should instead consider using the stable APIs provided in org.apache.spark.sql.sources

    Annotations

    @DeveloperApi()

13. class UDFRegistration extends Logging

    

    Functions for registering user-defined functions.

    Functions for registering user-defined functions. Use SQLContext.udf to access this.

    Since

    1.3.0

14. case class UserDefinedFunction(f: AnyRef, dataType: DataType, inputTypes: Seq[DataType] = Nil) extends Product with Serializable

    

    A user-defined function.

    A user-defined function. To create one, use the udf functions in functions. As an example:

    // Defined a UDF that returns true or false based on some numeric score.
    val predict = udf((score: Double) => if (score > 0.5) true else false)
    
    // Projects a column that adds a prediction column based on the score column.
    df.select( predict(df("score")) )

    Annotations

    @Experimental()

    Since

    1.3.0

15. type SchemaRDD = DataFrame

    

    Type alias for DataFrame.

    Type alias for DataFrame. Kept here for backward source compatibility for Scala.

    Annotations

    @deprecated

    Deprecated

    (Since version 1.3.0) use DataFrame