org.apache.spark.sql

execution

package execution

:: DeveloperApi :: An execution engine for relational query plans that runs on top Spark and returns RDDs.

Note that the operators in this package are created automatically by a query planner using a SQLContext and are not intended to be used directly by end users of Spark SQL. They are documented here in order to make it easier for others to understand the performance characteristics of query plans that are generated by Spark SQL.

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. execution
  2. AnyRef
  3. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Type Members

  1. case class Aggregate(partial: Boolean, groupingExpressions: Seq[Expression], aggregateExpressions: Seq[NamedExpression], child: SparkPlan) extends SparkPlan with UnaryNode with Product with Serializable

    :: DeveloperApi :: Groups input data by groupingExpressions and computes the aggregateExpressions for each group.

    :: DeveloperApi :: Groups input data by groupingExpressions and computes the aggregateExpressions for each group.

    partial

    if true then aggregation is done partially on local data without shuffling to ensure all values where groupingExpressions are equal are present.

    groupingExpressions

    expressions that are evaluated to determine grouping.

    aggregateExpressions

    expressions that are computed for each group.

    child

    the input data source.

    Annotations
    @DeveloperApi()

  2. case class AggregateEvaluation(schema: Seq[Attribute], initialValues: Seq[Expression], update: Seq[Expression], result: Expression) extends Product with Serializable

  3. case class BatchPythonEvaluation(udf: PythonUDF, output: Seq[Attribute], child: SparkPlan) extends SparkPlan with Product with Serializable

    :: DeveloperApi :: Uses PythonRDD to evaluate a PythonUDF, one partition of tuples at a time.

    :: DeveloperApi :: Uses PythonRDD to evaluate a PythonUDF, one partition of tuples at a time. The input data is zipped with the result of the udf evaluation.

    Annotations
    @DeveloperApi()

  4. case class CacheTableCommand(tableName: String, plan: Option[LogicalPlan], isLazy: Boolean) extends LogicalPlan with RunnableCommand with Product with Serializable

    :: DeveloperApi ::

    :: DeveloperApi ::

    Annotations
    @DeveloperApi()

  5. case class DescribeCommand(child: SparkPlan, output: Seq[Attribute], isExtended: Boolean) extends LogicalPlan with RunnableCommand with Product with Serializable

    :: DeveloperApi ::

    :: DeveloperApi ::

    Annotations
    @DeveloperApi()

  6. case class Distinct(partial: Boolean, child: SparkPlan) extends SparkPlan with UnaryNode with Product with Serializable

    :: DeveloperApi :: Computes the set of distinct input rows using a HashSet.

    :: DeveloperApi :: Computes the set of distinct input rows using a HashSet.

    partial

    when true the distinct operation is performed partially, per partition, without shuffling the data.

    child

    the input query plan.

    Annotations
    @DeveloperApi()

  7. case class EvaluatePython(udf: PythonUDF, child: LogicalPlan, resultAttribute: AttributeReference) extends catalyst.plans.logical.UnaryNode with Product with Serializable

    :: DeveloperApi :: Evaluates a PythonUDF, appending the result to the end of the input tuple.

    :: DeveloperApi :: Evaluates a PythonUDF, appending the result to the end of the input tuple.

    Annotations
    @DeveloperApi()

  8. case class Except(left: SparkPlan, right: SparkPlan) extends SparkPlan with BinaryNode with Product with Serializable

    :: DeveloperApi :: Returns a table with the elements from left that are not in right using the built-in spark subtract function.

    :: DeveloperApi :: Returns a table with the elements from left that are not in right using the built-in spark subtract function.

    Annotations
    @DeveloperApi()

  9. case class Exchange(newPartitioning: Partitioning, newOrdering: Seq[SortOrder], child: SparkPlan) extends SparkPlan with UnaryNode with Product with Serializable

    :: DeveloperApi :: Performs a shuffle that will result in the desired newPartitioning.

    :: DeveloperApi :: Performs a shuffle that will result in the desired newPartitioning. Optionally sorts each resulting partition based on expressions from the partition key. It is invalid to construct an exchange operator with a newOrdering that cannot be calculated using the partitioning key.

    Annotations
    @DeveloperApi()

  10. case class Expand(projections: Seq[GroupExpression], output: Seq[Attribute], child: SparkPlan) extends SparkPlan with UnaryNode with Product with Serializable

    Apply the all of the GroupExpressions to every input row, hence we will get multiple output rows for a input row.

    Apply the all of the GroupExpressions to every input row, hence we will get multiple output rows for a input row.

    projections

    The group of expressions, all of the group expressions should output the same schema specified bye the parameter output

    output

    The output Schema

    child

    Child operator

    Annotations
    @DeveloperApi()

  11. case class ExplainCommand(logicalPlan: LogicalPlan, output: Seq[Attribute] = ..., extended: Boolean = false) extends LogicalPlan with RunnableCommand with Product with Serializable

    An explain command for users to see how a command will be executed.

    An explain command for users to see how a command will be executed.

    Note that this command takes in a logical plan, runs the optimizer on the logical plan (but do NOT actually execute it).

    :: DeveloperApi ::

    Annotations
    @DeveloperApi()

  12. case class ExternalSort(sortOrder: Seq[SortOrder], global: Boolean, child: SparkPlan) extends SparkPlan with UnaryNode with Product with Serializable

    :: DeveloperApi :: Performs a sort, spilling to disk as needed.

    :: DeveloperApi :: Performs a sort, spilling to disk as needed.

    global

    when true performs a global sort of all partitions by shuffling the data first if necessary.

    Annotations
    @DeveloperApi()

  13. case class Filter(condition: Expression, child: SparkPlan) extends SparkPlan with UnaryNode with Product with Serializable

    :: DeveloperApi ::

    :: DeveloperApi ::

    Annotations
    @DeveloperApi()

  14. case class Generate(generator: Generator, join: Boolean, outer: Boolean, output: Seq[Attribute], child: SparkPlan) extends SparkPlan with UnaryNode with Product with Serializable

    :: DeveloperApi :: Applies a Generator to a stream of input rows, combining the output of each into a new stream of rows.

    :: DeveloperApi :: Applies a Generator to a stream of input rows, combining the output of each into a new stream of rows. This operation is similar to a flatMap in functional programming with one important additional feature, which allows the input rows to be joined with their output.

    generator

    the generator expression

    join

    when true, each output row is implicitly joined with the input tuple that produced it.

    outer

    when true, each input row will be output at least once, even if the output of the given generator is empty. outer has no effect when join is false.

    output

    the output attributes of this node, which constructed in analysis phase, and we can not change it, as the parent node bound with it already.

    Annotations
    @DeveloperApi()

  15. case class GeneratedAggregate(partial: Boolean, groupingExpressions: Seq[Expression], aggregateExpressions: Seq[NamedExpression], unsafeEnabled: Boolean, child: SparkPlan) extends SparkPlan with UnaryNode with Product with Serializable

    :: DeveloperApi :: Alternate version of aggregation that leverages projection and thus code generation.

    :: DeveloperApi :: Alternate version of aggregation that leverages projection and thus code generation. Aggregations are converted into a set of projections from a aggregation buffer tuple back onto itself. Currently only used for simple aggregations like SUM, COUNT, or AVERAGE are supported.

    partial

    if true then aggregation is done partially on local data without shuffling to ensure all values where groupingExpressions are equal are present.

    groupingExpressions

    expressions that are evaluated to determine grouping.

    aggregateExpressions

    expressions that are computed for each group.

    unsafeEnabled

    whether to allow Unsafe-based aggregation buffers to be used.

    child

    the input data source.

    Annotations
    @DeveloperApi()

  16. case class Intersect(left: SparkPlan, right: SparkPlan) extends SparkPlan with BinaryNode with Product with Serializable

    :: DeveloperApi :: Returns the rows in left that also appear in right using the built in spark intersection function.

    :: DeveloperApi :: Returns the rows in left that also appear in right using the built in spark intersection function.

    Annotations
    @DeveloperApi()

  17. case class Limit(limit: Int, child: SparkPlan) extends SparkPlan with UnaryNode with Product with Serializable

    :: DeveloperApi :: Take the first limit elements.

    :: DeveloperApi :: Take the first limit elements. Note that the implementation is different depending on whether this is a terminal operator or not. If it is terminal and is invoked using executeCollect, this operator uses something similar to Spark's take method on the Spark driver. If it is not terminal or is invoked using execute, we first take the limit on each partition, and then repartition all the data to a single partition to compute the global limit.

    Annotations
    @DeveloperApi()

  18. case class OutputFaker(output: Seq[Attribute], child: SparkPlan) extends SparkPlan with Product with Serializable

    :: DeveloperApi :: A plan node that does nothing but lie about the output of its child.

    :: DeveloperApi :: A plan node that does nothing but lie about the output of its child. Used to spice a (hopefully structurally equivalent) tree from a different optimization sequence into an already resolved tree.

    Annotations
    @DeveloperApi()

  19. case class Project(projectList: Seq[NamedExpression], child: SparkPlan) extends SparkPlan with UnaryNode with Product with Serializable

    :: DeveloperApi ::

    :: DeveloperApi ::

    Annotations
    @DeveloperApi()

  20. class QueryExecutionException extends Exception

  21. case class Repartition(numPartitions: Int, shuffle: Boolean, child: SparkPlan) extends SparkPlan with UnaryNode with Product with Serializable

    :: DeveloperApi :: Return a new RDD that has exactly numPartitions partitions.

    :: DeveloperApi :: Return a new RDD that has exactly numPartitions partitions.

    Annotations
    @DeveloperApi()

  22. case class Sample(lowerBound: Double, upperBound: Double, withReplacement: Boolean, seed: Long, child: SparkPlan) extends SparkPlan with UnaryNode with Product with Serializable

    :: DeveloperApi :: Sample the dataset.

    :: DeveloperApi :: Sample the dataset.

    lowerBound

    Lower-bound of the sampling probability (usually 0.0)

    upperBound

    Upper-bound of the sampling probability. The expected fraction sampled will be ub - lb.

    withReplacement

    Whether to sample with replacement.

    seed

    the random seed

    child

    the QueryPlan

    Annotations
    @DeveloperApi()

  23. case class SetCommand(kv: Option[(String, Option[String])], output: Seq[Attribute]) extends LogicalPlan with RunnableCommand with Logging with Product with Serializable

    :: DeveloperApi ::

    :: DeveloperApi ::

    Annotations
    @DeveloperApi()

  24. case class ShowTablesCommand(databaseName: Option[String]) extends LogicalPlan with RunnableCommand with Product with Serializable

    A command for users to get tables in the given database.

    A command for users to get tables in the given database. If a databaseName is not given, the current database will be used. The syntax of using this command in SQL is:

    SHOW TABLES [IN databaseName]

    :: DeveloperApi ::

    Annotations
    @DeveloperApi()

  25. case class Sort(sortOrder: Seq[SortOrder], global: Boolean, child: SparkPlan) extends SparkPlan with UnaryNode with Product with Serializable

    :: DeveloperApi :: Performs a sort on-heap.

    :: DeveloperApi :: Performs a sort on-heap.

    global

    when true performs a global sort of all partitions by shuffling the data first if necessary.

    Annotations
    @DeveloperApi()

  26. abstract class SparkPlan extends QueryPlan[SparkPlan] with Logging with Serializable

    :: DeveloperApi ::

    :: DeveloperApi ::

    Annotations
    @DeveloperApi()

  27. case class TakeOrdered(limit: Int, sortOrder: Seq[SortOrder], child: SparkPlan) extends SparkPlan with UnaryNode with Product with Serializable

    :: DeveloperApi :: Take the first limit elements as defined by the sortOrder.

    :: DeveloperApi :: Take the first limit elements as defined by the sortOrder. This is logically equivalent to having a Limit operator after a Sort operator. This could have been named TopK, but Spark's top operator does the opposite in ordering so we name it TakeOrdered to avoid confusion.

    Annotations
    @DeveloperApi()

  28. case class UncacheTableCommand(tableName: String) extends LogicalPlan with RunnableCommand with Product with Serializable

    :: DeveloperApi ::

    :: DeveloperApi ::

    Annotations
    @DeveloperApi()

  29. case class Union(children: Seq[SparkPlan]) extends SparkPlan with Product with Serializable

    :: DeveloperApi ::

    :: DeveloperApi ::

    Annotations
    @DeveloperApi()

  30. case class Window(projectList: Seq[Attribute], windowExpression: Seq[NamedExpression], windowSpec: WindowSpecDefinition, child: SparkPlan) extends SparkPlan with UnaryNode with Product with Serializable

    :: DeveloperApi :: For every row, evaluates windowExpression containing Window Functions and attaches the results with other regular expressions (presented by projectList).

    :: DeveloperApi :: For every row, evaluates windowExpression containing Window Functions and attaches the results with other regular expressions (presented by projectList). Evert operator handles a single Window Specification, windowSpec.

Value Members

  1. object ClearCacheCommand extends LogicalPlan with RunnableCommand with Product with Serializable

    :: DeveloperApi :: Clear all cached data from the in-memory cache.

    :: DeveloperApi :: Clear all cached data from the in-memory cache.

    Annotations
    @DeveloperApi()

  2. object EvaluatePython extends Serializable

  3. object Exchange extends Serializable

  4. object RDDConversions

    :: DeveloperApi ::

    :: DeveloperApi ::

    Annotations
    @DeveloperApi()

  5. object SparkPlan extends Serializable

  6. package debug

    :: DeveloperApi :: Contains methods for debugging query execution.

    :: DeveloperApi :: Contains methods for debugging query execution.

    Usage:

    import org.apache.spark.sql.execution.debug._
sql("SELECT key FROM src").debug()
dataFrame.typeCheck()

  7. package joins

    :: DeveloperApi :: Physical execution operators for join operations.

Inherited from AnyRef

Inherited from Any

Ungrouped