execution

Type Members

1. case class AppendColumnsExec(func: (Any) ⇒ Any, deserializer: Expression, serializer: Seq[NamedExpression], child: SparkPlan) extends SparkPlan with UnaryExecNode with Product with Serializable

   Applies the given function to each input row, appending the encoded result at the end of the row.

2. case class AppendColumnsWithObjectExec(func: (Any) ⇒ Any, inputSerializer: Seq[NamedExpression], newColumnsSerializer: Seq[NamedExpression], child: SparkPlan) extends SparkPlan with ObjectConsumerExec with Product with Serializable

   An optimized version of AppendColumnsExec, that can be executed on deserialized object directly.

3. trait BaseLimitExec extends SparkPlan with UnaryExecNode with CodegenSupport

   Helper trait which defines methods that are shared by both LocalLimitExec and GlobalLimitExec.

4. case class BatchedDataSourceScanExec(output: Seq[Attribute], rdd: RDD[InternalRow], relation: BaseRelation, outputPartitioning: Partitioning, metadata: Map[String, String], metastoreTableIdentifier: Option[TableIdentifier]) extends SparkPlan with DataSourceScanExec with CodegenSupport with Product with Serializable

   Physical plan node for scanning data from a batched relation.

5. trait BinaryExecNode extends SparkPlan

6. abstract class BufferedRowIterator extends AnyRef

7. class CacheManager extends Logging

   Provides support in a SQLContext for caching query results and automatically using these cached results when subsequent queries are executed.

8. case class CachedData(plan: LogicalPlan, cachedRepresentation: InMemoryRelation) extends Product with Serializable

   Holds a cached logical plan and its data

9. case class CoGroupExec(func: (Any, Iterator[Any], Iterator[Any]) ⇒ TraversableOnce[Any], keyDeserializer: Expression, leftDeserializer: Expression, rightDeserializer: Expression, leftGroup: Seq[Attribute], rightGroup: Seq[Attribute], leftAttr: Seq[Attribute], rightAttr: Seq[Attribute], outputObjAttr: Attribute, left: SparkPlan, right: SparkPlan) extends SparkPlan with BinaryExecNode with ObjectProducerExec with Product with Serializable

   Co-groups the data from left and right children, and calls the function with each group and 2 iterators containing all elements in the group from left and right side.

10. class CoGroupedIterator extends Iterator[(InternalRow, Iterator[InternalRow], Iterator[InternalRow])]

    Iterates over GroupedIterators and returns the cogrouped data, i.

11. case class CoalesceExec(numPartitions: Int, child: SparkPlan) extends SparkPlan with UnaryExecNode with Product with Serializable

    Physical plan for returning a new RDD that has exactly numPartitions partitions.

12. class CoalescedPartitioner extends Partitioner

    A Partitioner that might group together one or more partitions from the parent.

13. trait CodegenSupport extends SparkPlan

    An interface for those physical operators that support codegen.

14. case class CollapseCodegenStages(conf: SQLConf) extends Rule[SparkPlan] with Product with Serializable

    Find the chained plans that support codegen, collapse them together as WholeStageCodegen.

15. case class CollectLimitExec(limit: Int, child: SparkPlan) extends SparkPlan with UnaryExecNode with Product with Serializable

    Take the first limit elements and collect them to a single partition.

16. trait DataSourceScanExec extends SparkPlan with LeafExecNode

17. case class DeserializeToObjectExec(deserializer: Expression, outputObjAttr: Attribute, child: SparkPlan) extends SparkPlan with UnaryExecNode with ObjectProducerExec with CodegenSupport with Product with Serializable

    Takes the input row from child and turns it into object using the given deserializer expression.

18. case class ExpandExec(projections: Seq[Seq[Expression]], output: Seq[Attribute], child: SparkPlan) extends SparkPlan with UnaryExecNode with CodegenSupport with Product with Serializable

    Apply all of the GroupExpressions to every input row, hence we will get multiple output rows for an input row.

19. trait FileRelation extends AnyRef

    An interface for relations that are backed by files.

20. case class FilterExec(condition: Expression, child: SparkPlan) extends SparkPlan with UnaryExecNode with CodegenSupport with PredicateHelper with Product with Serializable

    Physical plan for Filter.

21. case class FlatMapGroupsInRExec(func: Array[Byte], packageNames: Array[Byte], broadcastVars: Array[Broadcast[AnyRef]], inputSchema: StructType, outputSchema: StructType, keyDeserializer: Expression, valueDeserializer: Expression, groupingAttributes: Seq[Attribute], dataAttributes: Seq[Attribute], outputObjAttr: Attribute, child: SparkPlan) extends SparkPlan with UnaryExecNode with ObjectProducerExec with Product with Serializable

    Groups the input rows together and calls the R function with each group and an iterator containing all elements in the group.

22. case class GenerateExec(generator: Generator, join: Boolean, outer: Boolean, output: Seq[Attribute], child: SparkPlan) extends SparkPlan with UnaryExecNode with Product with Serializable

    Applies a Generator to a stream of input rows, combining the output of each into a new stream of rows.

23. case class GlobalLimitExec(limit: Int, child: SparkPlan) extends SparkPlan with BaseLimitExec with Product with Serializable

    Take the first limit elements of the child's single output partition.

24. class GroupedIterator extends Iterator[(InternalRow, Iterator[InternalRow])]

    Iterates over a presorted set of rows, chunking it up by the grouping expression.

25. case class InputAdapter(child: SparkPlan) extends SparkPlan with UnaryExecNode with CodegenSupport with Product with Serializable

    InputAdapter is used to hide a SparkPlan from a subtree that support codegen.

26. trait LeafExecNode extends SparkPlan

27. case class LocalLimitExec(limit: Int, child: SparkPlan) extends SparkPlan with BaseLimitExec with Product with Serializable

    Take the first limit elements of each child partition, but do not collect or shuffle them.

28. case class LocalTableScanExec(output: Seq[Attribute], rows: Seq[InternalRow]) extends SparkPlan with LeafExecNode with Product with Serializable

    Physical plan node for scanning data from a local collection.

29. case class LogicalRDD(output: Seq[Attribute], rdd: RDD[InternalRow])(session: SparkSession) extends LogicalPlan with MultiInstanceRelation with Product with Serializable

    Logical plan node for scanning data from an RDD.

30. case class MapElementsExec(func: AnyRef, outputObjAttr: Attribute, child: SparkPlan) extends SparkPlan with ObjectConsumerExec with ObjectProducerExec with CodegenSupport with Product with Serializable

    Applies the given function to each input object.

31. case class MapGroupsExec(func: (Any, Iterator[Any]) ⇒ TraversableOnce[Any], keyDeserializer: Expression, valueDeserializer: Expression, groupingAttributes: Seq[Attribute], dataAttributes: Seq[Attribute], outputObjAttr: Attribute, child: SparkPlan) extends SparkPlan with UnaryExecNode with ObjectProducerExec with Product with Serializable

    Groups the input rows together and calls the function with each group and an iterator containing all elements in the group.

32. case class MapPartitionsExec(func: (Iterator[Any]) ⇒ Iterator[Any], outputObjAttr: Attribute, child: SparkPlan) extends SparkPlan with ObjectConsumerExec with ObjectProducerExec with Product with Serializable

    Applies the given function to input object iterator.

33. trait ObjectConsumerExec extends SparkPlan with UnaryExecNode

    Physical version of ObjectConsumer.

34. trait ObjectProducerExec extends SparkPlan

    Physical version of ObjectProducer.

35. case class OutputFakerExec(output: Seq[Attribute], child: SparkPlan) extends SparkPlan with Product with Serializable

    A plan node that does nothing but lie about the output of its child.

36. case class PlanLater(plan: LogicalPlan) extends SparkPlan with LeafExecNode with Product with Serializable

37. case class PlanSubqueries(sparkSession: SparkSession) extends Rule[SparkPlan] with Product with Serializable

    Plans scalar subqueries from that are present in the given SparkPlan.

38. case class ProjectExec(projectList: Seq[NamedExpression], child: SparkPlan) extends SparkPlan with UnaryExecNode with CodegenSupport with Product with Serializable

    Physical plan for Project.

39. class QueryExecution extends AnyRef

    The primary workflow for executing relational queries using Spark.

40. class QueryExecutionException extends Exception

41. case class RDDScanExec(output: Seq[Attribute], rdd: RDD[InternalRow], nodeName: String) extends SparkPlan with LeafExecNode with Product with Serializable

    Physical plan node for scanning data from an RDD.

42. case class RangeExec(range: Range) extends SparkPlan with LeafExecNode with CodegenSupport with Product with Serializable

    Physical plan for range (generating a range of 64 bit numbers).

43. case class RowDataSourceScanExec(output: Seq[Attribute], rdd: RDD[InternalRow], relation: BaseRelation, outputPartitioning: Partitioning, metadata: Map[String, String], metastoreTableIdentifier: Option[TableIdentifier]) extends SparkPlan with DataSourceScanExec with CodegenSupport with Product with Serializable

    Physical plan node for scanning data from a relation.

44. abstract class RowIterator extends AnyRef

    An internal iterator interface which presents a more restrictive API than scala.collection.Iterator.

45. case class SampleExec(lowerBound: Double, upperBound: Double, withReplacement: Boolean, seed: Long, child: SparkPlan) extends SparkPlan with UnaryExecNode with CodegenSupport with Product with Serializable

    Physical plan for sampling the dataset.

46. case class ScalarSubquery(executedPlan: SparkPlan, exprId: ExprId) extends SubqueryExpression with Product with Serializable

    A subquery that will return only one row and one column.

47. case class SerializeFromObjectExec(serializer: Seq[NamedExpression], child: SparkPlan) extends SparkPlan with ObjectConsumerExec with CodegenSupport with Product with Serializable

    Takes the input object from child and turns in into unsafe row using the given serializer expression.

48. class ShuffledRowRDD extends RDD[InternalRow]

    This is a specialized version of org.apache.spark.rdd.ShuffledRDD that is optimized for shuffling rows instead of Java key-value pairs.

49. case class SortExec(sortOrder: Seq[SortOrder], global: Boolean, child: SparkPlan, testSpillFrequency: Int = 0) extends SparkPlan with UnaryExecNode with CodegenSupport with Product with Serializable

    Performs (external) sorting.

50. class SparkOptimizer extends Optimizer

51. abstract class SparkPlan extends QueryPlan[SparkPlan] with Logging with Serializable

    The base class for physical operators.

52. class SparkPlanInfo extends AnyRef

    :: DeveloperApi :: Stores information about a SQL SparkPlan.

53. class SparkPlanner extends SparkStrategies

54. class SparkSqlAstBuilder extends AstBuilder

    Builder that converts an ANTLR ParseTree into a LogicalPlan/Expression/TableIdentifier.

55. class SparkSqlParser extends AbstractSqlParser

    Concrete parser for Spark SQL statements.

56. abstract class SparkStrategies extends QueryPlanner[SparkPlan]

57. abstract class SparkStrategy extends GenericStrategy[SparkPlan]

    Converts a logical plan into zero or more SparkPlans.

58. case class SubqueryExec(name: String, child: SparkPlan) extends SparkPlan with UnaryExecNode with Product with Serializable

    Physical plan for a subquery.

59. case class TakeOrderedAndProjectExec(limit: Int, sortOrder: Seq[SortOrder], projectList: Seq[NamedExpression], child: SparkPlan) extends SparkPlan with UnaryExecNode with Product with Serializable

    Take the first limit elements as defined by the sortOrder, and do projection if needed.

60. trait UnaryExecNode extends SparkPlan

61. case class UnionExec(children: Seq[SparkPlan]) extends SparkPlan with Product with Serializable

    Physical plan for unioning two plans, without a distinct.

62. final class UnsafeFixedWidthAggregationMap extends AnyRef

63. final class UnsafeKVExternalSorter extends AnyRef

64. class UnsafeRowSerializer extends Serializer with Serializable

    Serializer for serializing UnsafeRows during shuffle.

65. case class WholeStageCodegenExec(child: SparkPlan) extends SparkPlan with UnaryExecNode with CodegenSupport with Product with Serializable

    WholeStageCodegen compile a subtree of plans that support codegen together into single Java function.

66. case class WindowExec(windowExpression: Seq[NamedExpression], partitionSpec: Seq[Expression], orderSpec: Seq[SortOrder], child: SparkPlan) extends SparkPlan with UnaryExecNode with Product with Serializable

    This class calculates and outputs (windowed) aggregates over the rows in a single (sorted) partition.