Packages

class SparkPlanner extends SparkStrategies with SQLConfHelper

Linear Supertypes
SQLConfHelper, SparkStrategies, QueryPlanner[SparkPlan], AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. SparkPlanner
  2. SQLConfHelper
  3. SparkStrategies
  4. QueryPlanner
  5. AnyRef
  6. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new SparkPlanner(session: SparkSession, experimentalMethods: ExperimentalMethods)

Type Members

  1. case class StreamingGlobalLimitStrategy(outputMode: OutputMode) extends Strategy with Product with Serializable

    Used to plan the streaming global limit operator for streams in append mode.

    Used to plan the streaming global limit operator for streams in append mode. We need to check for either a direct Limit or a Limit wrapped in a ReturnAnswer operator, following the example of the SpecialLimits Strategy above.

    Definition Classes
    SparkStrategies

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @native()
  6. def collectPlaceholders(plan: SparkPlan): Seq[(SparkPlan, LogicalPlan)]
    Attributes
    protected
    Definition Classes
    SparkPlanner → QueryPlanner
  7. def conf: SQLConf
    Definition Classes
    SQLConfHelper
  8. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  9. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  10. val experimentalMethods: ExperimentalMethods
  11. def extraPlanningStrategies: Seq[Strategy]

    Override to add extra planning strategies to the planner.

    Override to add extra planning strategies to the planner. These strategies are tried after the strategies defined in ExperimentalMethods, and before the regular strategies.

  12. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable])
  13. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  14. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  15. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  16. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  17. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  18. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  19. def numPartitions: Int
  20. def plan(plan: LogicalPlan): Iterator[SparkPlan]
    Definition Classes
    SparkStrategies → QueryPlanner
  21. def pruneFilterProject(projectList: Seq[NamedExpression], filterPredicates: Seq[Expression], prunePushedDownFilters: (Seq[Expression]) => Seq[Expression], scanBuilder: (Seq[Attribute]) => SparkPlan): SparkPlan

    Used to build table scan operators where complex projection and filtering are done using separate physical operators.

    Used to build table scan operators where complex projection and filtering are done using separate physical operators. This function returns the given scan operator with Project and Filter nodes added only when needed. For example, a Project operator is only used when the final desired output requires complex expressions to be evaluated or when columns can be further eliminated out after filtering has been done.

    The prunePushedDownFilters parameter is used to remove those filters that can be optimized away by the filter pushdown optimization.

    The required attributes for both filtering and expression evaluation are passed to the provided scanBuilder function so that it can avoid unnecessary column materialization.

  22. def prunePlans(plans: Iterator[SparkPlan]): Iterator[SparkPlan]
    Attributes
    protected
    Definition Classes
    SparkPlanner → QueryPlanner
  23. val session: SparkSession
  24. lazy val singleRowRdd: RDD[InternalRow]
    Attributes
    protected
    Definition Classes
    SparkStrategies
  25. def strategies: Seq[Strategy]
    Definition Classes
    SparkPlanner → QueryPlanner
  26. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  27. def toString(): String
    Definition Classes
    AnyRef → Any
  28. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  29. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  30. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  31. object Aggregation extends Strategy

    Used to plan the aggregate operator for expressions based on the AggregateFunction2 interface.

    Used to plan the aggregate operator for expressions based on the AggregateFunction2 interface.

    Definition Classes
    SparkStrategies
  32. object BasicOperators extends Strategy
    Definition Classes
    SparkStrategies
  33. object FlatMapGroupsWithStateStrategy extends Strategy

    Strategy to convert FlatMapGroupsWithState logical operator to physical operator in streaming plans.

    Strategy to convert FlatMapGroupsWithState logical operator to physical operator in streaming plans. Conversion for batch plans is handled by BasicOperators.

    Definition Classes
    SparkStrategies
  34. object InMemoryScans extends Strategy
    Definition Classes
    SparkStrategies
  35. object JoinSelection extends Strategy with PredicateHelper with JoinSelectionHelper

    Select the proper physical plan for join based on join strategy hints, the availability of equi-join keys and the sizes of joining relations.

    Select the proper physical plan for join based on join strategy hints, the availability of equi-join keys and the sizes of joining relations. Below are the existing join strategies, their characteristics and their limitations.

    - Broadcast hash join (BHJ): Only supported for equi-joins, while the join keys do not need to be sortable. Supported for all join types except full outer joins. BHJ usually performs faster than the other join algorithms when the broadcast side is small. However, broadcasting tables is a network-intensive operation and it could cause OOM or perform badly in some cases, especially when the build/broadcast side is big.

    - Shuffle hash join: Only supported for equi-joins, while the join keys do not need to be sortable. Supported for all join types. Building hash map from table is a memory-intensive operation and it could cause OOM when the build side is big.

    - Shuffle sort merge join (SMJ): Only supported for equi-joins and the join keys have to be sortable. Supported for all join types.

    - Broadcast nested loop join (BNLJ): Supports both equi-joins and non-equi-joins. Supports all the join types, but the implementation is optimized for: 1) broadcasting the left side in a right outer join; 2) broadcasting the right side in a left outer, left semi, left anti or existence join; 3) broadcasting either side in an inner-like join. For other cases, we need to scan the data multiple times, which can be rather slow.

    - Shuffle-and-replicate nested loop join (a.k.a. cartesian product join): Supports both equi-joins and non-equi-joins. Supports only inner like joins.

    Definition Classes
    SparkStrategies
  36. object PythonEvals extends Strategy

    Strategy to convert EvalPython logical operator to physical operator.

    Strategy to convert EvalPython logical operator to physical operator.

    Definition Classes
    SparkStrategies
  37. object SparkScripts extends Strategy
    Definition Classes
    SparkStrategies
  38. object SpecialLimits extends Strategy

    Plans special cases of limit operators.

    Plans special cases of limit operators.

    Definition Classes
    SparkStrategies
  39. object StatefulAggregationStrategy extends Strategy

    Used to plan streaming aggregation queries that are computed incrementally as part of a org.apache.spark.sql.streaming.StreamingQuery.

    Used to plan streaming aggregation queries that are computed incrementally as part of a org.apache.spark.sql.streaming.StreamingQuery. Currently this rule is injected into the planner on-demand, only when planning in a org.apache.spark.sql.execution.streaming.StreamExecution

    Definition Classes
    SparkStrategies
  40. object StreamingDeduplicationStrategy extends Strategy

    Used to plan the streaming deduplicate operator.

    Used to plan the streaming deduplicate operator.

    Definition Classes
    SparkStrategies
  41. object StreamingJoinStrategy extends Strategy
    Definition Classes
    SparkStrategies
  42. object StreamingRelationStrategy extends Strategy

    This strategy is just for explaining Dataset/DataFrame created by spark.readStream.

    This strategy is just for explaining Dataset/DataFrame created by spark.readStream. It won't affect the execution, because StreamingRelation will be replaced with StreamingExecutionRelation in StreamingQueryManager and StreamingExecutionRelation will be replaced with the real relation using the Source in StreamExecution.

    Definition Classes
    SparkStrategies
  43. object Window extends Strategy
    Definition Classes
    SparkStrategies

Inherited from SQLConfHelper

Inherited from SparkStrategies

Inherited from QueryPlanner[SparkPlan]

Inherited from AnyRef

Inherited from Any

Ungrouped