Packages

c

org.apache.spark.sql.execution

SparkStrategies

abstract class SparkStrategies extends QueryPlanner[SparkPlan]

Self Type
SparkPlanner
Linear Supertypes
QueryPlanner[SparkPlan], AnyRef, Any
Known Subclasses
SparkPlanner
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. SparkStrategies
  2. QueryPlanner
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new SparkStrategies()

Type Members

  1. case class StreamingGlobalLimitStrategy(outputMode: OutputMode) extends Strategy with Product with Serializable

    Used to plan the streaming global limit operator for streams in append mode.

    Used to plan the streaming global limit operator for streams in append mode. We need to check for either a direct Limit or a Limit wrapped in a ReturnAnswer operator, following the example of the SpecialLimits Strategy above.

Abstract Value Members

  1. abstract def collectPlaceholders(plan: SparkPlan): Seq[(SparkPlan, LogicalPlan)]
    Attributes
    protected
    Definition Classes
    QueryPlanner
  2. abstract def prunePlans(plans: Iterator[SparkPlan]): Iterator[SparkPlan]
    Attributes
    protected
    Definition Classes
    QueryPlanner
  3. abstract def strategies: Seq[GenericStrategy[SparkPlan]]
    Definition Classes
    QueryPlanner

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  8. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable])
  9. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  10. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  11. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  12. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  13. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  14. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  15. def plan(plan: LogicalPlan): Iterator[SparkPlan]
    Definition Classes
    SparkStrategies → QueryPlanner
  16. lazy val singleRowRdd: RDD[InternalRow]
    Attributes
    protected
  17. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  18. def toString(): String
    Definition Classes
    AnyRef → Any
  19. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  20. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  21. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  22. object Aggregation extends Strategy

    Used to plan the aggregate operator for expressions based on the AggregateFunction2 interface.

  23. object BasicOperators extends Strategy
  24. object FlatMapGroupsWithStateStrategy extends Strategy

    Strategy to convert FlatMapGroupsWithState logical operator to physical operator in streaming plans.

    Strategy to convert FlatMapGroupsWithState logical operator to physical operator in streaming plans. Conversion for batch plans is handled by BasicOperators.

  25. object InMemoryScans extends Strategy
  26. object JoinSelection extends Strategy with PredicateHelper with JoinSelectionHelper

    Select the proper physical plan for join based on join strategy hints, the availability of equi-join keys and the sizes of joining relations.

    Select the proper physical plan for join based on join strategy hints, the availability of equi-join keys and the sizes of joining relations. Below are the existing join strategies, their characteristics and their limitations.

    - Broadcast hash join (BHJ): Only supported for equi-joins, while the join keys do not need to be sortable. Supported for all join types except full outer joins. BHJ usually performs faster than the other join algorithms when the broadcast side is small. However, broadcasting tables is a network-intensive operation and it could cause OOM or perform badly in some cases, especially when the build/broadcast side is big.

    - Shuffle hash join: Only supported for equi-joins, while the join keys do not need to be sortable. Supported for all join types. Building hash map from table is a memory-intensive operation and it could cause OOM when the build side is big.

    - Shuffle sort merge join (SMJ): Only supported for equi-joins and the join keys have to be sortable. Supported for all join types.

    - Broadcast nested loop join (BNLJ): Supports both equi-joins and non-equi-joins. Supports all the join types, but the implementation is optimized for: 1) broadcasting the left side in a right outer join; 2) broadcasting the right side in a left outer, left semi, left anti or existence join; 3) broadcasting either side in an inner-like join. For other cases, we need to scan the data multiple times, which can be rather slow.

    - Shuffle-and-replicate nested loop join (a.k.a. cartesian product join): Supports both equi-joins and non-equi-joins. Supports only inner like joins.

  27. object PythonEvals extends Strategy

    Strategy to convert EvalPython logical operator to physical operator.

  28. object SparkScripts extends Strategy
  29. object SpecialLimits extends Strategy

    Plans special cases of limit operators.

  30. object StatefulAggregationStrategy extends Strategy

    Used to plan streaming aggregation queries that are computed incrementally as part of a org.apache.spark.sql.streaming.StreamingQuery.

    Used to plan streaming aggregation queries that are computed incrementally as part of a org.apache.spark.sql.streaming.StreamingQuery. Currently this rule is injected into the planner on-demand, only when planning in a org.apache.spark.sql.execution.streaming.StreamExecution

  31. object StreamingDeduplicationStrategy extends Strategy

    Used to plan the streaming deduplicate operator.

  32. object StreamingJoinStrategy extends Strategy
  33. object StreamingRelationStrategy extends Strategy

    This strategy is just for explaining Dataset/DataFrame created by spark.readStream.

    This strategy is just for explaining Dataset/DataFrame created by spark.readStream. It won't affect the execution, because StreamingRelation will be replaced with StreamingExecutionRelation in StreamingQueryManager and StreamingExecutionRelation will be replaced with the real relation using the Source in StreamExecution.

  34. object Window extends Strategy

Inherited from QueryPlanner[SparkPlan]

Inherited from AnyRef

Inherited from Any

Ungrouped