Packages

package adaptive

Ordering
  1. Alphabetic
Visibility
  1. Public
  2. Protected

Type Members

  1. class AQEOptimizer extends RuleExecutor[LogicalPlan]

    The optimizer for re-optimizing the logical plan used by AdaptiveSparkPlanExec.

  2. case class AQEShuffleReadExec extends SparkPlan with UnaryExecNode with Product with Serializable

    A wrapper of shuffle query stage, which follows the given partition arrangement.

  3. trait AQEShuffleReadRule extends Rule[SparkPlan]

    A rule that may create AQEShuffleReadExec on top of ShuffleQueryStageExec and change the plan output partitioning.

    A rule that may create AQEShuffleReadExec on top of ShuffleQueryStageExec and change the plan output partitioning. The AQE framework will skip the rule if it leads to extra shuffles.

  4. case class AdaptiveExecutionContext(session: SparkSession, qe: QueryExecution) extends Product with Serializable

    The execution context shared between the main query and all sub-queries.

  5. case class AdaptiveSparkPlanExec(inputPlan: SparkPlan, context: AdaptiveExecutionContext, preprocessingRules: Seq[Rule[SparkPlan]], isSubquery: Boolean, supportsColumnar: Boolean = false) extends SparkPlan with LeafExecNode with Product with Serializable

    A root node to execute the query plan adaptively.

    A root node to execute the query plan adaptively. It splits the query plan into independent stages and executes them in order according to their dependencies. The query stage materializes its output at the end. When one stage completes, the data statistics of the materialized output will be used to optimize the remainder of the query.

    To create query stages, we traverse the query tree bottom up. When we hit an exchange node, and if all the child query stages of this exchange node are materialized, we create a new query stage for this exchange node. The new stage is then materialized asynchronously once it is created.

    When one query stage finishes materialization, the rest query is re-optimized and planned based on the latest statistics provided by all materialized stages. Then we traverse the query plan again and create more stages if possible. After all stages have been materialized, we execute the rest of the plan.

  6. trait AdaptiveSparkPlanHelper extends AnyRef

    This class provides utility methods related to tree traversal of an AdaptiveSparkPlanExec plan.

    This class provides utility methods related to tree traversal of an AdaptiveSparkPlanExec plan. Unlike their counterparts in org.apache.spark.sql.catalyst.trees.TreeNode or org.apache.spark.sql.catalyst.plans.QueryPlan, these methods traverse down leaf nodes of adaptive plans, i.e., AdaptiveSparkPlanExec and QueryStageExec.

  7. case class BroadcastQueryStageExec(id: Int, plan: SparkPlan, _canonicalized: SparkPlan) extends QueryStageExec with Product with Serializable

    A broadcast query stage whose child is a BroadcastExchangeLike or ReusedExchangeExec.

    A broadcast query stage whose child is a BroadcastExchangeLike or ReusedExchangeExec.

    id

    the query stage id.

    plan

    the underlying plan.

    _canonicalized

    the canonicalized plan before applying query stage optimizer rules.

  8. case class CoalesceShufflePartitions(session: SparkSession) extends Rule[SparkPlan] with AQEShuffleReadRule with Product with Serializable

    A rule to coalesce the shuffle partitions based on the map output statistics, which can avoid many small reduce tasks that hurt performance.

  9. trait Cost extends Ordered[Cost]

    An interface to represent the cost of a plan.

    An interface to represent the cost of a plan.

    Annotations
    @Unstable()
    Note

    This class is subject to be changed and/or moved in the near future.

  10. trait CostEvaluator extends AnyRef

    An interface to evaluate the cost of a physical plan.

    An interface to evaluate the cost of a physical plan.

    Annotations
    @Unstable()
    Note

    This class is subject to be changed and/or moved in the near future.

  11. case class InsertAdaptiveSparkPlan(adaptiveExecutionContext: AdaptiveExecutionContext) extends Rule[SparkPlan] with Product with Serializable

    This rule wraps the query plan with an AdaptiveSparkPlanExec, which executes the query plan and re-optimize the plan during execution based on runtime data statistics.

    This rule wraps the query plan with an AdaptiveSparkPlanExec, which executes the query plan and re-optimize the plan during execution based on runtime data statistics.

    Note that this rule is stateful and thus should not be reused across query executions.

  12. case class LogicalQueryStage(logicalPlan: LogicalPlan, physicalPlan: SparkPlan) extends LogicalPlan with LeafNode with Product with Serializable

    The LogicalPlan wrapper for a QueryStageExec, or a snippet of physical plan containing a QueryStageExec, in which all ancestor nodes of the QueryStageExec are linked to the same logical node.

    The LogicalPlan wrapper for a QueryStageExec, or a snippet of physical plan containing a QueryStageExec, in which all ancestor nodes of the QueryStageExec are linked to the same logical node.

    For example, a logical Aggregate can be transformed into FinalAgg - Shuffle - PartialAgg, in which the Shuffle will be wrapped into a QueryStageExec, thus the LogicalQueryStage will have FinalAgg - QueryStageExec as its physical plan.

  13. case class OptimizeSkewedJoin(ensureRequirements: EnsureRequirements) extends Rule[SparkPlan] with Product with Serializable

    A rule to optimize skewed joins to avoid straggler tasks whose share of data are significantly larger than those of the rest of the tasks.

    A rule to optimize skewed joins to avoid straggler tasks whose share of data are significantly larger than those of the rest of the tasks.

    The general idea is to divide each skew partition into smaller partitions and replicate its matching partition on the other side of the join so that they can run in parallel tasks. Note that when matching partitions from the left side and the right side both have skew, it will become a cartesian product of splits from left and right joining together.

    For example, assume the Sort-Merge join has 4 partitions: left: [L1, L2, L3, L4] right: [R1, R2, R3, R4]

    Let's say L2, L4 and R3, R4 are skewed, and each of them get split into 2 sub-partitions. This is scheduled to run 4 tasks at the beginning: (L1, R1), (L2, R2), (L3, R3), (L4, R4). This rule expands it to 9 tasks to increase parallelism: (L1, R1), (L2-1, R2), (L2-2, R2), (L3, R3-1), (L3, R3-2), (L4-1, R4-1), (L4-2, R4-1), (L4-1, R4-2), (L4-2, R4-2)

  14. case class PlanAdaptiveDynamicPruningFilters(rootPlan: AdaptiveSparkPlanExec) extends Rule[SparkPlan] with AdaptiveSparkPlanHelper with Product with Serializable

    A rule to insert dynamic pruning predicates in order to reuse the results of broadcast.

  15. case class PlanAdaptiveSubqueries(subqueryMap: Map[Long, BaseSubqueryExec]) extends Rule[SparkPlan] with Product with Serializable
  16. abstract class QueryStageExec extends SparkPlan with LeafExecNode

    A query stage is an independent subgraph of the query plan.

    A query stage is an independent subgraph of the query plan. Query stage materializes its output before proceeding with further operators of the query plan. The data statistics of the materialized output can be used to optimize subsequent query stages.

    There are 2 kinds of query stages:

    1. Shuffle query stage. This stage materializes its output to shuffle files, and Spark launches another job to execute the further operators. 2. Broadcast query stage. This stage materializes its output to an array in driver JVM. Spark broadcasts the array before executing the further operators.
  17. case class ReuseAdaptiveSubquery(reuseMap: TrieMap[SparkPlan, BaseSubqueryExec]) extends Rule[SparkPlan] with Product with Serializable
  18. case class ShuffleQueryStageExec(id: Int, plan: SparkPlan, _canonicalized: SparkPlan) extends QueryStageExec with Product with Serializable

    A shuffle query stage whose child is a ShuffleExchangeLike or ReusedExchangeExec.

    A shuffle query stage whose child is a ShuffleExchangeLike or ReusedExchangeExec.

    id

    the query stage id.

    plan

    the underlying plan.

    _canonicalized

    the canonicalized plan before applying query stage optimizer rules.

  19. case class SimpleCost(value: Long) extends Cost with Product with Serializable

    A simple implementation of Cost, which takes a number of Long as the cost value.

  20. case class SimpleCostEvaluator(forceOptimizeSkewedJoin: Boolean) extends CostEvaluator with Product with Serializable

    A skew join aware implementation of CostEvaluator, which counts the number of ShuffleExchangeLike nodes and skew join nodes in the plan.

  21. case class SkewJoinChildWrapper(plan: SparkPlan) extends SparkPlan with LeafExecNode with Product with Serializable
  22. case class StageFailure(stage: QueryStageExec, error: Throwable) extends StageMaterializationEvent with Product with Serializable

    The materialization of a query stage hit an error and failed.

  23. sealed trait StageMaterializationEvent extends AnyRef

    The event type for stage materialization.

  24. case class StageSuccess(stage: QueryStageExec, result: Any) extends StageMaterializationEvent with Product with Serializable

    The materialization of a query stage completed with success.

Value Members

  1. object AQEPropagateEmptyRelation extends PropagateEmptyRelationBase

    This rule runs in the AQE optimizer and optimizes more cases compared to PropagateEmptyRelationBase: 1.

    This rule runs in the AQE optimizer and optimizes more cases compared to PropagateEmptyRelationBase: 1. Join is single column NULL-aware anti join (NAAJ) Broadcasted HashedRelation is HashedRelationWithAllNullKeys. Eliminate join to an empty LocalRelation.

  2. object AQEUtils
  3. object AdaptiveSparkPlanExec extends Serializable
  4. object CostEvaluator extends Logging
  5. object DynamicJoinSelection extends Rule[LogicalPlan] with JoinSelectionHelper

    This optimization rule includes three join selection:

    This optimization rule includes three join selection:

    1. detects a join child that has a high ratio of empty partitions and adds a NO_BROADCAST_HASH hint to avoid it being broadcast, as shuffle join is faster in this case: many tasks complete immediately since one join side is empty. 2. detects a join child that every partition size is less than local map threshold and adds a PREFER_SHUFFLE_HASH hint to encourage being shuffle hash join instead of sort merge join. 3. if a join satisfies both NO_BROADCAST_HASH and PREFER_SHUFFLE_HASH, then add a SHUFFLE_HASH hint.
  6. object LogicalQueryStageStrategy extends Strategy with PredicateHelper

    Strategy for plans containing LogicalQueryStage nodes: 1.

    Strategy for plans containing LogicalQueryStage nodes: 1. Transforms LogicalQueryStage to its corresponding physical plan that is either being executed or has already completed execution. 2. Transforms Join which has one child relation already planned and executed as a BroadcastQueryStageExec. This is to prevent reversing a broadcast stage into a shuffle stage in case of the larger join child relation finishes before the smaller relation. Note that this rule needs to be applied before regular join strategies.

  7. object OptimizeShuffleWithLocalRead extends Rule[SparkPlan] with AQEShuffleReadRule

    A rule to optimize the shuffle read to local read iff no additional shuffles will be introduced: 1.

    A rule to optimize the shuffle read to local read iff no additional shuffles will be introduced: 1. if the input plan is a shuffle, add local read directly as we can never introduce extra shuffles in this case. 2. otherwise, add local read to the probe side of broadcast hash join and then run EnsureRequirements to check whether additional shuffle introduced. If introduced, we will revert all the local reads.

  8. object OptimizeSkewInRebalancePartitions extends Rule[SparkPlan] with AQEShuffleReadRule

    A rule to optimize the skewed shuffle partitions in RebalancePartitions based on the map output statistics, which can avoid data skew that hurt performance.

    A rule to optimize the skewed shuffle partitions in RebalancePartitions based on the map output statistics, which can avoid data skew that hurt performance.

    We use ADVISORY_PARTITION_SIZE_IN_BYTES size to decide if a partition should be optimized. Let's say we have 3 maps with 3 shuffle partitions, and assuming r1 has data skew issue. the map side looks like: m0:[b0, b1, b2], m1:[b0, b1, b2], m2:[b0, b1, b2] and the reduce side looks like: (without this rule) r1[m0-b1, m1-b1, m2-b1] / \ r0:[m0-b0, m1-b0, m2-b0], r1-0:[m0-b1], r1-1:[m1-b1], r1-2:[m2-b1], r2[m0-b2, m1-b2, m2-b2]

  9. object ShufflePartitionsUtil extends Logging

Ungrouped