Packages

object PartitionPruning extends Rule[LogicalPlan] with PredicateHelper

Dynamic partition pruning optimization is performed based on the type and selectivity of the join operation. During query optimization, we insert a predicate on the filterable table using the filter from the other side of the join and a custom wrapper called DynamicPruning.

The basic mechanism for DPP inserts a duplicated subquery with the filter from the other side, when the following conditions are met: (1) the table to prune is filterable by the JOIN key (2) the join operation is one of the following types: INNER, LEFT SEMI, LEFT OUTER (partitioned on right), or RIGHT OUTER (partitioned on left)

In order to enable partition pruning directly in broadcasts, we use a custom DynamicPruning clause that incorporates the In clause with the subquery and the benefit estimation. During query planning, when the join type is known, we use the following mechanism: (1) if the join is a broadcast hash join, we replace the duplicated subquery with the reused results of the broadcast, (2) else if the estimated benefit of partition pruning outweighs the overhead of running the subquery query twice, we keep the duplicated subquery (3) otherwise, we drop the subquery.

Linear Supertypes
PredicateHelper, AliasHelper, Rule[LogicalPlan], Logging, SQLConfHelper, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. PartitionPruning
  2. PredicateHelper
  3. AliasHelper
  4. Rule
  5. Logging
  6. SQLConfHelper
  7. AnyRef
  8. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Value Members

  1. def apply(plan: LogicalPlan): LogicalPlan
    Definition Classes
    PartitionPruning → Rule
  2. def conf: SQLConf
    Definition Classes
    SQLConfHelper
  3. def findExpressionAndTrackLineageDown(exp: Expression, plan: LogicalPlan): Option[(Expression, LogicalPlan)]
    Definition Classes
    PredicateHelper
  4. def getFilterableTableScan(a: Expression, plan: LogicalPlan): Option[LogicalPlan]

    Searches for a table scan that can be filtered for a given column in a logical plan.

    Searches for a table scan that can be filtered for a given column in a logical plan.

    This methods tries to find either a v1 partitioned scan for a given partition column or a v2 scan that support runtime filtering on a given attribute.

  5. val ruleName: String
    Definition Classes
    Rule