optimizer

Value Members

1. object BooleanSimplification extends Rule[LogicalPlan] with PredicateHelper

   

   Simplifies boolean expressions: 1.

   Simplifies boolean expressions: 1. Simplifies expressions whose answer can be determined without evaluating both sides. 2. Eliminates / extracts common factors. 3. Merge same expressions 4. Removes Not operator.

2. object ColumnPruning extends Rule[LogicalPlan]

   

   Attempts to eliminate the reading of unneeded columns from the query plan using the following transformations:

   Attempts to eliminate the reading of unneeded columns from the query plan using the following transformations:

   • Inserting Projections beneath the following operators:
     • Aggregate
     • Generate
     • Project <- Join
     • LeftSemiJoin

3. object CombineFilters extends Rule[LogicalPlan]

   

   Combines two adjacent Filter operators into one, merging the conditions into one conjunctive predicate.

4. object CombineLimits extends Rule[LogicalPlan]

   

   Combines two adjacent Limit operators into one, merging the expressions into one single expression.

5. object ConstantFolding extends Rule[LogicalPlan]

   

   Replaces Expressions that can be statically evaluated with equivalent Literal values.

6. object ConvertToLocalRelation extends Rule[LogicalPlan]

   

   Converts local operations (i.e.

   Converts local operations (i.e. ones that don't require data exchange) on LocalRelation to another LocalRelation.

   This is relatively simple as it currently handles only a single case: Project.

7. object DecimalAggregates extends Rule[LogicalPlan]

   

   Speeds up aggregates on fixed-precision decimals by executing them on unscaled Long values.

   Speeds up aggregates on fixed-precision decimals by executing them on unscaled Long values.

   This uses the same rules for increasing the precision and scale of the output as org.apache.spark.sql.catalyst.analysis.HiveTypeCoercion.DecimalPrecision.

8. object DefaultOptimizer extends Optimizer

   

9. object LikeSimplification extends Rule[LogicalPlan]

   

   Simplifies LIKE expressions that do not need full regular expressions to evaluate the condition.

   Simplifies LIKE expressions that do not need full regular expressions to evaluate the condition. For example, when the expression is just checking to see if a string starts with a given pattern.

10. object NullPropagation extends Rule[LogicalPlan]

    

    Replaces Expressions that can be statically evaluated with equivalent Literal values.

    Replaces Expressions that can be statically evaluated with equivalent Literal values. This rule is more specific with Null value propagation from bottom to top of the expression tree.

11. object OptimizeIn extends Rule[LogicalPlan]

    

    Replaces (value, seq[Literal]) with optimized version(value, HashSet[Literal]) which is much faster

12. object ProjectCollapsing extends Rule[LogicalPlan]

    

    Combines two adjacent Project operators into one and perform alias substitution, merging the expressions into one single expression.

13. object PushPredicateThroughGenerate extends Rule[LogicalPlan] with PredicateHelper

    

    Push Filter operators through Generate operators.

    Push Filter operators through Generate operators. Parts of the predicate that reference attributes generated in Generate will remain above, and the rest should be pushed beneath.

14. object PushPredicateThroughJoin extends Rule[LogicalPlan] with PredicateHelper

    

    Pushes down Filter operators where the condition can be evaluated using only the attributes of the left or right side of a join.

    Pushes down Filter operators where the condition can be evaluated using only the attributes of the left or right side of a join. Other Filter conditions are moved into the condition of the Join.

    And also Pushes down the join filter, where the condition can be evaluated using only the attributes of the left or right side of sub query when applicable.

    Check https://cwiki.apache.org/confluence/display/Hive/OuterJoinBehavior for more details

15. object PushPredicateThroughProject extends Rule[LogicalPlan] with PredicateHelper

    

    Pushes Filter operators through Project operators, in-lining any Aliases that were defined in the projection.

    Pushes Filter operators through Project operators, in-lining any Aliases that were defined in the projection.

    This heuristic is valid assuming the expression evaluation cost is minimal.

16. object RemoveLiteralFromGroupExpressions extends Rule[LogicalPlan]

    

    Removes literals from group expressions in Aggregate, as they have no effect to the result but only makes the grouping key bigger.

17. object RemovePositive extends Rule[LogicalPlan]

    

    Removes UnaryPositive identify function

18. object ReplaceDistinctWithAggregate extends Rule[LogicalPlan]

    

    Replaces logical Distinct operator with an Aggregate operator.

    Replaces logical Distinct operator with an Aggregate operator.

    SELECT DISTINCT f1, f2 FROM t  ==>  SELECT f1, f2 FROM t GROUP BY f1, f2

19. object SamplePushDown extends Rule[LogicalPlan]

    

    Pushes operations down into a Sample.

20. object SetOperationPushDown extends Rule[LogicalPlan] with PredicateHelper

    

    Pushes certain operations to both sides of a Union, Intersect or Except operator.

    Pushes certain operations to both sides of a Union, Intersect or Except operator. Operations that are safe to pushdown are listed as follows. Union: Right now, Union means UNION ALL, which does not de-duplicate rows. So, it is safe to pushdown Filters and Projections through it. Once we add UNION DISTINCT, we will not be able to pushdown Projections.

    Intersect: It is not safe to pushdown Projections through it because we need to get the intersect of rows by comparing the entire rows. It is fine to pushdown Filters with deterministic condition.

    Except: It is not safe to pushdown Projections through it because we need to get the intersect of rows by comparing the entire rows. It is fine to pushdown Filters with deterministic condition.

21. object SimplifyCaseConversionExpressions extends Rule[LogicalPlan]

    

    Removes the inner case conversion expressions that are unnecessary because the inner conversion is overwritten by the outer one.

22. object SimplifyCasts extends Rule[LogicalPlan]

    

    Removes Casts that are unnecessary because the input is already the correct type.

23. object SimplifyFilters extends Rule[LogicalPlan]

    

    Removes filters that can be evaluated trivially.

    Removes filters that can be evaluated trivially. This is done either by eliding the filter for cases where it will always evaluate to true, or substituting a dummy empty relation when the filter will always evaluate to false.