org.apache.spark.sql.catalyst.optimizer
SetOperationPushDown
Related Doc:
package optimizer
Returns true if expr can be evaluated using only the output of plan.
Returns true if expr can be evaluated using only the output of plan. This method
can be used to determine when it is acceptable to move expression evaluation within a query
plan.
For example consider a join between two relations R(a, b) and S(c, d).
canEvaluate(EqualTo(a,b), R) returns true where as canEvaluate(EqualTo(a,c), R) returns
false.
Name for this rule, automatically inferred based on class name.
Name for this rule, automatically inferred based on class name.
Pushes certain operations to both sides of a Union, Intersect or Except operator. Operations that are safe to pushdown are listed as follows. Union: Right now, Union means UNION ALL, which does not de-duplicate rows. So, it is safe to pushdown Filters and Projections through it. Once we add UNION DISTINCT, we will not be able to pushdown Projections.
Intersect: It is not safe to pushdown Projections through it because we need to get the intersect of rows by comparing the entire rows. It is fine to pushdown Filters with deterministic condition.
Except: It is not safe to pushdown Projections through it because we need to get the intersect of rows by comparing the entire rows. It is fine to pushdown Filters with deterministic condition.