package exchange
- Alphabetic
- Public
- Protected
Type Members
- case class BroadcastExchangeExec(mode: BroadcastMode, child: SparkPlan) extends Exchange with BroadcastExchangeLike with Product with Serializable
A BroadcastExchangeExec collects, transforms and finally broadcasts the result of a transformed SparkPlan.
- trait BroadcastExchangeLike extends Exchange
Common trait for all broadcast exchange implementations to facilitate pattern matching.
- case class EnsureRequirements(optimizeOutRepartition: Boolean = true, requiredDistribution: Option[Distribution] = None) extends Rule[SparkPlan] with Product with Serializable
Ensures that the Partitioning of input data meets the Distribution requirements for each operator by inserting ShuffleExchangeExec Operators where required.
Ensures that the Partitioning of input data meets the Distribution requirements for each operator by inserting ShuffleExchangeExec Operators where required. Also ensure that the input partition ordering requirements are met.
- optimizeOutRepartition
A flag to indicate that if this rule should optimize out user-specified repartition shuffles or not. This is mostly true, but can be false in AQE when AQE optimization may change the plan output partitioning and need to retain the user-specified repartition shuffles in the plan.
- requiredDistribution
The root required distribution we should ensure. This value is used in AQE in case we change final stage output partitioning.
- abstract class Exchange extends SparkPlan with UnaryExecNode
Base class for operators that exchange data among multiple threads or processes.
Base class for operators that exchange data among multiple threads or processes.
Exchanges are the key class of operators that enable parallelism. Although the implementation differs significantly, the concept is similar to the exchange operator described in "Volcano -- An Extensible and Parallel Query Evaluation System" by Goetz Graefe.
- case class ReusedExchangeExec(output: Seq[Attribute], child: Exchange) extends SparkPlan with LeafExecNode with Product with Serializable
A wrapper for reused exchange to have different output, because two exchanges which produce logically identical output will have distinct sets of output attribute ids, so we need to preserve the original ids because they're what downstream operators are expecting.
- case class ShuffleExchangeExec(outputPartitioning: Partitioning, child: SparkPlan, shuffleOrigin: ShuffleOrigin = ENSURE_REQUIREMENTS) extends Exchange with ShuffleExchangeLike with Product with Serializable
Performs a shuffle that will result in the desired partitioning.
- trait ShuffleExchangeLike extends Exchange
Common trait for all shuffle exchange implementations to facilitate pattern matching.
- sealed trait ShuffleOrigin extends AnyRef
Value Members
- object BroadcastExchangeExec extends Serializable
- case object ENSURE_REQUIREMENTS extends ShuffleOrigin with Product with Serializable
- case object REBALANCE_PARTITIONS_BY_COL extends ShuffleOrigin with Product with Serializable
- case object REBALANCE_PARTITIONS_BY_NONE extends ShuffleOrigin with Product with Serializable
- case object REPARTITION_BY_COL extends ShuffleOrigin with Product with Serializable
- case object REPARTITION_BY_NUM extends ShuffleOrigin with Product with Serializable
- object ShuffleExchangeExec extends Serializable
- object ValidateRequirements extends Logging
Validates that the Partitioning of input data meets the Distribution requirements for each operator, and so are the ordering requirements.