package exchange
- Alphabetic
- Public
- All
Type Members
-
case class
BroadcastExchangeExec(mode: BroadcastMode, child: SparkPlan) extends Exchange with BroadcastExchangeLike with Product with Serializable
A BroadcastExchangeExec collects, transforms and finally broadcasts the result of a transformed SparkPlan.
-
trait
BroadcastExchangeLike extends Exchange
Common trait for all broadcast exchange implementations to facilitate pattern matching.
-
case class
EnsureRequirements(optimizeOutRepartition: Boolean = true) extends Rule[SparkPlan] with Product with Serializable
Ensures that the Partitioning of input data meets the Distribution requirements for each operator by inserting ShuffleExchangeExec Operators where required.
Ensures that the Partitioning of input data meets the Distribution requirements for each operator by inserting ShuffleExchangeExec Operators where required. Also ensure that the input partition ordering requirements are met.
- optimizeOutRepartition
A flag to indicate that if this rule should optimize out user-specified repartition shuffles or not. This is mostly true, but can be false in AQE when AQE optimization may change the plan output partitioning and need to retain the user-specified repartition shuffles in the plan.
-
abstract
class
Exchange extends SparkPlan with UnaryExecNode
Base class for operators that exchange data among multiple threads or processes.
Base class for operators that exchange data among multiple threads or processes.
Exchanges are the key class of operators that enable parallelism. Although the implementation differs significantly, the concept is similar to the exchange operator described in "Volcano -- An Extensible and Parallel Query Evaluation System" by Goetz Graefe.
-
case class
ReusedExchangeExec(output: Seq[Attribute], child: Exchange) extends SparkPlan with LeafExecNode with Product with Serializable
A wrapper for reused exchange to have different output, because two exchanges which produce logically identical output will have distinct sets of output attribute ids, so we need to preserve the original ids because they're what downstream operators are expecting.
-
case class
ShuffleExchangeExec(outputPartitioning: Partitioning, child: SparkPlan, shuffleOrigin: ShuffleOrigin = ENSURE_REQUIREMENTS) extends Exchange with ShuffleExchangeLike with Product with Serializable
Performs a shuffle that will result in the desired partitioning.
-
trait
ShuffleExchangeLike extends Exchange
Common trait for all shuffle exchange implementations to facilitate pattern matching.
- sealed trait ShuffleOrigin extends AnyRef
Value Members
- object BroadcastExchangeExec extends Serializable
- object ENSURE_REQUIREMENTS extends ShuffleOrigin with Product with Serializable
- object REBALANCE_PARTITIONS_BY_COL extends ShuffleOrigin with Product with Serializable
- object REBALANCE_PARTITIONS_BY_NONE extends ShuffleOrigin with Product with Serializable
- object REPARTITION_BY_COL extends ShuffleOrigin with Product with Serializable
- object REPARTITION_BY_NUM extends ShuffleOrigin with Product with Serializable
- object ShuffleExchangeExec extends Serializable
-
object
ValidateRequirements extends Logging
Validates that the Partitioning of input data meets the Distribution requirements for each operator, and so are the ordering requirements.