package exchange
- Alphabetic
- Public
- All
Type Members
-
case class
BroadcastExchangeExec(mode: BroadcastMode, child: SparkPlan) extends Exchange with BroadcastExchangeLike with Product with Serializable
A BroadcastExchangeExec collects, transforms and finally broadcasts the result of a transformed SparkPlan.
-
trait
BroadcastExchangeLike extends Exchange
Common trait for all broadcast exchange implementations to facilitate pattern matching.
-
abstract
class
Exchange extends SparkPlan with UnaryExecNode
Base class for operators that exchange data among multiple threads or processes.
Base class for operators that exchange data among multiple threads or processes.
Exchanges are the key class of operators that enable parallelism. Although the implementation differs significantly, the concept is similar to the exchange operator described in "Volcano -- An Extensible and Parallel Query Evaluation System" by Goetz Graefe.
-
case class
ReusedExchangeExec(output: Seq[Attribute], child: Exchange) extends SparkPlan with LeafExecNode with Product with Serializable
A wrapper for reused exchange to have different output, because two exchanges which produce logically identical output will have distinct sets of output attribute ids, so we need to preserve the original ids because they're what downstream operators are expecting.
-
case class
ShuffleExchangeExec(outputPartitioning: Partitioning, child: SparkPlan, shuffleOrigin: ShuffleOrigin = ENSURE_REQUIREMENTS) extends Exchange with ShuffleExchangeLike with Product with Serializable
Performs a shuffle that will result in the desired partitioning.
-
trait
ShuffleExchangeLike extends Exchange
Common trait for all shuffle exchange implementations to facilitate pattern matching.
- sealed trait ShuffleOrigin extends AnyRef
Value Members
- object BroadcastExchangeExec extends Serializable
- object ENSURE_REQUIREMENTS extends ShuffleOrigin with Product with Serializable
-
object
EnsureRequirements extends Rule[SparkPlan]
Ensures that the Partitioning of input data meets the Distribution requirements for each operator by inserting ShuffleExchangeExec Operators where required.
Ensures that the Partitioning of input data meets the Distribution requirements for each operator by inserting ShuffleExchangeExec Operators where required. Also ensure that the input partition ordering requirements are met.
- object REPARTITION extends ShuffleOrigin with Product with Serializable
- object REPARTITION_WITH_NUM extends ShuffleOrigin with Product with Serializable
-
object
ReuseExchange extends Rule[SparkPlan]
Find out duplicated exchanges in the spark plan, then use the same exchange for all the references.
- object ShuffleExchangeExec extends Serializable