Class

org.apache.spark.sql.execution.joins

HashJoinExec

Related Doc: package joins

Permalink

case class HashJoinExec(leftKeys: Seq[Expression], rightKeys: Seq[Expression], buildSide: BuildSide, condition: Option[Expression], joinType: JoinType, left: SparkPlan, right: SparkPlan, leftSizeInBytes: BigInt, rightSizeInBytes: BigInt, replicatedTableJoin: Boolean) extends NonRecursivePlans with BinaryExecNode with HashJoin with SnappyJoinLike with BatchConsumer with Product with Serializable

:: DeveloperApi :: Performs a local hash join of two child relations. If a relation (out of a datasource) is already replicated across all nodes then rather than doing a Broadcast join which can be expensive, this join just scans through the single partition of the replicated relation while streaming through the other relation.

Annotations
@DeveloperApi()
Linear Supertypes
BatchConsumer, CodegenSupport, SnappyJoinLike, HashJoin, BinaryExecNode, NonRecursivePlans, SparkPlan, Serializable, Serializable, internal.Logging, QueryPlan[SparkPlan], TreeNode[SparkPlan], Product, Equals, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. HashJoinExec
  2. BatchConsumer
  3. CodegenSupport
  4. SnappyJoinLike
  5. HashJoin
  6. BinaryExecNode
  7. NonRecursivePlans
  8. SparkPlan
  9. Serializable
  10. Serializable
  11. Logging
  12. QueryPlan
  13. TreeNode
  14. Product
  15. Equals
  16. AnyRef
  17. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new HashJoinExec(leftKeys: Seq[Expression], rightKeys: Seq[Expression], buildSide: BuildSide, condition: Option[Expression], joinType: JoinType, left: SparkPlan, right: SparkPlan, leftSizeInBytes: BigInt, rightSizeInBytes: BigInt, replicatedTableJoin: Boolean)

    Permalink

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  4. lazy val allAttributes: AttributeSeq

    Permalink
    Definition Classes
    QueryPlan
  5. def apply(number: Int): TreeNode[_]

    Permalink
    Definition Classes
    TreeNode
  6. def argString: String

    Permalink
    Definition Classes
    TreeNode
  7. def asCode: String

    Permalink
    Definition Classes
    TreeNode
  8. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  9. def batchConsume(ctx: CodegenContext, plan: SparkPlan, input: Seq[ExprCode]): String

    Permalink

    Generate Java source code to do any processing before a batch is consumed by a DataSourceScanExec that does batch processing (e.g.

    Generate Java source code to do any processing before a batch is consumed by a DataSourceScanExec that does batch processing (e.g. per-batch optimizations, initializations etc).

    Implementations should use this for additional optimizations that can be done at batch level when a batched scan is being done. They should not depend on this being invoked since many scans will not be batched.

    Definition Classes
    HashJoinExecBatchConsumer
  10. def beforeStop(ctx: CodegenContext, plan: SparkPlan, input: Seq[ExprCode]): String

    Permalink

    Generate Java source code to do any processing before return after current row processing i.e.

    Generate Java source code to do any processing before return after current row processing i.e. when shouldStop() returns true.

    Definition Classes
    BatchConsumer
  11. lazy val buildKeys: Seq[Expression]

    Permalink
    Attributes
    protected
    Definition Classes
    HashJoin
  12. lazy val buildPlan: SparkPlan

    Permalink
    Attributes
    protected
    Definition Classes
    HashJoin
  13. val buildSide: BuildSide

    Permalink
    Definition Classes
    HashJoinExec → HashJoin
  14. def buildSideKeyGenerator(): Projection

    Permalink
    Attributes
    protected
    Definition Classes
    HashJoin
  15. lazy val buildSideKeys: Seq[Expression]

    Permalink
    Attributes
    protected
  16. def canConsume(plan: SparkPlan): Boolean

    Permalink

    Returns true if the given plan returning batches of data can be consumed by this plan.

    Returns true if the given plan returning batches of data can be consumed by this plan.

    Definition Classes
    HashJoinExecBatchConsumer
  17. lazy val canonicalized: SparkPlan

    Permalink
    Attributes
    protected
    Definition Classes
    QueryPlan
  18. final def children: Seq[SparkPlan]

    Permalink
    Definition Classes
    BinaryExecNode → TreeNode
  19. lazy val cleanArgs: Seq[Any]

    Permalink
    Attributes
    protected
    Definition Classes
    QueryPlan
  20. def cleanExpression(e: Expression): Expression

    Permalink
    Attributes
    protected
    Definition Classes
    QueryPlan
  21. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  22. def collect[B](pf: PartialFunction[SparkPlan, B]): Seq[B]

    Permalink
    Definition Classes
    TreeNode
  23. def collectFirst[B](pf: PartialFunction[SparkPlan, B]): Option[B]

    Permalink
    Definition Classes
    TreeNode
  24. def collectLeaves(): Seq[SparkPlan]

    Permalink
    Definition Classes
    TreeNode
  25. val condition: Option[Expression]

    Permalink
    Definition Classes
    HashJoinExec → HashJoin
  26. lazy val constraints: ExpressionSet

    Permalink
    Definition Classes
    QueryPlan
  27. final def consume(ctx: CodegenContext, outputVars: Seq[ExprCode], row: String): String

    Permalink
    Definition Classes
    CodegenSupport
  28. lazy val containsChild: Set[TreeNode[_]]

    Permalink
    Definition Classes
    TreeNode
  29. def createResultProjection(): (InternalRow) ⇒ InternalRow

    Permalink
    Attributes
    protected
    Definition Classes
    HashJoin
  30. def doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: ExprCode): String

    Permalink
    Definition Classes
    HashJoinExec → CodegenSupport
  31. def doExecute(): RDD[InternalRow]

    Permalink
    Attributes
    protected
    Definition Classes
    NonRecursivePlans → SparkPlan
  32. def doExecuteBroadcast[T](): Broadcast[T]

    Permalink
    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    SparkPlan
  33. def doPrepare(): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    SparkPlan
  34. def doProduce(ctx: CodegenContext): String

    Permalink
    Definition Classes
    HashJoinExec → CodegenSupport
  35. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  36. def evaluateRequiredVariables(attributes: Seq[Attribute], variables: Seq[ExprCode], required: AttributeSet): String

    Permalink
    Attributes
    protected
    Definition Classes
    CodegenSupport
  37. def evaluateVariables(variables: Seq[ExprCode]): String

    Permalink
    Attributes
    protected
    Definition Classes
    CodegenSupport
  38. final def execute(): RDD[InternalRow]

    Permalink
    Definition Classes
    SparkPlan
  39. final def executeBroadcast[T](): Broadcast[T]

    Permalink
    Definition Classes
    SparkPlan
  40. def executeCollect(): Array[InternalRow]

    Permalink
    Definition Classes
    SparkPlan
  41. def executeCollectPublic(): Array[Row]

    Permalink
    Definition Classes
    SparkPlan
  42. final def executeQuery[T](query: ⇒ T): T

    Permalink
    Attributes
    protected
    Definition Classes
    SparkPlan
  43. def executeTake(n: Int): Array[InternalRow]

    Permalink
    Definition Classes
    SparkPlan
  44. def executeToIterator(): Iterator[InternalRow]

    Permalink
    Definition Classes
    SparkPlan
  45. final def expressions: Seq[Expression]

    Permalink
    Definition Classes
    QueryPlan
  46. def fastEquals(other: TreeNode[_]): Boolean

    Permalink
    Definition Classes
    TreeNode
  47. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  48. def find(f: (SparkPlan) ⇒ Boolean): Option[SparkPlan]

    Permalink
    Definition Classes
    TreeNode
  49. def flatMap[A](f: (SparkPlan) ⇒ TraversableOnce[A]): Seq[A]

    Permalink
    Definition Classes
    TreeNode
  50. def foreach(f: (SparkPlan) ⇒ Unit): Unit

    Permalink
    Definition Classes
    TreeNode
  51. def foreachUp(f: (SparkPlan) ⇒ Unit): Unit

    Permalink
    Definition Classes
    TreeNode
  52. def generateTreeString(depth: Int, lastChildren: Seq[Boolean], builder: StringBuilder, verbose: Boolean, prefix: String): StringBuilder

    Permalink
    Definition Classes
    TreeNode
  53. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  54. def getRelevantConstraints(constraints: Set[Expression]): Set[Expression]

    Permalink
    Attributes
    protected
    Definition Classes
    QueryPlan
  55. def getSubsetsAndIndices(partitioning: Partitioning, keys: Seq[Expression], child: SparkPlan): Seq[(Seq[Expression], Seq[Int])]

    Permalink

    Optionally return result if partitioning is a subset of given join keys, and if so then return the subset as well as the indices of subset keys in the join keys (in order).

    Optionally return result if partitioning is a subset of given join keys, and if so then return the subset as well as the indices of subset keys in the join keys (in order). Also unwraps aliases in the keys for matching against partitioning and returns a boolean indicating whether alias was unwrapped or not.

    Attributes
    protected
    Definition Classes
    SnappyJoinLike
  56. def hashCode(): Int

    Permalink
    Definition Classes
    TreeNode → AnyRef → Any
  57. def initializeLogIfNecessary(isInterpreter: Boolean): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  58. def innerChildren: Seq[QueryPlan[_]]

    Permalink
    Attributes
    protected
    Definition Classes
    QueryPlan → TreeNode
  59. def inputRDDs(): Seq[RDD[InternalRow]]

    Permalink
    Definition Classes
    HashJoinExec → CodegenSupport
  60. def inputSet: AttributeSet

    Permalink
    Definition Classes
    QueryPlan
  61. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  62. def isTraceEnabled(): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  63. def join(streamedIter: Iterator[InternalRow], hashed: HashedRelation, numOutputRows: SQLMetric): Iterator[InternalRow]

    Permalink
    Attributes
    protected
    Definition Classes
    HashJoin
  64. val joinType: JoinType

    Permalink
    Definition Classes
    HashJoinExecSnappyJoinLike → HashJoin
  65. def jsonFields: List[JField]

    Permalink
    Attributes
    protected
    Definition Classes
    TreeNode
  66. val left: SparkPlan

    Permalink
    Definition Classes
    HashJoinExecSnappyJoinLike → HashJoin → BinaryExecNode
  67. val leftKeys: Seq[Expression]

    Permalink
    Definition Classes
    HashJoinExecSnappyJoinLike → HashJoin
  68. val leftSizeInBytes: BigInt

    Permalink
    Definition Classes
    HashJoinExecSnappyJoinLike
  69. def log: Logger

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  70. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  71. def logDebug(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  72. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  73. def logError(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  74. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  75. def logInfo(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  76. def logName: String

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  77. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  78. def logTrace(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  79. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  80. def logWarning(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  81. def longMetric(name: String): SQLMetric

    Permalink
    Definition Classes
    SparkPlan
  82. def makeCopy(newArgs: Array[AnyRef]): NonRecursivePlans

    Permalink
    Definition Classes
    NonRecursivePlans → SparkPlan → TreeNode
  83. def map[A](f: (SparkPlan) ⇒ A): Seq[A]

    Permalink
    Definition Classes
    TreeNode
  84. def mapChildren(f: (SparkPlan) ⇒ SparkPlan): SparkPlan

    Permalink
    Definition Classes
    TreeNode
  85. def mapExpressions(f: (Expression) ⇒ Expression): HashJoinExec.this.type

    Permalink
    Definition Classes
    QueryPlan
  86. def mapProductIterator[B](f: (Any) ⇒ B)(implicit arg0: ClassTag[B]): Array[B]

    Permalink
    Attributes
    protected
    Definition Classes
    TreeNode
  87. def metadata: Map[String, String]

    Permalink
    Definition Classes
    SparkPlan
  88. val metricAdd: (String) ⇒ String

    Permalink
  89. def metricTerm(ctx: CodegenContext, name: String): String

    Permalink
    Definition Classes
    CodegenSupport
  90. lazy val metrics: Map[String, SQLMetric]

    Permalink
    Definition Classes
    HashJoinExec → SparkPlan
  91. def missingInput: AttributeSet

    Permalink
    Definition Classes
    QueryPlan
  92. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  93. def newMutableProjection(expressions: Seq[Expression], inputSchema: Seq[Attribute], useSubexprElimination: Boolean): MutableProjection

    Permalink
    Attributes
    protected
    Definition Classes
    SparkPlan
  94. def newNaturalAscendingOrdering(dataTypes: Seq[DataType]): Ordering[InternalRow]

    Permalink
    Attributes
    protected
    Definition Classes
    SparkPlan
  95. def newOrdering(order: Seq[SortOrder], inputSchema: Seq[Attribute]): Ordering[InternalRow]

    Permalink
    Attributes
    protected
    Definition Classes
    SparkPlan
  96. def newPredicate(expression: Expression, inputSchema: Seq[Attribute]): Predicate

    Permalink
    Attributes
    protected
    Definition Classes
    SparkPlan
  97. def nodeName: String

    Permalink
    Definition Classes
    HashJoinExec → TreeNode
  98. final var nonCodeGeneratedPlanCalls: Int

    Permalink

    Variable to disallow recursive generation so will mark the case of non-codegenerated case and throw back exception to use CodegenSparkFallback.

    Variable to disallow recursive generation so will mark the case of non-codegenerated case and throw back exception to use CodegenSparkFallback.

    Attributes
    protected
    Definition Classes
    NonRecursivePlans
  99. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  100. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  101. def numberedTreeString: String

    Permalink
    Definition Classes
    TreeNode
  102. val origin: Origin

    Permalink
    Definition Classes
    TreeNode
  103. def otherCopyArgs: Seq[AnyRef]

    Permalink
    Attributes
    protected
    Definition Classes
    TreeNode
  104. def output: Seq[Attribute]

    Permalink
    Definition Classes
    HashJoin
  105. def outputOrdering: Seq[SortOrder]

    Permalink
    Definition Classes
    HashJoinExec → SparkPlan
  106. def outputPartitioning: Partitioning

    Permalink
    Definition Classes
    HashJoinExec → HashJoin → SparkPlan
  107. def outputSet: AttributeSet

    Permalink
    Definition Classes
    QueryPlan
  108. def p(number: Int): SparkPlan

    Permalink
    Definition Classes
    TreeNode
  109. var parent: CodegenSupport

    Permalink
    Attributes
    protected
    Definition Classes
    CodegenSupport
  110. final def prepare(): Unit

    Permalink
    Definition Classes
    SparkPlan
  111. def prepareSubqueries(): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    SparkPlan
  112. def prettyJson: String

    Permalink
    Definition Classes
    TreeNode
  113. def printSchema(): Unit

    Permalink
    Definition Classes
    QueryPlan
  114. final def produce(ctx: CodegenContext, parent: CodegenSupport): String

    Permalink
    Definition Classes
    CodegenSupport
  115. def producedAttributes: AttributeSet

    Permalink
    Definition Classes
    QueryPlan
  116. def references: AttributeSet

    Permalink
    Definition Classes
    QueryPlan
  117. val replicatedTableJoin: Boolean

    Permalink
  118. def requiredChildDistribution: Seq[Distribution]

    Permalink
    Definition Classes
    HashJoinExecSnappyJoinLike → SparkPlan
  119. def requiredChildOrdering: Seq[Seq[SortOrder]]

    Permalink
    Definition Classes
    SparkPlan
  120. def resetMetrics(): Unit

    Permalink
    Definition Classes
    SparkPlan
  121. val right: SparkPlan

    Permalink
    Definition Classes
    HashJoinExecSnappyJoinLike → HashJoin → BinaryExecNode
  122. val rightKeys: Seq[Expression]

    Permalink
    Definition Classes
    HashJoinExecSnappyJoinLike → HashJoin
  123. val rightSizeInBytes: BigInt

    Permalink
    Definition Classes
    HashJoinExecSnappyJoinLike
  124. def sameResult(plan: SparkPlan): Boolean

    Permalink
    Definition Classes
    QueryPlan
  125. lazy val schema: StructType

    Permalink
    Definition Classes
    QueryPlan
  126. def schemaString: String

    Permalink
    Definition Classes
    QueryPlan
  127. def simpleString: String

    Permalink
    Definition Classes
    QueryPlan → TreeNode
  128. def sparkContext: SparkContext

    Permalink
    Attributes
    protected
    Definition Classes
    SparkPlan
  129. final val sqlContext: SQLContext

    Permalink
    Definition Classes
    SparkPlan
  130. def statePrefix: String

    Permalink
    Attributes
    protected
    Definition Classes
    QueryPlan
  131. def streamSideKeyGenerator(): UnsafeProjection

    Permalink
    Attributes
    protected
    Definition Classes
    HashJoin
  132. lazy val streamSideKeys: Seq[Expression]

    Permalink
    Attributes
    protected
  133. lazy val streamedKeys: Seq[Expression]

    Permalink
    Attributes
    protected
    Definition Classes
    HashJoin
  134. lazy val streamedPlan: SparkPlan

    Permalink
    Attributes
    protected
    Definition Classes
    HashJoin
  135. def stringArgs: Iterator[Any]

    Permalink
    Attributes
    protected
    Definition Classes
    TreeNode
  136. val subexpressionEliminationEnabled: Boolean

    Permalink
    Definition Classes
    SparkPlan
  137. def subqueries: Seq[SparkPlan]

    Permalink
    Definition Classes
    QueryPlan
  138. def supportCodegen: Boolean

    Permalink
    Definition Classes
    CodegenSupport
  139. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  140. def toJSON: String

    Permalink
    Definition Classes
    TreeNode
  141. def toString(): String

    Permalink
    Definition Classes
    TreeNode → AnyRef → Any
  142. def transform(rule: PartialFunction[SparkPlan, SparkPlan]): SparkPlan

    Permalink
    Definition Classes
    TreeNode
  143. def transformAllExpressions(rule: PartialFunction[Expression, Expression]): HashJoinExec.this.type

    Permalink
    Definition Classes
    QueryPlan
  144. def transformDown(rule: PartialFunction[SparkPlan, SparkPlan]): SparkPlan

    Permalink
    Definition Classes
    TreeNode
  145. def transformExpressions(rule: PartialFunction[Expression, Expression]): HashJoinExec.this.type

    Permalink
    Definition Classes
    QueryPlan
  146. def transformExpressionsDown(rule: PartialFunction[Expression, Expression]): HashJoinExec.this.type

    Permalink
    Definition Classes
    QueryPlan
  147. def transformExpressionsUp(rule: PartialFunction[Expression, Expression]): HashJoinExec.this.type

    Permalink
    Definition Classes
    QueryPlan
  148. def transformUp(rule: PartialFunction[SparkPlan, SparkPlan]): SparkPlan

    Permalink
    Definition Classes
    TreeNode
  149. def treeString(verbose: Boolean): String

    Permalink
    Definition Classes
    TreeNode
  150. def treeString: String

    Permalink
    Definition Classes
    TreeNode
  151. def usedInputs: AttributeSet

    Permalink
    Definition Classes
    HashJoinExec → CodegenSupport
  152. def validConstraints: Set[Expression]

    Permalink
    Attributes
    protected
    Definition Classes
    QueryPlan
  153. def verboseString: String

    Permalink
    Definition Classes
    QueryPlan → TreeNode
  154. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  155. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  156. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  157. def waitForSubqueries(): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    SparkPlan
  158. def withNewChildren(newChildren: Seq[SparkPlan]): SparkPlan

    Permalink
    Definition Classes
    TreeNode

Inherited from BatchConsumer

Inherited from CodegenSupport

Inherited from SnappyJoinLike

Inherited from HashJoin

Inherited from BinaryExecNode

Inherited from NonRecursivePlans

Inherited from SparkPlan

Inherited from Serializable

Inherited from Serializable

Inherited from internal.Logging

Inherited from QueryPlan[SparkPlan]

Inherited from TreeNode[SparkPlan]

Inherited from Product

Inherited from Equals

Inherited from AnyRef

Inherited from Any

Ungrouped