TungstenSort

Instance Constructors

new TungstenSort(sortOrder: Seq[SortOrder], global: Boolean, child: SparkPlan, testSpillFrequency: Int = 0)

global
when true performs a global sort of all partitions by shuffling the data first if necessary.
testSpillFrequency
Method for configuring periodic spilling in unit tests. If set, will spill every frequency records.

Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def apply(number: Int): SparkPlan

Definition Classes
TreeNode
def argString: String

Definition Classes
TreeNode
def asCode: String

Definition Classes
TreeNode
final def asInstanceOf[T0]: T0

Definition Classes
Any
def canProcessSafeRows: Boolean

Specifies whether this operator is capable of processing Java-object-based Rows (i.e.
Specifies whether this operator is capable of processing Java-object-based Rows (i.e. rows that are not UnsafeRows).

Definition Classes
TungstenSort → SparkPlan
def canProcessUnsafeRows: Boolean

Specifies whether this operator is capable of processing UnsafeRows
Specifies whether this operator is capable of processing UnsafeRows

Definition Classes
TungstenSort → SparkPlan
val child: SparkPlan

Definition Classes
TungstenSort → UnaryNode
def children: Seq[SparkPlan]

Definition Classes
UnaryNode → TreeNode
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
val codegenEnabled: Boolean

Definition Classes
SparkPlan
def collect[B](pf: PartialFunction[SparkPlan, B]): Seq[B]

Definition Classes
TreeNode
def collectFirst[B](pf: PartialFunction[SparkPlan, B]): Option[B]

Definition Classes
TreeNode
lazy val containsChild: Set[TreeNode[_]]

Definition Classes
TreeNode
def doExecute(): RDD[InternalRow]

Overridden by concrete implementations of SparkPlan.
Overridden by concrete implementations of SparkPlan. Produces the result of the query as an RDD[InternalRow]

Attributes
protected
Definition Classes
TungstenSort → SparkPlan
def doPrepare(): Unit

Overridden by concrete implementations of SparkPlan.
Overridden by concrete implementations of SparkPlan. It is guaranteed to run before any execute of SparkPlan. This is helpful if we want to set up some state before executing the query, e.g., BroadcastHashJoin uses it to broadcast asynchronously.
Note: the prepare method has already walked down the tree, so the implementation doesn't need to call children's prepare methods.

Attributes
protected
Definition Classes
SparkPlan
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def execute(): RDD[InternalRow]

Returns the result of this query as an RDD[InternalRow] by delegating to doExecute after adding query plan information to created RDDs for visualization.
Returns the result of this query as an RDD[InternalRow] by delegating to doExecute after adding query plan information to created RDDs for visualization. Concrete implementations of SparkPlan should override doExecute instead.

Definition Classes
SparkPlan
def executeCollect(): Array[Row]

Runs this query returning the result as an array.
Runs this query returning the result as an array.

Definition Classes
SparkPlan
def executeTake(n: Int): Array[Row]

Runs this query returning the first n rows as an array.
Runs this query returning the first n rows as an array.
This is modeled after RDD.take but never runs any job locally on the driver.

Definition Classes
SparkPlan
def expressions: Seq[Expression]

Definition Classes
QueryPlan
def fastEquals(other: TreeNode[_]): Boolean

Definition Classes
TreeNode
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
def find(f: (SparkPlan) ⇒ Boolean): Option[SparkPlan]

Definition Classes
TreeNode
def flatMap[A](f: (SparkPlan) ⇒ TraversableOnce[A]): Seq[A]

Definition Classes
TreeNode
def foreach(f: (SparkPlan) ⇒ Unit): Unit

Definition Classes
TreeNode
def foreachUp(f: (SparkPlan) ⇒ Unit): Unit

Definition Classes
TreeNode
def generateTreeString(depth: Int, builder: StringBuilder): StringBuilder

Attributes
protected
Definition Classes
TreeNode
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
def getNodeNumbered(number: MutableInt): SparkPlan

Attributes
protected
Definition Classes
TreeNode
val global: Boolean

when true performs a global sort of all partitions by shuffling the data first if necessary.
def inputSet: AttributeSet

Definition Classes
QueryPlan
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
def isTraceEnabled(): Boolean

Attributes
protected
Definition Classes
Logging
def log: Logger

Attributes
protected
Definition Classes
Logging
def logDebug(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logDebug(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logError(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logError(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logInfo(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logInfo(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logName: String

Attributes
protected
Definition Classes
Logging
def logTrace(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logTrace(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logWarning(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logWarning(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def makeCopy(newArgs: Array[AnyRef]): SparkPlan

Overridden make copy also propogates sqlContext to copied plan.
Overridden make copy also propogates sqlContext to copied plan.

Definition Classes
SparkPlan → TreeNode
def map[A](f: (SparkPlan) ⇒ A): Seq[A]

Definition Classes
TreeNode
def mapChildren(f: (SparkPlan) ⇒ SparkPlan): SparkPlan

Definition Classes
TreeNode
def missingInput: AttributeSet

Definition Classes
QueryPlan
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def newMutableProjection(expressions: Seq[Expression], inputSchema: Seq[Attribute]): () ⇒ MutableProjection

Attributes
protected
Definition Classes
SparkPlan
def newNaturalAscendingOrdering(dataTypes: Seq[DataType]): Ordering[InternalRow]

Creates a row ordering for the given schema, in natural ascending order.
Creates a row ordering for the given schema, in natural ascending order.

Attributes
protected
Definition Classes
SparkPlan
def newOrdering(order: Seq[SortOrder], inputSchema: Seq[Attribute]): Ordering[InternalRow]

Attributes
protected
Definition Classes
SparkPlan
def newPredicate(expression: Expression, inputSchema: Seq[Attribute]): (InternalRow) ⇒ Boolean

Attributes
protected
Definition Classes
SparkPlan
def newProjection(expressions: Seq[Expression], inputSchema: Seq[Attribute]): Projection

Attributes
protected
Definition Classes
SparkPlan
def nodeName: String

Definition Classes
TreeNode
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
def numberedTreeString: String

Definition Classes
TreeNode
val origin: Origin

Definition Classes
TreeNode
def otherCopyArgs: Seq[AnyRef]

Attributes
protected
Definition Classes
TreeNode
def output: Seq[Attribute]

Definition Classes
TungstenSort → QueryPlan
def outputOrdering: Seq[SortOrder]

Specifies how data is ordered in each partition.
Specifies how data is ordered in each partition.

Definition Classes
TungstenSort → SparkPlan
def outputPartitioning: Partitioning

Specifies how data is partitioned across different nodes in the cluster.
Specifies how data is partitioned across different nodes in the cluster.

Definition Classes
UnaryNode → SparkPlan
def outputSet: AttributeSet

Definition Classes
QueryPlan
def outputsUnsafeRows: Boolean

Specifies whether this operator outputs UnsafeRows
Specifies whether this operator outputs UnsafeRows

Definition Classes
TungstenSort → SparkPlan
final def prepare(): Unit

Prepare a SparkPlan for execution.
Prepare a SparkPlan for execution. It's idempotent.

Definition Classes
SparkPlan
def printSchema(): Unit

Definition Classes
QueryPlan
def references: AttributeSet

Definition Classes
QueryPlan
def requiredChildDistribution: Seq[Distribution]

Specifies any partition requirements on the input data for this operator.
Specifies any partition requirements on the input data for this operator.

Definition Classes
TungstenSort → SparkPlan
def requiredChildOrdering: Seq[Seq[SortOrder]]

Specifies sort order for each partition requirements on the input data for this operator.
Specifies sort order for each partition requirements on the input data for this operator.

Definition Classes
SparkPlan
lazy val schema: StructType

Definition Classes
QueryPlan
def schemaString: String

Definition Classes
QueryPlan
def simpleString: String

Definition Classes
QueryPlan → TreeNode
val sortOrder: Seq[SortOrder]
def sparkContext: SparkContext

Attributes
protected
Definition Classes
SparkPlan
final val sqlContext: SQLContext

A handle to the SQL Context that was used to create this plan.
A handle to the SQL Context that was used to create this plan. Since many operators need access to the sqlContext for RDD operations or configuration this field is automatically populated by the query planning infrastructure.

Attributes
protected[org.apache.spark]
Definition Classes
SparkPlan
def statePrefix: String

Attributes
protected
Definition Classes
QueryPlan
def stringArgs: Iterator[Any]

Attributes
protected
Definition Classes
TreeNode
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
val testSpillFrequency: Int

Method for configuring periodic spilling in unit tests.
Method for configuring periodic spilling in unit tests. If set, will spill every frequency records.
def toString(): String

Definition Classes
TreeNode → AnyRef → Any
def transform(rule: PartialFunction[SparkPlan, SparkPlan]): SparkPlan

Definition Classes
TreeNode
def transformAllExpressions(rule: PartialFunction[Expression, Expression]): TungstenSort.this.type

Definition Classes
QueryPlan
def transformChildren(rule: PartialFunction[SparkPlan, SparkPlan], nextOperation: (SparkPlan, PartialFunction[SparkPlan, SparkPlan]) ⇒ SparkPlan): SparkPlan

Attributes
protected
Definition Classes
TreeNode
def transformDown(rule: PartialFunction[SparkPlan, SparkPlan]): SparkPlan

Definition Classes
TreeNode
def transformExpressions(rule: PartialFunction[Expression, Expression]): TungstenSort.this.type

Definition Classes
QueryPlan
def transformExpressionsDown(rule: PartialFunction[Expression, Expression]): TungstenSort.this.type

Definition Classes
QueryPlan
def transformExpressionsUp(rule: PartialFunction[Expression, Expression]): TungstenSort.this.type

Definition Classes
QueryPlan
def transformUp(rule: PartialFunction[SparkPlan, SparkPlan]): SparkPlan

Definition Classes
TreeNode
def treeString: String

Definition Classes
TreeNode
val unsafeEnabled: Boolean

Definition Classes
SparkPlan
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
def withNewChildren(newChildren: Seq[SparkPlan]): SparkPlan

Definition Classes
TreeNode

Related Docs: object TungstenSort | package execution

case class TungstenSort(sortOrder: Seq[SortOrder], global: Boolean, child: SparkPlan, testSpillFrequency: Int = 0) extends SparkPlan with UnaryNode with Product with Serializable

Instance Constructors

new TungstenSort(sortOrder: Seq[SortOrder], global: Boolean, child: SparkPlan, testSpillFrequency: Int = 0)

Value Members

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: Any): Boolean

def apply(number: Int): SparkPlan

def argString: String

def asCode: String

final def asInstanceOf[T0]: T0

def canProcessSafeRows: Boolean

def canProcessUnsafeRows: Boolean

val child: SparkPlan

def children: Seq[SparkPlan]

def clone(): AnyRef

val codegenEnabled: Boolean

def collect[B](pf: PartialFunction[SparkPlan, B]): Seq[B]

def collectFirst[B](pf: PartialFunction[SparkPlan, B]): Option[B]

lazy val containsChild: Set[TreeNode[_]]

def doExecute(): RDD[InternalRow]

def doPrepare(): Unit

final def eq(arg0: AnyRef): Boolean

final def execute(): RDD[InternalRow]

def executeCollect(): Array[Row]

def executeTake(n: Int): Array[Row]

def expressions: Seq[Expression]

def fastEquals(other: TreeNode[_]): Boolean

def finalize(): Unit

def find(f: (SparkPlan) ⇒ Boolean): Option[SparkPlan]

def flatMap[A](f: (SparkPlan) ⇒ TraversableOnce[A]): Seq[A]

def foreach(f: (SparkPlan) ⇒ Unit): Unit

def foreachUp(f: (SparkPlan) ⇒ Unit): Unit

def generateTreeString(depth: Int, builder: StringBuilder): StringBuilder

final def getClass(): Class[_]

def getNodeNumbered(number: MutableInt): SparkPlan

val global: Boolean

def inputSet: AttributeSet

final def isInstanceOf[T0]: Boolean

def isTraceEnabled(): Boolean

def log: Logger

def logDebug(msg: ⇒ String, throwable: Throwable): Unit

def logDebug(msg: ⇒ String): Unit

def logError(msg: ⇒ String, throwable: Throwable): Unit

def logError(msg: ⇒ String): Unit

def logInfo(msg: ⇒ String, throwable: Throwable): Unit

def logInfo(msg: ⇒ String): Unit

def logName: String

def logTrace(msg: ⇒ String, throwable: Throwable): Unit

def logTrace(msg: ⇒ String): Unit

def logWarning(msg: ⇒ String, throwable: Throwable): Unit

def logWarning(msg: ⇒ String): Unit

def makeCopy(newArgs: Array[AnyRef]): SparkPlan

def map[A](f: (SparkPlan) ⇒ A): Seq[A]

def mapChildren(f: (SparkPlan) ⇒ SparkPlan): SparkPlan

def missingInput: AttributeSet

final def ne(arg0: AnyRef): Boolean

def newMutableProjection(expressions: Seq[Expression], inputSchema: Seq[Attribute]): () ⇒ MutableProjection

def newNaturalAscendingOrdering(dataTypes: Seq[DataType]): Ordering[InternalRow]

def newOrdering(order: Seq[SortOrder], inputSchema: Seq[Attribute]): Ordering[InternalRow]

def newPredicate(expression: Expression, inputSchema: Seq[Attribute]): (InternalRow) ⇒ Boolean

def newProjection(expressions: Seq[Expression], inputSchema: Seq[Attribute]): Projection

def nodeName: String

final def notify(): Unit

final def notifyAll(): Unit

def numberedTreeString: String

val origin: Origin

def otherCopyArgs: Seq[AnyRef]

def output: Seq[Attribute]

def outputOrdering: Seq[SortOrder]

def outputPartitioning: Partitioning

def outputSet: AttributeSet

def outputsUnsafeRows: Boolean

final def prepare(): Unit

def printSchema(): Unit

def references: AttributeSet

def requiredChildDistribution: Seq[Distribution]

def requiredChildOrdering: Seq[Seq[SortOrder]]