Object

org.apache.spark.sql.catalyst.plans.physical

SinglePartition

Related Doc: package physical

Permalink

object SinglePartition extends Partitioning with Product with Serializable

Linear Supertypes
Serializable, Serializable, Product, Equals, Partitioning, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. SinglePartition
  2. Serializable
  3. Serializable
  4. Product
  5. Equals
  6. Partitioning
  7. AnyRef
  8. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  5. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  6. def compatibleWith(other: Partitioning): Boolean

    Permalink

    Returns true iff we can say that the partitioning scheme of this Partitioning guarantees the same partitioning scheme described by other.

    Returns true iff we can say that the partitioning scheme of this Partitioning guarantees the same partitioning scheme described by other.

    Compatibility of partitionings is only checked for operators that have multiple children and that require a specific child output Distribution, such as joins.

    Intuitively, partitionings are compatible if they route the same partitioning key to the same partition. For instance, two hash partitionings are only compatible if they produce the same number of output partitionings and hash records according to the same hash function and same partitioning key schema.

    Put another way, two partitionings are compatible with each other if they satisfy all of the same distribution guarantees.

    Definition Classes
    SinglePartitionPartitioning
  7. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  8. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  9. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  10. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  11. def guarantees(other: Partitioning): Boolean

    Permalink

    Returns true iff we can say that the partitioning scheme of this Partitioning guarantees the same partitioning scheme described by other.

    Returns true iff we can say that the partitioning scheme of this Partitioning guarantees the same partitioning scheme described by other. If a A.guarantees(B), then repartitioning the child's output according to B will be unnecessary. guarantees is used as a performance optimization to allow the exchange planner to avoid redundant repartitionings. By default, a partitioning only guarantees partitionings that are equal to itself (i.e. the same number of partitions, same strategy (range or hash), etc).

    In order to enable more aggressive optimization, this strict equality check can be relaxed. For example, say that the planner needs to repartition all of an operator's children so that they satisfy the AllTuples distribution. One way to do this is to repartition all children to have the SinglePartition partitioning. If one of the operator's children already happens to be hash-partitioned with a single partition then we do not need to re-shuffle this child; this repartitioning can be avoided if a single-partition HashPartitioning guarantees SinglePartition.

    The SinglePartition example given above is not particularly interesting; guarantees' real value occurs for more advanced partitioning strategies. SPARK-7871 will introduce a notion of null-safe partitionings, under which partitionings can specify whether rows whose partitioning keys contain null values will be grouped into the same partition or whether they will have an unknown / random distribution. If a partitioning does not require nulls to be clustered then a partitioning which _does_ cluster nulls will guarantee the null clustered partitioning. The converse is not true, however: a partitioning which clusters nulls cannot be guaranteed by one which does not cluster them. Thus, in general guarantees is not a symmetric relation.

    Another way to think about guarantees: if A.guarantees(B), then any partitioning of rows produced by A could have also been produced by B.

    Definition Classes
    SinglePartitionPartitioning
  12. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  13. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  14. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  15. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  16. val numPartitions: Int

    Permalink

    Returns the number of partitions that the data is split across

    Returns the number of partitions that the data is split across

    Definition Classes
    SinglePartitionPartitioning
  17. def satisfies(required: Distribution): Boolean

    Permalink

    Returns true iff the guarantees made by this Partitioning are sufficient to satisfy the partitioning scheme mandated by the required Distribution, i.e.

    Returns true iff the guarantees made by this Partitioning are sufficient to satisfy the partitioning scheme mandated by the required Distribution, i.e. the current dataset does not need to be re-partitioned for the required Distribution (it is possible that tuples within a partition need to be reorganized).

    Definition Classes
    SinglePartitionPartitioning
  18. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  19. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  20. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  21. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from Partitioning

Inherited from AnyRef

Inherited from Any

Ungrouped