Class

com.spotify.scio.values

SCollectionWithHotKeyFanout

Related Doc: package values

Permalink

class SCollectionWithHotKeyFanout[K, V] extends TransformNameable

An enhanced SCollection that uses an intermediate node to combine "hot" keys partially before performing the full combine.

Linear Supertypes
TransformNameable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. SCollectionWithHotKeyFanout
  2. TransformNameable
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new SCollectionWithHotKeyFanout(self: PairSCollectionFunctions[K, V], hotKeyFanout: Either[(K) ⇒ Int, Int])(implicit arg0: ClassTag[K], arg1: ClassTag[V])

    Permalink

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  4. def aggregateByKey[U](zeroValue: U)(seqOp: (U, V) ⇒ U, combOp: (U, U) ⇒ U)(implicit arg0: ClassTag[U]): SCollection[(K, U)]

    Permalink

    Aggregate the values of each key, using given combine functions and a neutral "zero value".

    Aggregate the values of each key, using given combine functions and a neutral "zero value". This function can return a different result type, U, than the type of the values in this SCollection, V. Thus, we need one operation for merging a V into a U and one operation for merging two U's. To avoid memory allocation, both of these functions are allowed to modify and return their first argument instead of creating a new U.

  5. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  6. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  7. def combineByKey[C](createCombiner: (V) ⇒ C)(mergeValue: (C, V) ⇒ C)(mergeCombiners: (C, C) ⇒ C)(implicit arg0: ClassTag[C]): SCollection[(K, C)]

    Permalink

    Generic function to combine the elements for each key using a custom set of aggregation functions.

    Generic function to combine the elements for each key using a custom set of aggregation functions. Turns an SCollection[(K, V)] into a result of type SCollection[(K, C)], for a "combined type" C Note that V and C can be different -- for example, one might group an SCollection of type (Int, Int) into an RDD of type (Int, Seq[Int]). Users provide three functions:

    - createCombiner, which turns a V into a C (e.g., creates a one-element list)

    - mergeValue, to merge a V into a C (e.g., adds it to the end of a list)

    - mergeCombiners, to combine two C's into a single one.

  8. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  9. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  10. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  11. def foldByKey(implicit mon: Monoid[V]): SCollection[(K, V)]

    Permalink

    Fold by key with Monoid, which defines the associative function and "zero value" for V.

    Fold by key with Monoid, which defines the associative function and "zero value" for V. This could be more powerful and better optimized in some cases.

  12. def foldByKey(zeroValue: V)(op: (V, V) ⇒ V): SCollection[(K, V)]

    Permalink

    Merge the values for each key using an associative function and a neutral "zero value" which may be added to the result an arbitrary number of times, and must not change the result (e.g., Nil for list concatenation, 0 for addition, or 1 for multiplication.).

  13. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  14. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  15. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  16. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  17. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  18. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  19. def reduceByKey(op: (V, V) ⇒ V): SCollection[(K, V)]

    Permalink

    Merge the values for each key using an associative reduce function.

    Merge the values for each key using an associative reduce function. This will also perform the merging locally on each mapper before sending results to a reducer, similarly to a "combiner" in MapReduce.

  20. def sumByKey(implicit sg: Semigroup[V]): SCollection[(K, V)]

    Permalink

    Reduce by key with Semigroup.

    Reduce by key with Semigroup. This could be more powerful and better optimized in some cases.

  21. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  22. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any
  23. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  24. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  25. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  26. def withName(name: String): SCollectionWithHotKeyFanout.this.type

    Permalink

    Set a custom name for the next transform to be applied.

    Set a custom name for the next transform to be applied.

    Definition Classes
    SCollectionWithHotKeyFanoutTransformNameable

Inherited from TransformNameable

Inherited from AnyRef

Inherited from Any

per_key

Ungrouped