com.twitter.scalding

StreamOperations

trait StreamOperations[+Self <: StreamOperations[Self]] extends Sortable[Self] with Serializable

Implements reductions on top of a simple abstraction for the Fields-API We use the f-bounded polymorphism trick to return the type called Self in each operation.

Linear Supertypes
Serializable, Sortable[Self], AnyRef, Any
Known Subclasses
CoGroupBuilder, GroupBuilder
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. StreamOperations
  2. Serializable
  3. Sortable
  4. AnyRef
  5. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Abstract Value Members

  1. abstract def mapStream[T, X](fieldDef: (Fields, Fields))(mapfn: (Iterator[T]) ⇒ TraversableOnce[X])(implicit conv: TupleConverter[T], setter: TupleSetter[X]): Self

    Corresponds to a Cascading Buffer which allows you to stream through the data, keeping some, dropping, scanning, etc.

    Corresponds to a Cascading Buffer which allows you to stream through the data, keeping some, dropping, scanning, etc... The iterator you are passed is lazy, and mapping will not trigger the entire evaluation. If you convert to a list (i.e. to reverse), you need to be aware that memory constraints may become an issue.

    WARNING: Any fields not referenced by the input fields will be aligned to the first output, and the final hadoop stream will have a length of the maximum of the output of this, and the input stream. So, if you change the length of your inputs, the other fields won't be aligned. YOU NEED TO INCLUDE ALL THE FIELDS YOU WANT TO KEEP ALIGNED IN THIS MAPPING! POB: This appears to be a Cascading design decision.

    WARNING: mapfn needs to be stateless. Multiple calls needs to be safe (no mutable state captured)

  2. abstract def sortBy(innerSort: Fields): Self

    Definition Classes
    Sortable

  3. abstract def sorting: Option[Fields]

    Definition Classes
    Sortable

Concrete Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any

  3. final def ##(): Int

    Definition Classes
    AnyRef → Any

  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any

  6. final def asInstanceOf[T0]: T0

    Definition Classes
    Any

  7. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  8. def drop(cnt: Int): Self

    Remove the first cnt elements

  9. def dropWhile[T](f: Fields)(fn: (T) ⇒ Boolean)(implicit conv: TupleConverter[T]): Self

    Drop while the predicate is true, starting at the first false, output all

  10. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  11. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any

  12. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  13. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any

  14. def hashCode(): Int

    Definition Classes
    AnyRef → Any

  15. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any

  16. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  17. final def notify(): Unit

    Definition Classes
    AnyRef

  18. final def notifyAll(): Unit

    Definition Classes
    AnyRef

  19. def scanLeft[X, T](fieldDef: (Fields, Fields))(init: X)(fn: (X, T) ⇒ X)(implicit setter: TupleSetter[X], conv: TupleConverter[T]): Self

  20. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef

  21. def take(cnt: Int): Self

    Only keep the first cnt elements

  22. def takeWhile[T](f: Fields)(fn: (T) ⇒ Boolean)(implicit conv: TupleConverter[T]): Self

    Take while the predicate is true, stopping at the first false.

    Take while the predicate is true, stopping at the first false. Output all taken elements.

  23. def toString(): String

    Definition Classes
    AnyRef → Any

  24. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  25. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  26. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from Serializable

Inherited from Sortable[Self]

Inherited from AnyRef

Inherited from Any

Ungrouped