typed

Type Members

1. class BijectedSourceSink[T, U] extends TypedSource[U] with TypedSink[U]

2. trait CoGroupable[K, +R] extends HasReducers with Serializable

   Represents something than can be CoGrouped with another CoGroupable

3. trait CoGrouped[K, +R] extends KeyedListLike[K, R, CoGrouped] with CoGroupable[K, R] with WithReducers[CoGrouped[K, R]]

4. abstract class CoGroupedJoiner[K] extends Joiner

5. case class ComputedValue[T](toTypedPipe: TypedPipe[T]) extends ValuePipe[T] with Product with Serializable

6. case class Converter[R](conv: TupleConverter[R]) extends FlatMapFn[R] with Product with Serializable

7. class DistinctCoGroupJoiner[K] extends CoGroupedJoiner[K]

8. case class FilteredFn[R](fmap: FlatMapFn[R], fn: (R) ⇒ Boolean) extends FlatMapFn[R] with Product with Serializable

9. sealed trait FlatMapFn[+R] extends (TupleEntry) ⇒ TraversableOnce[R] with Serializable

   Closures are difficult for serialization.

10. case class FlatMappedFn[T, R](fmap: FlatMapFn[T], fn: (T) ⇒ TraversableOnce[R]) extends FlatMapFn[R] with Product with Serializable

11. trait Grouped[K, +V] extends KeyedListLike[K, V, UnsortedGrouped] with HashJoinable[K, V] with Sortable[V, [+x]SortedGrouped[K, x] with Reversable[SortedGrouped[K, x]]] with WithReducers[Grouped[K, V]]

    This encodes the rules that 1) sorting is only possible before doing any reduce, 2) reversing is only possible after sorting.

12. trait HasReducers extends AnyRef

    used for types that may know how many reducers they need e.

13. trait HashJoinable[K, +V] extends CoGroupable[K, V] with KeyedPipe[K]

    If we can HashJoin, then we can CoGroup, but not vice-versa i.

14. class HashJoiner[K, V, W, R] extends Joiner

    Only intended to be use to implement the hashCogroup on TypedPipe/Grouped

15. case class IdentityReduce[K, V1](keyOrdering: Ordering[K], mapped: TypedPipe[(K, V1)], reducers: Option[Int]) extends ReduceStep[K, V1] with Grouped[K, V1] with Product with Serializable

16. case class IdentityValueSortedReduce[K, V1](keyOrdering: Ordering[K], mapped: TypedPipe[(K, V1)], valueSort: Ordering[_ >: V1], reducers: Option[Int]) extends ReduceStep[K, V1] with SortedGrouped[K, V1] with Reversable[IdentityValueSortedReduce[K, V1]] with Product with Serializable

17. final case class IterablePipe[T](iterable: Iterable[T]) extends TypedPipe[T] with Product with Serializable

    You should use a view here If you avoid toPipe, this class is more efficient than IterableSource.

18. case class IteratorMappedReduce[K, V1, V2](keyOrdering: Ordering[K], mapped: TypedPipe[(K, V1)], reduceFn: (K, Iterator[V1]) ⇒ Iterator[V2], reducers: Option[Int]) extends ReduceStep[K, V1] with UnsortedGrouped[K, V2] with Product with Serializable

19. trait KeyedList[K, +T] extends KeyedListLike[K, T, KeyedList]

    This is for the case where you don't want to expose any structure but the ability to operate on an iterator of the values

20. trait KeyedListLike[K, +T, +This[K, +T] <: KeyedListLike[K, T, This]] extends Serializable

    Represents sharded lists of items of type T There are exactly two the fundamental operations: toTypedPipe: marks the end of the grouped-on-key operations.

21. trait KeyedPipe[K] extends AnyRef

    Represents anything that starts as a TypedPipe of Key Value, where the value type has been erased.

22. case class LiteralValue[T](value: T) extends ValuePipe[T] with Product with Serializable

23. case class MapFn[T, R](fmap: FlatMapFn[T], fn: (T) ⇒ R) extends FlatMapFn[R] with Product with Serializable

24. class MappablePipeJoinEnrichment[T] extends AnyRef

25. class MemorySink[T] extends TypedSink[T]

26. final case class MergedTypedPipe[T](left: TypedPipe[T], right: TypedPipe[T]) extends TypedPipe[T] with Product with Serializable

27. sealed trait NoStackAndThen[-A, +B] extends Serializable

    This type is used to implement .

28. trait PartitionSchemed[P, T] extends SchemedSource with TypedSink[(P, T)] with Mappable[(P, T)]

    Trait to assist with creating partitioned sources.

29. trait PartitionedDelimited extends Serializable

    Trait to assist with creating objects such as PartitionedTsv to read from separated files.

30. case class PartitionedDelimitedSource[P, T](path: String, template: String, separator: String, fields: Fields, skipHeader: Boolean = false, writeHeader: Boolean = false, quote: String = "\"", strict: Boolean = true, safe: Boolean = true)(implicit mt: Manifest[T], valueSetter: TupleSetter[T], valueConverter: TupleConverter[T], partitionSetter: TupleSetter[P], partitionConverter: TupleConverter[P]) extends SchemedSource with PartitionSchemed[P, T] with Serializable with Product with Serializable

    Scalding source to read or write partitioned delimited text.

31. case class PartitionedTextLine[P](path: String, template: String, encoding: String = ...)(implicit valueSetter: TupleSetter[String], valueConverter: TupleConverter[(Long, String)], partitionSetter: TupleSetter[P], partitionConverter: TupleConverter[P]) extends SchemedSource with TypedSink[(P, String)] with Mappable[(P, (Long, String))] with Serializable with Product with Serializable

    Scalding source to read or write partitioned text.

32. class PipeTExtensions extends Serializable

33. sealed trait ReduceStep[K, V1] extends KeyedPipe[K]

    This is a class that models the logical portion of the reduce step.

34. trait Reversable[+R] extends AnyRef

35. case class SketchJoined[K, V, V2, R](left: Sketched[K, V], right: TypedPipe[(K, V2)], numReducers: Int)(joiner: (K, V, Iterable[V2]) ⇒ Iterator[R])(implicit evidence$1: Ordering[K]) extends HasReducers with Product with Serializable

36. case class Sketched[K, V](pipe: TypedPipe[(K, V)], numReducers: Int, delta: Double, eps: Double, seed: Int)(implicit serialization: (K) ⇒ Array[Byte], ordering: Ordering[K]) extends HasReducers with Product with Serializable

37. trait Sortable[+T, +Sorted[+_]] extends AnyRef

    All sorting methods defined here trigger Hadoop secondary sort on key + value.

38. trait SortedGrouped[K, +V] extends KeyedListLike[K, V, SortedGrouped] with WithReducers[SortedGrouped[K, V]]

    After sorting, we are no longer CoGroupable, and we can only call reverse in the initial SortedGrouped created from the Sortable: .

39. case class TemplatePartition(partitionFields: Fields, template: String) extends Partition with Product with Serializable

    Creates a partition using the given template string.

40. trait TypedPipe[+T] extends Serializable

    Think of a TypedPipe as a distributed unordered list that may or may not yet have been materialized in memory or disk.

41. class TypedPipeFactory[T] extends TypedPipe[T]

    This is a TypedPipe that delays having access to the FlowDef and Mode until toPipe is called

42. class TypedPipeInst[T] extends TypedPipe[T]

    This is an instance of a TypedPipe that wraps a cascading Pipe

43. trait TypedSink[-T] extends Serializable

    Opposite of TypedSource, used for writing into

44. trait TypedSource[+T] extends Serializable

45. trait UnsortedGrouped[K, +V] extends KeyedListLike[K, V, UnsortedGrouped] with HashJoinable[K, V] with WithReducers[UnsortedGrouped[K, V]]

    This is the state after we have done some reducing.

46. case class UnsortedIdentityReduce[K, V1](keyOrdering: Ordering[K], mapped: TypedPipe[(K, V1)], reducers: Option[Int]) extends ReduceStep[K, V1] with UnsortedGrouped[K, V1] with Product with Serializable

47. sealed trait ValuePipe[+T] extends Serializable

    ValuePipe is special case of a TypedPipe of just a optional single element.

48. case class ValueSortedReduce[K, V1, V2](keyOrdering: Ordering[K], mapped: TypedPipe[(K, V1)], valueSort: Ordering[_ >: V1], reduceFn: (K, Iterator[V1]) ⇒ Iterator[V2], reducers: Option[Int]) extends ReduceStep[K, V1] with SortedGrouped[K, V2] with Product with Serializable

49. class WithOnComplete[T] extends TypedPipe[T]

50. trait WithReducers[+This <: WithReducers[This]] extends HasReducers

    used for objects that may _set_ how many reducers they need e.