scalding

Type Members

1. class BatchedDeltaService[K, V] extends BatchedService[K, V]

   Use this when you have written JUST BEFORE the store.

2. class BatchedOperations extends AnyRef

   Services and Stores are very similar, but not exact.

3. trait BatchedScaldingSink[T] extends ScaldingSink[T]

4. trait BatchedScaldingStore[K, V] extends ScaldingStore[K, V]

5. trait BatchedService[K, V] extends ScaldingService[K, V]

6. trait BatchedWindowService[K, V] extends BatchedService[K, V]

   This is a service that has a finite memory.

7. class EmptyService[K, V] extends ScaldingService[K, V]

8. type FactoryInput = (Interval[Time], Mode)

   The Platform recursively passes this input around to describe a step forward: requested input time span, and scalding Mode

9. type FailureReason = String

   These are printed/logged only when we can't make any progress

10. type FlowInput = (FlowDef, Mode)

    When it is time to run build the final flow, this is what scalding needs.

    When it is time to run build the final flow, this is what scalding needs. It is modified in the Reader[FlowInput, T]

11. case class FlowPlanException(errs: List[FailureReason]) extends Exception with Product with Serializable

    Puts a type on the final exception thrown by the scalding Planner

12. type FlowProducer[+T] = Reader[(FlowDef, Mode), T]

    This is a function that modifies a flow to return T generally T will be some kind of TypedPipe

13. type FlowToPipe[+T] = Reader[(FlowDef, Mode), TimedPipe[T]]

    We so commonly talk about producing TimedPipe we define this

14. class InitialBatchedStore[K, V] extends BatchedScaldingStore[K, V]

    For (firstNonZero - 1) we read empty.

15. type KeyValuePipe[+K, +V] = TypedPipe[(Time, (K, V))]

16. class Memo[T] extends Serializable

17. type PipeFactory[+T] = StateWithError[(Interval[Time], Mode), List[FailureReason], FlowToPipe[T]]

    We are usually producing Pipes in the Planner

18. type PlannerOutput[+T] = StateWithError[(Interval[Time], Mode), List[FailureReason], T]

    The recursive planner produces these objects which are Monads

19. trait RunningState[T] extends AnyRef

20. class Scalding extends Platform[Scalding]

21. trait ScaldingService[K, +V] extends Serializable

22. trait ScaldingSink[T] extends AnyRef

23. trait ScaldingStore[K, V] extends Serializable

24. trait SimpleService[K, V] extends ScaldingService[K, V]

    A UniqueKeyedService covers the case where Keys are globally unique and either are not present or have one value.

25. trait SimpleWindowedService[K, V] extends BatchedWindowService[K, V]

    More familiar interface to scalding users that creates the Reader from two other methods

26. trait SourceUniqueKeyedService[S <: scalding.Source, K, V] extends UniqueKeyedService[K, V]

27. class SummingbirdKryoHadoop extends KryoHadoop

28. type Time = Long

    We represent time as Long Millis

29. type TimedPipe[+T] = TypedPipe[(Time, T)]

    How we represent the streams in scalding

30. type Try[+T] = Either[List[FailureReason], T]

31. trait UniqueKeyedService[K, V] extends SimpleService[K, V]

    A UniqueKeyedService covers the case where Keys are globally unique and either are not present or have one value.

32. class VersionedBatchStore[K, V, K2, V2] extends VersionedBatchStoreBase[K, V]

33. abstract class VersionedBatchStoreBase[K, V] extends BatchedScaldingStore[K, V]

    Allows subclasses to share the means of reading version numbers but plug in methods to actually read or write the data.

34. trait WaitingState[T] extends AnyRef

    Job state models the memory of when the next job should try to cover

35. case class WriteDot(filename: String) extends Product with Serializable

    Use this option to write the logical graph that cascading produces before Map/Reduce planning.

36. case class WriteStepsDot(filename: String) extends Product with Serializable

    Use this option to write map/reduce graph that cascading produces Use the job name as the key