scalding

Type Members

1. class BatchedDeltaService[K, V] extends BatchedService[K, V]

   Use this when you have written JUST BEFORE the store.

2. class BatchedOperations extends AnyRef

   Services and Stores are very similar, but not exact.

3. trait BatchedScaldingSink[T] extends ScaldingSink[T]

4. trait BatchedScaldingStore[K, V] extends ScaldingStore[K, V]

5. trait BatchedService[K, V] extends ScaldingService[K, V]

6. trait BatchedWindowService[K, V] extends BatchedService[K, V]

   This is a service that has a finite memory.

7. class DirectoryBatchedStore[K <: Writable, V <: Writable] extends BatchedScaldingStore[K, V]

   DirectoryBatched Scalding Store, which only contains (K, V) data pairs in the data.

8. class EmptyService[K, V] extends ScaldingService[K, V]

9. type FactoryInput = (Interval[Time], Mode)

   The Platform recursively passes this input around to describe a step forward: requested input time span, and scalding Mode

10. type FailureReason = String

    These are printed/logged only when we can't make any progress

11. type FlowInput = (FlowDef, Mode)

    When it is time to run build the final flow, this is what scalding needs.

    When it is time to run build the final flow, this is what scalding needs. It is modified in the Reader[FlowInput, T]

12. case class FlowPlanException(errs: List[FailureReason]) extends Exception with Product with Serializable

    Puts a type on the final exception thrown by the scalding Planner

13. type FlowProducer[+T] = Reader[(FlowDef, Mode), T]

    This is a function that modifies a flow to return T generally T will be some kind of TypedPipe

14. type FlowToPipe[+T] = Reader[(FlowDef, Mode), TimedPipe[T]]

    We so commonly talk about producing TimedPipe we define this

15. class InitialBatchedStore[K, V] extends BatchedScaldingStore[K, V]

    For (firstNonZero - 1) we read empty.

16. type KeyValuePipe[+K, +V] = TypedPipe[(Time, (K, V))]

17. class Memo[T] extends Serializable

18. type PipeFactory[+T] = StateWithError[(Interval[Time], Mode), List[FailureReason], FlowToPipe[T]]

    We are usually producing Pipes in the Planner

19. type PlannerOutput[+T] = StateWithError[(Interval[Time], Mode), List[FailureReason], T]

    The recursive planner produces these objects which are Monads

20. trait PrepareState[T] extends AnyRef

21. trait RunningState[T] extends AnyRef

22. class Scalding extends Platform[Scalding]

23. trait ScaldingExecutionConfig extends ChillExecutionConfig[Scalding]

24. trait ScaldingService[K, +V] extends Serializable

25. trait ScaldingSink[T] extends AnyRef

26. trait ScaldingStore[K, V] extends Serializable

27. trait SimpleService[K, V] extends ScaldingService[K, V]

    A UniqueKeyedService covers the case where Keys are globally unique and either are not present or have one value.

28. trait SimpleWindowedService[K, V] extends BatchedWindowService[K, V]

    More familiar interface to scalding users that creates the Reader from two other methods

29. trait SourceUniqueKeyedService[S <: scalding.Source, K, V] extends UniqueKeyedService[K, V]

30. type Time = Long

    We represent time as Long Millis

31. type TimedPipe[+T] = TypedPipe[(Time, T)]

    How we represent the streams in scalding

32. type Try[+T] = Either[List[FailureReason], T]

33. trait UniqueKeyedService[K, V] extends SimpleService[K, V]

    A UniqueKeyedService covers the case where Keys are globally unique and either are not present or have one value.

34. class VersionedBatchStore[K, V, K2, V2] extends VersionedBatchStoreBase[K, V]

35. abstract class VersionedBatchStoreBase[K, V] extends BatchedScaldingStore[K, V]

    Allows subclasses to share the means of reading version numbers but plug in methods to actually read or write the data.

36. trait WaitingState[T] extends AnyRef

    Job state models the memory of when the next job should try to cover

37. class WrappedHadoopConfig extends ReadableMap

38. case class WriteDot(filename: String) extends Product with Serializable

    Use this option to write the logical graph that cascading produces before Map/Reduce planning.

39. case class WriteStepsDot(filename: String) extends Product with Serializable

    Use this option to write map/reduce graph that cascading produces Use the job name as the key