scalding

Type Members

1. trait ExternalService[K, +V] extends Service[K, V]

   

   This represents a service that is *external* to the current job.

   This represents a service that is *external* to the current job. This does not include joins for data that is generated in the same Producer graph

2. type FactoryInput = (Interval[Timestamp], Mode)

   

   The Platform recursively passes this input around to describe a step forward: requested input time span, and scalding Mode

3. type FailureReason = String

   

   These are printed/logged only when we can't make any progress

4. type FlowInput = (FlowDef, Mode)

   

   When it is time to run build the final flow, this is what scalding needs.

   When it is time to run build the final flow, this is what scalding needs. It is modified in the Reader[FlowInput, T]

5. case class FlowPlanException(errs: List[FailureReason]) extends Exception with Product with Serializable

   

   Puts a type on the final exception thrown by the scalding Planner

6. type FlowProducer[+T] = Reader[FlowInput, T]

   

   This is a function that modifies a flow to return T generally T will be some kind of TypedPipe

7. type FlowToPipe[+T] = Reader[FlowInput, TimedPipe[T]]

   

   We so commonly talk about producing TimedPipe we define this

8. sealed trait InternalService[K, +V] extends Service[K, V]

   

   This represents a join against data that is materialized by a store in the current job

9. type KeyValuePipe[+K, +V] = TypedPipe[(Timestamp, (K, V))]

   

10. class Memo[T] extends Serializable

    

11. type PipeFactory[+T] = StateWithError[FactoryInput, List[FailureReason], FlowToPipe[T]]

    

    We are usually producing Pipes in the Planner

12. class PipeFactoryOps[+T] extends AnyRef

    

13. type PlannerOutput[+T] = StateWithError[FactoryInput, List[FailureReason], T]

    

    The recursive planner produces these objects which are Monads

14. class Scalding extends Platform[Scalding] with Serializable

    

15. trait ScaldingExecutionConfig extends ChillExecutionConfig[Scalding]

    

16. sealed trait Service[K, +V] extends Serializable

    

17. trait Sink[T] extends AnyRef

    

18. trait Store[K, V] extends Serializable

    

19. case class StoreService[K, V](store: BatchedStore[K, V]) extends InternalService[K, V] with Store[K, V] with Product with Serializable

    

20. type TimedPipe[+T] = TypedPipe[(Timestamp, T)]

    

    How we represent the streams in scalding

21. type Try[+T] = Either[List[FailureReason], T]

    

22. case class WriteDot(filename: String) extends Product with Serializable

    

    Use this option to write the logical graph that cascading produces before Map/Reduce planning.

    Use this option to write the logical graph that cascading produces before Map/Reduce planning. Use the job name as the key

23. case class WriteStepsDot(filename: String) extends Product with Serializable

    

    Use this option to write map/reduce graph that cascading produces Use the job name as the key