This creates a new ExecutionContext, passes to the reader, builds the flow and cleans up the state of the FlowDef
Here is the recommended way to run scalding as a library Put all your logic is calls like this: import ExecutionContext.
Here is the recommended way to run scalding as a library Put all your logic is calls like this: import ExecutionContext._
Reader(implicit ec: ExecutionContext => //job here ) you can compose these readers in flatMaps: for { firstPipe <- job1 secondPipe <- job2 } yield firstPipe.group.join(secondPipe.join)
Note that the only config considered is in conf. The caller is responsible for setting up the Config completely.
Here is a minimal example: val future = Execution.run(Local(true), Config.default) { implicit ec: ExecutionContext => //do logic here } Or one for Hadoop: val jobConf = new JobConf val future = Execution.run(Hdfs(jobConf, true), Config.hadoopWithDefaults(jobConf)) { implicit ec: ExecutionContext => //do logic here } If you want to be synchronous, use waitFor instead of run