Access to mist-specific job parameters
Get access to mist-extras in job definition Example:
Scala api - root class for jobs definition
Scala api - root class for jobs definition
Example:
import mist.api._ import org.apache.spark.SparkContext object MyJob extends MistFn { override def handle = { withArgs(arg[Int]("number").onSparkContext((i: Int, sc: SparkContext) => { sc.parallelize(1 to i).map(_ * 2).collect() }) } }
Provide context combinators to complete job definition, that can take some another arguments + spark computational context.
Provide context combinators to complete job definition, that can take some another arguments + spark computational context.
Available contexts:
There are two ways how to define job using that standard combinators:
For job which doesn't require any external argument except context use one of bellow functions:
In case when you have arguments you can call that functions on them
withArgs(arg[Int]("x") & arg[String]("str")).onSparkContext( (x: Int, str: String, sc: SparkContext) => { ... })
Scala dsl to start job definition like withArgs(a,b...n).onSparkContext(..
For java
Get access to mist-extras in job definition Example: