api

Type Members

1. trait ArgDef[A] extends AnyRef

   

2. sealed trait ArgInfo extends AnyRef

   

3. sealed trait ArgType extends AnyRef

   

4. trait ArgsInstances extends AnyRef

   

5. final case class Extracted[+A](value: A) extends Extraction[A] with Product with Serializable

   

6. sealed trait Extraction[+A] extends AnyRef

   

7. sealed trait Failed extends Extraction[Nothing]

   

8. sealed trait FnContext extends AnyRef

   

9. trait FnEntryPoint extends AnyRef

   

10. case class FullFnContext(sc: SparkContext, streamingDuration: Duration, info: RuntimeJobInfo, params: JsMap) extends FnContext with Product with Serializable

    

11. trait Handle extends RawHandle[JsData]

    

12. case class InternalArgument(tags: Seq[String] = Seq.empty) extends ArgInfo with Product with Serializable

    

13. trait Logging extends AnyRef

    

14. final case class MList(v: ArgType) extends ArgType with Product with Serializable

    

15. final case class MMap(k: ArgType, v: ArgType) extends RootArgType with Product with Serializable

    

16. final case class MObj(fields: Seq[(String, ArgType)]) extends RootArgType with Product with Serializable

    

17. final case class MOption(v: ArgType) extends ArgType with Product with Serializable

    

18. class MistExtras extends AnyRef

    

    Access to mist-specific job parameters

19. trait MistExtrasDef extends AnyRef

    

    Get access to mist-extras in job definition Example:

    Get access to mist-extras in job definition Example:

    withMistExtras.onSparkContext((extras: MistExtras, sc: SparkContext) => {
       val jobId = extras.jobId
    })

20. abstract class MistFn extends FnEntryPoint

    

    Scala api - root class for jobs definition

    Scala api - root class for jobs definition

    Example:

    import mist.api._
    import org.apache.spark.SparkContext
    
    object MyJob extends MistFn {
      override def handle = {
        withArgs(arg[Int]("number").onSparkContext((i: Int, sc: SparkContext) => {
          sc.parallelize(1 to i).map(_ * 2).collect()
        })
      }
    }

21. trait MistFnSyntax extends AnyRef

    

    Provide context combinators to complete job definition, that can take some another arguments + spark computational context.

    Provide context combinators to complete job definition, that can take some another arguments + spark computational context.

    Available contexts:

    • org.apache.spark.SparkContext
    • org.apache.spark.streaming.StreamingContext
    • org.apache.spark.sql.SQLContext
    • org.apache.spark.sql.hive.HiveContext

    There are two ways how to define job using that standard combinators:

    For job which doesn't require any external argument except context use one of bellow functions:

    • onSparkContext((spark: SparkContext) => {...})
    • onStreamingContext((ssc: StreamingContext) => {...})
    • onSqlContext((sqlCtx: SQLContext) => {...})
    • onHiveContext((hiveCtx: HiveContext) => {...})

    In case when you have arguments you can call that functions on them

    withArgs(arg[Int]("x") & arg[String]("str")).onSparkContext(
      (x: Int, str: String, sc: SparkContext) => {
        ...
    })

22. class NamedUserArg[A] extends UserArg[A]

    

23. class NamedUserArgWithDefault[A] extends UserArg[A]

    

24. trait RawHandle[A] extends AnyRef

    

25. sealed trait RootArgType extends ArgType

    

26. case class RuntimeJobInfo(id: String, workerId: String) extends Product with Serializable

    

27. trait SparkArgs extends AnyRef

    

28. trait SystemArg[A] extends ArgDef[A]

    

29. trait UserArg[A] extends ArgDef[A]

    

30. case class UserInputArgument(name: String, t: ArgType) extends ArgInfo with Product with Serializable

    

31. trait WithArgsScala extends AnyRef

    

    Scala dsl to start job definition like withArgs(a,b...n).onSparkContext(..