Package

mist

api

Permalink

package api

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. api
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Type Members

  1. trait BaseContexts extends AnyRef

    Permalink

    Provide context combinators to complete job definition, that can take some another arguments + spark computational context.

    Provide context combinators to complete job definition, that can take some another arguments + spark computational context.

    Available contexts:

    • org.apache.spark.SparkContext
    • org.apache.spark.streaming.StreamingContext
    • org.apache.spark.sql.SQLContext
    • org.apache.spark.sql.hive.HiveContext

    There are two ways how to define job using that standard combinators:

    For job which doesn't require any external argument except context use one of bellow functions:

    • onSparkContext((spark: SparkContext) => {...})
    • onStreamingContext((ssc: StreamingContext) => {...})
    • onSqlContext((sqlCtx: SQLContext) => {...})
    • onHiveContext((hiveCtx: HiveContext) => {...})

    In case when you have arguments you can call that functions on them

    withArgs(arg[Int]("x") & arg[String]("str")).onSparkContext(
      (x: Int, str: String, sc: SparkContext) => {
        ...
    })
  2. trait Contexts extends BaseContexts

    Permalink
  3. sealed trait FnContext extends AnyRef

    Permalink
  4. case class FullFnContext(setupConf: SetupConfiguration, params: Map[String, Any]) extends FnContext with Product with Serializable

    Permalink
  5. trait Handle[A] extends AnyRef

    Permalink
  6. final case class JobFailure[+A](e: Throwable) extends JobResult[A] with Product with Serializable

    Permalink
  7. sealed trait JobResult[+A] extends AnyRef

    Permalink
  8. final case class JobSuccess[+A](value: A) extends JobResult[A] with Product with Serializable

    Permalink
  9. type MLogger = io.hydrosphere.mist.api.MLogger

    Permalink
  10. class MistExtras extends AnyRef

    Permalink

    Access to mist-specific job parameters + logger

  11. trait MistExtrasDef extends AnyRef

    Permalink

    Get access to mist-extras in job definition Example:

    Get access to mist-extras in job definition Example:

    withMistExtras.onSparkContext((extras: MistExtras, sc: SparkContext) => {
       val jobId = extras.jobId
       extras.logger.info(s"Hello from my job $jobId")
    })
  12. abstract class MistFn[A] extends ArgsInstances with Contexts with MistExtrasDef with WithArgsScala

    Permalink

    Scala api - root class for jobs definition

    Scala api - root class for jobs definition

    Example:

    import mist.api._
    import mist.api.DefaultEncoders._
    import org.apache.spark.SparkContext
    
    object MyJob extends MistJob[Array[Int]] {
      override def handle = {
        withArgs(arg[Int]("number").onSparkContext((i: Int, sc: SparkContext) => {
          sc.parallelize(1 to i).map(_ * 2).collect()
        })
      }
    }

Value Members

  1. object BaseContexts extends BaseContexts

    Permalink
  2. object BaseContextsArgs

    Permalink

    Arguments for constructing contexts

  3. object Contexts extends Contexts

    Permalink
  4. object FnContext

    Permalink
  5. object Handle

    Permalink
  6. object JobResult

    Permalink
  7. object MistExtras

    Permalink
  8. object MistExtrasDef extends MistExtrasDef

    Permalink
  9. object SessionArgs

    Permalink
  10. package args

    Permalink
  11. package data

    Permalink
  12. package encoding

    Permalink
  13. package internal

    Permalink
  14. package jdsl

    Permalink

Inherited from AnyRef

Inherited from Any

Ungrouped