org.apache.spark.scheduler.cluster.mesos

MesosSchedulerUtils

trait MesosSchedulerUtils extends Logging

Shared trait for implementing a Mesos Scheduler. This holds common state and helper methods and Mesos scheduler will use.

Linear Supertypes
Logging, AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. MesosSchedulerUtils
  2. Logging
  3. AnyRef
  4. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any

  3. final def ##(): Int

    Definition Classes
    AnyRef → Any

  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any

  6. final def asInstanceOf[T0]: T0

    Definition Classes
    Any

  7. def checkPorts(conf: SparkConf, ports: List[(Long, Long)]): Boolean

    Checks executor ports if they are within some range of the offered list of ports ranges,

    Checks executor ports if they are within some range of the offered list of ports ranges,

    conf

    the Spark Config

    ports

    the list of ports to check

    returns

    true if ports are within range false otherwise

    Attributes
    protected

  8. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  9. def createResource(resourceName: String, quantity: Double): Resource

    Build a Mesos resource protobuf object

    Build a Mesos resource protobuf object

    Attributes
    protected

  10. def createResource(name: String, amount: Double, role: Option[String] = None): Resource

  11. def createSchedulerDriver(masterUrl: String, scheduler: Scheduler, sparkUser: String, appName: String, conf: SparkConf, webuiUrl: Option[String] = None, checkpoint: Option[Boolean] = None, failoverTimeout: Option[Double] = None, frameworkId: Option[String] = None): SchedulerDriver

    Creates a new MesosSchedulerDriver that communicates to the Mesos master.

    Creates a new MesosSchedulerDriver that communicates to the Mesos master.

    masterUrl

    The url to connect to Mesos master

    scheduler

    the scheduler class to receive scheduler callbacks

    sparkUser

    User to impersonate with when running tasks

    appName

    The framework name to display on the Mesos UI

    conf

    Spark configuration

    webuiUrl

    The WebUI url to link from Mesos UI

    checkpoint

    Option to checkpoint tasks for failover

    failoverTimeout

    Duration Mesos master expect scheduler to reconnect on disconnect

    frameworkId

    The id of the new framework

    Attributes
    protected

  12. def declineOffer(driver: SchedulerDriver, offer: Offer, reason: Option[String] = None, refuseSeconds: Option[Long] = None): Unit

    Attributes
    protected

  13. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  14. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any

  15. def executorMemory(sc: SparkContext): Int

    Return the amount of memory to allocate to each executor, taking into account container overheads.

    Return the amount of memory to allocate to each executor, taking into account container overheads.

    sc

    SparkContext to use to get spark.mesos.executor.memoryOverhead value

    returns

    memory requirement as (0.1 * memoryOverhead) or MEMORY_OVERHEAD_MINIMUM (whichever is larger)

  16. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  17. def getAttribute(attr: Attribute): (String, Set[String])

    Helper method to get the key,value-set pair for a Mesos Attribute protobuf

    Helper method to get the key,value-set pair for a Mesos Attribute protobuf

    Attributes
    protected

  18. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any

  19. def getRangeResource(res: List[Resource], name: String): List[(Long, Long)]

    Transforms a range resource to a list of ranges

    Transforms a range resource to a list of ranges

    res

    the mesos resource list

    name

    the name of the resource

    returns

    the list of ranges returned

    Attributes
    protected

  20. def getRejectOfferDuration(conf: SparkConf): Long

    Attributes
    protected

  21. def getRejectOfferDurationForReachedMaxCores(conf: SparkConf): Long

    Attributes
    protected

  22. def getRejectOfferDurationForUnmetConstraints(conf: SparkConf): Long

    Attributes
    protected

  23. def getResource(res: List[Resource], name: String): Double

  24. def hashCode(): Int

    Definition Classes
    AnyRef → Any

  25. def initializeLogIfNecessary(isInterpreter: Boolean): Unit

    Attributes
    protected
    Definition Classes
    Logging

  26. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any

  27. def isTraceEnabled(): Boolean

    Attributes
    protected
    Definition Classes
    Logging

  28. def log: Logger

    Attributes
    protected
    Definition Classes
    Logging

  29. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  30. def logDebug(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  31. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  32. def logError(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  33. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  34. def logInfo(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  35. def logName: String

    Attributes
    protected
    Definition Classes
    Logging

  36. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  37. def logTrace(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  38. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  39. def logWarning(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  40. val managedPortNames: List[String]

  41. def markErr(): Unit

    Attributes
    protected

  42. def markRegistered(): Unit

    Signal that the scheduler has registered with Mesos.

    Signal that the scheduler has registered with Mesos.

    Attributes
    protected

  43. def matchesAttributeRequirements(slaveOfferConstraints: Map[String, Set[String]], offerAttributes: Map[String, GeneratedMessage]): Boolean

    Match the requirements (if any) to the offer attributes.

    Match the requirements (if any) to the offer attributes. if attribute requirements are not specified - return true else if attribute is defined and no values are given, simple attribute presence is performed else if attribute name and value is specified, subset match is performed on slave attributes

  44. def mesosToTaskState(state: mesos.Protos.TaskState): TaskState.TaskState

  45. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  46. def nonZeroPortValuesFromConfig(conf: SparkConf): List[Long]

    The values of the non-zero ports to be used by the executor process.

    The values of the non-zero ports to be used by the executor process.

    conf

    the spark config to use

    returns

    the ono-zero values of the ports

  47. final def notify(): Unit

    Definition Classes
    AnyRef

  48. final def notifyAll(): Unit

    Definition Classes
    AnyRef

  49. def parseConstraintString(constraintsVal: String): Map[String, Set[String]]

    Parses the attributes constraints provided to spark and build a matching data struct: Map[<attribute-name>, Set[values-to-match] The constraints are specified as ';' separated key-value pairs where keys and values are separated by ':'.

    Parses the attributes constraints provided to spark and build a matching data struct: Map[<attribute-name>, Set[values-to-match] The constraints are specified as ';' separated key-value pairs where keys and values are separated by ':'. The ':' implies equality (for singular values) and "is one of" for multiple values (comma separated). For example:

    parseConstraintString("os:centos7;zone:us-east-1a,us-east-1b")
// would result in
<code>
Map(
  "os" -> Set("centos7"),
  "zone":   -> Set("us-east-1a", "us-east-1b")
)

    Mesos documentation: http://mesos.apache.org/documentation/attributes-resources/ https://github.com/apache/mesos/blob/master/src/common/values.cpp https://github.com/apache/mesos/blob/master/src/common/attributes.cpp

    constraintsVal

    constaints string consisting of ';' separated key-value pairs (separated by ':')

    returns

    Map of constraints to match resources offers.

  50. def partitionPortResources(requestedPorts: List[Long], offeredResources: List[Resource]): (List[Resource], List[Resource])

    Partitions port resources.

    Partitions port resources.

    requestedPorts

    non-zero ports to assign

    offeredResources

    the resources offered

    returns

    resources left, port resources to be used.

  51. def partitionResources(resources: List[Resource], resourceName: String, amountToUse: Double): (List[Resource], List[Resource])

    Partition the existing set of resources into two groups, those remaining to be scheduled and those requested to be used for a new task.

    Partition the existing set of resources into two groups, those remaining to be scheduled and those requested to be used for a new task.

    resources

    The full list of available resources

    resourceName

    The name of the resource to take from the available resources

    amountToUse

    The amount of resources to take from the available resources

    returns

    The remaining resources list and the used resources list.

  52. def setupUris(uris: String, builder: Builder, useFetcherCache: Boolean = false): Unit

  53. def startScheduler(newDriver: SchedulerDriver): Unit

    Starts the MesosSchedulerDriver and stores the current running driver to this new instance.

    Starts the MesosSchedulerDriver and stores the current running driver to this new instance. This driver is expected to not be running. This method returns only after the scheduler has registered with Mesos.

  54. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef

  55. def taskStateToMesos(state: TaskState.TaskState): mesos.Protos.TaskState

  56. def toAttributeMap(offerAttributes: List[Attribute]): Map[String, GeneratedMessage]

    Converts the attributes from the resource offer into a Map of name to Attribute Value The attribute values are the mesos attribute types and they are

    Converts the attributes from the resource offer into a Map of name to Attribute Value The attribute values are the mesos attribute types and they are

    offerAttributes

    the attributes offered

    returns

    Attributes
    protected

  57. def toString(): String

    Definition Classes
    AnyRef → Any

  58. def unsetFrameworkID(sc: SparkContext): Unit

    spark.

    spark.mesos.driver.frameworkId is set by the cluster dispatcher to correlate driver submissions with frameworkIDs. However, this causes issues when a driver process launches more than one framework (more than one SparkContext(, because they all try to register with the same frameworkID. To enforce that only the first driver registers with the configured framework ID, the driver calls this method after the first registration.

  59. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  60. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  61. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from Logging

Inherited from AnyRef

Inherited from Any

Ungrouped