cluster

Type Members

1. trait SchedulerExtensionService extends AnyRef

   

   An extension service that can be loaded into a Spark YARN scheduler.

   An extension service that can be loaded into a Spark YARN scheduler. A Service that can be started and stopped.

   1. For implementations to be loadable by SchedulerExtensionServices, they must provide an empty constructor. 2. The stop() operation MUST be idempotent, and succeed even if start() was never invoked.

2. case class SchedulerExtensionServiceBinding(sparkContext: SparkContext, applicationId: ApplicationId, attemptId: Option[ApplicationAttemptId] = None) extends Product with Serializable

   

   Binding information for a SchedulerExtensionService.

   Binding information for a SchedulerExtensionService.

   The attempt ID will be set if the service is started within a YARN application master; there is then a different attempt ID for every time that AM is restarted. When the service binding is instantiated in client mode, there's no attempt ID, as it lacks this information.

   sparkContext

   current spark context

   applicationId

   YARN application ID

   attemptId

   YARN attemptID. This will always be unset in client mode, and always set in cluster mode.