package resource
- Alphabetic
- Public
- All
Type Members
-
class
ExecutorResourceRequest extends Serializable
An Executor resource request.
An Executor resource request. This is used in conjunction with the ResourceProfile to programmatically specify the resources needed for an RDD that will be applied at the stage level.
This is used to specify what the resource requirements are for an Executor and how Spark can find out specific details about those resources. Not all the parameters are required for every resource type. Resources like GPUs are supported and have same limitations as using the global spark configs spark.executor.resource.gpu.*. The amount, discoveryScript, and vendor parameters for resources are all the same parameters a user would specify through the configs: spark.executor.resource.{resourceName}.{amount, discoveryScript, vendor}.
For instance, a user wants to allocate an Executor with GPU resources on YARN. The user has to specify the resource name (gpu), the amount or number of GPUs per Executor, the discovery script would be specified so that when the Executor starts up it can discovery what GPU addresses are available for it to use because YARN doesn't tell Spark that, then vendor would not be used because its specific for Kubernetes.
See the configuration and cluster specific docs for more details.
Use ExecutorResourceRequests class as a convenience API.
- Annotations
- @Evolving() @Since( "3.1.0" )
-
class
ExecutorResourceRequests extends Serializable
A set of Executor resource requests.
A set of Executor resource requests. This is used in conjunction with the ResourceProfile to programmatically specify the resources needed for an RDD that will be applied at the stage level.
- Annotations
- @Evolving() @Since( "3.1.0" )
-
class
ResourceDiscoveryScriptPlugin extends ResourceDiscoveryPlugin with Logging
The default plugin that is loaded into a Spark application to control how custom resources are discovered.
The default plugin that is loaded into a Spark application to control how custom resources are discovered. This executes the discovery script specified by the user and gets the json output back and constructs ResourceInformation objects from that. If the user specifies custom plugins, this is the last one to be executed and throws if the resource isn't discovered.
- Annotations
- @DeveloperApi()
- Since
3.0.0
-
class
ResourceID extends AnyRef
Resource identifier.
Resource identifier.
- Annotations
- @DeveloperApi()
- Since
3.0.0
-
class
ResourceInformation extends Serializable
Class to hold information about a type of Resource.
Class to hold information about a type of Resource. A resource could be a GPU, FPGA, etc. The array of addresses are resource specific and its up to the user to interpret the address.
One example is GPUs, where the addresses would be the indices of the GPUs
- Annotations
- @Evolving()
- Since
3.0.0
-
class
ResourceProfile extends Serializable with Logging
Resource profile to associate with an RDD.
Resource profile to associate with an RDD. A ResourceProfile allows the user to specify executor and task requirements for an RDD that will get applied during a stage. This allows the user to change the resource requirements between stages. This is meant to be immutable so user can't change it after building. Users should use ResourceProfileBuilder to build it.
- Annotations
- @Evolving() @Since( "3.1.0" )
-
class
ResourceProfileBuilder extends AnyRef
Resource profile builder to build a ResourceProfile to associate with an RDD.
Resource profile builder to build a ResourceProfile to associate with an RDD. A ResourceProfile allows the user to specify executor and task resource requirements for an RDD that will get applied during a stage. This allows the user to change the resource requirements between stages.
- Annotations
- @Evolving() @Since( "3.1.0" )
-
class
ResourceRequest extends AnyRef
Class that represents a resource request.
Class that represents a resource request.
The class used when discovering resources (using the discovery script), or via the context as it is parsing configuration for the ResourceID.
- Annotations
- @DeveloperApi()
- Since
3.0.0
-
class
TaskResourceRequest extends Serializable
A task resource request.
A task resource request. This is used in conjunction with the ResourceProfile to programmatically specify the resources needed for an RDD that will be applied at the stage level.
Use TaskResourceRequests class as a convenience API.
- Annotations
- @Evolving() @Since( "3.1.0" )
-
class
TaskResourceRequests extends Serializable
A set of task resource requests.
A set of task resource requests. This is used in conjunction with the ResourceProfile to programmatically specify the resources needed for an RDD that will be applied at the stage level.
- Annotations
- @Evolving() @Since( "3.1.0" )
Value Members
- object ResourceProfile extends Logging with Serializable