org.elasticsearch.spark.deploy.yarn.security

Class EsServiceCredentialProvider

java.lang.Object

org.elasticsearch.spark.deploy.yarn.security.EsServiceCredentialProvider

All Implemented Interfaces:

org.apache.spark.deploy.yarn.security.ServiceCredentialProvider

public class EsServiceCredentialProvider
extends Object
implements org.apache.spark.deploy.yarn.security.ServiceCredentialProvider

A provider interface in Spark's Yarn library that obtains tokens for an application.

When a job is submitted to a YARN cluster, the credential providers are constructed using a service loader. Each provider is queried for if its service requires a token, and if so, Spark requests that it obtains one.

In client deployment mode, these tokens are retrieved on the driver when the YARN application is first initialized and started by the main program.

In cluster deployment mode, these tokens are retrieved initially on the driver before submitting the application master that runs the main program.

If a principal and keytab are provided to a job, a credentials file is created, and a background thread is started on the application master that will obtain new tokens when they get close to expiring. Those tokens are written to an HDFS directory which the worker nodes will regularly poll to get updated tokens. If the job is launched in client mode, the client will also receive updated tokens.

Constructor Summary

Constructors

Constructor and Description
EsServiceCredentialProvider()

Method Summary

Modifier and Type	Method and Description
boolean	credentialsRequired(Configuration hadoopConf) Given a configuration, check to see if tokens would be required.
boolean	credentialsRequired(SparkConf sparkConf, Configuration hadoopConf) Given a configuration, check to see if tokens would be required.
scala.Option<Object>	obtainCredentials(Configuration hadoopConf, SparkConf sparkConf, Credentials creds) Obtains api key tokens from Elasticsearch and stashes them in the given credentials object
String	serviceName() Name of the service for logging purposes and for the purpose of determining if the service is disabled in the settings using the property spark.security.credentials.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

EsServiceCredentialProvider

public EsServiceCredentialProvider()

Method Detail

credentialsRequired

public boolean credentialsRequired(Configuration hadoopConf)

Given a configuration, check to see if tokens would be required.

Specified by:

credentialsRequired in interface org.apache.spark.deploy.yarn.security.ServiceCredentialProvider

Parameters:

hadoopConf - the current Hadoop configuration

Returns:

true if tokens should be gathered, false if they should not be

credentialsRequired

public boolean credentialsRequired(SparkConf sparkConf,
                                   Configuration hadoopConf)

Given a configuration, check to see if tokens would be required.

Parameters:

sparkConf - the current Spark configuration - used by Cloudera's CDS Spark fork (#1301)

hadoopConf - the current Hadoop configuration

Returns:

true if tokens should be gathered, false if they should not be

obtainCredentials

public scala.Option<Object> obtainCredentials(Configuration hadoopConf,
                                              SparkConf sparkConf,
                                              Credentials creds)

Obtains api key tokens from Elasticsearch and stashes them in the given credentials object

Specified by:

obtainCredentials in interface org.apache.spark.deploy.yarn.security.ServiceCredentialProvider

Parameters:

hadoopConf - Hadoop configuration, picking up all Hadoop specific settings

sparkConf - All settings that exist in Spark

creds - The credentials object that will be shared between all workers

Returns:

The expiration time for the token

serviceName

public String serviceName()

Name of the service for logging purposes and for the purpose of determining if the service is disabled in the settings using the property spark.security.credentials.[serviceName].enabled

Specified by:

serviceName in interface org.apache.spark.deploy.yarn.security.ServiceCredentialProvider

Returns:

the service name this provider corresponds to