org.apache.spark.deploy.yarn

ClientDistributedCacheManager

class ClientDistributedCacheManager extends Logging

Client side methods to setup the Hadoop distributed cache

Linear Supertypes
Logging, AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. ClientDistributedCacheManager
  2. Logging
  3. AnyRef
  4. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new ClientDistributedCacheManager()

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any

  3. final def ##(): Int

    Definition Classes
    AnyRef → Any

  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any

  6. def addResource(fs: FileSystem, conf: Configuration, destPath: Path, localResources: HashMap[String, LocalResource], resourceType: LocalResourceType, link: String, statCache: Map[URI, FileStatus], appMasterOnly: Boolean = false): Unit

    Add a resource to the list of distributed cache resources.

    Add a resource to the list of distributed cache resources. This list can be sent to the ApplicationMaster and possibly the executors so that it can be downloaded into the Hadoop distributed cache for use by this application. Adds the LocalResource to the localResources HashMap passed in and saves the stats of the resources to they can be sent to the executors and verified.

    fs

    FileSystem

    conf

    Configuration

    destPath

    path to the resource

    localResources

    localResource hashMap to insert the resource into

    resourceType

    LocalResourceType

    link

    link presented in the distributed cache to the destination

    statCache

    cache to store the file/directory stats

    appMasterOnly

    Whether to only add the resource to the app master

  7. def ancestorsHaveExecutePermissions(fs: FileSystem, path: Path, statCache: Map[URI, FileStatus]): Boolean

    Returns true if all ancestors of the specified path have the 'execute' permission set for all users (i.

    Returns true if all ancestors of the specified path have the 'execute' permission set for all users (i.e. that other users can traverse the directory heirarchy to the given path)

    fs
    path
    statCache
    returns

    true if all ancestors have the 'execute' permission set for all users

  8. final def asInstanceOf[T0]: T0

    Definition Classes
    Any

  9. def checkPermissionOfOther(fs: FileSystem, path: Path, action: FsAction, statCache: Map[URI, FileStatus]): Boolean

    Checks for a given path whether the Other permissions on it imply the permission in the passed FsAction

    Checks for a given path whether the Other permissions on it imply the permission in the passed FsAction

    fs
    path
    action
    statCache
    returns

    true if the path in the uri is visible to all, false otherwise

  10. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  11. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  12. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any

  13. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  14. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any

  15. def getFileStatus(fs: FileSystem, uri: URI, statCache: Map[URI, FileStatus]): FileStatus

    Checks to see if the given uri exists in the cache, if it does it returns the existing FileStatus, otherwise it stats the uri, stores it in the cache, and returns the FileStatus.

    Checks to see if the given uri exists in the cache, if it does it returns the existing FileStatus, otherwise it stats the uri, stores it in the cache, and returns the FileStatus.

    fs
    uri
    statCache
    returns

    FileStatus

  16. def getVisibility(conf: Configuration, uri: URI, statCache: Map[URI, FileStatus]): LocalResourceVisibility

    Returns the local resource visibility depending on the cache file permissions

    Returns the local resource visibility depending on the cache file permissions

    conf
    uri
    statCache
    returns

    LocalResourceVisibility

  17. def hashCode(): Int

    Definition Classes
    AnyRef → Any

  18. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any

  19. def isPublic(conf: Configuration, uri: URI, statCache: Map[URI, FileStatus]): Boolean

    Returns a boolean to denote whether a cache file is visible to all(public) or not

    Returns a boolean to denote whether a cache file is visible to all(public) or not

    conf
    uri
    statCache
    returns

    true if the path in the uri is visible to all, false otherwise

  20. def isTraceEnabled(): Boolean

    Attributes
    protected
    Definition Classes
    Logging

  21. def log: Logger

    Attributes
    protected
    Definition Classes
    Logging

  22. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  23. def logDebug(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  24. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  25. def logError(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  26. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  27. def logInfo(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  28. def logName: String

    Attributes
    protected
    Definition Classes
    Logging

  29. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  30. def logTrace(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  31. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging

  32. def logWarning(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging

  33. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  34. final def notify(): Unit

    Definition Classes
    AnyRef

  35. final def notifyAll(): Unit

    Definition Classes
    AnyRef

  36. def setDistArchivesEnv(env: Map[String, String]): Unit

    Adds the necessary cache archive env variables to the env passed in

    Adds the necessary cache archive env variables to the env passed in

    env

  37. def setDistFilesEnv(env: Map[String, String]): Unit

    Adds the necessary cache file env variables to the env passed in

    Adds the necessary cache file env variables to the env passed in

    env

  38. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef

  39. def toString(): String

    Definition Classes
    AnyRef → Any

  40. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  41. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  42. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from Logging

Inherited from AnyRef

Inherited from Any

Ungrouped