org.apache.hadoop.hbase.util

类 CommonFSUtils

java.lang.Object

org.apache.hadoop.hbase.util.CommonFSUtils

@InterfaceAudience.Private
public abstract class CommonFSUtils
extends Object

Utility methods for interacting with the underlying file system.

嵌套类概要

嵌套类

限定符和类型	类和说明
static class	CommonFSUtils.StreamLacksCapabilityException Helper exception for those cases where the place where we need to check a stream capability is not where we have the needed context to explain the impact and mitigation for a lack.

字段概要

字段

限定符和类型	字段和说明
static String	FULL_RWX_PERMISSIONS Full access permissions (starting point for a umask)
static String	HBASE_WAL_DIR Parameter name for HBase WAL directory
static String	UNSAFE_STREAM_CAPABILITY_ENFORCE Parameter to disable stream capability enforcement checks

构造器概要

构造器

限定符	构造器和说明
protected	CommonFSUtils()

方法概要

限定符和类型	方法和说明
static void	checkShortCircuitReadBufferSize(org.apache.hadoop.conf.Configuration conf) Check if short circuit read buffer size is set and if not, set it to hbase value.
static org.apache.hadoop.fs.FSDataOutputStream	create(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.permission.FsPermission perm, boolean overwrite) Create the specified file on the filesystem.
static boolean	delete(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, boolean recursive) Calls fs.delete() and returns the value returned by the fs.delete()
static boolean	deleteDirectory(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path dir) Delete if exists.
static org.apache.hadoop.fs.FileSystem	getCurrentFileSystem(org.apache.hadoop.conf.Configuration conf)
static long	getDefaultBlockSize(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path) Return the number of bytes that large input files should be optimally be split into to minimize i/o time.
static int	getDefaultBufferSize(org.apache.hadoop.fs.FileSystem fs) Returns the default buffer size to use during writes.
static short	getDefaultReplication(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path)
static org.apache.hadoop.fs.permission.FsPermission	getFilePermissions(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.conf.Configuration conf, String permssionConfKey) Get the file permissions specified in the configuration, if they are enabled.
static org.apache.hadoop.fs.Path	getNamespaceDir(org.apache.hadoop.fs.Path rootdir, String namespace) Returns the Path object representing the namespace directory under path rootdir
static String	getPath(org.apache.hadoop.fs.Path p) Return the 'path' component of a Path.
static org.apache.hadoop.fs.Path	getRootDir(org.apache.hadoop.conf.Configuration c)
static org.apache.hadoop.fs.FileSystem	getRootDirFileSystem(org.apache.hadoop.conf.Configuration c)
static org.apache.hadoop.fs.Path	getTableDir(org.apache.hadoop.fs.Path rootdir, TableName tableName) Returns the Path object representing the table directory under path rootdir
static TableName	getTableName(org.apache.hadoop.fs.Path tablePath) Returns the TableName object representing the table directory under path rootdir
static org.apache.hadoop.fs.FileSystem	getWALFileSystem(org.apache.hadoop.conf.Configuration c)
static org.apache.hadoop.fs.Path	getWALRootDir(org.apache.hadoop.conf.Configuration c)
static boolean	hasCapability(org.apache.hadoop.fs.FSDataOutputStream stream, String capability) If our FileSystem version includes the StreamCapabilities class, check if the given stream has a particular capability.
static boolean	isExists(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path) Calls fs.exists().
static boolean	isHDFS(org.apache.hadoop.conf.Configuration conf)
static boolean	isMatchingTail(org.apache.hadoop.fs.Path pathToSearch, org.apache.hadoop.fs.Path pathTail) Compare path component of the Path URI; e.g. if hdfs://a/b/c and /a/b/c, it will compare the '/a/b/c' part.
static boolean	isMatchingTail(org.apache.hadoop.fs.Path pathToSearch, String pathTail) Compare path component of the Path URI; e.g. if hdfs://a/b/c and /a/b/c, it will compare the '/a/b/c' part.
static boolean	isRecoveredEdits(org.apache.hadoop.fs.Path path) Checks if the given path is the one with 'recovered.edits' dir.
static boolean	isStartingWithPath(org.apache.hadoop.fs.Path rootPath, String path) Compare of path component.
static List<org.apache.hadoop.fs.LocatedFileStatus>	listLocatedStatus(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path dir) Calls fs.listFiles() to get FileStatus and BlockLocations together for reducing rpc call
static org.apache.hadoop.fs.FileStatus[]	listStatus(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path dir) Calls fs.listStatus() and treats FileNotFoundException as non-fatal This would accommodates differences between hadoop versions
static org.apache.hadoop.fs.FileStatus[]	listStatus(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path dir, org.apache.hadoop.fs.PathFilter filter) Calls fs.listStatus() and treats FileNotFoundException as non-fatal This accommodates differences between hadoop versions, where hadoop 1 does not throw a FileNotFoundException, and return an empty FileStatus[] while Hadoop 2 will throw FileNotFoundException.
static void	logFileSystemState(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path root, org.slf4j.Logger LOG) Log the current state of the filesystem from a certain root directory
static String	removeWALRootPath(org.apache.hadoop.fs.Path path, org.apache.hadoop.conf.Configuration conf) Checks for the presence of the WAL log root path (using the provided conf object) in the given path.
static boolean	renameAndSetModifyTime(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dest)
static void	setFsDefault(org.apache.hadoop.conf.Configuration c, org.apache.hadoop.fs.Path root)
static void	setRootDir(org.apache.hadoop.conf.Configuration c, org.apache.hadoop.fs.Path root)
static void	setStoragePolicy(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, String storagePolicy) Sets storage policy for given path.
static void	setupShortCircuitRead(org.apache.hadoop.conf.Configuration conf) Do our short circuit read setup.
static void	setWALRootDir(org.apache.hadoop.conf.Configuration c, org.apache.hadoop.fs.Path root)
static org.apache.hadoop.fs.Path	validateRootPath(org.apache.hadoop.fs.Path root) Verifies root directory path is a valid URI with a scheme

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

字段详细资料

HBASE_WAL_DIR

public static final String HBASE_WAL_DIR

Parameter name for HBase WAL directory

另请参阅:

常量字段值

UNSAFE_STREAM_CAPABILITY_ENFORCE

public static final String UNSAFE_STREAM_CAPABILITY_ENFORCE

Parameter to disable stream capability enforcement checks

另请参阅:

常量字段值

FULL_RWX_PERMISSIONS

public static final String FULL_RWX_PERMISSIONS

Full access permissions (starting point for a umask)

另请参阅:

常量字段值

构造器详细资料

CommonFSUtils

protected CommonFSUtils()

方法详细资料

isStartingWithPath

public static boolean isStartingWithPath(org.apache.hadoop.fs.Path rootPath,
                                         String path)

Compare of path component. Does not consider schema; i.e. if schemas different but path starts with rootPath, then the function returns true

参数:

rootPath - value to check for

path - subject to check

返回:

True if path starts with rootPath

isMatchingTail

public static boolean isMatchingTail(org.apache.hadoop.fs.Path pathToSearch,
                                     String pathTail)

Compare path component of the Path URI; e.g. if hdfs://a/b/c and /a/b/c, it will compare the '/a/b/c' part. Does not consider schema; i.e. if schemas different but path or subpath matches, the two will equate.

参数:

pathToSearch - Path we will be trying to match against.

pathTail - what to match

返回:

True if pathTail is tail on the path of pathToSearch

isMatchingTail

public static boolean isMatchingTail(org.apache.hadoop.fs.Path pathToSearch,
                                     org.apache.hadoop.fs.Path pathTail)

Compare path component of the Path URI; e.g. if hdfs://a/b/c and /a/b/c, it will compare the '/a/b/c' part. If you passed in 'hdfs://a/b/c and b/c, it would return true. Does not consider schema; i.e. if schemas different but path or subpath matches, the two will equate.

参数:

pathToSearch - Path we will be trying to match agains against

pathTail - what to match

返回:

True if pathTail is tail on the path of pathToSearch

deleteDirectory

public static boolean deleteDirectory(org.apache.hadoop.fs.FileSystem fs,
                                      org.apache.hadoop.fs.Path dir)
                               throws IOException

Delete if exists.

参数:

fs - filesystem object

dir - directory to delete

返回:

True if deleted dir

抛出:

IOException - e

getDefaultBlockSize

public static long getDefaultBlockSize(org.apache.hadoop.fs.FileSystem fs,
                                       org.apache.hadoop.fs.Path path)
                                throws IOException

Return the number of bytes that large input files should be optimally be split into to minimize i/o time. use reflection to search for getDefaultBlockSize(Path f) if the method doesn't exist, fall back to using getDefaultBlockSize()

参数:

fs - filesystem object

返回:

the default block size for the path's filesystem

抛出:

IOException - e

getDefaultReplication

public static short getDefaultReplication(org.apache.hadoop.fs.FileSystem fs,
                                          org.apache.hadoop.fs.Path path)
                                   throws IOException

抛出:

IOException

getDefaultBufferSize

public static int getDefaultBufferSize(org.apache.hadoop.fs.FileSystem fs)

Returns the default buffer size to use during writes. The size of the buffer should probably be a multiple of hardware page size (4096 on Intel x86), and it determines how much data is buffered during read and write operations.

参数:

fs - filesystem object

返回:

default buffer size to use during writes

create

public static org.apache.hadoop.fs.FSDataOutputStream create(org.apache.hadoop.fs.FileSystem fs,
                                                             org.apache.hadoop.fs.Path path,
                                                             org.apache.hadoop.fs.permission.FsPermission perm,
                                                             boolean overwrite)
                                                      throws IOException

Create the specified file on the filesystem. By default, this will:

1. apply the umask in the configuration (if it is enabled)
2. use the fs configured buffer size (or 4096 if not set)
3. use the default replication
4. use the default block size
5. not track progress

参数:

fs - FileSystem on which to write the file

path - Path to the file to write

perm - intial permissions

overwrite - Whether or not the created file should be overwritten.

返回:

output stream to the created file

抛出:

IOException - if the file cannot be created

getFilePermissions

public static org.apache.hadoop.fs.permission.FsPermission getFilePermissions(org.apache.hadoop.fs.FileSystem fs,
                                                                              org.apache.hadoop.conf.Configuration conf,
                                                                              String permssionConfKey)

Get the file permissions specified in the configuration, if they are enabled.

参数:

fs - filesystem that the file will be created on.

conf - configuration to read for determining if permissions are enabled and which to use

permssionConfKey - property key in the configuration to use when finding the permission

返回:

the permission to use when creating a new file on the fs. If special permissions are not specified in the configuration, then the default permissions on the the fs will be returned.

validateRootPath

public static org.apache.hadoop.fs.Path validateRootPath(org.apache.hadoop.fs.Path root)
                                                  throws IOException

Verifies root directory path is a valid URI with a scheme

参数:

root - root directory path

返回:

Passed root argument.

抛出:

IOException - if not a valid URI with a scheme

removeWALRootPath

public static String removeWALRootPath(org.apache.hadoop.fs.Path path,
                                       org.apache.hadoop.conf.Configuration conf)
                                throws IOException

Checks for the presence of the WAL log root path (using the provided conf object) in the given path. If it exists, this method removes it and returns the String representation of remaining relative path.

参数:

path - must not be null

conf - must not be null

返回:

String representation of the remaining relative path

抛出:

IOException - from underlying filesystem

getPath

public static String getPath(org.apache.hadoop.fs.Path p)

Return the 'path' component of a Path. In Hadoop, Path is an URI. This method returns the 'path' component of a Path's URI: e.g. If a Path is hdfs://example.org:9000/hbase_trunk/TestTable/compaction.dir, this method returns /hbase_trunk/TestTable/compaction.dir. This method is useful if you want to print out a Path without qualifying Filesystem instance.

参数:

p - Filesystem Path whose 'path' component we are to return.

返回:

Path portion of the Filesystem

getRootDir

public static org.apache.hadoop.fs.Path getRootDir(org.apache.hadoop.conf.Configuration c)
                                            throws IOException

参数:

c - configuration

返回:

Path to hbase root directory from configuration as a qualified Path.

抛出:

IOException - e

setRootDir

public static void setRootDir(org.apache.hadoop.conf.Configuration c,
                              org.apache.hadoop.fs.Path root)
                       throws IOException

抛出:

IOException

setFsDefault

public static void setFsDefault(org.apache.hadoop.conf.Configuration c,
                                org.apache.hadoop.fs.Path root)
                         throws IOException

抛出:

IOException

getRootDirFileSystem

public static org.apache.hadoop.fs.FileSystem getRootDirFileSystem(org.apache.hadoop.conf.Configuration c)
                                                            throws IOException

抛出:

IOException

getWALRootDir

public static org.apache.hadoop.fs.Path getWALRootDir(org.apache.hadoop.conf.Configuration c)
                                               throws IOException

参数:

c - configuration

返回:

Path to hbase log root directory: e.g. "hbase.wal.dir" from configuration as a qualified Path. Defaults to HBase root dir.

抛出:

IOException - e

setWALRootDir

public static void setWALRootDir(org.apache.hadoop.conf.Configuration c,
                                 org.apache.hadoop.fs.Path root)
                          throws IOException

抛出:

IOException

getWALFileSystem

public static org.apache.hadoop.fs.FileSystem getWALFileSystem(org.apache.hadoop.conf.Configuration c)
                                                        throws IOException

抛出:

IOException

getTableDir

public static org.apache.hadoop.fs.Path getTableDir(org.apache.hadoop.fs.Path rootdir,
                                                    TableName tableName)

Returns the Path object representing the table directory under path rootdir

参数:

rootdir - qualified path of HBase root directory

tableName - name of table

返回:

Path for table

getTableName

public static TableName getTableName(org.apache.hadoop.fs.Path tablePath)

Returns the TableName object representing the table directory under path rootdir

参数:

tablePath - path of table

返回:

Path for table

getNamespaceDir

public static org.apache.hadoop.fs.Path getNamespaceDir(org.apache.hadoop.fs.Path rootdir,
                                                        String namespace)

Returns the Path object representing the namespace directory under path rootdir

参数:

rootdir - qualified path of HBase root directory

namespace - namespace name

返回:

Path for table

setStoragePolicy

public static void setStoragePolicy(org.apache.hadoop.fs.FileSystem fs,
                                    org.apache.hadoop.fs.Path path,
                                    String storagePolicy)

Sets storage policy for given path. If the passed path is a directory, we'll set the storage policy for all files created in the future in said directory. Note that this change in storage policy takes place at the FileSystem level; it will persist beyond this RS's lifecycle. If we're running on a version of FileSystem that doesn't support the given storage policy (or storage policies at all), then we'll issue a log message and continue. See http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html

参数:

fs - We only do anything it implements a setStoragePolicy method

path - the Path whose storage policy is to be set

storagePolicy - Policy to set on path; see hadoop 2.6+ org.apache.hadoop.hdfs.protocol.HdfsConstants for possible list e.g 'COLD', 'WARM', 'HOT', 'ONE_SSD', 'ALL_SSD', 'LAZY_PERSIST'.

isHDFS

public static boolean isHDFS(org.apache.hadoop.conf.Configuration conf)
                      throws IOException

参数:

conf - must not be null

返回:

True if this filesystem whose scheme is 'hdfs'.

抛出:

IOException - from underlying FileSystem

isRecoveredEdits

public static boolean isRecoveredEdits(org.apache.hadoop.fs.Path path)

Checks if the given path is the one with 'recovered.edits' dir.

参数:

path - must not be null

返回:

True if we recovered edits

getCurrentFileSystem

public static org.apache.hadoop.fs.FileSystem getCurrentFileSystem(org.apache.hadoop.conf.Configuration conf)
                                                            throws IOException

参数:

conf - must not be null

返回:

Returns the filesystem of the hbase rootdir.

抛出:

IOException - from underlying FileSystem

listStatus

public static org.apache.hadoop.fs.FileStatus[] listStatus(org.apache.hadoop.fs.FileSystem fs,
                                                           org.apache.hadoop.fs.Path dir,
                                                           org.apache.hadoop.fs.PathFilter filter)
                                                    throws IOException

Calls fs.listStatus() and treats FileNotFoundException as non-fatal This accommodates differences between hadoop versions, where hadoop 1 does not throw a FileNotFoundException, and return an empty FileStatus[] while Hadoop 2 will throw FileNotFoundException. Where possible, prefer FSUtils#listStatusWithStatusFilter(FileSystem, Path, FileStatusFilter) instead.

参数:

fs - file system

dir - directory

filter - path filter

返回:

null if dir is empty or doesn't exist, otherwise FileStatus array

抛出:

IOException

listStatus

public static org.apache.hadoop.fs.FileStatus[] listStatus(org.apache.hadoop.fs.FileSystem fs,
                                                           org.apache.hadoop.fs.Path dir)
                                                    throws IOException

Calls fs.listStatus() and treats FileNotFoundException as non-fatal This would accommodates differences between hadoop versions

参数:

fs - file system

dir - directory

返回:

null if dir is empty or doesn't exist, otherwise FileStatus array

抛出:

IOException

listLocatedStatus

public static List<org.apache.hadoop.fs.LocatedFileStatus> listLocatedStatus(org.apache.hadoop.fs.FileSystem fs,
                                                                             org.apache.hadoop.fs.Path dir)
                                                                      throws IOException

Calls fs.listFiles() to get FileStatus and BlockLocations together for reducing rpc call

参数:

fs - file system

dir - directory

返回:

LocatedFileStatus list

抛出:

IOException

delete

public static boolean delete(org.apache.hadoop.fs.FileSystem fs,
                             org.apache.hadoop.fs.Path path,
                             boolean recursive)
                      throws IOException

Calls fs.delete() and returns the value returned by the fs.delete()

参数:

fs - must not be null

path - must not be null

recursive - delete tree rooted at path

返回:

the value returned by the fs.delete()

抛出:

IOException - from underlying FileSystem

isExists

public static boolean isExists(org.apache.hadoop.fs.FileSystem fs,
                               org.apache.hadoop.fs.Path path)
                        throws IOException

Calls fs.exists(). Checks if the specified path exists

参数:

fs - must not be null

path - must not be null

返回:

the value returned by fs.exists()

抛出:

IOException - from underlying FileSystem

logFileSystemState

public static void logFileSystemState(org.apache.hadoop.fs.FileSystem fs,
                                      org.apache.hadoop.fs.Path root,
                                      org.slf4j.Logger LOG)
                               throws IOException

Log the current state of the filesystem from a certain root directory

参数:

fs - filesystem to investigate

root - root file/directory to start logging from

LOG - log to output information

抛出:

IOException - if an unexpected exception occurs

renameAndSetModifyTime

public static boolean renameAndSetModifyTime(org.apache.hadoop.fs.FileSystem fs,
                                             org.apache.hadoop.fs.Path src,
                                             org.apache.hadoop.fs.Path dest)
                                      throws IOException

抛出:

IOException

setupShortCircuitRead

public static void setupShortCircuitRead(org.apache.hadoop.conf.Configuration conf)

Do our short circuit read setup. Checks buffer size to use and whether to do checksumming in hbase or hdfs.

参数:

conf - must not be null

checkShortCircuitReadBufferSize

public static void checkShortCircuitReadBufferSize(org.apache.hadoop.conf.Configuration conf)

Check if short circuit read buffer size is set and if not, set it to hbase value.

参数:

conf - must not be null

hasCapability

public static boolean hasCapability(org.apache.hadoop.fs.FSDataOutputStream stream,
                                    String capability)

If our FileSystem version includes the StreamCapabilities class, check if the given stream has a particular capability.

参数:

stream - capabilities are per-stream instance, so check this one specifically. must not be null

capability - what to look for, per Hadoop Common's FileSystem docs

返回:

true if there are no StreamCapabilities. false if there are, but this stream doesn't implement it. return result of asking the stream otherwise.