com.google.cloud.hadoop.fs.gcs

Class GoogleHadoopFileSystemBase

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.fs.FileSystem

com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase

All Implemented Interfaces:

FileSystemDescriptor, Closeable, AutoCloseable, org.apache.hadoop.conf.Configurable

Direct Known Subclasses:

GoogleHadoopFileSystem

public abstract class GoogleHadoopFileSystemBase
extends org.apache.hadoop.fs.FileSystem
implements FileSystemDescriptor

This class provides a Hadoop compatible File System on top of Google Cloud Storage (GCS).

It is implemented as a thin abstraction layer on top of GCS. The layer hides any specific characteristics of the underlying store and exposes FileSystem interface understood by the Hadoop engine.

Users interact with the files in the storage using fully qualified URIs. The file system exposed by this class is identified using the 'gs' scheme. For example, gs://dir1/dir2/file1.txt.

This implementation translates paths between hadoop Path and GCS URI with the convention that the Hadoop root directly corresponds to the GCS "root", e.g. gs:/. This is convenient for many reasons, such as data portability and close equivalence to gsutil paths, but imposes certain inherited constraints, such as files not being allowed in root (only 'directories' can be placed in root), and directory names inside root have a more limited set of allowed characters.

One of the main goals of this implementation is to maintain compatibility with behavior of HDFS implementation when accessed through FileSystem interface. HDFS implementation is not very consistent about the cases when it throws versus the cases when methods return false. We run GHFS tests and HDFS tests against the same test data and use that as a guide to decide whether to throw or to return false.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	GoogleHadoopFileSystemBase.Counter Defines names of counters we track for each operation.
static class	GoogleHadoopFileSystemBase.GcsFileChecksumType Available GCS checksum types for use with GoogleHadoopFileSystemConfiguration.GCS_FILE_CHECKSUM_TYPE.
static class	GoogleHadoopFileSystemBase.OutputStreamType Available types for use with GoogleHadoopFileSystemConfiguration.GCS_OUTPUT_STREAM_TYPE.

Nested classes/interfaces inherited from class org.apache.hadoop.fs.FileSystem

org.apache.hadoop.fs.FileSystem.DirectoryEntries, org.apache.hadoop.fs.FileSystem.Statistics

Field Summary

Fields

Modifier and Type	Field and Description
static String	AUTHENTICATION_PREFIX Prefix to use for common authentication keys.
protected com.google.common.collect.ImmutableMap<GoogleHadoopFileSystemBase.Counter,AtomicLong>	counters Map of counter values
static org.apache.hadoop.fs.PathFilter	DEFAULT_FILTER Default PathFilter that accepts all paths.
protected long	defaultBlockSize Default block size.
protected GcsDelegationTokens	delegationTokens Delegation token support
static String	GHFS_ID Identifies this version of the GoogleHadoopFileSystemBase library.
protected URI	initUri The URI the File System is passed in initialize.
static String	PATH_CODEC_USE_LEGACY_ENCODING Use LEGACY_PATH_CODEC.
static String	PATH_CODEC_USE_URI_ENCODING Use new URI_ENCODED_PATH_CODEC.
protected PathCodec	pathCodec
static String	PROPERTIES_FILE A resource file containing GCS related build properties.
static short	REPLICATION_FACTOR_DEFAULT Default value of replication factor.
static String	UNKNOWN_VERSION The version returned when one cannot be found in properties.
static String	VERSION Current version.
static String	VERSION_PROPERTY The key in the PROPERTIES_FILE that contains the version built.

Fields inherited from class org.apache.hadoop.fs.FileSystem

DEFAULT_FS, FS_DEFAULT_NAME_KEY, LOG, SHUTDOWN_HOOK_PRIORITY, statistics, TRASH_PREFIX, USER_HOME_PREFIX

Constructor Summary

Constructors

Constructor and Description
GoogleHadoopFileSystemBase() Constructs an instance of GoogleHadoopFileSystemBase; the internal GoogleCloudStorageFileSystem will be set up with config settings when initialize() is called.

Method Summary

Modifier and Type	Method and Description
org.apache.hadoop.fs.FSDataOutputStream	append(org.apache.hadoop.fs.Path hadoopPath, int bufferSize, org.apache.hadoop.util.Progressable progress) Appends to an existing file (optional operation).
protected void	checkPath(org.apache.hadoop.fs.Path path)
void	close()
void	completeLocalOutput(org.apache.hadoop.fs.Path fsOutputFile, org.apache.hadoop.fs.Path tmpLocalFile)
void	concat(org.apache.hadoop.fs.Path tgt, org.apache.hadoop.fs.Path[] srcs) Concat existing files into one file.
protected abstract void	configureBuckets(GoogleCloudStorageFileSystem gcsFs) Validates and possibly creates buckets needed by subclass.
void	copyFromLocalFile(boolean delSrc, boolean overwrite, org.apache.hadoop.fs.Path[] srcs, org.apache.hadoop.fs.Path dst)
void	copyFromLocalFile(boolean delSrc, boolean overwrite, org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst)
void	copyToLocalFile(boolean delSrc, org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst)
org.apache.hadoop.fs.FSDataOutputStream	create(org.apache.hadoop.fs.Path hadoopPath, org.apache.hadoop.fs.permission.FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress) Opens the given file for writing.
protected com.google.common.collect.ImmutableMap<GoogleHadoopFileSystemBase.Counter,AtomicLong>	createCounterMap()
org.apache.hadoop.fs.FSDataOutputStream	createNonRecursive(org.apache.hadoop.fs.Path hadoopPath, org.apache.hadoop.fs.permission.FsPermission permission, EnumSet<org.apache.hadoop.fs.CreateFlag> flags, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress)
boolean	delete(org.apache.hadoop.fs.Path hadoopPath, boolean recursive) Deletes the given file or directory.
boolean	deleteOnExit(org.apache.hadoop.fs.Path f)
String	getCanonicalServiceName()
org.apache.hadoop.fs.ContentSummary	getContentSummary(org.apache.hadoop.fs.Path f)
long	getDefaultBlockSize()
protected int	getDefaultPort() The default port is listed as -1 as an indication that ports are not used.
short	getDefaultReplication() Gets the default replication factor.
abstract org.apache.hadoop.fs.Path	getDefaultWorkingDirectory() Gets the default value of working directory.
org.apache.hadoop.security.token.Token<?>	getDelegationToken(String renewer)
org.apache.hadoop.fs.FileChecksum	getFileChecksum(org.apache.hadoop.fs.Path hadoopPath)
org.apache.hadoop.fs.FileStatus	getFileStatus(org.apache.hadoop.fs.Path hadoopPath) Gets status of the given path item.
abstract org.apache.hadoop.fs.Path	getFileSystemRoot() Returns the Hadoop path representing the root of the FileSystem associated with this FileSystemDescriptor.
GoogleCloudStorageFileSystem	getGcsFs() Gets GCS FS instance.
abstract URI	getGcsPath(org.apache.hadoop.fs.Path hadoopPath) Gets GCS path corresponding to the given Hadoop path, which can be relative or absolute, and can have either gs://<path> or gs:/<path> forms.
abstract org.apache.hadoop.fs.Path	getHadoopPath(URI gcsPath) Gets Hadoop path corresponding to the given GCS path.
org.apache.hadoop.fs.Path	getHomeDirectory() Returns home directory of the current user.
protected abstract String	getHomeDirectorySubpath() Returns an unqualified path without any leading slash, relative to the filesystem root, which serves as the home directory of the current user; see getHomeDirectory for a description of what the home directory means.
abstract String	getScheme() Returns the URI scheme for the Hadoop FileSystem associated with this FileSystemDescriptor.
URI	getUri() Returns a URI of the root of this FileSystem.
long	getUsed()
org.apache.hadoop.fs.Path	getWorkingDirectory() Gets the current working directory.
byte[]	getXAttr(org.apache.hadoop.fs.Path path, String name)
Map<String,byte[]>	getXAttrs(org.apache.hadoop.fs.Path path)
Map<String,byte[]>	getXAttrs(org.apache.hadoop.fs.Path path, List<String> names)
org.apache.hadoop.fs.FileStatus[]	globStatus(org.apache.hadoop.fs.Path pathPattern) Returns an array of FileStatus objects whose path names match pathPattern.
org.apache.hadoop.fs.FileStatus[]	globStatus(org.apache.hadoop.fs.Path pathPattern, org.apache.hadoop.fs.PathFilter filter) Returns an array of FileStatus objects whose path names match pathPattern and is accepted by the user-supplied path filter.
void	initialize(URI path, org.apache.hadoop.conf.Configuration config) See initialize(URI, Configuration, boolean) for details; calls with third arg defaulting to 'true' for initializing the superclass.
void	initialize(URI path, org.apache.hadoop.conf.Configuration config, boolean initSuperclass) Initializes this file system instance.
org.apache.hadoop.fs.FileStatus[]	listStatus(org.apache.hadoop.fs.Path hadoopPath) Lists file status.
List<String>	listXAttrs(org.apache.hadoop.fs.Path path)
org.apache.hadoop.fs.Path	makeQualified(org.apache.hadoop.fs.Path path) Overridden to make root its own parent.
boolean	mkdirs(org.apache.hadoop.fs.Path hadoopPath, org.apache.hadoop.fs.permission.FsPermission permission) Makes the given path and all non-existent parents directories.
org.apache.hadoop.fs.FSDataInputStream	open(org.apache.hadoop.fs.Path hadoopPath, int bufferSize) Opens the given file for reading.
protected void	processDeleteOnExit()
void	removeXAttr(org.apache.hadoop.fs.Path path, String name)
boolean	rename(org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst) Renames src to dst.
void	setOwner(org.apache.hadoop.fs.Path p, String username, String groupname)
void	setPermission(org.apache.hadoop.fs.Path p, org.apache.hadoop.fs.permission.FsPermission permission)
void	setTimes(org.apache.hadoop.fs.Path p, long mtime, long atime)
void	setVerifyChecksum(boolean verifyChecksum)
void	setWorkingDirectory(org.apache.hadoop.fs.Path hadoopPath) Sets the current working directory to the given path.
void	setXAttr(org.apache.hadoop.fs.Path path, String name, byte[] value, EnumSet<org.apache.hadoop.fs.XAttrSetFlag> flags)
org.apache.hadoop.fs.Path	startLocalOutput(org.apache.hadoop.fs.Path fsOutputFile, org.apache.hadoop.fs.Path tmpLocalFile)

Methods inherited from class org.apache.hadoop.fs.FileSystem

access, addDelegationTokens, append, append, appendFile, areSymlinksEnabled, cancelDeleteOnExit, canonicalizeUri, clearStatistics, closeAll, closeAllForUGI, copyFromLocalFile, copyFromLocalFile, copyToLocalFile, copyToLocalFile, create, create, create, create, create, create, create, create, create, create, create, create, createFile, createNewFile, createNonRecursive, createNonRecursive, createSnapshot, createSnapshot, createSymlink, delete, deleteSnapshot, enableSymlinks, exists, fixRelativePart, get, get, get, getAclStatus, getAllStatistics, getAllStoragePolicies, getBlockSize, getCanonicalUri, getChildFileSystems, getDefaultBlockSize, getDefaultReplication, getDefaultUri, getFileBlockLocations, getFileBlockLocations, getFileChecksum, getFileLinkStatus, getFileSystemClass, getFSofPath, getGlobalStorageStatistics, getInitialWorkingDirectory, getLength, getLinkTarget, getLocal, getName, getNamed, getQuotaUsage, getReplication, getServerDefaults, getServerDefaults, getStatistics, getStatistics, getStatus, getStatus, getStoragePolicy, getStorageStatistics, getTrashRoot, getTrashRoots, getUsed, isDirectory, isFile, listCorruptFileBlocks, listFiles, listLocatedStatus, listLocatedStatus, listStatus, listStatus, listStatus, listStatusBatch, listStatusIterator, mkdirs, mkdirs, modifyAclEntries, moveFromLocalFile, moveFromLocalFile, moveToLocalFile, newInstance, newInstance, newInstance, newInstanceLocal, open, primitiveCreate, primitiveMkdir, primitiveMkdir, printStatistics, removeAcl, removeAclEntries, removeDefaultAcl, rename, renameSnapshot, resolveLink, resolvePath, setAcl, setDefaultUri, setDefaultUri, setReplication, setStoragePolicy, setWriteChecksum, setXAttr, supportsSymlinks, truncate, unsetStoragePolicy

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PATH_CODEC_USE_URI_ENCODING

public static final String PATH_CODEC_USE_URI_ENCODING

Use new URI_ENCODED_PATH_CODEC.

See Also:

Constant Field Values

PATH_CODEC_USE_LEGACY_ENCODING

public static final String PATH_CODEC_USE_LEGACY_ENCODING

Use LEGACY_PATH_CODEC.

See Also:

Constant Field Values

REPLICATION_FACTOR_DEFAULT

public static final short REPLICATION_FACTOR_DEFAULT

Default value of replication factor.

See Also:

Constant Field Values

DEFAULT_FILTER

public static final org.apache.hadoop.fs.PathFilter DEFAULT_FILTER

Default PathFilter that accepts all paths.

AUTHENTICATION_PREFIX

public static final String AUTHENTICATION_PREFIX

Prefix to use for common authentication keys.

See Also:

Constant Field Values

PROPERTIES_FILE

public static final String PROPERTIES_FILE

A resource file containing GCS related build properties.

See Also:

Constant Field Values

VERSION_PROPERTY

public static final String VERSION_PROPERTY

The key in the PROPERTIES_FILE that contains the version built.

See Also:

Constant Field Values

UNKNOWN_VERSION

public static final String UNKNOWN_VERSION

The version returned when one cannot be found in properties.

See Also:

Constant Field Values

VERSION

public static final String VERSION

Current version.

GHFS_ID

public static final String GHFS_ID

Identifies this version of the GoogleHadoopFileSystemBase library.

initUri

protected URI initUri

The URI the File System is passed in initialize.

delegationTokens

protected GcsDelegationTokens delegationTokens

Delegation token support

pathCodec

protected PathCodec pathCodec

defaultBlockSize

protected long defaultBlockSize

Default block size. Note that this is the size that is reported to Hadoop FS clients. It does not modify the actual block size of an underlying GCS object, because GCS JSON API does not allow modifying or querying the value. Modifying this value allows one to control how many mappers are used to process a given file.

counters

protected final com.google.common.collect.ImmutableMap<GoogleHadoopFileSystemBase.Counter,AtomicLong> counters

Map of counter values

Constructor Detail

GoogleHadoopFileSystemBase

public GoogleHadoopFileSystemBase()

Constructs an instance of GoogleHadoopFileSystemBase; the internal GoogleCloudStorageFileSystem will be set up with config settings when initialize() is called.

Method Detail

createCounterMap

protected com.google.common.collect.ImmutableMap<GoogleHadoopFileSystemBase.Counter,AtomicLong> createCounterMap()

getHomeDirectorySubpath

protected abstract String getHomeDirectorySubpath()

Returns an unqualified path without any leading slash, relative to the filesystem root, which serves as the home directory of the current user; see getHomeDirectory for a description of what the home directory means.

getHadoopPath

public abstract org.apache.hadoop.fs.Path getHadoopPath(URI gcsPath)

Gets Hadoop path corresponding to the given GCS path.

Parameters:

gcsPath - Fully-qualified GCS path, of the form gs://bucket/object-path.

getGcsPath

public abstract URI getGcsPath(org.apache.hadoop.fs.Path hadoopPath)

Gets GCS path corresponding to the given Hadoop path, which can be relative or absolute, and can have either gs://<path> or gs:/<path> forms.

Parameters:

hadoopPath - Hadoop path.

getDefaultWorkingDirectory

public abstract org.apache.hadoop.fs.Path getDefaultWorkingDirectory()

Gets the default value of working directory.

getFileSystemRoot

public abstract org.apache.hadoop.fs.Path getFileSystemRoot()

Description copied from interface: FileSystemDescriptor

Returns the Hadoop path representing the root of the FileSystem associated with this FileSystemDescriptor.

Specified by:

getFileSystemRoot in interface FileSystemDescriptor

getScheme

public abstract String getScheme()

Description copied from interface: FileSystemDescriptor

Returns the URI scheme for the Hadoop FileSystem associated with this FileSystemDescriptor.

Specified by:

getScheme in interface FileSystemDescriptor

Overrides:

getScheme in class org.apache.hadoop.fs.FileSystem

makeQualified

public org.apache.hadoop.fs.Path makeQualified(org.apache.hadoop.fs.Path path)

Overridden to make root its own parent. This is POSIX compliant, but more importantly guards against poor directory accounting in the PathData class of Hadoop 2's FsShell.

Overrides:

makeQualified in class org.apache.hadoop.fs.FileSystem

checkPath

protected void checkPath(org.apache.hadoop.fs.Path path)

Overrides:

checkPath in class org.apache.hadoop.fs.FileSystem

initialize

public void initialize(URI path,
                       org.apache.hadoop.conf.Configuration config)
                throws IOException

See initialize(URI, Configuration, boolean) for details; calls with third arg defaulting to 'true' for initializing the superclass.

Overrides:

initialize in class org.apache.hadoop.fs.FileSystem

Parameters:

path - URI of a file/directory within this file system.

config - Hadoop configuration.

Throws:

IOException

initialize

public void initialize(URI path,
                       org.apache.hadoop.conf.Configuration config,
                       boolean initSuperclass)
                throws IOException

Initializes this file system instance. Note: The path passed to this method could be path of any file/directory. It does not matter because the only thing we check is whether it uses 'gs' scheme. The rest is ignored.

Parameters:

path - URI of a file/directory within this file system.

config - Hadoop configuration.

initSuperclass - if false, doesn't call super.initialize(path, config); avoids registering a global Statistics object for this instance.

Throws:

IOException

getUri

public URI getUri()

Returns a URI of the root of this FileSystem.

Specified by:

getUri in class org.apache.hadoop.fs.FileSystem

getDefaultPort

protected int getDefaultPort()

The default port is listed as -1 as an indication that ports are not used.

Overrides:

getDefaultPort in class org.apache.hadoop.fs.FileSystem

open

public org.apache.hadoop.fs.FSDataInputStream open(org.apache.hadoop.fs.Path hadoopPath,
                                                   int bufferSize)
                                            throws IOException

Opens the given file for reading.

Note: This function overrides the given bufferSize value with a higher number unless further overridden using configuration parameter fs.gs.inputstream.buffer.size.

Specified by:

open in class org.apache.hadoop.fs.FileSystem

Parameters:

hadoopPath - File to open.

bufferSize - Size of buffer to use for IO.

Returns:

A readable stream.

Throws:

FileNotFoundException - if the given path does not exist.

IOException - if an error occurs.

create

public org.apache.hadoop.fs.FSDataOutputStream create(org.apache.hadoop.fs.Path hadoopPath,
                                                      org.apache.hadoop.fs.permission.FsPermission permission,
                                                      boolean overwrite,
                                                      int bufferSize,
                                                      short replication,
                                                      long blockSize,
                                                      org.apache.hadoop.util.Progressable progress)
                                               throws IOException

Opens the given file for writing.

Note: This function overrides the given bufferSize value with a higher number unless further overridden using configuration parameter fs.gs.outputstream.buffer.size.

Specified by:

create in class org.apache.hadoop.fs.FileSystem

Parameters:

hadoopPath - The file to open.

permission - Permissions to set on the new file. Ignored.

overwrite - If a file with this name already exists, then if true, the file will be overwritten, and if false an error will be thrown.

bufferSize - The size of the buffer to use.

replication - Required block replication for the file. Ignored.

blockSize - The block-size to be used for the new file. Ignored.

progress - Progress is reported through this. Ignored.

Returns:

A writable stream.

Throws:

IOException - if an error occurs.

See Also:

setPermission(Path, FsPermission)

createNonRecursive

public org.apache.hadoop.fs.FSDataOutputStream createNonRecursive(org.apache.hadoop.fs.Path hadoopPath,
                                                                  org.apache.hadoop.fs.permission.FsPermission permission,
                                                                  EnumSet<org.apache.hadoop.fs.CreateFlag> flags,
                                                                  int bufferSize,
                                                                  short replication,
                                                                  long blockSize,
                                                                  org.apache.hadoop.util.Progressable progress)
                                                           throws IOException

Overrides:

createNonRecursive in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

append

public org.apache.hadoop.fs.FSDataOutputStream append(org.apache.hadoop.fs.Path hadoopPath,
                                                      int bufferSize,
                                                      org.apache.hadoop.util.Progressable progress)
                                               throws IOException

Appends to an existing file (optional operation). Not supported.

Specified by:

append in class org.apache.hadoop.fs.FileSystem

Parameters:

hadoopPath - The existing file to be appended.

bufferSize - The size of the buffer to be used.

progress - For reporting progress if it is not null.

Returns:

A writable stream.

Throws:

IOException - if an error occurs.

concat

public void concat(org.apache.hadoop.fs.Path tgt,
                   org.apache.hadoop.fs.Path[] srcs)
            throws IOException

Concat existing files into one file.

Overrides:

concat in class org.apache.hadoop.fs.FileSystem

Parameters:

tgt - the path to the target destination.

srcs - the paths to the sources to use for the concatenation.

Throws:

IOException - IO failure

rename

public boolean rename(org.apache.hadoop.fs.Path src,
                      org.apache.hadoop.fs.Path dst)
               throws IOException

Renames src to dst. Src must not be equal to the filesystem root.

Specified by:

rename in class org.apache.hadoop.fs.FileSystem

Parameters:

src - Source path.

dst - Destination path.

Returns:

true if successful, or false if the old name does not exist or if the new name already belongs to the namespace.

Throws:

IOException - if an error occurs.

delete

public boolean delete(org.apache.hadoop.fs.Path hadoopPath,
                      boolean recursive)
               throws IOException

Deletes the given file or directory.

Specified by:

delete in class org.apache.hadoop.fs.FileSystem

Parameters:

hadoopPath - The path to delete.

recursive - If path is a directory and set to true, the directory is deleted, else throws an exception. In case of a file, the recursive parameter is ignored.

Returns:

true if delete is successful else false.

Throws:

IOException - if an error occurs.

listStatus

public org.apache.hadoop.fs.FileStatus[] listStatus(org.apache.hadoop.fs.Path hadoopPath)
                                             throws IOException

Lists file status. If the given path points to a directory then the status of children is returned, otherwise the status of the given file is returned.

Specified by:

listStatus in class org.apache.hadoop.fs.FileSystem

Parameters:

hadoopPath - Given path.

Returns:

File status list or null if path does not exist.

Throws:

IOException - if an error occurs.

setWorkingDirectory

public void setWorkingDirectory(org.apache.hadoop.fs.Path hadoopPath)

Sets the current working directory to the given path.

Specified by:

setWorkingDirectory in class org.apache.hadoop.fs.FileSystem

Parameters:

hadoopPath - New working directory.

getWorkingDirectory

public org.apache.hadoop.fs.Path getWorkingDirectory()

Gets the current working directory.

Specified by:

getWorkingDirectory in class org.apache.hadoop.fs.FileSystem

Returns:

The current working directory.

mkdirs

public boolean mkdirs(org.apache.hadoop.fs.Path hadoopPath,
                      org.apache.hadoop.fs.permission.FsPermission permission)
               throws IOException

Makes the given path and all non-existent parents directories. Has the semantics of Unix 'mkdir -p'.

Specified by:

mkdirs in class org.apache.hadoop.fs.FileSystem

Parameters:

hadoopPath - Given path.

permission - Permissions to set on the given directory.

Returns:

true on success, false otherwise.

Throws:

IOException - if an error occurs.

getDefaultReplication

public short getDefaultReplication()

Gets the default replication factor.

Overrides:

getDefaultReplication in class org.apache.hadoop.fs.FileSystem

getFileStatus

public org.apache.hadoop.fs.FileStatus getFileStatus(org.apache.hadoop.fs.Path hadoopPath)
                                              throws IOException

Gets status of the given path item.

Specified by:

getFileStatus in class org.apache.hadoop.fs.FileSystem

Parameters:

hadoopPath - The path we want information about.

Returns:

A FileStatus object for the given path.

Throws:

FileNotFoundException - when the path does not exist;

IOException - on other errors.

globStatus

public org.apache.hadoop.fs.FileStatus[] globStatus(org.apache.hadoop.fs.Path pathPattern)
                                             throws IOException

Returns an array of FileStatus objects whose path names match pathPattern. Return null if pathPattern has no glob and the path does not exist. Return an empty array if pathPattern has a glob and no path matches it.

Overrides:

globStatus in class org.apache.hadoop.fs.FileSystem

Parameters:

pathPattern - A regular expression specifying the path pattern.

Returns:

An array of FileStatus objects.

Throws:

IOException - if an error occurs.

globStatus

public org.apache.hadoop.fs.FileStatus[] globStatus(org.apache.hadoop.fs.Path pathPattern,
                                                    org.apache.hadoop.fs.PathFilter filter)
                                             throws IOException

Returns an array of FileStatus objects whose path names match pathPattern and is accepted by the user-supplied path filter. Results are sorted by their path names.

Return null if pathPattern has no glob and the path does not exist. Return an empty array if pathPattern has a glob and no path matches it.

Overrides:

globStatus in class org.apache.hadoop.fs.FileSystem

Parameters:

pathPattern - A regular expression specifying the path pattern.

filter - A user-supplied path filter.

Returns:

An array of FileStatus objects.

Throws:

IOException - if an error occurs.

getHomeDirectory

public org.apache.hadoop.fs.Path getHomeDirectory()

Returns home directory of the current user. Note: This directory is only used for Hadoop purposes. It is not the same as a user's OS home directory.

Overrides:

getHomeDirectory in class org.apache.hadoop.fs.FileSystem

getCanonicalServiceName

public String getCanonicalServiceName()

Returns the service if delegation tokens are configured, otherwise, null.

Overrides:

getCanonicalServiceName in class org.apache.hadoop.fs.FileSystem

getGcsFs

public GoogleCloudStorageFileSystem getGcsFs()

Gets GCS FS instance.

configureBuckets

protected abstract void configureBuckets(GoogleCloudStorageFileSystem gcsFs)
                                  throws IOException

Validates and possibly creates buckets needed by subclass.

Parameters:

gcsFs - GoogleCloudStorageFileSystem to configure buckets

Throws:

IOException - if bucket name is invalid or cannot be found.

deleteOnExit

public boolean deleteOnExit(org.apache.hadoop.fs.Path f)
                     throws IOException

Overrides:

deleteOnExit in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

processDeleteOnExit

protected void processDeleteOnExit()

Overrides:

processDeleteOnExit in class org.apache.hadoop.fs.FileSystem

getContentSummary

public org.apache.hadoop.fs.ContentSummary getContentSummary(org.apache.hadoop.fs.Path f)
                                                      throws IOException

Overrides:

getContentSummary in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

getDelegationToken

public org.apache.hadoop.security.token.Token<?> getDelegationToken(String renewer)
                                                             throws IOException

Overrides:

getDelegationToken in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

copyFromLocalFile

public void copyFromLocalFile(boolean delSrc,
                              boolean overwrite,
                              org.apache.hadoop.fs.Path[] srcs,
                              org.apache.hadoop.fs.Path dst)
                       throws IOException

Overrides:

copyFromLocalFile in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

copyFromLocalFile

public void copyFromLocalFile(boolean delSrc,
                              boolean overwrite,
                              org.apache.hadoop.fs.Path src,
                              org.apache.hadoop.fs.Path dst)
                       throws IOException

Overrides:

copyFromLocalFile in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

copyToLocalFile

public void copyToLocalFile(boolean delSrc,
                            org.apache.hadoop.fs.Path src,
                            org.apache.hadoop.fs.Path dst)
                     throws IOException

Overrides:

copyToLocalFile in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

startLocalOutput

public org.apache.hadoop.fs.Path startLocalOutput(org.apache.hadoop.fs.Path fsOutputFile,
                                                  org.apache.hadoop.fs.Path tmpLocalFile)
                                           throws IOException

Overrides:

startLocalOutput in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

completeLocalOutput

public void completeLocalOutput(org.apache.hadoop.fs.Path fsOutputFile,
                                org.apache.hadoop.fs.Path tmpLocalFile)
                         throws IOException

Overrides:

completeLocalOutput in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Overrides:

close in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

getUsed

public long getUsed()
             throws IOException

Overrides:

getUsed in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

getDefaultBlockSize

public long getDefaultBlockSize()

Overrides:

getDefaultBlockSize in class org.apache.hadoop.fs.FileSystem

getFileChecksum

public org.apache.hadoop.fs.FileChecksum getFileChecksum(org.apache.hadoop.fs.Path hadoopPath)
                                                  throws IOException

Overrides:

getFileChecksum in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

setVerifyChecksum

public void setVerifyChecksum(boolean verifyChecksum)

Overrides:

setVerifyChecksum in class org.apache.hadoop.fs.FileSystem

setPermission

public void setPermission(org.apache.hadoop.fs.Path p,
                          org.apache.hadoop.fs.permission.FsPermission permission)
                   throws IOException

Overrides:

setPermission in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

setOwner

public void setOwner(org.apache.hadoop.fs.Path p,
                     String username,
                     String groupname)
              throws IOException

Overrides:

setOwner in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

setTimes

public void setTimes(org.apache.hadoop.fs.Path p,
                     long mtime,
                     long atime)
              throws IOException

Overrides:

setTimes in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

getXAttr

public byte[] getXAttr(org.apache.hadoop.fs.Path path,
                       String name)
                throws IOException

Overrides:

getXAttr in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

getXAttrs

public Map<String,byte[]> getXAttrs(org.apache.hadoop.fs.Path path)
                             throws IOException

Overrides:

getXAttrs in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

getXAttrs

public Map<String,byte[]> getXAttrs(org.apache.hadoop.fs.Path path,
                                    List<String> names)
                             throws IOException

Overrides:

getXAttrs in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

listXAttrs

public List<String> listXAttrs(org.apache.hadoop.fs.Path path)
                        throws IOException

Overrides:

listXAttrs in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

setXAttr

public void setXAttr(org.apache.hadoop.fs.Path path,
                     String name,
                     byte[] value,
                     EnumSet<org.apache.hadoop.fs.XAttrSetFlag> flags)
              throws IOException

Overrides:

setXAttr in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

removeXAttr

public void removeXAttr(org.apache.hadoop.fs.Path path,
                        String name)
                 throws IOException

Overrides:

removeXAttr in class org.apache.hadoop.fs.FileSystem

Throws:

IOException