com.google.cloud.hadoop.fs.gcs

Class GoogleHadoopFileSystemBase

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.fs.FileSystem

com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase

All Implemented Interfaces:

FileSystemDescriptor, java.io.Closeable, java.lang.AutoCloseable, org.apache.hadoop.conf.Configurable

Direct Known Subclasses:

GoogleHadoopFileSystem

public abstract class GoogleHadoopFileSystemBase
extends org.apache.hadoop.fs.FileSystem
implements FileSystemDescriptor

This class provides a Hadoop compatible File System on top of Google Cloud Storage (GCS).

It is implemented as a thin abstraction layer on top of GCS. The layer hides any specific characteristics of the underlying store and exposes FileSystem interface understood by the Hadoop engine.

Users interact with the files in the storage using fully qualified URIs. The file system exposed by this class is identified using the 'gs' scheme. For example, gs://dir1/dir2/file1.txt.

This implementation translates paths between hadoop Path and GCS URI with the convention that the Hadoop root directly corresponds to the GCS "root", e.g. gs:/. This is convenient for many reasons, such as data portability and close equivalence to gsutil paths, but imposes certain inherited constraints, such as files not being allowed in root (only 'directories' can be placed in root), and directory names inside root have a more limited set of allowed characters.

One of the main goals of this implementation is to maintain compatibility with behavior of HDFS implementation when accessed through FileSystem interface. HDFS implementation is not very consistent about the cases when it throws versus the cases when methods return false. We run GHFS tests and HDFS tests against the same test data and use that as a guide to decide whether to throw or to return false.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	GoogleHadoopFileSystemBase.Counter Defines names of counters we track for each operation.
static class	GoogleHadoopFileSystemBase.GcsFileChecksumType Available GCS checksum types for use with GoogleHadoopFileSystemConfiguration.GCS_FILE_CHECKSUM_TYPE.
static class	GoogleHadoopFileSystemBase.OutputStreamType Available types for use with GoogleHadoopFileSystemConfiguration.GCS_OUTPUT_STREAM_TYPE.
static class	GoogleHadoopFileSystemBase.ParentTimestampUpdateIncludePredicate A predicate that processes individual directory paths and evaluates the conditions set in fs.gs.parent.timestamp.update.enable, fs.gs.parent.timestamp.update.substrings.include and fs.gs.parent.timestamp.update.substrings.exclude to determine if a path should be ignored when running directory timestamp updates.

Nested classes/interfaces inherited from class org.apache.hadoop.fs.FileSystem

org.apache.hadoop.fs.FileSystem.DirectoryEntries, org.apache.hadoop.fs.FileSystem.Statistics

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	AUTHENTICATION_PREFIX Prefix to use for common authentication keys.
protected com.google.common.collect.ImmutableMap<GoogleHadoopFileSystemBase.Counter,java.util.concurrent.atomic.AtomicLong>	counters Map of counter values
static org.apache.hadoop.fs.PathFilter	DEFAULT_FILTER Default PathFilter that accepts all paths.
protected long	defaultBlockSize Default block size.
protected GcsDelegationTokens	delegationTokens Delegation token support
static java.lang.String	GHFS_ID Identifies this version of the GoogleHadoopFileSystemBase library.
protected java.net.URI	initUri The URI the File System is passed in initialize.
static java.lang.String	PATH_CODEC_USE_LEGACY_ENCODING Use LEGACY_PATH_CODEC.
static java.lang.String	PATH_CODEC_USE_URI_ENCODING Use new URI_ENCODED_PATH_CODEC.
protected PathCodec	pathCodec
static java.lang.String	PROPERTIES_FILE A resource file containing GCS related build properties.
static short	REPLICATION_FACTOR_DEFAULT Default value of replication factor.
static java.lang.String	UNKNOWN_VERSION The version returned when one cannot be found in properties.
static java.lang.String	VERSION Current version.
static java.lang.String	VERSION_PROPERTY The key in the PROPERTIES_FILE that contains the version built.

Fields inherited from class org.apache.hadoop.fs.FileSystem

DEFAULT_FS, FS_DEFAULT_NAME_KEY, LOG, SHUTDOWN_HOOK_PRIORITY, statistics, TRASH_PREFIX, USER_HOME_PREFIX

Constructor Summary

Constructors

Constructor and Description
GoogleHadoopFileSystemBase() Constructs an instance of GoogleHadoopFileSystemBase; the internal GoogleCloudStorageFileSystem will be set up with config settings when initialize() is called.

Method Summary

Modifier and Type	Method and Description
org.apache.hadoop.fs.FSDataOutputStream	append(org.apache.hadoop.fs.Path hadoopPath, int bufferSize, org.apache.hadoop.util.Progressable progress) Appends to an existing file (optional operation).
protected void	checkPath(org.apache.hadoop.fs.Path path)
void	close()
void	completeLocalOutput(org.apache.hadoop.fs.Path fsOutputFile, org.apache.hadoop.fs.Path tmpLocalFile)
void	concat(org.apache.hadoop.fs.Path trg, org.apache.hadoop.fs.Path[] psrcs) Concat existing files into one file.
protected abstract void	configureBuckets(GoogleCloudStorageFileSystem gcsFs) Validates and possibly creates buckets needed by subclass.
void	copyFromLocalFile(boolean delSrc, boolean overwrite, org.apache.hadoop.fs.Path[] srcs, org.apache.hadoop.fs.Path dst)
void	copyFromLocalFile(boolean delSrc, boolean overwrite, org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst)
void	copyToLocalFile(boolean delSrc, org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst)
org.apache.hadoop.fs.FSDataOutputStream	create(org.apache.hadoop.fs.Path hadoopPath, org.apache.hadoop.fs.permission.FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress) Opens the given file for writing.
protected com.google.common.collect.ImmutableMap<GoogleHadoopFileSystemBase.Counter,java.util.concurrent.atomic.AtomicLong>	createCounterMap()
org.apache.hadoop.fs.FSDataOutputStream	createNonRecursive(org.apache.hadoop.fs.Path hadoopPath, org.apache.hadoop.fs.permission.FsPermission permission, java.util.EnumSet<org.apache.hadoop.fs.CreateFlag> flags, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress)
boolean	delete(org.apache.hadoop.fs.Path hadoopPath, boolean recursive) Deletes the given file or directory.
boolean	deleteOnExit(org.apache.hadoop.fs.Path f)
java.lang.String	getCanonicalServiceName()
org.apache.hadoop.fs.ContentSummary	getContentSummary(org.apache.hadoop.fs.Path f)
long	getDefaultBlockSize()
protected int	getDefaultPort() The default port is listed as -1 as an indication that ports are not used.
short	getDefaultReplication() Gets the default replication factor.
abstract org.apache.hadoop.fs.Path	getDefaultWorkingDirectory() Gets the default value of working directory.
org.apache.hadoop.security.token.Token<?>	getDelegationToken(java.lang.String renewer)
org.apache.hadoop.fs.FileChecksum	getFileChecksum(org.apache.hadoop.fs.Path hadoopPath)
org.apache.hadoop.fs.FileStatus	getFileStatus(org.apache.hadoop.fs.Path hadoopPath) Gets status of the given path item.
abstract org.apache.hadoop.fs.Path	getFileSystemRoot() Returns the Hadoop path representing the root of the FileSystem associated with this FileSystemDescriptor.
GoogleCloudStorageFileSystem	getGcsFs() Gets GCS FS instance.
abstract java.net.URI	getGcsPath(org.apache.hadoop.fs.Path hadoopPath) Gets GCS path corresponding to the given Hadoop path, which can be relative or absolute, and can have either gs:// or gs:/ forms.
abstract org.apache.hadoop.fs.Path	getHadoopPath(java.net.URI gcsPath) Gets Hadoop path corresponding to the given GCS path.
org.apache.hadoop.fs.Path	getHomeDirectory() Returns home directory of the current user.
protected abstract java.lang.String	getHomeDirectorySubpath() Returns an unqualified path without any leading slash, relative to the filesystem root, which serves as the home directory of the current user; see getHomeDirectory for a description of what the home directory means.
abstract java.lang.String	getScheme() Returns the URI scheme for the Hadoop FileSystem associated with this FileSystemDescriptor.
java.net.URI	getUri() Returns a URI of the root of this FileSystem.
long	getUsed()
org.apache.hadoop.fs.Path	getWorkingDirectory() Gets the current working directory.
byte[]	getXAttr(org.apache.hadoop.fs.Path path, java.lang.String name)
java.util.Map<java.lang.String,byte[]>	getXAttrs(org.apache.hadoop.fs.Path path)
java.util.Map<java.lang.String,byte[]>	getXAttrs(org.apache.hadoop.fs.Path path, java.util.List<java.lang.String> names)
org.apache.hadoop.fs.FileStatus[]	globStatus(org.apache.hadoop.fs.Path pathPattern) Returns an array of FileStatus objects whose path names match pathPattern.
org.apache.hadoop.fs.FileStatus[]	globStatus(org.apache.hadoop.fs.Path pathPattern, org.apache.hadoop.fs.PathFilter filter) Returns an array of FileStatus objects whose path names match pathPattern and is accepted by the user-supplied path filter.
void	initialize(java.net.URI path, org.apache.hadoop.conf.Configuration config) See initialize(URI, Configuration, boolean) for details; calls with third arg defaulting to 'true' for initializing the superclass.
void	initialize(java.net.URI path, org.apache.hadoop.conf.Configuration config, boolean initSuperclass) Initializes this file system instance.
org.apache.hadoop.fs.FileStatus[]	listStatus(org.apache.hadoop.fs.Path hadoopPath) Lists file status.
java.util.List<java.lang.String>	listXAttrs(org.apache.hadoop.fs.Path path)
org.apache.hadoop.fs.Path	makeQualified(org.apache.hadoop.fs.Path path) Overridden to make root it's own parent.
boolean	mkdirs(org.apache.hadoop.fs.Path hadoopPath, org.apache.hadoop.fs.permission.FsPermission permission) Makes the given path and all non-existent parents directories.
org.apache.hadoop.fs.FSDataInputStream	open(org.apache.hadoop.fs.Path hadoopPath, int bufferSize) Opens the given file for reading.
protected void	processDeleteOnExit()
void	removeXAttr(org.apache.hadoop.fs.Path path, java.lang.String name)
boolean	rename(org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst) Renames src to dst.
void	setOwner(org.apache.hadoop.fs.Path p, java.lang.String username, java.lang.String groupname)
void	setPermission(org.apache.hadoop.fs.Path p, org.apache.hadoop.fs.permission.FsPermission permission)
void	setTimes(org.apache.hadoop.fs.Path p, long mtime, long atime)
void	setVerifyChecksum(boolean verifyChecksum)
void	setWorkingDirectory(org.apache.hadoop.fs.Path hadoopPath) Sets the current working directory to the given path.
void	setXAttr(org.apache.hadoop.fs.Path path, java.lang.String name, byte[] value, java.util.EnumSet<org.apache.hadoop.fs.XAttrSetFlag> flags)
org.apache.hadoop.fs.Path	startLocalOutput(org.apache.hadoop.fs.Path fsOutputFile, org.apache.hadoop.fs.Path tmpLocalFile)

Methods inherited from class org.apache.hadoop.fs.FileSystem

access, addDelegationTokens, append, append, appendFile, areSymlinksEnabled, cancelDeleteOnExit, canonicalizeUri, clearStatistics, closeAll, closeAllForUGI, copyFromLocalFile, copyFromLocalFile, copyToLocalFile, copyToLocalFile, create, create, create, create, create, create, create, create, create, create, create, create, createFile, createNewFile, createNonRecursive, createNonRecursive, createSnapshot, createSnapshot, createSymlink, delete, deleteSnapshot, enableSymlinks, exists, fixRelativePart, get, get, get, getAclStatus, getAllStatistics, getAllStoragePolicies, getBlockSize, getCanonicalUri, getChildFileSystems, getDefaultBlockSize, getDefaultReplication, getDefaultUri, getFileBlockLocations, getFileBlockLocations, getFileChecksum, getFileLinkStatus, getFileSystemClass, getFSofPath, getGlobalStorageStatistics, getInitialWorkingDirectory, getLength, getLinkTarget, getLocal, getName, getNamed, getQuotaUsage, getReplication, getServerDefaults, getServerDefaults, getStatistics, getStatistics, getStatus, getStatus, getStoragePolicy, getStorageStatistics, getTrashRoot, getTrashRoots, getUsed, isDirectory, isFile, listCorruptFileBlocks, listFiles, listLocatedStatus, listLocatedStatus, listStatus, listStatus, listStatus, listStatusBatch, listStatusIterator, mkdirs, mkdirs, modifyAclEntries, moveFromLocalFile, moveFromLocalFile, moveToLocalFile, newInstance, newInstance, newInstance, newInstanceLocal, open, primitiveCreate, primitiveMkdir, primitiveMkdir, printStatistics, removeAcl, removeAclEntries, removeDefaultAcl, rename, renameSnapshot, resolveLink, resolvePath, setAcl, setDefaultUri, setDefaultUri, setReplication, setStoragePolicy, setWriteChecksum, setXAttr, supportsSymlinks, truncate, unsetStoragePolicy

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PATH_CODEC_USE_URI_ENCODING

public static final java.lang.String PATH_CODEC_USE_URI_ENCODING

Use new URI_ENCODED_PATH_CODEC.

See Also:

Constant Field Values

PATH_CODEC_USE_LEGACY_ENCODING

public static final java.lang.String PATH_CODEC_USE_LEGACY_ENCODING

Use LEGACY_PATH_CODEC.

See Also:

Constant Field Values

REPLICATION_FACTOR_DEFAULT

public static final short REPLICATION_FACTOR_DEFAULT

Default value of replication factor.

See Also:

Constant Field Values

DEFAULT_FILTER

public static final org.apache.hadoop.fs.PathFilter DEFAULT_FILTER

Default PathFilter that accepts all paths.

AUTHENTICATION_PREFIX

public static final java.lang.String AUTHENTICATION_PREFIX

Prefix to use for common authentication keys.

See Also:

Constant Field Values

PROPERTIES_FILE

public static final java.lang.String PROPERTIES_FILE

A resource file containing GCS related build properties.

See Also:

Constant Field Values

VERSION_PROPERTY

public static final java.lang.String VERSION_PROPERTY

The key in the PROPERTIES_FILE that contains the version built.

See Also:

Constant Field Values

UNKNOWN_VERSION

public static final java.lang.String UNKNOWN_VERSION

The version returned when one cannot be found in properties.

See Also:

Constant Field Values

VERSION

public static final java.lang.String VERSION

Current version.

GHFS_ID

public static final java.lang.String GHFS_ID

Identifies this version of the GoogleHadoopFileSystemBase library.

initUri

protected java.net.URI initUri

The URI the File System is passed in initialize.

delegationTokens

protected GcsDelegationTokens delegationTokens

Delegation token support

pathCodec

protected PathCodec pathCodec

defaultBlockSize

protected long defaultBlockSize

Default block size. Note that this is the size that is reported to Hadoop FS clients. It does not modify the actual block size of an underlying GCS object, because GCS JSON API does not allow modifying or querying the value. Modifying this value allows one to control how many mappers are used to process a given file.

counters

protected final com.google.common.collect.ImmutableMap<GoogleHadoopFileSystemBase.Counter,java.util.concurrent.atomic.AtomicLong> counters

Map of counter values

Constructor Detail

GoogleHadoopFileSystemBase

public GoogleHadoopFileSystemBase()

Constructs an instance of GoogleHadoopFileSystemBase; the internal GoogleCloudStorageFileSystem will be set up with config settings when initialize() is called.

Method Detail

createCounterMap

protected com.google.common.collect.ImmutableMap<GoogleHadoopFileSystemBase.Counter,java.util.concurrent.atomic.AtomicLong> createCounterMap()

getHomeDirectorySubpath

protected abstract java.lang.String getHomeDirectorySubpath()

Returns an unqualified path without any leading slash, relative to the filesystem root, which serves as the home directory of the current user; see getHomeDirectory for a description of what the home directory means.

getHadoopPath

public abstract org.apache.hadoop.fs.Path getHadoopPath(java.net.URI gcsPath)

Gets Hadoop path corresponding to the given GCS path.

Parameters:

gcsPath - Fully-qualified GCS path, of the form gs:///