com.google.cloud.hadoop.fs.gcs

Class GoogleHadoopGlobalRootedFileSystem

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.fs.FileSystem

com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase

com.google.cloud.hadoop.fs.gcs.GoogleHadoopGlobalRootedFileSystem

All Implemented Interfaces:

FileSystemDescriptor, Closeable, AutoCloseable, org.apache.hadoop.conf.Configurable

Deprecated.

This implementation is deprecated and relies on the deprecated systemBucket property. Use GoogleHadoopFileSystem instead.

@Deprecated
public class GoogleHadoopGlobalRootedFileSystem
extends GoogleHadoopFileSystemBase

This class provides a Hadoop compatible File System on top of Google Cloud Storage (GCS). It is implemented as a thin abstraction layer on top of GCS. The layer hides any specific characteristics of the underlying store and exposes FileSystem interface understood by the Hadoop engine.

Users interact with the files in the storage using fully qualified URIs. The file system exposed by this class is identified using the 'gs' scheme. For example, gs://dir1/dir2/file1.txt.

This implementation translates paths between hadoop Path and GCS URI with the convention that the Hadoop root directly corresponds to the GCS "root", e.g. gs:/. This is convenient for many reasons, such as data portability and close equivalence to gsutil paths, but imposes certain inherited constraints, such as files not being allowed in root (only 'directories' can be placed in root), and directory names inside root have a more limited set of allowed characters.

One of the main goals of this implementation is to maintain compatibility with behavior of HDFS implementation when accessed through FileSystem interface. HDFS implementation is not very consistent about the cases when it throws versus the cases when methods return false. We run GHFS tests and HDFS tests against the same test data and use that as a guide to decide whether to throw or to return false.

Nested Class Summary

Nested classes/interfaces inherited from class com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase

GoogleHadoopFileSystemBase.Counter, GoogleHadoopFileSystemBase.ListStatusFileNotFoundBehavior, GoogleHadoopFileSystemBase.OutputStreamType, GoogleHadoopFileSystemBase.ParentTimestampUpdateIncludePredicate

Nested classes/interfaces inherited from class org.apache.hadoop.fs.FileSystem

org.apache.hadoop.fs.FileSystem.Statistics

Field Summary

Fields inherited from class com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase

AUTHENTICATION_PREFIX, BLOCK_SIZE_DEFAULT, BLOCK_SIZE_KEY, BUFFERSIZE_DEFAULT, BUFFERSIZE_KEY, counters, DEFAULT_FILTER, defaultBlockSize, ENABLE_GCE_SERVICE_ACCOUNT_AUTH_KEY, GCE_BUCKET_DELETE_ENABLE_DEFAULT, GCE_BUCKET_DELETE_ENABLE_KEY, GCS_APPLICATION_NAME_SUFFIX_DEFAULT, GCS_APPLICATION_NAME_SUFFIX_KEY, GCS_CLIENT_ID_KEY, GCS_CLIENT_SECRET_KEY, GCS_CREATE_SYSTEM_BUCKET_DEFAULT, GCS_CREATE_SYSTEM_BUCKET_KEY, GCS_ENABLE_FLAT_GLOB_DEFAULT, GCS_ENABLE_FLAT_GLOB_KEY, GCS_ENABLE_INFER_IMPLICIT_DIRECTORIES_DEFAULT, GCS_ENABLE_INFER_IMPLICIT_DIRECTORIES_KEY, GCS_ENABLE_MARKER_FILE_CREATION_DEFAULT, GCS_ENABLE_MARKER_FILE_CREATION_KEY, GCS_ENABLE_METADATA_CACHE_DEFAULT, GCS_ENABLE_METADATA_CACHE_KEY, GCS_ENABLE_PERFORMANCE_CACHE_DEFAULT, GCS_ENABLE_PERFORMANCE_CACHE_KEY, GCS_ENABLE_REPAIR_IMPLICIT_DIRECTORIES_DEFAULT, GCS_ENABLE_REPAIR_IMPLICIT_DIRECTORIES_KEY, GCS_FILE_SIZE_LIMIT_250GB, GCS_FILE_SIZE_LIMIT_250GB_DEFAULT, GCS_HTTP_TRANSPORT_DEFAULT, GCS_HTTP_TRANSPORT_KEY, GCS_INPUTSTREAM_FAST_FAIL_ON_NOT_FOUND_ENABLE_DEFAULT, GCS_INPUTSTREAM_FAST_FAIL_ON_NOT_FOUND_ENABLE_KEY, GCS_INPUTSTREAM_INPLACE_SEEK_LIMIT_DEFAULT, GCS_INPUTSTREAM_INPLACE_SEEK_LIMIT_KEY, GCS_INPUTSTREAM_INTERNALBUFFER_ENABLE_DEFAULT, GCS_INPUTSTREAM_INTERNALBUFFER_ENABLE_KEY, GCS_INPUTSTREAM_SUPPORT_CONTENT_ENCODING_ENABLE_DEFAULT, GCS_INPUTSTREAM_SUPPORT_CONTENT_ENCODING_ENABLE_KEY, GCS_MAX_LIST_ITEMS_PER_CALL, GCS_MAX_LIST_ITEMS_PER_CALL_DEFAULT, GCS_METADATA_CACHE_DIRECTORY_DEFAULT, GCS_METADATA_CACHE_DIRECTORY_KEY, GCS_METADATA_CACHE_MAX_ENTRY_AGE_DEFAULT, GCS_METADATA_CACHE_MAX_ENTRY_AGE_KEY, GCS_METADATA_CACHE_MAX_INFO_AGE_DEFAULT, GCS_METADATA_CACHE_MAX_INFO_AGE_KEY, GCS_METADATA_CACHE_TYPE_DEFAULT, GCS_METADATA_CACHE_TYPE_KEY, GCS_OUTPUTSTREAM_TYPE_DEFAULT, GCS_OUTPUTSTREAM_TYPE_KEY, GCS_PARENT_TIMESTAMP_UPDATE_ENABLE_DEFAULT, GCS_PARENT_TIMESTAMP_UPDATE_ENABLE_KEY, GCS_PARENT_TIMESTAMP_UPDATE_EXCLUDES_DEFAULT, GCS_PARENT_TIMESTAMP_UPDATE_EXCLUDES_KEY, GCS_PARENT_TIMESTAMP_UPDATE_INCLUDES_DEFAULT, GCS_PARENT_TIMESTAMP_UPDATE_INCLUDES_KEY, GCS_PERFORMANCE_CACHE_LIST_CACHING_ENABLE_DEFAULT, GCS_PERFORMANCE_CACHE_LIST_CACHING_ENABLE_KEY, GCS_PERFORMANCE_CACHE_MAX_ENTRY_AGE_MILLIS_DEFAULT, GCS_PERFORMANCE_CACHE_MAX_ENTRY_AGE_MILLIS_KEY, GCS_PROJECT_ID_KEY, GCS_PROXY_ADDRESS_DEFAULT, GCS_PROXY_ADDRESS_KEY, GCS_SYSTEM_BUCKET_KEY, GCS_WORKING_DIRECTORY_KEY, gcsfs, GHFS_ID, initUri, listStatusFileNotFoundBehavior, LOG, MR_JOB_HISTORY_DONE_DIR_KEY, MR_JOB_HISTORY_INTERMEDIATE_DONE_DIR_KEY, PATH_CODEC_DEFAULT, PATH_CODEC_KEY, PATH_CODEC_USE_LEGACY_ENCODING, PATH_CODEC_USE_URI_ENCODING, PERMISSIONS_TO_REPORT_DEFAULT, PERMISSIONS_TO_REPORT_KEY, PROPERTIES_FILE, REPLICATION_FACTOR_DEFAULT, SERVICE_ACCOUNT_AUTH_EMAIL_KEY, SERVICE_ACCOUNT_AUTH_KEYFILE_KEY, systemBucket, UNKNOWN_VERSION, VERSION, VERSION_PROPERTY, WRITE_BUFFERSIZE_DEFAULT, WRITE_BUFFERSIZE_KEY

Fields inherited from class org.apache.hadoop.fs.FileSystem

DEFAULT_FS, FS_DEFAULT_NAME_KEY, SHUTDOWN_HOOK_PRIORITY, statistics

Constructor Summary

Constructors

Constructor and Description
GoogleHadoopGlobalRootedFileSystem() Deprecated. Constructs an instance of GoogleHadoopGlobalRootedFileSystem; the internal GoogleCloudStorageFileSystem will be set up with config settings when initialize() is called.
GoogleHadoopGlobalRootedFileSystem(GoogleCloudStorageFileSystem gcsfs) Deprecated. Constructs an instance of GoogleHadoopGlobalRootedFileSystem using the provided GoogleCloudStorageFileSystem; initialize() will not re-initialize it.

Method Summary

Modifier and Type	Method and Description
org.apache.hadoop.fs.Path	getDefaultWorkingDirectory() Deprecated. Gets the default value of working directory.
org.apache.hadoop.fs.Path	getFileSystemRoot() Deprecated. Returns the Hadoop path representing the root of the FileSystem associated with this FileSystemDescriptor.
URI	getGcsPath(org.apache.hadoop.fs.Path hadoopPath) Deprecated. Gets GCS path corresponding to the given Hadoop path, which can be relative or absolute, and can have either gs:// or gs:/ forms.
org.apache.hadoop.fs.Path	getHadoopPath(URI gcsPath) Deprecated. Gets Hadoop path corresponding to the given GCS path.
protected String	getHomeDirectorySubpath() Deprecated. Returns an unqualified path without any leading slash, relative to the filesystem root, which serves as the home directory of the current user; see getHomeDirectory for a description of what the home directory means.
String	getScheme() Deprecated. As the global-rooted FileSystem, our hadoop-path "scheme" is distinct from GCS's scheme.

Methods inherited from class com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase

append, checkPath, close, completeLocalOutput, configureBuckets, copyFromLocalFile, copyFromLocalFile, copyToLocalFile, create, createCounterMap, delete, delete, deleteOnExit, getCanonicalServiceName, getContentSummary, getDefaultBlockSize, getDefaultPort, getDefaultReplication, getDelegationToken, getFileChecksum, getFileStatus, getGcsFs, getHadoopScheme, getHomeDirectory, getUri, getUsed, getWorkingDirectory, globStatus, globStatus, initialize, initialize, listStatus, makeQualified, mkdirs, open, processDeleteOnExit, rename, setListStatusFileNotFoundBehavior, setOwner, setPermission, setTimes, setVerifyChecksum, setWorkingDirectory, startLocalOutput

Methods inherited from class org.apache.hadoop.fs.FileSystem

access, addDelegationTokens, append, append, areSymlinksEnabled, cancelDeleteOnExit, canonicalizeUri, clearStatistics, closeAll, closeAllForUGI, concat, copyFromLocalFile, copyFromLocalFile, copyToLocalFile, copyToLocalFile, create, create, create, create, create, create, create, create, create, create, create, create, createNewFile, createNonRecursive, createNonRecursive, createNonRecursive, createSnapshot, createSnapshot, createSymlink, deleteSnapshot, enableSymlinks, exists, fixRelativePart, get, get, get, getAclStatus, getAllStatistics, getBlockSize, getCanonicalUri, getChildFileSystems, getDefaultBlockSize, getDefaultReplication, getDefaultUri, getFileBlockLocations, getFileBlockLocations, getFileChecksum, getFileLinkStatus, getFileSystemClass, getFSofPath, getInitialWorkingDirectory, getLength, getLinkTarget, getLocal, getName, getNamed, getReplication, getServerDefaults, getServerDefaults, getStatistics, getStatistics, getStatus, getStatus, getXAttr, getXAttrs, getXAttrs, isDirectory, isFile, listCorruptFileBlocks, listFiles, listLocatedStatus, listLocatedStatus, listStatus, listStatus, listStatus, listStatusIterator, listXAttrs, mkdirs, mkdirs, modifyAclEntries, moveFromLocalFile, moveFromLocalFile, moveToLocalFile, newInstance, newInstance, newInstance, newInstanceLocal, open, primitiveCreate, primitiveMkdir, primitiveMkdir, printStatistics, removeAcl, removeAclEntries, removeDefaultAcl, removeXAttr, rename, renameSnapshot, resolveLink, resolvePath, setAcl, setDefaultUri, setDefaultUri, setReplication, setWriteChecksum, setXAttr, setXAttr, supportsSymlinks, truncate

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

GoogleHadoopGlobalRootedFileSystem

public GoogleHadoopGlobalRootedFileSystem()

Deprecated.

Constructs an instance of GoogleHadoopGlobalRootedFileSystem; the internal GoogleCloudStorageFileSystem will be set up with config settings when initialize() is called.

GoogleHadoopGlobalRootedFileSystem

public GoogleHadoopGlobalRootedFileSystem(GoogleCloudStorageFileSystem gcsfs)

Deprecated.

Constructs an instance of GoogleHadoopGlobalRootedFileSystem using the provided GoogleCloudStorageFileSystem; initialize() will not re-initialize it.

Method Detail

getHomeDirectorySubpath

protected String getHomeDirectorySubpath()

Deprecated.

Returns an unqualified path without any leading slash, relative to the filesystem root, which serves as the home directory of the current user; see getHomeDirectory for a description of what the home directory means.

Specified by:

getHomeDirectorySubpath in class GoogleHadoopFileSystemBase

getHadoopPath

public org.apache.hadoop.fs.Path getHadoopPath(URI gcsPath)

Deprecated.

Gets Hadoop path corresponding to the given GCS path.

Specified by:

getHadoopPath in class GoogleHadoopFileSystemBase

Parameters:

gcsPath - Fully-qualified GCS path, of the form gs:///