com.google.cloud.hadoop.fs.gcs

Class GoogleHadoopFileSystemConfiguration

java.lang.Object

com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemConfiguration

public class GoogleHadoopFileSystemConfiguration
extends Object

This class provides a configuration for the GoogleHadoopFileSystem implementations.

Field Summary

Fields

Modifier and Type	Field and Description
static GoogleHadoopFileSystemConfigurationProperty<String>	AUTH_CLIENT_ID Configuration key for GCS client ID.
static GoogleHadoopFileSystemConfigurationProperty<String>	AUTH_CLIENT_SECRET Configuration key for GCS client secret.
static GoogleHadoopFileSystemConfigurationProperty<String>	AUTH_SERVICE_ACCOUNT_EMAIL Configuration key specifying the email address of the service-account with which to authenticate.
static GoogleHadoopFileSystemConfigurationProperty<String>	AUTH_SERVICE_ACCOUNT_ENABLE Configuration key for enabling GCE service account authentication.
static GoogleHadoopFileSystemConfigurationProperty<String>	AUTH_SERVICE_ACCOUNT_KEY_FILE Configuration key specifying local file containing a service-account private .p12 keyfile.
static GoogleHadoopFileSystemConfigurationProperty<Long>	BLOCK_SIZE Configuration key for default block size of a file.
static GoogleHadoopFileSystemConfigurationProperty<String>	DELEGATION_TOKEN_BINDING_CLASS Configuration key for Delegation Token binding class.
static GoogleHadoopFileSystemConfigurationProperty<Boolean>	GCE_BUCKET_DELETE_ENABLE If true, recursive delete on a path that refers to a GCS bucket itself ('/' for any bucket-rooted GoogleHadoopFileSystem) or delete on that path when it's empty will result in fully deleting the GCS bucket.
static GoogleHadoopFileSystemConfigurationProperty<String>	GCS_APPLICATION_NAME_SUFFIX Configuration key for adding a suffix to the GHFS application name sent to GCS.
static GoogleHadoopFileSystemConfigurationProperty<Integer>	GCS_BATCH_THREADS Configuration key for a number of threads to execute batch requests.
static GoogleHadoopFileSystemConfigurationProperty<Boolean>	GCS_CONCURRENT_GLOB_ENABLE Configuration key for enabling the use of flat and regular glob search algorithms in two parallel threads.
static GoogleHadoopFileSystemConfigurationProperty<String>	GCS_CONFIG_OVERRIDE_FILE Override configuration file path.
static GoogleHadoopFileSystemConfigurationProperty<Boolean>	GCS_COOPERATIVE_LOCKING_ENABLE Configuration key for using cooperative locking to achieve a directory mutation operations isolation.
static GoogleHadoopFileSystemConfigurationProperty<Long>	GCS_COOPERATIVE_LOCKING_EXPIRATION_TIMEOUT_MS Configuration key for lock expiration when using cooperative locking.
static GoogleHadoopFileSystemConfigurationProperty<Integer>	GCS_COOPERATIVE_LOCKING_MAX_CONCURRENT_OPERATIONS Configuration key for maximum allowed concurrent operations when using cooperative locking.
static GoogleHadoopFileSystemConfigurationProperty<Integer>	GCS_COPY_BATCH_THREADS Configuration key for a number of threads to execute batch requests for copy operations.
static GoogleHadoopFileSystemConfigurationProperty<Long>	GCS_COPY_MAX_REQUESTS_PER_BATCH Configuration key for a max number of GCS RPCs in batch request for copy operations.
static GoogleHadoopFileSystemConfigurationProperty<Boolean>	GCS_COPY_WITH_REWRITE_ENABLE Configuration key for enabling the use of Rewrite requests for copy operations.
static GoogleHadoopFileSystemConfigurationProperty<GoogleHadoopFileSystemBase.GcsFileChecksumType>	GCS_FILE_CHECKSUM_TYPE Configuration key for which type of FileChecksum to return; if a particular file doesn't support the requested type, then getFileChecksum() will return null for that file.
static GoogleHadoopFileSystemConfigurationProperty<Boolean>	GCS_FLAT_GLOB_ENABLE Configuration key for enabling the use of a large flat listing to pre-populate possible glob matches in a single API call before running the core globbing logic in-memory rather than sequentially and recursively performing API calls.
static GoogleHadoopFileSystemConfigurationProperty<GoogleCloudStorageReadOptions.GenerationReadConsistency>	GCS_GENERATION_READ_CONSISTENCY Configuration key for the generation consistency read model.
static GoogleHadoopFileSystemConfigurationProperty<Integer>	GCS_HTTP_CONNECT_TIMEOUT Configuration key for the connect timeout (in millisecond) for HTTP request to GCS.
static GoogleHadoopFileSystemConfigurationProperty<Integer>	GCS_HTTP_MAX_RETRY Configuration key for the max number of retries for failed HTTP request to GCS.
static GoogleHadoopFileSystemConfigurationProperty<Integer>	GCS_HTTP_READ_TIMEOUT Configuration key for the connect timeout (in millisecond) for HTTP request to GCS.
static GoogleHadoopFileSystemConfigurationProperty<HttpTransportFactory.HttpTransportType>	GCS_HTTP_TRANSPORT Configuration key for the name of HttpTransport class to use for connecting to GCS.
static GoogleHadoopFileSystemConfigurationProperty<Boolean>	GCS_INFER_IMPLICIT_DIRECTORIES_ENABLE Configuration key for enabling automatic inference of implicit directories.
static GoogleHadoopFileSystemConfigurationProperty<Integer>	GCS_INPUT_STREAM_BUFFER_SIZE Configuration key for setting read buffer size.
static GoogleHadoopFileSystemConfigurationProperty<GoogleCloudStorageReadOptions.Fadvise>	GCS_INPUT_STREAM_FADVISE Tunes reading objects behavior to optimize HTTP GET requests for various use cases.
static GoogleHadoopFileSystemConfigurationProperty<Boolean>	GCS_INPUT_STREAM_FAST_FAIL_ON_NOT_FOUND_ENABLE If true, on opening a file we will proactively perform a metadata GET to check whether the object exists, even though the underlying channel will not open a data stream until read() is actually called so that streams can seek to nonzero file positions without incurring an extra stream creation.
static GoogleHadoopFileSystemConfigurationProperty<Long>	GCS_INPUT_STREAM_INPLACE_SEEK_LIMIT If forward seeks are within this many bytes of the current position, seeks are performed by reading and discarding bytes in-place rather than opening a new underlying stream.
static GoogleHadoopFileSystemConfigurationProperty<Integer>	GCS_INPUT_STREAM_MIN_RANGE_REQUEST_SIZE Minimum size in bytes of the HTTP Range header set in GCS request when opening new stream to read an object.
static GoogleHadoopFileSystemConfigurationProperty<Boolean>	GCS_INPUT_STREAM_SUPPORT_GZIP_ENCODING_ENABLE If true, reading a file with GZIP content encoding (HTTP header "Content-Encoding: gzip") will result in failure (IOException is thrown).
static GoogleHadoopFileSystemConfigurationProperty<Boolean>	GCS_LAZY_INITIALIZATION_ENABLE Configuration key for enabling lazy initialization of GCS FS instance.
static GoogleHadoopFileSystemConfigurationProperty<String>	GCS_MARKER_FILE_PATTERN Configuration key for marker file pattern.
static GoogleHadoopFileSystemConfigurationProperty<Long>	GCS_MAX_LIST_ITEMS_PER_CALL Configuration key for number of items to return per call to the list* GCS RPCs.
static GoogleHadoopFileSystemConfigurationProperty<Long>	GCS_MAX_REQUESTS_PER_BATCH Configuration key for a max number of GCS RPCs in batch request.
static GoogleHadoopFileSystemConfigurationProperty<Integer>	GCS_MAX_WAIT_MILLIS_EMPTY_OBJECT_CREATE Configuration key for modifying the maximum amount of time to wait for empty object creation.
static GoogleHadoopFileSystemConfigurationProperty<Integer>	GCS_OUTPUT_STREAM_BUFFER_SIZE Configuration key for setting write buffer size.
static GoogleHadoopFileSystemConfigurationProperty<Boolean>	GCS_OUTPUT_STREAM_DIRECT_UPLOAD_ENABLE Configuration key for enabling GCS direct upload.
static GoogleHadoopFileSystemConfigurationProperty<Integer>	GCS_OUTPUT_STREAM_PIPE_BUFFER_SIZE Configuration key for setting pipe buffer size.
static GoogleHadoopFileSystemConfigurationProperty<GoogleHadoopFileSystemBase.OutputStreamType>	GCS_OUTPUT_STREAM_TYPE Configuration key for which type of output stream to use; different options may have different degrees of support for advanced features like hsync() and different performance characteristics.
static GoogleHadoopFileSystemConfigurationProperty<Integer>	GCS_OUTPUT_STREAM_UPLOAD_CHUNK_SIZE Configuration key for setting GCS upload chunk size.
static GoogleHadoopFileSystemConfigurationProperty<Boolean>	GCS_PERFORMANCE_CACHE_ENABLE Configuration key for using a local item cache to supplement GCS API "getFile" results.
static GoogleHadoopFileSystemConfigurationProperty<Boolean>	GCS_PERFORMANCE_CACHE_LIST_CACHING_ENABLE Configuration key for whether or not to enable list caching for the performance cache.
static GoogleHadoopFileSystemConfigurationProperty<Long>	GCS_PERFORMANCE_CACHE_MAX_ENTRY_AGE_MILLIS Configuration key for maximum number of milliseconds a GoogleCloudStorageItemInfo will remain "valid" in the performance cache before it's invalidated.
static GoogleHadoopFileSystemConfigurationProperty<String>	GCS_PROJECT_ID Configuration key for GCS project ID.
static GoogleHadoopFileSystemConfigurationProperty<String>	GCS_PROXY_ADDRESS Configuration key for setting a proxy for the connector to use to connect to GCS.
static GoogleHadoopFileSystemConfigurationProperty<String>	GCS_PROXY_PASSWORD Configuration key for setting a proxy password for the connector to use to authenticate with proxy used to connect to GCS.
static GoogleHadoopFileSystemConfigurationProperty<String>	GCS_PROXY_USERNAME Configuration key for setting a proxy username for the connector to use to authenticate with proxy used to connect to GCS.
static GoogleHadoopFileSystemConfigurationProperty<Boolean>	GCS_REPAIR_IMPLICIT_DIRECTORIES_ENABLE Configuration key for enabling automatic repair of implicit directories whenever detected inside delete and rename calls.
static GoogleHadoopFileSystemConfigurationProperty<Collection<String>>	GCS_REQUESTER_PAYS_BUCKETS Configuration key for GCS Requester Pays Buckets.
static GoogleHadoopFileSystemConfigurationProperty<RequesterPaysOptions.RequesterPaysMode>	GCS_REQUESTER_PAYS_MODE Configuration key for GCS project ID.
static GoogleHadoopFileSystemConfigurationProperty<String>	GCS_REQUESTER_PAYS_PROJECT_ID Configuration key for GCS Requester Pays Project ID.
static GoogleHadoopFileSystemConfigurationProperty<Long>	GCS_REWRITE_MAX_BYTES_PER_CALL Configuration key for specifying max number of bytes rewritten in a single rewrite request when fs.gs.copy.with.rewrite.enable is set to 'true'.
static GoogleHadoopFileSystemConfigurationProperty<Boolean>	GCS_STATUS_PARALLEL_ENABLE If true, executes GCS requests in listStatus and getFileStatus methods in parallel to reduce latency.
static GoogleHadoopFileSystemConfigurationProperty<String>	GCS_WORKING_DIRECTORY Configuration key for initial working directory of a GHFS instance.
static String	MR_JOB_HISTORY_DONE_DIR_KEY Configuration key of the MR done directory.
static String	MR_JOB_HISTORY_INTERMEDIATE_DONE_DIR_KEY Configuration key for the MR intermediate done dir.
static GoogleHadoopFileSystemConfigurationProperty<String>	PATH_CODEC Configuration key for changing the path codec from legacy to 'uri path encoding'.
static GoogleHadoopFileSystemConfigurationProperty<String>	PERMISSIONS_TO_REPORT Key for the permissions that we report a file or directory to have.

Constructor Summary

Constructors

Constructor and Description
GoogleHadoopFileSystemConfiguration()

Method Summary

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MR_JOB_HISTORY_INTERMEDIATE_DONE_DIR_KEY

public static final String MR_JOB_HISTORY_INTERMEDIATE_DONE_DIR_KEY

Configuration key for the MR intermediate done dir.

See Also:

Constant Field Values

MR_JOB_HISTORY_DONE_DIR_KEY

public static final String MR_JOB_HISTORY_DONE_DIR_KEY

Configuration key of the MR done directory.

See Also:

Constant Field Values

PERMISSIONS_TO_REPORT

public static final GoogleHadoopFileSystemConfigurationProperty<String> PERMISSIONS_TO_REPORT

Key for the permissions that we report a file or directory to have. Can either be octal or symbolic mode accepted by FsPermission.FsPermission(String)

Default value for the permissions that we report a file or directory to have. Note: We do not really support file/dir permissions but we need to report some permission value when Hadoop calls getFileStatus(). A MapReduce job fails if we report permissions more relaxed than the value below and this is the default File System.

BLOCK_SIZE

public static final GoogleHadoopFileSystemConfigurationProperty<Long> BLOCK_SIZE

Configuration key for default block size of a file.

Note that this is the size that is reported to Hadoop FS clients. It does not modify the actual block size of an underlying GCS object, because GCS JSON API does not allow modifying or querying the value. Modifying this value allows one to control how many mappers are used to process a given file.

AUTH_SERVICE_ACCOUNT_ENABLE

public static final GoogleHadoopFileSystemConfigurationProperty<String> AUTH_SERVICE_ACCOUNT_ENABLE

Configuration key for enabling GCE service account authentication. This key is deprecated. See HadoopCredentialConfiguration for current key names.

AUTH_SERVICE_ACCOUNT_EMAIL

public static final GoogleHadoopFileSystemConfigurationProperty<String> AUTH_SERVICE_ACCOUNT_EMAIL

Configuration key specifying the email address of the service-account with which to authenticate. Only required if AUTH_SERVICE_ACCOUNT_ENABLE is true AND we're using fs.gs.service.account.auth.keyfile to authenticate with a private keyfile. NB: Once GCE supports setting multiple service account email addresses for metadata auth, this key will also be used in the metadata auth flow. This key is deprecated. See HadoopCredentialConfiguration for current key names.

AUTH_SERVICE_ACCOUNT_KEY_FILE

public static final GoogleHadoopFileSystemConfigurationProperty<String> AUTH_SERVICE_ACCOUNT_KEY_FILE

Configuration key specifying local file containing a service-account private .p12 keyfile. Only used if AUTH_SERVICE_ACCOUNT_ENABLE is true; if provided, the keyfile will be used for service-account authentication. Otherwise, it is assumed that we are on a GCE VM with metadata-authentication for service-accounts enabled, and the metadata server will be used instead. Default value: none This key is deprecated. See HadoopCredentialConfiguration for current key names.

AUTH_CLIENT_ID

public static final GoogleHadoopFileSystemConfigurationProperty<String> AUTH_CLIENT_ID

Configuration key for GCS client ID. Required if AUTH_SERVICE_ACCOUNT_ENABLE == false. Default value: none This key is deprecated. See HadoopCredentialConfiguration for current key names.

AUTH_CLIENT_SECRET

public static final GoogleHadoopFileSystemConfigurationProperty<String> AUTH_CLIENT_SECRET

Configuration key for GCS client secret. Required if AUTH_SERVICE_ACCOUNT_ENABLE == false. Default value: none This key is deprecated. See HadoopCredentialConfiguration for current key names.

DELEGATION_TOKEN_BINDING_CLASS

public static final GoogleHadoopFileSystemConfigurationProperty<String> DELEGATION_TOKEN_BINDING_CLASS

Configuration key for Delegation Token binding class. Default value: none

GCS_PROJECT_ID

public static final GoogleHadoopFileSystemConfigurationProperty<String> GCS_PROJECT_ID

Configuration key for GCS project ID. Default value: none

GCS_WORKING_DIRECTORY

public static final GoogleHadoopFileSystemConfigurationProperty<String> GCS_WORKING_DIRECTORY

Configuration key for initial working directory of a GHFS instance. Default value: '/'

GCE_BUCKET_DELETE_ENABLE

public static final GoogleHadoopFileSystemConfigurationProperty<Boolean> GCE_BUCKET_DELETE_ENABLE

If true, recursive delete on a path that refers to a GCS bucket itself ('/' for any bucket-rooted GoogleHadoopFileSystem) or delete on that path when it's empty will result in fully deleting the GCS bucket. If false, any operation that normally would have deleted the bucket will be ignored instead. Setting to 'false' preserves the typical behavior of "rm -rf /" which translates to deleting everything inside of root, but without clobbering the filesystem authority corresponding to that root path in the process.

GCS_REQUESTER_PAYS_MODE

public static final GoogleHadoopFileSystemConfigurationProperty<RequesterPaysOptions.RequesterPaysMode> GCS_REQUESTER_PAYS_MODE

Configuration key for GCS project ID. Default value: "DISABLED"

GCS_REQUESTER_PAYS_PROJECT_ID

public static final GoogleHadoopFileSystemConfigurationProperty<String> GCS_REQUESTER_PAYS_PROJECT_ID

Configuration key for GCS Requester Pays Project ID. Default value: none

GCS_REQUESTER_PAYS_BUCKETS

public static final GoogleHadoopFileSystemConfigurationProperty<Collection<String>> GCS_REQUESTER_PAYS_BUCKETS

Configuration key for GCS Requester Pays Buckets. Default value: none

GCS_FILE_CHECKSUM_TYPE

public static final GoogleHadoopFileSystemConfigurationProperty<GoogleHadoopFileSystemBase.GcsFileChecksumType> GCS_FILE_CHECKSUM_TYPE

Configuration key for which type of FileChecksum to return; if a particular file doesn't support the requested type, then getFileChecksum() will return null for that file.

GCS_PERFORMANCE_CACHE_ENABLE

public static final GoogleHadoopFileSystemConfigurationProperty<Boolean> GCS_PERFORMANCE_CACHE_ENABLE

Configuration key for using a local item cache to supplement GCS API "getFile" results. This provides faster access to recently queried data. Because the data is cached, modifications made outside of this instance may not be immediately reflected. The performance cache can be used in conjunction with other caching options.

GCS_PERFORMANCE_CACHE_MAX_ENTRY_AGE_MILLIS

public static final GoogleHadoopFileSystemConfigurationProperty<Long> GCS_PERFORMANCE_CACHE_MAX_ENTRY_AGE_MILLIS

Configuration key for maximum number of milliseconds a GoogleCloudStorageItemInfo will remain "valid" in the performance cache before it's invalidated.

GCS_PERFORMANCE_CACHE_LIST_CACHING_ENABLE

public static final GoogleHadoopFileSystemConfigurationProperty<Boolean> GCS_PERFORMANCE_CACHE_LIST_CACHING_ENABLE

Configuration key for whether or not to enable list caching for the performance cache.

GCS_STATUS_PARALLEL_ENABLE

public static final GoogleHadoopFileSystemConfigurationProperty<Boolean> GCS_STATUS_PARALLEL_ENABLE

If true, executes GCS requests in listStatus and getFileStatus methods in parallel to reduce latency.

GCS_LAZY_INITIALIZATION_ENABLE

public static final GoogleHadoopFileSystemConfigurationProperty<Boolean> GCS_LAZY_INITIALIZATION_ENABLE

Configuration key for enabling lazy initialization of GCS FS instance.

GCS_REPAIR_IMPLICIT_DIRECTORIES_ENABLE

public static final GoogleHadoopFileSystemConfigurationProperty<Boolean> GCS_REPAIR_IMPLICIT_DIRECTORIES_ENABLE

Configuration key for enabling automatic repair of implicit directories whenever detected inside delete and rename calls.

PATH_CODEC

public static final GoogleHadoopFileSystemConfigurationProperty<String> PATH_CODEC

Configuration key for changing the path codec from legacy to 'uri path encoding'.

GCS_INFER_IMPLICIT_DIRECTORIES_ENABLE

public static final GoogleHadoopFileSystemConfigurationProperty<Boolean> GCS_INFER_IMPLICIT_DIRECTORIES_ENABLE

Configuration key for enabling automatic inference of implicit directories. If set, we create and return in-memory directory objects on the fly when no backing object exists, but we know there are files with the same prefix.

GCS_FLAT_GLOB_ENABLE

public static final GoogleHadoopFileSystemConfigurationProperty<Boolean> GCS_FLAT_GLOB_ENABLE

Configuration key for enabling the use of a large flat listing to pre-populate possible glob matches in a single API call before running the core globbing logic in-memory rather than sequentially and recursively performing API calls.

GCS_CONCURRENT_GLOB_ENABLE

public static final GoogleHadoopFileSystemConfigurationProperty<Boolean> GCS_CONCURRENT_GLOB_ENABLE

Configuration key for enabling the use of flat and regular glob search algorithms in two parallel threads. After the first one returns result, the other one will be interrupted.

GCS_MARKER_FILE_PATTERN

public static final GoogleHadoopFileSystemConfigurationProperty<String> GCS_MARKER_FILE_PATTERN

Configuration key for marker file pattern. Default value: none

GCS_MAX_REQUESTS_PER_BATCH

public static final GoogleHadoopFileSystemConfigurationProperty<Long> GCS_MAX_REQUESTS_PER_BATCH

Configuration key for a max number of GCS RPCs in batch request.

GCS_BATCH_THREADS

public static final GoogleHadoopFileSystemConfigurationProperty<Integer> GCS_BATCH_THREADS

Configuration key for a number of threads to execute batch requests.

GCS_COPY_MAX_REQUESTS_PER_BATCH

public static final GoogleHadoopFileSystemConfigurationProperty<Long> GCS_COPY_MAX_REQUESTS_PER_BATCH

Configuration key for a max number of GCS RPCs in batch request for copy operations.

GCS_COPY_BATCH_THREADS

public static final GoogleHadoopFileSystemConfigurationProperty<Integer> GCS_COPY_BATCH_THREADS

Configuration key for a number of threads to execute batch requests for copy operations.

GCS_COPY_WITH_REWRITE_ENABLE

public static final GoogleHadoopFileSystemConfigurationProperty<Boolean> GCS_COPY_WITH_REWRITE_ENABLE

Configuration key for enabling the use of Rewrite requests for copy operations. Rewrite request has the same effect as Copy request, but it can handle moving large objects that may potentially timeout a Copy request.

GCS_REWRITE_MAX_BYTES_PER_CALL

public static final GoogleHadoopFileSystemConfigurationProperty<Long> GCS_REWRITE_MAX_BYTES_PER_CALL

Configuration key for specifying max number of bytes rewritten in a single rewrite request when fs.gs.copy.with.rewrite.enable is set to 'true'.

GCS_MAX_LIST_ITEMS_PER_CALL

public static final GoogleHadoopFileSystemConfigurationProperty<Long> GCS_MAX_LIST_ITEMS_PER_CALL

Configuration key for number of items to return per call to the list* GCS RPCs.

GCS_HTTP_MAX_RETRY

public static final GoogleHadoopFileSystemConfigurationProperty<Integer> GCS_HTTP_MAX_RETRY

Configuration key for the max number of retries for failed HTTP request to GCS. Note that the connector will retry *up to* the number of times as specified, using a default ExponentialBackOff strategy.

Also, note that this number will only control the number of retries in the low level HTTP request implementation.

GCS_HTTP_CONNECT_TIMEOUT

public static final GoogleHadoopFileSystemConfigurationProperty<Integer> GCS_HTTP_CONNECT_TIMEOUT

Configuration key for the connect timeout (in millisecond) for HTTP request to GCS.

GCS_HTTP_READ_TIMEOUT

public static final GoogleHadoopFileSystemConfigurationProperty<Integer> GCS_HTTP_READ_TIMEOUT

Configuration key for the connect timeout (in millisecond) for HTTP request to GCS.

GCS_PROXY_ADDRESS

public static final GoogleHadoopFileSystemConfigurationProperty<String> GCS_PROXY_ADDRESS

Configuration key for setting a proxy for the connector to use to connect to GCS. The proxy must be an HTTP proxy of the form "host:port".

GCS_PROXY_USERNAME

public static final GoogleHadoopFileSystemConfigurationProperty<String> GCS_PROXY_USERNAME

Configuration key for setting a proxy username for the connector to use to authenticate with proxy used to connect to GCS.

GCS_PROXY_PASSWORD

public static final GoogleHadoopFileSystemConfigurationProperty<String> GCS_PROXY_PASSWORD

Configuration key for setting a proxy password for the connector to use to authenticate with proxy used to connect to GCS.

GCS_HTTP_TRANSPORT

public static final GoogleHadoopFileSystemConfigurationProperty<HttpTransportFactory.HttpTransportType> GCS_HTTP_TRANSPORT

Configuration key for the name of HttpTransport class to use for connecting to GCS. Must be the name of an HttpTransportFactory.HttpTransportType (APACHE or JAVA_NET).

GCS_APPLICATION_NAME_SUFFIX

public static final GoogleHadoopFileSystemConfigurationProperty<String> GCS_APPLICATION_NAME_SUFFIX

Configuration key for adding a suffix to the GHFS application name sent to GCS.

GCS_MAX_WAIT_MILLIS_EMPTY_OBJECT_CREATE

public static final GoogleHadoopFileSystemConfigurationProperty<Integer> GCS_MAX_WAIT_MILLIS_EMPTY_OBJECT_CREATE

Configuration key for modifying the maximum amount of time to wait for empty object creation.

GCS_OUTPUT_STREAM_TYPE

public static final GoogleHadoopFileSystemConfigurationProperty<GoogleHadoopFileSystemBase.OutputStreamType> GCS_OUTPUT_STREAM_TYPE

Configuration key for which type of output stream to use; different options may have different degrees of support for advanced features like hsync() and different performance characteristics. Options:

BASIC: Stream is closest analogue to direct wrapper around low-level HTTP stream into GCS.

SYNCABLE_COMPOSITE: Stream behaves similarly to BASIC when used with basic create/write/close patterns, but supports hsync() by creating discrete temporary GCS objects which are composed onto the destination object.

GCS_OUTPUT_STREAM_BUFFER_SIZE

public static final GoogleHadoopFileSystemConfigurationProperty<Integer> GCS_OUTPUT_STREAM_BUFFER_SIZE

Configuration key for setting write buffer size.

GCS_OUTPUT_STREAM_PIPE_BUFFER_SIZE

public static final GoogleHadoopFileSystemConfigurationProperty<Integer> GCS_OUTPUT_STREAM_PIPE_BUFFER_SIZE

Configuration key for setting pipe buffer size.

GCS_OUTPUT_STREAM_UPLOAD_CHUNK_SIZE

public static final GoogleHadoopFileSystemConfigurationProperty<Integer> GCS_OUTPUT_STREAM_UPLOAD_CHUNK_SIZE

Configuration key for setting GCS upload chunk size.

GCS_OUTPUT_STREAM_DIRECT_UPLOAD_ENABLE

public static final GoogleHadoopFileSystemConfigurationProperty<Boolean> GCS_OUTPUT_STREAM_DIRECT_UPLOAD_ENABLE

Configuration key for enabling GCS direct upload.

GCS_GENERATION_READ_CONSISTENCY

public static final GoogleHadoopFileSystemConfigurationProperty<GoogleCloudStorageReadOptions.GenerationReadConsistency> GCS_GENERATION_READ_CONSISTENCY

Configuration key for the generation consistency read model.

GCS_INPUT_STREAM_BUFFER_SIZE

public static final GoogleHadoopFileSystemConfigurationProperty<Integer> GCS_INPUT_STREAM_BUFFER_SIZE

Configuration key for setting read buffer size.

GCS_INPUT_STREAM_FAST_FAIL_ON_NOT_FOUND_ENABLE

public static final GoogleHadoopFileSystemConfigurationProperty<Boolean> GCS_INPUT_STREAM_FAST_FAIL_ON_NOT_FOUND_ENABLE

If true, on opening a file we will proactively perform a metadata GET to check whether the object exists, even though the underlying channel will not open a data stream until read() is actually called so that streams can seek to nonzero file positions without incurring an extra stream creation. This is necessary to technically match the expected behavior of Hadoop filesystems, but incurs extra latency overhead on open(). If the calling code can handle late failures on not-found errors, or has independently already ensured that a file exists before calling open(), then set this to false for more efficient reads.

GCS_INPUT_STREAM_SUPPORT_GZIP_ENCODING_ENABLE

public static final GoogleHadoopFileSystemConfigurationProperty<Boolean> GCS_INPUT_STREAM_SUPPORT_GZIP_ENCODING_ENABLE

If true, reading a file with GZIP content encoding (HTTP header "Content-Encoding: gzip") will result in failure (IOException is thrown).

GCS_INPUT_STREAM_INPLACE_SEEK_LIMIT

public static final GoogleHadoopFileSystemConfigurationProperty<Long> GCS_INPUT_STREAM_INPLACE_SEEK_LIMIT

If forward seeks are within this many bytes of the current position, seeks are performed by reading and discarding bytes in-place rather than opening a new underlying stream.

GCS_INPUT_STREAM_FADVISE

public static final GoogleHadoopFileSystemConfigurationProperty<GoogleCloudStorageReadOptions.Fadvise> GCS_INPUT_STREAM_FADVISE

Tunes reading objects behavior to optimize HTTP GET requests for various use cases.

GCS_INPUT_STREAM_MIN_RANGE_REQUEST_SIZE

public static final GoogleHadoopFileSystemConfigurationProperty<Integer> GCS_INPUT_STREAM_MIN_RANGE_REQUEST_SIZE

Minimum size in bytes of the HTTP Range header set in GCS request when opening new stream to read an object.

GCS_CONFIG_OVERRIDE_FILE

public static final GoogleHadoopFileSystemConfigurationProperty<String> GCS_CONFIG_OVERRIDE_FILE

Override configuration file path. This file must be a valid Hadoop configuration file.

GCS_COOPERATIVE_LOCKING_ENABLE

public static final GoogleHadoopFileSystemConfigurationProperty<Boolean> GCS_COOPERATIVE_LOCKING_ENABLE

Configuration key for using cooperative locking to achieve a directory mutation operations isolation.

GCS_COOPERATIVE_LOCKING_EXPIRATION_TIMEOUT_MS

public static final GoogleHadoopFileSystemConfigurationProperty<Long> GCS_COOPERATIVE_LOCKING_EXPIRATION_TIMEOUT_MS

Configuration key for lock expiration when using cooperative locking.

GCS_COOPERATIVE_LOCKING_MAX_CONCURRENT_OPERATIONS

public static final GoogleHadoopFileSystemConfigurationProperty<Integer> GCS_COOPERATIVE_LOCKING_MAX_CONCURRENT_OPERATIONS

Configuration key for maximum allowed concurrent operations when using cooperative locking.

Constructor Detail

GoogleHadoopFileSystemConfiguration

public GoogleHadoopFileSystemConfiguration()