Package com.google.cloud.hadoop.fs.gcs
Class GoogleHadoopFS
- java.lang.Object
-
- org.apache.hadoop.fs.AbstractFileSystem
-
- com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS
-
- All Implemented Interfaces:
org.apache.hadoop.fs.PathCapabilities
public class GoogleHadoopFS extends org.apache.hadoop.fs.AbstractFileSystem
GoogleHadoopFS provides a YARN compatible Abstract File System on top of Google Cloud Storage (GCS).It is implemented as a thin abstraction layer on top of GoogleHadoopFileSystem, but will soon be refactored to share a common base.
-
-
Constructor Summary
Constructors Constructor Description GoogleHadoopFS(GoogleHadoopFileSystem ghfs, URI uri, org.apache.hadoop.conf.Configuration conf)
GoogleHadoopFS(URI uri, org.apache.hadoop.conf.Configuration conf)
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description void
checkPath(org.apache.hadoop.fs.Path path)
Only accept valid AbstractFileSystem and GoogleHadoopFileSystem Paths.org.apache.hadoop.fs.FSDataOutputStream
createInternal(org.apache.hadoop.fs.Path file, EnumSet<org.apache.hadoop.fs.CreateFlag> flag, org.apache.hadoop.fs.permission.FsPermission absolutePermission, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress, org.apache.hadoop.fs.Options.ChecksumOpt checksumOpt, boolean createParent)
boolean
delete(org.apache.hadoop.fs.Path f, boolean recursive)
org.apache.hadoop.fs.BlockLocation[]
getFileBlockLocations(org.apache.hadoop.fs.Path f, long start, long len)
org.apache.hadoop.fs.FileChecksum
getFileChecksum(org.apache.hadoop.fs.Path f)
org.apache.hadoop.fs.FileStatus
getFileStatus(org.apache.hadoop.fs.Path f)
org.apache.hadoop.fs.FsStatus
getFsStatus()
org.apache.hadoop.fs.FsServerDefaults
getServerDefaults()
URI
getUri()
This is overridden to use GoogleHadoopFileSystem's URI, because AbstractFileSystem appends the default port to the authority.int
getUriDefaultPort()
boolean
isValidName(String src)
Follow HDFS conventions except allow for ':' in paths.org.apache.hadoop.fs.FileStatus[]
listStatus(org.apache.hadoop.fs.Path f)
void
mkdir(org.apache.hadoop.fs.Path dir, org.apache.hadoop.fs.permission.FsPermission permission, boolean createParent)
org.apache.hadoop.fs.FSDataInputStream
open(org.apache.hadoop.fs.Path f, int bufferSize)
void
renameInternal(org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst)
void
setOwner(org.apache.hadoop.fs.Path f, String username, String groupname)
void
setPermission(org.apache.hadoop.fs.Path f, org.apache.hadoop.fs.permission.FsPermission permission)
boolean
setReplication(org.apache.hadoop.fs.Path f, short replication)
void
setTimes(org.apache.hadoop.fs.Path f, long mtime, long atime)
void
setVerifyChecksum(boolean verifyChecksum)
-
Methods inherited from class org.apache.hadoop.fs.AbstractFileSystem
access, checkScheme, clearStatistics, create, createFileSystem, createMultipartUploader, createSnapshot, createSymlink, deleteSnapshot, equals, get, getAclStatus, getAllStatistics, getAllStoragePolicies, getCanonicalServiceName, getDelegationTokens, getFileLinkStatus, getFsStatus, getHomeDirectory, getInitialWorkingDirectory, getLinkTarget, getServerDefaults, getStatistics, getStatistics, getStoragePolicy, getUriPath, getXAttr, getXAttrs, getXAttrs, hashCode, hasPathCapability, listCorruptFileBlocks, listLocatedStatus, listStatusIterator, listXAttrs, makeQualified, methodNotSupported, modifyAclEntries, msync, open, openFileWithOptions, printStatistics, removeAcl, removeAclEntries, removeDefaultAcl, removeXAttr, rename, renameInternal, renameSnapshot, resolvePath, satisfyStoragePolicy, setAcl, setStoragePolicy, setXAttr, setXAttr, supportsSymlinks, truncate, unsetStoragePolicy
-
-
-
-
Constructor Detail
-
GoogleHadoopFS
public GoogleHadoopFS(URI uri, org.apache.hadoop.conf.Configuration conf) throws URISyntaxException, IOException
- Throws:
URISyntaxException
IOException
-
GoogleHadoopFS
public GoogleHadoopFS(GoogleHadoopFileSystem ghfs, URI uri, org.apache.hadoop.conf.Configuration conf) throws URISyntaxException, IOException
- Throws:
URISyntaxException
IOException
-
-
Method Detail
-
createInternal
public org.apache.hadoop.fs.FSDataOutputStream createInternal(org.apache.hadoop.fs.Path file, EnumSet<org.apache.hadoop.fs.CreateFlag> flag, org.apache.hadoop.fs.permission.FsPermission absolutePermission, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress, org.apache.hadoop.fs.Options.ChecksumOpt checksumOpt, boolean createParent) throws IOException
- Specified by:
createInternal
in classorg.apache.hadoop.fs.AbstractFileSystem
- Throws:
IOException
-
getUriDefaultPort
public int getUriDefaultPort()
- Specified by:
getUriDefaultPort
in classorg.apache.hadoop.fs.AbstractFileSystem
-
getUri
public URI getUri()
This is overridden to use GoogleHadoopFileSystem's URI, because AbstractFileSystem appends the default port to the authority.- Overrides:
getUri
in classorg.apache.hadoop.fs.AbstractFileSystem
-
isValidName
public boolean isValidName(String src)
Follow HDFS conventions except allow for ':' in paths.- Overrides:
isValidName
in classorg.apache.hadoop.fs.AbstractFileSystem
-
checkPath
public void checkPath(org.apache.hadoop.fs.Path path)
Only accept valid AbstractFileSystem and GoogleHadoopFileSystem Paths.- Overrides:
checkPath
in classorg.apache.hadoop.fs.AbstractFileSystem
-
getServerDefaults
public org.apache.hadoop.fs.FsServerDefaults getServerDefaults() throws IOException
- Specified by:
getServerDefaults
in classorg.apache.hadoop.fs.AbstractFileSystem
- Throws:
IOException
-
mkdir
public void mkdir(org.apache.hadoop.fs.Path dir, org.apache.hadoop.fs.permission.FsPermission permission, boolean createParent) throws IOException
- Specified by:
mkdir
in classorg.apache.hadoop.fs.AbstractFileSystem
- Throws:
IOException
-
delete
public boolean delete(org.apache.hadoop.fs.Path f, boolean recursive) throws IOException
- Specified by:
delete
in classorg.apache.hadoop.fs.AbstractFileSystem
- Throws:
IOException
-
open
public org.apache.hadoop.fs.FSDataInputStream open(org.apache.hadoop.fs.Path f, int bufferSize) throws IOException
- Specified by:
open
in classorg.apache.hadoop.fs.AbstractFileSystem
- Throws:
IOException
-
setReplication
public boolean setReplication(org.apache.hadoop.fs.Path f, short replication) throws IOException
- Specified by:
setReplication
in classorg.apache.hadoop.fs.AbstractFileSystem
- Throws:
IOException
-
renameInternal
public void renameInternal(org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst) throws IOException
- Specified by:
renameInternal
in classorg.apache.hadoop.fs.AbstractFileSystem
- Throws:
IOException
-
setPermission
public void setPermission(org.apache.hadoop.fs.Path f, org.apache.hadoop.fs.permission.FsPermission permission) throws IOException
- Specified by:
setPermission
in classorg.apache.hadoop.fs.AbstractFileSystem
- Throws:
IOException
-
setOwner
public void setOwner(org.apache.hadoop.fs.Path f, String username, String groupname) throws IOException
- Specified by:
setOwner
in classorg.apache.hadoop.fs.AbstractFileSystem
- Throws:
IOException
-
setTimes
public void setTimes(org.apache.hadoop.fs.Path f, long mtime, long atime) throws IOException
- Specified by:
setTimes
in classorg.apache.hadoop.fs.AbstractFileSystem
- Throws:
IOException
-
getFileChecksum
public org.apache.hadoop.fs.FileChecksum getFileChecksum(org.apache.hadoop.fs.Path f) throws IOException
- Specified by:
getFileChecksum
in classorg.apache.hadoop.fs.AbstractFileSystem
- Throws:
IOException
-
getFileStatus
public org.apache.hadoop.fs.FileStatus getFileStatus(org.apache.hadoop.fs.Path f) throws IOException
- Specified by:
getFileStatus
in classorg.apache.hadoop.fs.AbstractFileSystem
- Throws:
IOException
-
getFileBlockLocations
public org.apache.hadoop.fs.BlockLocation[] getFileBlockLocations(org.apache.hadoop.fs.Path f, long start, long len) throws IOException
- Specified by:
getFileBlockLocations
in classorg.apache.hadoop.fs.AbstractFileSystem
- Throws:
IOException
-
getFsStatus
public org.apache.hadoop.fs.FsStatus getFsStatus() throws IOException
- Specified by:
getFsStatus
in classorg.apache.hadoop.fs.AbstractFileSystem
- Throws:
IOException
-
listStatus
public org.apache.hadoop.fs.FileStatus[] listStatus(org.apache.hadoop.fs.Path f) throws IOException
- Specified by:
listStatus
in classorg.apache.hadoop.fs.AbstractFileSystem
- Throws:
IOException
-
setVerifyChecksum
public void setVerifyChecksum(boolean verifyChecksum)
- Specified by:
setVerifyChecksum
in classorg.apache.hadoop.fs.AbstractFileSystem
-
-