org.apache.hadoop.fs.sftp

Class SFTPFileSystem

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.fs.FileSystem

org.apache.hadoop.fs.sftp.SFTPFileSystem

All Implemented Interfaces:

Closeable, AutoCloseable, Configurable, PathCapabilities, org.apache.hadoop.security.token.DelegationTokenIssuer

public class SFTPFileSystem
extends FileSystem

SFTP FileSystem.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.fs.FileSystem

FileSystem.DirListingIterator<T extends FileStatus>, FileSystem.Statistics

Field Summary

Fields

Modifier and Type	Field and Description
static int	DEFAULT_BLOCK_SIZE
static int	DEFAULT_BUFFER_SIZE
static String	E_CREATE_DIR
static String	E_DIR_CREATE_FROMFILE
static String	E_DIR_NOTEMPTY
static String	E_DPATH_EXIST
static String	E_FAILED_DISCONNECT
static String	E_FAILED_GETHOME
static String	E_FILE_CHECK_FAILED
static String	E_FILE_EXIST
static String	E_FILE_NOTFOUND
static String	E_FILE_STATUS
static String	E_FS_CLOSED
static String	E_HOST_NULL
static String	E_MAKE_DIR_FORPATH
static String	E_PATH_DIR
static String	E_SAME_DIRECTORY_ONLY
static String	E_SPATH_NOTEXIST
static String	E_USER_NULL
static String	FS_SFTP_CONNECTION_MAX
static String	FS_SFTP_HOST
static String	FS_SFTP_HOST_PORT
static String	FS_SFTP_KEYFILE
static String	FS_SFTP_PASSWORD_PREFIX
static String	FS_SFTP_USER_PREFIX
static org.slf4j.Logger	LOG

Fields inherited from class org.apache.hadoop.fs.FileSystem

DEFAULT_FS, FS_DEFAULT_NAME_KEY, SHUTDOWN_HOOK_PRIORITY, statistics, TRASH_PREFIX, USER_HOME_PREFIX

Constructor Summary

Constructors

Constructor and Description
SFTPFileSystem()

Method Summary

Modifier and Type	Method and Description
FSDataOutputStream	append(Path f, int bufferSize, Progressable progress) Append to an existing file (optional operation).
void	close() Close this FileSystem instance.
FSDataOutputStream	create(Path f, FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress) A stream obtained via this call must be closed before using other APIs of this class or else the invocation will block.
boolean	delete(Path f, boolean recursive) Delete a file.
FileStatus	getFileStatus(Path f) Return a file status object that represents the path.
Path	getHomeDirectory() Return the current user's home directory in this FileSystem.
URI	getUri() Returns a URI which identifies this FileSystem.
Path	getWorkingDirectory() Get the current working directory for the given FileSystem
void	initialize(URI uriInfo, Configuration conf) Initialize a FileSystem.
FileStatus[]	listStatus(Path f) List the statuses of the files/directories in the given path if the path is a directory.
boolean	mkdirs(Path f, FsPermission permission) Make the given file and all non-existent parents into directories.
FSDataInputStream	open(Path f, int bufferSize) Opens an FSDataInputStream at the indicated Path.
boolean	rename(Path src, Path dst) Renames Path src to Path dst.
void	setWorkingDirectory(Path newDir) Set the current working directory for the given FileSystem.

Methods inherited from class org.apache.hadoop.fs.FileSystem

append, append, appendFile, areSymlinksEnabled, cancelDeleteOnExit, canonicalizeUri, checkPath, clearStatistics, closeAll, closeAllForUGI, completeLocalOutput, concat, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyToLocalFile, copyToLocalFile, copyToLocalFile, create, create, create, create, create, create, create, create, create, create, create, create, createDataOutputStreamBuilder, createFile, createMultipartUploader, createNewFile, createNonRecursive, createNonRecursive, createNonRecursive, createPathHandle, createSnapshot, createSnapshot, createSymlink, delete, deleteOnExit, deleteSnapshot, enableSymlinks, exists, fixRelativePart, get, get, get, getAclStatus, getAllStatistics, getAllStoragePolicies, getBlockSize, getCanonicalServiceName, getCanonicalUri, getContentSummary, getDefaultBlockSize, getDefaultBlockSize, getDefaultPort, getDefaultReplication, getDefaultReplication, getDefaultUri, getFileBlockLocations, getFileBlockLocations, getFileChecksum, getFileChecksum, getFileLinkStatus, getFileSystemClass, getFSofPath, getGlobalStorageStatistics, getInitialWorkingDirectory, getLength, getLinkTarget, getLocal, getName, getNamed, getPathHandle, getQuotaUsage, getReplication, getScheme, getServerDefaults, getServerDefaults, getStatistics, getStatistics, getStatus, getStatus, getStoragePolicy, getStorageStatistics, getTrashRoot, getTrashRoots, getUsed, getUsed, getXAttr, getXAttrs, getXAttrs, globStatus, globStatus, hasPathCapability, isDirectory, isFile, listCorruptFileBlocks, listFiles, listLocatedStatus, listLocatedStatus, listStatus, listStatus, listStatus, listStatusIterator, listXAttrs, makeQualified, mkdirs, mkdirs, modifyAclEntries, moveFromLocalFile, moveFromLocalFile, moveToLocalFile, msync, newInstance, newInstance, newInstance, newInstanceLocal, open, open, open, openFile, openFile, openFileWithOptions, openFileWithOptions, primitiveCreate, primitiveMkdir, primitiveMkdir, printStatistics, processDeleteOnExit, removeAcl, removeAclEntries, removeDefaultAcl, removeXAttr, rename, renameSnapshot, resolveLink, resolvePath, satisfyStoragePolicy, setAcl, setDefaultUri, setDefaultUri, setOwner, setPermission, setQuota, setQuotaByStorageType, setReplication, setStoragePolicy, setTimes, setVerifyChecksum, setWriteChecksum, setXAttr, setXAttr, startLocalOutput, supportsSymlinks, truncate, unsetStoragePolicy

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.security.token.DelegationTokenIssuer

addDelegationTokens, getAdditionalTokenIssuers, getDelegationToken

Field Detail

LOG

public static final org.slf4j.Logger LOG

DEFAULT_BUFFER_SIZE

public static final int DEFAULT_BUFFER_SIZE

See Also:

Constant Field Values

DEFAULT_BLOCK_SIZE

public static final int DEFAULT_BLOCK_SIZE

See Also:

Constant Field Values

FS_SFTP_USER_PREFIX

public static final String FS_SFTP_USER_PREFIX

See Also:

Constant Field Values

FS_SFTP_PASSWORD_PREFIX

public static final String FS_SFTP_PASSWORD_PREFIX

See Also:

Constant Field Values

FS_SFTP_HOST

public static final String FS_SFTP_HOST

See Also:

Constant Field Values

FS_SFTP_HOST_PORT

public static final String FS_SFTP_HOST_PORT

See Also:

Constant Field Values

FS_SFTP_KEYFILE

public static final String FS_SFTP_KEYFILE

See Also:

Constant Field Values

FS_SFTP_CONNECTION_MAX

public static final String FS_SFTP_CONNECTION_MAX

See Also:

Constant Field Values

E_SAME_DIRECTORY_ONLY

public static final String E_SAME_DIRECTORY_ONLY

See Also:

Constant Field Values

E_HOST_NULL

public static final String E_HOST_NULL

See Also:

Constant Field Values

E_USER_NULL

public static final String E_USER_NULL

See Also:

Constant Field Values

E_PATH_DIR

public static final String E_PATH_DIR

See Also:

Constant Field Values

E_FILE_STATUS

public static final String E_FILE_STATUS

See Also:

Constant Field Values

E_FILE_NOTFOUND

public static final String E_FILE_NOTFOUND

See Also:

Constant Field Values

E_FILE_EXIST

public static final String E_FILE_EXIST

See Also:

Constant Field Values

E_CREATE_DIR

public static final String E_CREATE_DIR

See Also:

Constant Field Values

E_DIR_CREATE_FROMFILE

public static final String E_DIR_CREATE_FROMFILE

See Also:

Constant Field Values

E_MAKE_DIR_FORPATH

public static final String E_MAKE_DIR_FORPATH

See Also:

Constant Field Values

E_DIR_NOTEMPTY

public static final String E_DIR_NOTEMPTY

See Also:

Constant Field Values

E_FILE_CHECK_FAILED

public static final String E_FILE_CHECK_FAILED

See Also:

Constant Field Values

E_SPATH_NOTEXIST

public static final String E_SPATH_NOTEXIST

See Also:

Constant Field Values

E_DPATH_EXIST

public static final String E_DPATH_EXIST

See Also:

Constant Field Values

E_FAILED_GETHOME

public static final String E_FAILED_GETHOME

See Also:

Constant Field Values

E_FAILED_DISCONNECT

public static final String E_FAILED_DISCONNECT

See Also:

Constant Field Values

E_FS_CLOSED

public static final String E_FS_CLOSED

See Also:

Constant Field Values

Constructor Detail

SFTPFileSystem

public SFTPFileSystem()

Method Detail

initialize

public void initialize(URI uriInfo,
                       Configuration conf)
                throws IOException

Description copied from class: FileSystem

Initialize a FileSystem. Called after the new FileSystem instance is constructed, and before it is ready for use. FileSystem implementations overriding this method MUST forward it to their superclass, though the order in which it is done, and whether to alter the configuration before the invocation are options of the subclass.

Overrides:

initialize in class FileSystem

Parameters:

uriInfo - a URI whose authority section names the host, port, etc. for this FileSystem

conf - the configuration

Throws:

IOException - on any failure to initialize this instance.

getUri

public URI getUri()

Description copied from class: FileSystem

Returns a URI which identifies this FileSystem.

Specified by:

getUri in class FileSystem

Returns:

the URI of this filesystem.

open

public FSDataInputStream open(Path f,
                              int bufferSize)
                       throws IOException

Description copied from class: FileSystem

Opens an FSDataInputStream at the indicated Path.

Specified by:

open in class FileSystem

Parameters:

f - the file name to open

bufferSize - the size of the buffer to be used.

Throws:

IOException - IO failure

create

public FSDataOutputStream create(Path f,
                                 FsPermission permission,
                                 boolean overwrite,
                                 int bufferSize,
                                 short replication,
                                 long blockSize,
                                 Progressable progress)
                          throws IOException

A stream obtained via this call must be closed before using other APIs of this class or else the invocation will block.

Specified by:

create in class FileSystem

Parameters:

f - the file name to open

permission - file permission

overwrite - if a file with this name already exists, then if true, the file will be overwritten, and if false an error will be thrown.

bufferSize - the size of the buffer to be used.

replication - required block replication for the file.

blockSize - block size

progress - the progress reporter

Throws:

IOException - IO failure

See Also:

FileSystem.setPermission(Path, FsPermission)

append

public FSDataOutputStream append(Path f,
                                 int bufferSize,
                                 Progressable progress)
                          throws IOException

Description copied from class: FileSystem

Append to an existing file (optional operation).

Specified by:

append in class FileSystem

Parameters:

f - the existing file to be appended.

bufferSize - the size of the buffer to be used.

progress - for reporting progress if it is not null.

Throws:

IOException - IO failure

rename

public boolean rename(Path src,
                      Path dst)
               throws IOException

Description copied from class: FileSystem

Renames Path src to Path dst.

Specified by:

rename in class FileSystem

Parameters:

src - path to be renamed

dst - new path after rename

Returns:

true if rename is successful

Throws:

IOException - on failure

delete

public boolean delete(Path f,
                      boolean recursive)
               throws IOException

Description copied from class: FileSystem

Delete a file.

Specified by:

delete in class FileSystem

Parameters:

f - the path to delete.

recursive - if path is a directory and set to true, the directory is deleted else throws an exception. In case of a file the recursive can be set to either true or false.

Returns:

true if delete is successful else false.

Throws:

IOException - IO failure

listStatus

public FileStatus[] listStatus(Path f)
                        throws IOException

Description copied from class: FileSystem

List the statuses of the files/directories in the given path if the path is a directory.

Does not guarantee to return the List of files/directories status in a sorted order.

Will not return null. Expect IOException upon access error.

Specified by:

listStatus in class FileSystem

Parameters:

f - given path

Returns:

the statuses of the files/directories in the given patch

Throws:

FileNotFoundException - when the path does not exist

IOException - see specific implementation

setWorkingDirectory

public void setWorkingDirectory(Path newDir)

Description copied from class: FileSystem

Set the current working directory for the given FileSystem. All relative paths will be resolved relative to it.

Specified by:

setWorkingDirectory in class FileSystem

Parameters:

newDir - Path of new working directory

getWorkingDirectory

public Path getWorkingDirectory()

Description copied from class: FileSystem

Get the current working directory for the given FileSystem

Specified by:

getWorkingDirectory in class FileSystem

Returns:

the directory pathname

getHomeDirectory

public Path getHomeDirectory()

Description copied from class: FileSystem

Return the current user's home directory in this FileSystem. The default implementation returns "/user/$USER/".

Overrides:

getHomeDirectory in class FileSystem

mkdirs

public boolean mkdirs(Path f,
                      FsPermission permission)
               throws IOException

Description copied from class: FileSystem

Make the given file and all non-existent parents into directories. Has roughly the semantics of Unix @{code mkdir -p}. Existence of the directory hierarchy is not an error.

Specified by:

mkdirs in class FileSystem

Parameters:

f - path to create

permission - to apply to f

Throws:

IOException - IO failure

getFileStatus

public FileStatus getFileStatus(Path f)
                         throws IOException

Description copied from class: FileSystem

Return a file status object that represents the path.

Specified by:

getFileStatus in class FileSystem

Parameters:

f - The path we want information from

Returns:

a FileStatus object

Throws:

FileNotFoundException - when the path does not exist

IOException - see specific implementation

close

public void close()
           throws IOException

Description copied from class: FileSystem

Close this FileSystem instance. Will release any held locks, delete all files queued for deletion through calls to FileSystem.deleteOnExit(Path), and remove this FS instance from the cache, if cached. After this operation, the outcome of any method call on this FileSystem instance, or any input/output stream created by it is undefined.

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Overrides:

close in class FileSystem

Throws:

IOException - IO failure