Package org.apache.accumulo.server.fs

Class VolumeManagerImpl

java.lang.Object

org.apache.accumulo.server.fs.VolumeManagerImpl

All Implemented Interfaces:

AutoCloseable, VolumeManager

public class VolumeManagerImpl
extends Object
implements VolumeManager

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.accumulo.server.fs.VolumeManager

VolumeManager.FileType

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	VolumeManagerImpl(Map<String,Volume> volumes, AccumuloConfiguration conf, org.apache.hadoop.conf.Configuration hadoopConf)

Method Summary

Modifier and Type	Method	Description
void	bulkRename(Map<org.apache.hadoop.fs.Path,org.apache.hadoop.fs.Path> oldToNewPathMap, int poolSize, String poolName, String transactionId)	Rename lots of files at once in a thread pool and return once all the threads have completed.
boolean	canSyncAndFlush(org.apache.hadoop.fs.Path path)
Set<String>	choosable(VolumeChooserEnvironment env, Set<String> options)
String	choose(VolumeChooserEnvironment env, Set<String> options)
void	close()
org.apache.hadoop.fs.FSDataOutputStream	create(org.apache.hadoop.fs.Path path)
org.apache.hadoop.fs.FSDataOutputStream	create(org.apache.hadoop.fs.Path path, boolean overwrite, int bufferSize, short replication, long blockSize)
boolean	createNewFile(org.apache.hadoop.fs.Path path)
org.apache.hadoop.fs.FSDataOutputStream	createSyncable(org.apache.hadoop.fs.Path logPath, int bufferSize, short replication, long blockSize)
boolean	delete(org.apache.hadoop.fs.Path path)
boolean	deleteRecursively(org.apache.hadoop.fs.Path path)
protected void	ensureSyncIsEnabled()
boolean	exists(org.apache.hadoop.fs.Path path)
protected static Stream<Map.Entry<String,String>>	findVolumeOverridesMissingVolume(AccumuloConfiguration conf, Set<String> definedVolumes)
static VolumeManager	get(AccumuloConfiguration conf, org.apache.hadoop.conf.Configuration hadoopConf)
short	getDefaultReplication(org.apache.hadoop.fs.Path path)
org.apache.hadoop.fs.FileStatus	getFileStatus(org.apache.hadoop.fs.Path path)
org.apache.hadoop.fs.FileSystem	getFileSystemByPath(org.apache.hadoop.fs.Path path)
static VolumeManager	getLocalForTesting(String localBasePath)
protected static org.apache.hadoop.conf.Configuration	getVolumeManagerConfiguration(AccumuloConfiguration conf, org.apache.hadoop.conf.Configuration hadoopConf, String filesystemURI)	The Hadoop Configuration object does not currently allow for duplicate properties to be set in a single Configuration for different FileSystem URIs.
Collection<Volume>	getVolumes()	Fetch the configured instance Volumes
org.apache.hadoop.fs.FileStatus[]	globStatus(org.apache.hadoop.fs.Path pathPattern)
boolean	isReady()
org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus>	listFiles(org.apache.hadoop.fs.Path path, boolean recursive)
org.apache.hadoop.fs.FileStatus[]	listStatus(org.apache.hadoop.fs.Path path)
org.apache.hadoop.fs.Path	matchingFileSystem(org.apache.hadoop.fs.Path source, Set<String> options)
boolean	mkdirs(org.apache.hadoop.fs.Path path)
boolean	mkdirs(org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.permission.FsPermission permission)
boolean	moveToTrash(org.apache.hadoop.fs.Path path)
org.apache.hadoop.fs.FSDataInputStream	open(org.apache.hadoop.fs.Path path)
org.apache.hadoop.fs.FSDataOutputStream	overwrite(org.apache.hadoop.fs.Path path)
boolean	rename(org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.Path newPath)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.accumulo.server.fs.VolumeManager

getFirst

Constructor Details

VolumeManagerImpl

protected VolumeManagerImpl(Map<String,Volume> volumes,
 AccumuloConfiguration conf,
 org.apache.hadoop.conf.Configuration hadoopConf)

Method Details

getLocalForTesting

public static VolumeManager getLocalForTesting(String localBasePath)
                                        throws IOException

Throws:

IOException

close

public void close()
           throws IOException

Specified by:

close in interface AutoCloseable

Specified by:

close in interface VolumeManager

Throws:

IOException

create

public org.apache.hadoop.fs.FSDataOutputStream create(org.apache.hadoop.fs.Path path)
                                               throws IOException

Specified by:

create in interface VolumeManager

Throws:

IOException

overwrite

public org.apache.hadoop.fs.FSDataOutputStream overwrite(org.apache.hadoop.fs.Path path)
                                                  throws IOException

Specified by:

overwrite in interface VolumeManager

Throws:

IOException

create

public org.apache.hadoop.fs.FSDataOutputStream create(org.apache.hadoop.fs.Path path,
 boolean overwrite,
 int bufferSize,
 short replication,
 long blockSize)
                                               throws IOException

Specified by:

create in interface VolumeManager

Throws:

IOException

createNewFile

public boolean createNewFile(org.apache.hadoop.fs.Path path)
                      throws IOException

Specified by:

createNewFile in interface VolumeManager

Throws:

IOException

createSyncable

public org.apache.hadoop.fs.FSDataOutputStream createSyncable(org.apache.hadoop.fs.Path logPath,
 int bufferSize,
 short replication,
 long blockSize)
                                                       throws IOException

Specified by:

createSyncable in interface VolumeManager

Throws:

IOException

delete

public boolean delete(org.apache.hadoop.fs.Path path)
               throws IOException

Specified by:

delete in interface VolumeManager

Throws:

IOException

deleteRecursively

public boolean deleteRecursively(org.apache.hadoop.fs.Path path)
                          throws IOException

Specified by:

deleteRecursively in interface VolumeManager

Throws:

IOException

ensureSyncIsEnabled

protected void ensureSyncIsEnabled()

exists

public boolean exists(org.apache.hadoop.fs.Path path)
               throws IOException

Specified by:

exists in interface VolumeManager

Throws:

IOException

getFileStatus

public org.apache.hadoop.fs.FileStatus getFileStatus(org.apache.hadoop.fs.Path path)
                                              throws IOException

Specified by:

getFileStatus in interface VolumeManager

Throws:

IOException

getFileSystemByPath

public org.apache.hadoop.fs.FileSystem getFileSystemByPath(org.apache.hadoop.fs.Path path)

Specified by:

getFileSystemByPath in interface VolumeManager

listFiles

public org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> listFiles(org.apache.hadoop.fs.Path path,
 boolean recursive)
                                                                                      throws IOException

Specified by:

listFiles in interface VolumeManager

Throws:

IOException

listStatus

public org.apache.hadoop.fs.FileStatus[] listStatus(org.apache.hadoop.fs.Path path)
                                             throws IOException

Specified by:

listStatus in interface VolumeManager

Throws:

IOException

mkdirs

public boolean mkdirs(org.apache.hadoop.fs.Path path)
               throws IOException

Specified by:

mkdirs in interface VolumeManager

Throws:

IOException

mkdirs

public boolean mkdirs(org.apache.hadoop.fs.Path path,
 org.apache.hadoop.fs.permission.FsPermission permission)
               throws IOException

Specified by:

mkdirs in interface VolumeManager

Throws:

IOException

open

public org.apache.hadoop.fs.FSDataInputStream open(org.apache.hadoop.fs.Path path)
                                            throws IOException

Specified by:

open in interface VolumeManager

Throws:

IOException

rename

public boolean rename(org.apache.hadoop.fs.Path path,
 org.apache.hadoop.fs.Path newPath)
               throws IOException

Specified by:

rename in interface VolumeManager

Throws:

IOException

bulkRename

public void bulkRename(Map<org.apache.hadoop.fs.Path,org.apache.hadoop.fs.Path> oldToNewPathMap,
 int poolSize,
 String poolName,
 String transactionId)
                throws IOException

Description copied from interface: VolumeManager

Rename lots of files at once in a thread pool and return once all the threads have completed. This operation should be idempotent to allow calling multiple times in the case of a partial completion.

Specified by:

bulkRename in interface VolumeManager

Throws:

IOException

moveToTrash

public boolean moveToTrash(org.apache.hadoop.fs.Path path)
                    throws IOException

Specified by:

moveToTrash in interface VolumeManager

Throws:

IOException

getDefaultReplication

public short getDefaultReplication(org.apache.hadoop.fs.Path path)

Specified by:

getDefaultReplication in interface VolumeManager

getVolumeManagerConfiguration

protected static org.apache.hadoop.conf.Configuration getVolumeManagerConfiguration(AccumuloConfiguration conf,
 org.apache.hadoop.conf.Configuration hadoopConf,
 String filesystemURI)

The Hadoop Configuration object does not currently allow for duplicate properties to be set in a single Configuration for different FileSystem URIs. Here we will look for properties in the Accumulo configuration of the form:

 instance.volume.config.<volume-uri>.<hdfs-property>
 

We will use these properties to return a new Configuration object that can be used with the FileSystem URI to override properties in the original Configuration. If these properties are not set for a volume, then the original Configuration is returned. If they are set, a new Configuration is created with the overridden properties set. In either case, the returned Configuration is cached, to avoid unnecessary recomputation. This works because these override properties are instance properties and cannot change while the system is running.

Parameters:

conf - AccumuloConfiguration object

hadoopConf - Hadoop Configuration object

filesystemURI - Volume Filesystem URI

Returns:

Hadoop Configuration with custom overrides for this FileSystem

findVolumeOverridesMissingVolume

protected static Stream<Map.Entry<String,String>> findVolumeOverridesMissingVolume(AccumuloConfiguration conf,
 Set<String> definedVolumes)

get

public static VolumeManager get(AccumuloConfiguration conf,
 org.apache.hadoop.conf.Configuration hadoopConf)
                         throws IOException

Throws:

IOException

isReady

public boolean isReady()
                throws IOException

Specified by:

isReady in interface VolumeManager

Throws:

IOException

globStatus

public org.apache.hadoop.fs.FileStatus[] globStatus(org.apache.hadoop.fs.Path pathPattern)
                                             throws IOException

Specified by:

globStatus in interface VolumeManager

Throws:

IOException

matchingFileSystem

public org.apache.hadoop.fs.Path matchingFileSystem(org.apache.hadoop.fs.Path source,
 Set<String> options)

Specified by:

matchingFileSystem in interface VolumeManager

choose

public String choose(VolumeChooserEnvironment env,
 Set<String> options)

Specified by:

choose in interface VolumeManager

choosable

public Set<String> choosable(VolumeChooserEnvironment env,
 Set<String> options)

Specified by:

choosable in interface VolumeManager

canSyncAndFlush

public boolean canSyncAndFlush(org.apache.hadoop.fs.Path path)

Specified by:

canSyncAndFlush in interface VolumeManager

getVolumes

public Collection<Volume> getVolumes()

Description copied from interface: VolumeManager

Fetch the configured instance Volumes

Specified by:

getVolumes in interface VolumeManager