Package org.apache.accumulo.core.clientImpl.bulk

Class BulkImport

java.lang.Object

org.apache.accumulo.core.clientImpl.bulk.BulkImport

All Implemented Interfaces:

TableOperations.ImportDestinationArguments, TableOperations.ImportMappingOptions, TableOperations.ImportOptions

public class BulkImport
extends Object
implements TableOperations.ImportDestinationArguments, TableOperations.ImportMappingOptions

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static interface	BulkImport.KeyExtentCache

Field Summary

Fields inherited from interface org.apache.accumulo.core.client.admin.TableOperations.ImportMappingOptions

BULK_LOAD_THREADS_DEFAULT

Constructor Summary

Constructors

Constructor	Description
BulkImport(String directory, ClientContext context)

Method Summary

Modifier and Type	Method	Description
SortedMap<KeyExtent,Bulk.Files>	computeFileToTabletMappings(org.apache.hadoop.fs.FileSystem fs, TableId tableId, Map<String,String> tableProps, org.apache.hadoop.fs.Path dirPath, Executor executor, ClientContext context, int maxTablets)
static Map<KeyExtent,Long>	estimateSizes(AccumuloConfiguration acuConf, org.apache.hadoop.fs.Path mapFile, long fileSize, Collection<KeyExtent> extents, org.apache.hadoop.fs.FileSystem ns, com.google.common.cache.Cache<String,Long> fileLenCache, CryptoService cs)
TableOperations.ImportMappingOptions	executor(Executor service)	Files are examined to determine where to load them.
static List<org.apache.hadoop.fs.FileStatus>	filterInvalid(org.apache.hadoop.fs.FileStatus[] files)
static List<KeyExtent>	findOverlappingTablets(BulkImport.KeyExtentCache extentCache, FileSKVIterator reader)
static List<KeyExtent>	findOverlappingTablets(ClientContext context, BulkImport.KeyExtentCache extentCache, org.apache.hadoop.fs.Path file, org.apache.hadoop.fs.FileSystem fs, com.google.common.cache.Cache<String,Long> fileLenCache, CryptoService cs)
TableOperations.ImportMappingOptions	ignoreEmptyDir(boolean ignore)	Ignores empty bulk import source directory, rather than throwing an IllegalArgumentException.
void	load()	Loads the files into the table.
TableOperations.ImportMappingOptions	plan(LoadPlan plan)	Load files in the directory to the row ranges specified in the plan.
TableOperations.ImportMappingOptions	tableTime(boolean value)	Use table's next timestamp to override all timestamps in imported files.
TableOperations.ImportMappingOptions	threads(int numThreads)	Files are examined to determine where to load them.
TableOperations.ImportMappingOptions	to(String tableName)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BulkImport

public BulkImport(String directory,
                  ClientContext context)

Method Detail

tableTime

public TableOperations.ImportMappingOptions tableTime(boolean value)

Description copied from interface: TableOperations.ImportOptions

Use table's next timestamp to override all timestamps in imported files. The type of timestamp used depends on how the table was created.

Specified by:

tableTime in interface TableOperations.ImportOptions

Parameters:

value - override the time values in the input files, and use the current time for all mutations

See Also:

NewTableConfiguration.setTimeType(TimeType)

ignoreEmptyDir

public TableOperations.ImportMappingOptions ignoreEmptyDir(boolean ignore)

Description copied from interface: TableOperations.ImportOptions

Ignores empty bulk import source directory, rather than throwing an IllegalArgumentException.

Specified by:

ignoreEmptyDir in interface TableOperations.ImportOptions

load

public void load()
          throws TableNotFoundException,
                 IOException,
                 AccumuloException,
                 AccumuloSecurityException

Description copied from interface: TableOperations.ImportOptions

Loads the files into the table.

Specified by:

load in interface TableOperations.ImportOptions

Throws:

TableNotFoundException

IOException

AccumuloException

AccumuloSecurityException

executor

public TableOperations.ImportMappingOptions executor(Executor service)

Description copied from interface: TableOperations.ImportMappingOptions

Files are examined to determine where to load them. This examination is done in the current process using multiple threads. If this method is not called, then the client property bulk.threads is used to create a thread pool. This property defaults to "8C".

Specified by:

executor in interface TableOperations.ImportMappingOptions

Parameters:

service - Use this executor to run file examination task

threads

public TableOperations.ImportMappingOptions threads(int numThreads)

Description copied from interface: TableOperations.ImportMappingOptions

Files are examined to determine where to load them. This examination is done in the current process using multiple threads. If this method is not called, then the client property bulk.threads is used to create a thread pool. This property defaults to "8C".

Specified by:

threads in interface TableOperations.ImportMappingOptions

Parameters:

numThreads - Create a thread pool with this many thread to run file examination task.

plan

public TableOperations.ImportMappingOptions plan(LoadPlan plan)

Description copied from interface: TableOperations.ImportMappingOptions

Load files in the directory to the row ranges specified in the plan. The plan should contain at least one entry for every file in the directory. When this option is specified, the files are never examined so it is possible to send files to the wrong tablet.

Specified by:

plan in interface TableOperations.ImportMappingOptions

to

public TableOperations.ImportMappingOptions to(String tableName)

Specified by:

to in interface TableOperations.ImportDestinationArguments

Parameters:

tableName - Import files to this tableName

estimateSizes

public static Map<KeyExtent,Long> estimateSizes(AccumuloConfiguration acuConf,
                                                      org.apache.hadoop.fs.Path mapFile,
                                                      long fileSize,
                                                      Collection<KeyExtent> extents,
                                                      org.apache.hadoop.fs.FileSystem ns,
                                                      com.google.common.cache.Cache<String,Long> fileLenCache,
                                                      CryptoService cs)
                                               throws IOException

Throws:

IOException

findOverlappingTablets

public static List<KeyExtent> findOverlappingTablets(BulkImport.KeyExtentCache extentCache,
                                                     FileSKVIterator reader)
                                              throws IOException

Throws:

IOException

findOverlappingTablets

public static List<KeyExtent> findOverlappingTablets(ClientContext context,
                                                     BulkImport.KeyExtentCache extentCache,
                                                     org.apache.hadoop.fs.Path file,
                                                     org.apache.hadoop.fs.FileSystem fs,
                                                     com.google.common.cache.Cache<String,Long> fileLenCache,
                                                     CryptoService cs)
                                              throws IOException

Throws:

IOException

filterInvalid

public static List<org.apache.hadoop.fs.FileStatus> filterInvalid(org.apache.hadoop.fs.FileStatus[] files)

computeFileToTabletMappings

public SortedMap<KeyExtent,Bulk.Files> computeFileToTabletMappings(org.apache.hadoop.fs.FileSystem fs,
                                                                         TableId tableId,
                                                                         Map<String,String> tableProps,
                                                                         org.apache.hadoop.fs.Path dirPath,
                                                                         Executor executor,
                                                                         ClientContext context,
                                                                         int maxTablets)
                                                                  throws IOException,
                                                                         AccumuloException,
                                                                         AccumuloSecurityException

Throws:

IOException

AccumuloException

AccumuloSecurityException