|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Object org.apache.accumulo.core.client.mapred.AccumuloOutputFormat
public class AccumuloOutputFormat
This class allows MapReduce jobs to use Accumulo as the sink for data. This OutputFormat
accepts keys and values of type Text
(for a table
name) and Mutation
from the Map and Reduce functions.
The user must specify the following via static configurator methods:
setConnectorInfo(JobConf, String, AuthenticationToken)
setConnectorInfo(JobConf, String, String)
setZooKeeperInstance(JobConf, ClientConfiguration)
OR setMockInstance(JobConf, String)
Nested Class Summary | |
---|---|
protected static class |
AccumuloOutputFormat.AccumuloRecordWriter
A base class to be used to create RecordWriter instances that write to Accumulo. |
Field Summary | |
---|---|
protected static org.apache.log4j.Logger |
log
|
Constructor Summary | |
---|---|
AccumuloOutputFormat()
|
Method Summary | |
---|---|
protected static Boolean |
canCreateTables(org.apache.hadoop.mapred.JobConf job)
Determines whether tables are permitted to be created as needed. |
void |
checkOutputSpecs(org.apache.hadoop.fs.FileSystem ignored,
org.apache.hadoop.mapred.JobConf job)
|
protected static AuthenticationToken |
getAuthenticationToken(org.apache.hadoop.mapred.JobConf job)
Gets the authenticated token from either the specified token file or directly from the configuration, whichever was used when the job was configured. |
protected static BatchWriterConfig |
getBatchWriterOptions(org.apache.hadoop.mapred.JobConf job)
Gets the BatchWriterConfig settings. |
protected static String |
getDefaultTableName(org.apache.hadoop.mapred.JobConf job)
Gets the default table name from the configuration. |
protected static Instance |
getInstance(org.apache.hadoop.mapred.JobConf job)
Initializes an Accumulo Instance based on the configuration. |
protected static org.apache.log4j.Level |
getLogLevel(org.apache.hadoop.mapred.JobConf job)
Gets the log level from this configuration. |
protected static String |
getPrincipal(org.apache.hadoop.mapred.JobConf job)
Gets the principal from the configuration. |
org.apache.hadoop.mapred.RecordWriter<org.apache.hadoop.io.Text,Mutation> |
getRecordWriter(org.apache.hadoop.fs.FileSystem ignored,
org.apache.hadoop.mapred.JobConf job,
String name,
org.apache.hadoop.util.Progressable progress)
|
protected static Boolean |
getSimulationMode(org.apache.hadoop.mapred.JobConf job)
Determines whether this feature is enabled. |
protected static byte[] |
getToken(org.apache.hadoop.mapred.JobConf job)
Deprecated. since 1.6.0; Use getAuthenticationToken(JobConf) instead. |
protected static String |
getTokenClass(org.apache.hadoop.mapred.JobConf job)
Deprecated. since 1.6.0; Use getAuthenticationToken(JobConf) instead. |
protected static Boolean |
isConnectorInfoSet(org.apache.hadoop.mapred.JobConf job)
Determines if the connector has been configured. |
static void |
setBatchWriterOptions(org.apache.hadoop.mapred.JobConf job,
BatchWriterConfig bwConfig)
Sets the configuration for for the job's BatchWriter instances. |
static void |
setConnectorInfo(org.apache.hadoop.mapred.JobConf job,
String principal,
AuthenticationToken token)
Sets the connector information needed to communicate with Accumulo in this job. |
static void |
setConnectorInfo(org.apache.hadoop.mapred.JobConf job,
String principal,
String tokenFile)
Sets the connector information needed to communicate with Accumulo in this job. |
static void |
setCreateTables(org.apache.hadoop.mapred.JobConf job,
boolean enableFeature)
Sets the directive to create new tables, as necessary. |
static void |
setDefaultTableName(org.apache.hadoop.mapred.JobConf job,
String tableName)
Sets the default table name to use if one emits a null in place of a table name for a given mutation. |
static void |
setLogLevel(org.apache.hadoop.mapred.JobConf job,
org.apache.log4j.Level level)
Sets the log level for this job. |
static void |
setMockInstance(org.apache.hadoop.mapred.JobConf job,
String instanceName)
Configures a MockInstance for this job. |
static void |
setSimulationMode(org.apache.hadoop.mapred.JobConf job,
boolean enableFeature)
Sets the directive to use simulation mode for this job. |
static void |
setZooKeeperInstance(org.apache.hadoop.mapred.JobConf job,
ClientConfiguration clientConfig)
Configures a ZooKeeperInstance for this job. |
static void |
setZooKeeperInstance(org.apache.hadoop.mapred.JobConf job,
String instanceName,
String zooKeepers)
Deprecated. since 1.6.0; Use setZooKeeperInstance(JobConf, ClientConfiguration) instead. |
Methods inherited from class java.lang.Object |
---|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
Field Detail |
---|
protected static final org.apache.log4j.Logger log
Constructor Detail |
---|
public AccumuloOutputFormat()
Method Detail |
---|
public static void setConnectorInfo(org.apache.hadoop.mapred.JobConf job, String principal, AuthenticationToken token) throws AccumuloSecurityException
WARNING: The serialized token is stored in the configuration and shared with all MapReduce tasks. It is BASE64 encoded to provide a charset safe conversion to a string, and is not intended to be secure.
job
- the Hadoop job instance to be configuredprincipal
- a valid Accumulo user name (user must have Table.CREATE permission if setCreateTables(JobConf, boolean)
is set to true)token
- the user's password
AccumuloSecurityException
public static void setConnectorInfo(org.apache.hadoop.mapred.JobConf job, String principal, String tokenFile) throws AccumuloSecurityException
Stores the password in a file in HDFS and pulls that into the Distributed Cache in an attempt to be more secure than storing it in the Configuration.
job
- the Hadoop job instance to be configuredprincipal
- a valid Accumulo user name (user must have Table.CREATE permission if setCreateTables(JobConf, boolean)
is set to true)tokenFile
- the path to the password file
AccumuloSecurityException
protected static Boolean isConnectorInfoSet(org.apache.hadoop.mapred.JobConf job)
job
- the Hadoop context for the configured job
setConnectorInfo(JobConf, String, AuthenticationToken)
protected static String getPrincipal(org.apache.hadoop.mapred.JobConf job)
job
- the Hadoop context for the configured job
setConnectorInfo(JobConf, String, AuthenticationToken)
@Deprecated protected static String getTokenClass(org.apache.hadoop.mapred.JobConf job)
getAuthenticationToken(JobConf)
instead.
@Deprecated protected static byte[] getToken(org.apache.hadoop.mapred.JobConf job)
getAuthenticationToken(JobConf)
instead.
protected static AuthenticationToken getAuthenticationToken(org.apache.hadoop.mapred.JobConf job)
job
- the Hadoop job instance to be configured
setConnectorInfo(JobConf, String, AuthenticationToken)
,
setConnectorInfo(JobConf, String, String)
@Deprecated public static void setZooKeeperInstance(org.apache.hadoop.mapred.JobConf job, String instanceName, String zooKeepers)
setZooKeeperInstance(JobConf, ClientConfiguration)
instead.
ZooKeeperInstance
for this job.
job
- the Hadoop job instance to be configuredinstanceName
- the Accumulo instance namezooKeepers
- a comma-separated list of zookeeper serverspublic static void setZooKeeperInstance(org.apache.hadoop.mapred.JobConf job, ClientConfiguration clientConfig)
ZooKeeperInstance
for this job.
job
- the Hadoop job instance to be configuredclientConfig
- client configuration for specifying connection timeouts, SSL connection options, etc.public static void setMockInstance(org.apache.hadoop.mapred.JobConf job, String instanceName)
MockInstance
for this job.
job
- the Hadoop job instance to be configuredinstanceName
- the Accumulo instance nameprotected static Instance getInstance(org.apache.hadoop.mapred.JobConf job)
Instance
based on the configuration.
job
- the Hadoop context for the configured job
setZooKeeperInstance(JobConf, String, String)
,
setMockInstance(JobConf, String)
public static void setLogLevel(org.apache.hadoop.mapred.JobConf job, org.apache.log4j.Level level)
job
- the Hadoop job instance to be configuredlevel
- the logging levelprotected static org.apache.log4j.Level getLogLevel(org.apache.hadoop.mapred.JobConf job)
job
- the Hadoop context for the configured job
setLogLevel(JobConf, Level)
public static void setDefaultTableName(org.apache.hadoop.mapred.JobConf job, String tableName)
job
- the Hadoop job instance to be configuredtableName
- the table to use when the tablename is null in the write callprotected static String getDefaultTableName(org.apache.hadoop.mapred.JobConf job)
job
- the Hadoop context for the configured job
setDefaultTableName(JobConf, String)
public static void setBatchWriterOptions(org.apache.hadoop.mapred.JobConf job, BatchWriterConfig bwConfig)
BatchWriter
instances. If not set, a new BatchWriterConfig
, with sensible built-in defaults is
used. Setting the configuration multiple times overwrites any previous configuration.
job
- the Hadoop job instance to be configuredbwConfig
- the configuration for the BatchWriter
protected static BatchWriterConfig getBatchWriterOptions(org.apache.hadoop.mapred.JobConf job)
BatchWriterConfig
settings.
job
- the Hadoop context for the configured job
setBatchWriterOptions(JobConf, BatchWriterConfig)
public static void setCreateTables(org.apache.hadoop.mapred.JobConf job, boolean enableFeature)
By default, this feature is disabled.
job
- the Hadoop job instance to be configuredenableFeature
- the feature is enabled if true, disabled otherwiseprotected static Boolean canCreateTables(org.apache.hadoop.mapred.JobConf job)
job
- the Hadoop context for the configured job
setCreateTables(JobConf, boolean)
public static void setSimulationMode(org.apache.hadoop.mapred.JobConf job, boolean enableFeature)
By default, this feature is disabled.
job
- the Hadoop job instance to be configuredenableFeature
- the feature is enabled if true, disabled otherwiseprotected static Boolean getSimulationMode(org.apache.hadoop.mapred.JobConf job)
job
- the Hadoop context for the configured job
setSimulationMode(JobConf, boolean)
public void checkOutputSpecs(org.apache.hadoop.fs.FileSystem ignored, org.apache.hadoop.mapred.JobConf job) throws IOException
checkOutputSpecs
in interface org.apache.hadoop.mapred.OutputFormat<org.apache.hadoop.io.Text,Mutation>
IOException
public org.apache.hadoop.mapred.RecordWriter<org.apache.hadoop.io.Text,Mutation> getRecordWriter(org.apache.hadoop.fs.FileSystem ignored, org.apache.hadoop.mapred.JobConf job, String name, org.apache.hadoop.util.Progressable progress) throws IOException
getRecordWriter
in interface org.apache.hadoop.mapred.OutputFormat<org.apache.hadoop.io.Text,Mutation>
IOException
|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |