@InterfaceAudience.Private @InterfaceStability.Unstable public class Chain extends Object
ChainMapper
and the ChainReducer
classes.Modifier and Type | Field and Description |
---|---|
protected static String |
CHAIN_MAPPER |
protected static String |
CHAIN_MAPPER_CLASS |
protected static String |
CHAIN_MAPPER_CONFIG |
protected static String |
CHAIN_MAPPER_SIZE |
protected static String |
CHAIN_REDUCER |
protected static String |
CHAIN_REDUCER_CLASS |
protected static String |
CHAIN_REDUCER_CONFIG |
protected boolean |
isMap |
protected static String |
MAPPER_INPUT_KEY_CLASS |
protected static String |
MAPPER_INPUT_VALUE_CLASS |
protected static String |
MAPPER_OUTPUT_KEY_CLASS |
protected static String |
MAPPER_OUTPUT_VALUE_CLASS |
protected static String |
REDUCER_INPUT_KEY_CLASS |
protected static String |
REDUCER_INPUT_VALUE_CLASS |
protected static String |
REDUCER_OUTPUT_KEY_CLASS |
protected static String |
REDUCER_OUTPUT_VALUE_CLASS |
Modifier | Constructor and Description |
---|---|
protected |
Chain(boolean isMap)
Creates a Chain instance configured for a Mapper or a Reducer.
|
Modifier and Type | Method and Description |
---|---|
protected static void |
addMapper(boolean isMap,
Job job,
Class<? extends Mapper> klass,
Class<?> inputKeyClass,
Class<?> inputValueClass,
Class<?> outputKeyClass,
Class<?> outputValueClass,
org.apache.hadoop.conf.Configuration mapperConf)
Adds a Mapper class to the chain job.
|
protected static void |
checkReducerAlreadySet(boolean isMap,
org.apache.hadoop.conf.Configuration jobConf,
String prefix,
boolean shouldSet) |
protected static org.apache.hadoop.conf.Configuration |
getChainElementConf(org.apache.hadoop.conf.Configuration jobConf,
String confKey)
Creates a
Configuration for the Map or Reduce in the chain. |
protected static int |
getIndex(org.apache.hadoop.conf.Configuration conf,
String prefix) |
protected static String |
getPrefix(boolean isMap)
Returns the prefix to use for the configuration of the chain depending if
it is for a Mapper or a Reducer.
|
protected static void |
setMapperConf(boolean isMap,
org.apache.hadoop.conf.Configuration jobConf,
Class<?> inputKeyClass,
Class<?> inputValueClass,
Class<?> outputKeyClass,
Class<?> outputValueClass,
org.apache.hadoop.conf.Configuration mapperConf,
int index,
String prefix) |
protected static void |
setReducer(Job job,
Class<? extends Reducer> klass,
Class<?> inputKeyClass,
Class<?> inputValueClass,
Class<?> outputKeyClass,
Class<?> outputValueClass,
org.apache.hadoop.conf.Configuration reducerConf)
Sets the Reducer class to the chain job.
|
protected static void |
setReducerConf(org.apache.hadoop.conf.Configuration jobConf,
Class<?> inputKeyClass,
Class<?> inputValueClass,
Class<?> outputKeyClass,
Class<?> outputValueClass,
org.apache.hadoop.conf.Configuration reducerConf,
String prefix) |
protected static void |
validateKeyValueTypes(boolean isMap,
org.apache.hadoop.conf.Configuration jobConf,
Class<?> inputKeyClass,
Class<?> inputValueClass,
Class<?> outputKeyClass,
Class<?> outputValueClass,
int index,
String prefix) |
protected static final String CHAIN_MAPPER
protected static final String CHAIN_REDUCER
protected static final String CHAIN_MAPPER_SIZE
protected static final String CHAIN_MAPPER_CLASS
protected static final String CHAIN_MAPPER_CONFIG
protected static final String CHAIN_REDUCER_CLASS
protected static final String CHAIN_REDUCER_CONFIG
protected static final String MAPPER_INPUT_KEY_CLASS
protected static final String MAPPER_INPUT_VALUE_CLASS
protected static final String MAPPER_OUTPUT_KEY_CLASS
protected static final String MAPPER_OUTPUT_VALUE_CLASS
protected static final String REDUCER_INPUT_KEY_CLASS
protected static final String REDUCER_INPUT_VALUE_CLASS
protected static final String REDUCER_OUTPUT_KEY_CLASS
protected static final String REDUCER_OUTPUT_VALUE_CLASS
protected boolean isMap
protected Chain(boolean isMap)
isMap
- TRUE indicates the chain is for a Mapper, FALSE that is for a
Reducer.protected static String getPrefix(boolean isMap)
isMap
- TRUE for Mapper, FALSE for Reducer.protected static int getIndex(org.apache.hadoop.conf.Configuration conf, String prefix)
protected static org.apache.hadoop.conf.Configuration getChainElementConf(org.apache.hadoop.conf.Configuration jobConf, String confKey)
Configuration
for the Map or Reduce in the chain.
It creates a new Configuration using the chain job's Configuration as base and adds to it the configuration properties for the chain element. The keys of the chain element Configuration have precedence over the given Configuration.
jobConf
- the chain job's Configuration.confKey
- the key for chain element configuration serialized in the chain
job's Configuration.protected static void addMapper(boolean isMap, Job job, Class<? extends Mapper> klass, Class<?> inputKeyClass, Class<?> inputValueClass, Class<?> outputKeyClass, Class<?> outputValueClass, org.apache.hadoop.conf.Configuration mapperConf)
The configuration properties of the chain job have precedence over the configuration properties of the Mapper.
isMap
- indicates if the Chain is for a Mapper or for a Reducer.job
- chain job.klass
- the Mapper class to add.inputKeyClass
- mapper input key class.inputValueClass
- mapper input value class.outputKeyClass
- mapper output key class.outputValueClass
- mapper output value class.mapperConf
- a configuration for the Mapper class. It is recommended to use a
Configuration without default values using the
Configuration(boolean loadDefaults)
constructor with
FALSE.protected static void checkReducerAlreadySet(boolean isMap, org.apache.hadoop.conf.Configuration jobConf, String prefix, boolean shouldSet)
protected static void validateKeyValueTypes(boolean isMap, org.apache.hadoop.conf.Configuration jobConf, Class<?> inputKeyClass, Class<?> inputValueClass, Class<?> outputKeyClass, Class<?> outputValueClass, int index, String prefix)
protected static void setMapperConf(boolean isMap, org.apache.hadoop.conf.Configuration jobConf, Class<?> inputKeyClass, Class<?> inputValueClass, Class<?> outputKeyClass, Class<?> outputValueClass, org.apache.hadoop.conf.Configuration mapperConf, int index, String prefix)
protected static void setReducer(Job job, Class<? extends Reducer> klass, Class<?> inputKeyClass, Class<?> inputValueClass, Class<?> outputKeyClass, Class<?> outputValueClass, org.apache.hadoop.conf.Configuration reducerConf)
The configuration properties of the chain job have precedence over the configuration properties of the Reducer.
job
- the chain job.klass
- the Reducer class to add.inputKeyClass
- reducer input key class.inputValueClass
- reducer input value class.outputKeyClass
- reducer output key class.outputValueClass
- reducer output value class.reducerConf
- a configuration for the Reducer class. It is recommended to use a
Configuration without default values using the
Configuration(boolean loadDefaults)
constructor with
FALSE.Copyright © 2020 Apache Software Foundation. All rights reserved.