org.apache.hadoop.mapreduce.lib.chain

Class Chain

java.lang.Object

org.apache.hadoop.mapreduce.lib.chain.Chain

@InterfaceAudience.Private
@InterfaceStability.Unstable
public class Chain
extends Object

The Chain class provides all the common functionality for the ChainMapper and the ChainReducer classes.

Field Summary

Fields

Modifier and Type	Field and Description
protected static String	CHAIN_MAPPER
protected static String	CHAIN_MAPPER_CLASS
protected static String	CHAIN_MAPPER_CONFIG
protected static String	CHAIN_MAPPER_SIZE
protected static String	CHAIN_REDUCER
protected static String	CHAIN_REDUCER_CLASS
protected static String	CHAIN_REDUCER_CONFIG
protected boolean	isMap
protected static String	MAPPER_INPUT_KEY_CLASS
protected static String	MAPPER_INPUT_VALUE_CLASS
protected static String	MAPPER_OUTPUT_KEY_CLASS
protected static String	MAPPER_OUTPUT_VALUE_CLASS
protected static String	REDUCER_INPUT_KEY_CLASS
protected static String	REDUCER_INPUT_VALUE_CLASS
protected static String	REDUCER_OUTPUT_KEY_CLASS
protected static String	REDUCER_OUTPUT_VALUE_CLASS

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	Chain(boolean isMap) Creates a Chain instance configured for a Mapper or a Reducer.

Method Summary

Methods

Modifier and Type	Method and Description
protected static void	addMapper(boolean isMap, Job job, Class<? extends Mapper> klass, Class<?> inputKeyClass, Class<?> inputValueClass, Class<?> outputKeyClass, Class<?> outputValueClass, org.apache.hadoop.conf.Configuration mapperConf) Adds a Mapper class to the chain job.
protected static void	checkReducerAlreadySet(boolean isMap, org.apache.hadoop.conf.Configuration jobConf, String prefix, boolean shouldSet)
protected static org.apache.hadoop.conf.Configuration	getChainElementConf(org.apache.hadoop.conf.Configuration jobConf, String confKey) Creates a Configuration for the Map or Reduce in the chain.
protected static int	getIndex(org.apache.hadoop.conf.Configuration conf, String prefix)
protected static String	getPrefix(boolean isMap) Returns the prefix to use for the configuration of the chain depending if it is for a Mapper or a Reducer.
protected static void	setMapperConf(boolean isMap, org.apache.hadoop.conf.Configuration jobConf, Class<?> inputKeyClass, Class<?> inputValueClass, Class<?> outputKeyClass, Class<?> outputValueClass, org.apache.hadoop.conf.Configuration mapperConf, int index, String prefix)
protected static void	setReducer(Job job, Class<? extends Reducer> klass, Class<?> inputKeyClass, Class<?> inputValueClass, Class<?> outputKeyClass, Class<?> outputValueClass, org.apache.hadoop.conf.Configuration reducerConf) Sets the Reducer class to the chain job.
protected static void	setReducerConf(org.apache.hadoop.conf.Configuration jobConf, Class<?> inputKeyClass, Class<?> inputValueClass, Class<?> outputKeyClass, Class<?> outputValueClass, org.apache.hadoop.conf.Configuration reducerConf, String prefix)
protected static void	validateKeyValueTypes(boolean isMap, org.apache.hadoop.conf.Configuration jobConf, Class<?> inputKeyClass, Class<?> inputValueClass, Class<?> outputKeyClass, Class<?> outputValueClass, int index, String prefix)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

CHAIN_MAPPER

protected static final String CHAIN_MAPPER

See Also:

Constant Field Values

CHAIN_REDUCER

protected static final String CHAIN_REDUCER

See Also:

Constant Field Values

CHAIN_MAPPER_SIZE

protected static final String CHAIN_MAPPER_SIZE

See Also:

Constant Field Values

CHAIN_MAPPER_CLASS

protected static final String CHAIN_MAPPER_CLASS

See Also:

Constant Field Values

CHAIN_MAPPER_CONFIG

protected static final String CHAIN_MAPPER_CONFIG

See Also:

Constant Field Values

CHAIN_REDUCER_CLASS

protected static final String CHAIN_REDUCER_CLASS

See Also:

Constant Field Values

CHAIN_REDUCER_CONFIG

protected static final String CHAIN_REDUCER_CONFIG

See Also:

Constant Field Values

MAPPER_INPUT_KEY_CLASS

protected static final String MAPPER_INPUT_KEY_CLASS

See Also:

Constant Field Values

MAPPER_INPUT_VALUE_CLASS

protected static final String MAPPER_INPUT_VALUE_CLASS

See Also:

Constant Field Values

MAPPER_OUTPUT_KEY_CLASS

protected static final String MAPPER_OUTPUT_KEY_CLASS

See Also:

Constant Field Values

MAPPER_OUTPUT_VALUE_CLASS

protected static final String MAPPER_OUTPUT_VALUE_CLASS

See Also:

Constant Field Values

REDUCER_INPUT_KEY_CLASS

protected static final String REDUCER_INPUT_KEY_CLASS

See Also:

Constant Field Values

REDUCER_INPUT_VALUE_CLASS

protected static final String REDUCER_INPUT_VALUE_CLASS

See Also:

Constant Field Values

REDUCER_OUTPUT_KEY_CLASS

protected static final String REDUCER_OUTPUT_KEY_CLASS

See Also:

Constant Field Values

REDUCER_OUTPUT_VALUE_CLASS

protected static final String REDUCER_OUTPUT_VALUE_CLASS

See Also:

Constant Field Values

isMap

protected boolean isMap

Constructor Detail

Chain

protected Chain(boolean isMap)

Creates a Chain instance configured for a Mapper or a Reducer.

Parameters:

isMap - TRUE indicates the chain is for a Mapper, FALSE that is for a Reducer.

Method Detail

getPrefix

protected static String getPrefix(boolean isMap)

Returns the prefix to use for the configuration of the chain depending if it is for a Mapper or a Reducer.

Parameters:

isMap - TRUE for Mapper, FALSE for Reducer.

Returns:

the prefix to use.

getIndex

protected static int getIndex(org.apache.hadoop.conf.Configuration conf,
           String prefix)

getChainElementConf

protected static org.apache.hadoop.conf.Configuration getChainElementConf(org.apache.hadoop.conf.Configuration jobConf,
                                                       String confKey)

Creates a Configuration for the Map or Reduce in the chain.

It creates a new Configuration using the chain job's Configuration as base and adds to it the configuration properties for the chain element. The keys of the chain element Configuration have precedence over the given Configuration.

Parameters:

jobConf - the chain job's Configuration.

confKey - the key for chain element configuration serialized in the chain job's Configuration.

Returns:

a new Configuration aggregating the chain job's Configuration with the chain element configuration properties.

addMapper

protected static void addMapper(boolean isMap,
             Job job,
             Class<? extends Mapper> klass,
             Class<?> inputKeyClass,
             Class<?> inputValueClass,
             Class<?> outputKeyClass,
             Class<?> outputValueClass,
             org.apache.hadoop.conf.Configuration mapperConf)

Adds a Mapper class to the chain job.

The configuration properties of the chain job have precedence over the configuration properties of the Mapper.

Parameters:

isMap - indicates if the Chain is for a Mapper or for a Reducer.

job - chain job.

klass - the Mapper class to add.

inputKeyClass - mapper input key class.

inputValueClass - mapper input value class.

outputKeyClass - mapper output key class.

outputValueClass - mapper output value class.

mapperConf - a configuration for the Mapper class. It is recommended to use a Configuration without default values using the Configuration(boolean loadDefaults) constructor with FALSE.

checkReducerAlreadySet

protected static void checkReducerAlreadySet(boolean isMap,
                          org.apache.hadoop.conf.Configuration jobConf,
                          String prefix,
                          boolean shouldSet)

validateKeyValueTypes

protected static void validateKeyValueTypes(boolean isMap,
                         org.apache.hadoop.conf.Configuration jobConf,
                         Class<?> inputKeyClass,
                         Class<?> inputValueClass,
                         Class<?> outputKeyClass,
                         Class<?> outputValueClass,
                         int index,
                         String prefix)

setMapperConf

protected static void setMapperConf(boolean isMap,
                 org.apache.hadoop.conf.Configuration jobConf,
                 Class<?> inputKeyClass,
                 Class<?> inputValueClass,
                 Class<?> outputKeyClass,
                 Class<?> outputValueClass,
                 org.apache.hadoop.conf.Configuration mapperConf,
                 int index,
                 String prefix)

setReducer

protected static void setReducer(Job job,
              Class<? extends Reducer> klass,
              Class<?> inputKeyClass,
              Class<?> inputValueClass,
              Class<?> outputKeyClass,
              Class<?> outputValueClass,
              org.apache.hadoop.conf.Configuration reducerConf)

Sets the Reducer class to the chain job.

The configuration properties of the chain job have precedence over the configuration properties of the Reducer.

Parameters:

job - the chain job.

klass - the Reducer class to add.

inputKeyClass - reducer input key class.

inputValueClass - reducer input value class.

outputKeyClass - reducer output key class.

outputValueClass - reducer output value class.

reducerConf - a configuration for the Reducer class. It is recommended to use a Configuration without default values using the Configuration(boolean loadDefaults) constructor with FALSE.

setReducerConf

protected static void setReducerConf(org.apache.hadoop.conf.Configuration jobConf,
                  Class<?> inputKeyClass,
                  Class<?> inputValueClass,
                  Class<?> outputKeyClass,
                  Class<?> outputValueClass,
                  org.apache.hadoop.conf.Configuration reducerConf,
                  String prefix)