org.apache.hadoop.mapreduce.task.reduce

Class MergeManagerImpl<K,V>

java.lang.Object

org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl<K,V>

All Implemented Interfaces:

MergeManager<K,V>

@InterfaceAudience.LimitedPrivate(value="MapReduce")
@InterfaceStability.Unstable
public class MergeManagerImpl<K,V>
extends Object
implements MergeManager<K,V>

Field Summary

Fields

Modifier and Type	Field and Description
protected MapOutputFile	mapOutputFile

Constructor Summary

Constructors

Constructor and Description

MergeManagerImpl(TaskAttemptID reduceId, JobConf jobConf, org.apache.hadoop.fs.FileSystem localFS, org.apache.hadoop.fs.LocalDirAllocator localDirAllocator, Reporter reporter, org.apache.hadoop.io.compress.CompressionCodec codec, Class<? extends Reducer> combinerClass, Task.CombineOutputCollector<K,V> combineCollector, Counters.Counter spilledRecordsCounter, Counters.Counter reduceCombineInputCounter, Counters.Counter mergedMapOutputsCounter, ExceptionReporter exceptionReporter, org.apache.hadoop.util.Progress mergePhase, MapOutputFile mapOutputFile)

Method Summary

Methods

Modifier and Type	Method and Description
RawKeyValueIterator	close() Called at the end of shuffle.
void	closeInMemoryFile(org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput<K,V> mapOutput)
void	closeInMemoryMergedFile(org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput<K,V> mapOutput)
void	closeOnDiskFile(org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.CompressAwarePath file)
protected org.apache.hadoop.mapreduce.task.reduce.MergeThread<org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput<K,V>,K,V>	createInMemoryMerger()
protected org.apache.hadoop.mapreduce.task.reduce.MergeThread<org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.CompressAwarePath,K,V>	createOnDiskMerger()
MapOutput<K,V>	reserve(TaskAttemptID mapId, long requestedSize, int fetcher) To reserve resources for data to be shuffled.
void	waitForResource() To wait until merge has some freed resources available so that it can accept shuffled data.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

mapOutputFile

protected MapOutputFile mapOutputFile

Constructor Detail

MergeManagerImpl

public MergeManagerImpl(TaskAttemptID reduceId,
                JobConf jobConf,
                org.apache.hadoop.fs.FileSystem localFS,
                org.apache.hadoop.fs.LocalDirAllocator localDirAllocator,
                Reporter reporter,
                org.apache.hadoop.io.compress.CompressionCodec codec,
                Class<? extends Reducer> combinerClass,
                Task.CombineOutputCollector<K,V> combineCollector,
                Counters.Counter spilledRecordsCounter,
                Counters.Counter reduceCombineInputCounter,
                Counters.Counter mergedMapOutputsCounter,
                ExceptionReporter exceptionReporter,
                org.apache.hadoop.util.Progress mergePhase,
                MapOutputFile mapOutputFile)

Method Detail

createInMemoryMerger

protected org.apache.hadoop.mapreduce.task.reduce.MergeThread<org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput<K,V>,K,V> createInMemoryMerger()

createOnDiskMerger

protected org.apache.hadoop.mapreduce.task.reduce.MergeThread<org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.CompressAwarePath,K,V> createOnDiskMerger()

waitForResource

public void waitForResource()
                     throws InterruptedException

Description copied from interface: MergeManager

To wait until merge has some freed resources available so that it can accept shuffled data. This will be called before a network connection is established to get the map output.

Specified by:

waitForResource in interface MergeManager<K,V>

Throws:

InterruptedException

reserve

public MapOutput<K,V> reserve(TaskAttemptID mapId,
                     long requestedSize,
                     int fetcher)
                       throws IOException

Description copied from interface: MergeManager

To reserve resources for data to be shuffled. This will be called after a network connection is made to shuffle the data.

Specified by:

reserve in interface MergeManager<K,V>

Parameters:

mapId - mapper from which data will be shuffled.

requestedSize - size in bytes of data that will be shuffled.

fetcher - id of the map output fetcher that will shuffle the data.

Returns:

a MapOutput object that can be used by shuffle to shuffle data. If required resources cannot be reserved immediately, a null can be returned.

Throws:

IOException

closeInMemoryFile

public void closeInMemoryFile(org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput<K,V> mapOutput)

closeInMemoryMergedFile

public void closeInMemoryMergedFile(org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput<K,V> mapOutput)

closeOnDiskFile

public void closeOnDiskFile(org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.CompressAwarePath file)

close

public RawKeyValueIterator close()
                          throws Throwable

Description copied from interface: MergeManager

Called at the end of shuffle.

Specified by:

close in interface MergeManager<K,V>

Returns:

a key value iterator object.

Throws:

Throwable