org.apache.hadoop.mapreduce.task.reduce

Class ShuffleSchedulerImpl<K,V>

java.lang.Object

org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl<K,V>

All Implemented Interfaces:

ShuffleScheduler<K,V>

@InterfaceAudience.Private
@InterfaceStability.Unstable
public class ShuffleSchedulerImpl<K,V>
extends Object
implements ShuffleScheduler<K,V>

Constructor Summary

Constructors

Constructor and Description
ShuffleSchedulerImpl(JobConf job, TaskStatus status, TaskAttemptID reduceId, ExceptionReporter reporter, org.apache.hadoop.util.Progress progress, Counters.Counter shuffledMapsCounter, Counters.Counter reduceShuffleBytes, Counters.Counter failedShuffleCounter)

Method Summary

Methods

Modifier and Type	Method and Description
void	addKnownMapOutput(String hostName, String hostUrl, TaskAttemptID mapId)
void	close()
void	copyFailed(TaskAttemptID mapId, MapHost host, boolean readError, boolean connectExcpt)
void	copySucceeded(TaskAttemptID mapId, MapHost host, long bytes, long startMillis, long endMillis, MapOutput<K,V> output)
void	freeHost(MapHost host)
MapHost	getHost()
List<TaskAttemptID>	getMapsForHost(MapHost host)
int	getMaxHostFailures()
void	hostFailed(String hostname)
void	obsoleteMapOutput(TaskAttemptID mapId)
void	putBackKnownMapOutput(MapHost host, TaskAttemptID mapId)
void	reportLocalError(IOException ioe)
void	resetKnownMaps()
void	resolve(TaskCompletionEvent event) Interpret a TaskCompletionEvent from the event stream.
void	tipFailed(TaskID taskId)
boolean	waitUntilDone(int millis) Wait until the shuffle finishes or until the timeout.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ShuffleSchedulerImpl

public ShuffleSchedulerImpl(JobConf job,
                    TaskStatus status,
                    TaskAttemptID reduceId,
                    ExceptionReporter reporter,
                    org.apache.hadoop.util.Progress progress,
                    Counters.Counter shuffledMapsCounter,
                    Counters.Counter reduceShuffleBytes,
                    Counters.Counter failedShuffleCounter)

Method Detail

resolve

public void resolve(TaskCompletionEvent event)

Description copied from interface: ShuffleScheduler

Interpret a TaskCompletionEvent from the event stream.

Specified by:

resolve in interface ShuffleScheduler<K,V>

Parameters:

event - Intermediate output metadata

copySucceeded

public void copySucceeded(TaskAttemptID mapId,
                 MapHost host,
                 long bytes,
                 long startMillis,
                 long endMillis,
                 MapOutput<K,V> output)
                   throws IOException

Throws:

IOException

hostFailed

public void hostFailed(String hostname)

copyFailed

public void copyFailed(TaskAttemptID mapId,
              MapHost host,
              boolean readError,
              boolean connectExcpt)

reportLocalError

public void reportLocalError(IOException ioe)

tipFailed

public void tipFailed(TaskID taskId)

addKnownMapOutput

public void addKnownMapOutput(String hostName,
                     String hostUrl,
                     TaskAttemptID mapId)

obsoleteMapOutput

public void obsoleteMapOutput(TaskAttemptID mapId)

putBackKnownMapOutput

public void putBackKnownMapOutput(MapHost host,
                         TaskAttemptID mapId)

getHost

public MapHost getHost()
                throws InterruptedException

Throws:

InterruptedException

getMapsForHost

public List<TaskAttemptID> getMapsForHost(MapHost host)

freeHost

public void freeHost(MapHost host)

resetKnownMaps

public void resetKnownMaps()

waitUntilDone

public boolean waitUntilDone(int millis)
                      throws InterruptedException

Wait until the shuffle finishes or until the timeout.

Specified by:

waitUntilDone in interface ShuffleScheduler<K,V>

Parameters:

millis - maximum wait time

Returns:

true if the shuffle is done

Throws:

InterruptedException

close

public void close()
           throws InterruptedException

Specified by:

close in interface ShuffleScheduler<K,V>

Throws:

InterruptedException

getMaxHostFailures

public int getMaxHostFailures()