org.apache.hadoop.mapred

Class BackupStore<K,V>

java.lang.Object

org.apache.hadoop.mapred.BackupStore<K,V>

@InterfaceAudience.Private
@InterfaceStability.Unstable
public class BackupStore<K,V>
extends Object

BackupStore is an utility class that is used to support the mark-reset functionality of values iterator

It has two caches - a memory cache and a file cache where values are stored as they are iterated, after a mark. On reset, values are retrieved from these caches. Framework moves from the memory cache to the file cache when the memory cache becomes full.

Constructor Summary

Constructors

Constructor and Description
BackupStore(org.apache.hadoop.conf.Configuration conf, TaskAttemptID taskid)

Method Summary

Methods

Modifier and Type	Method and Description
void	clearMark()
void	exitResetMode() This function is called the ValuesIterator when a mark is called outside of a reset zone.
DataOutputStream	getOutputStream(int length) For writing the first key and value bytes directly from the value iterators, pass the current underlying output stream
boolean	hasNext()
void	mark()
void	next()
org.apache.hadoop.io.DataInputBuffer	nextKey()
org.apache.hadoop.io.DataInputBuffer	nextValue()
void	reinitialize()
void	reset()
void	updateCounters(int length) This method is called by the valueIterators after writing the first key and value bytes to the BackupStore
void	write(org.apache.hadoop.io.DataInputBuffer key, org.apache.hadoop.io.DataInputBuffer value) Write the given K,V to the cache.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BackupStore

public BackupStore(org.apache.hadoop.conf.Configuration conf,
           TaskAttemptID taskid)
            throws IOException

Throws:

IOException

Method Detail

write

public void write(org.apache.hadoop.io.DataInputBuffer key,
         org.apache.hadoop.io.DataInputBuffer value)
           throws IOException

Write the given K,V to the cache. Write to memcache if space is available, else write to the filecache

Parameters:

key -

value -

Throws:

IOException

mark

public void mark()
          throws IOException

Throws:

IOException

reset

public void reset()
           throws IOException

Throws:

IOException

hasNext

public boolean hasNext()
                throws IOException

Throws:

IOException

next

public void next()
          throws IOException

Throws:

IOException

nextValue

public org.apache.hadoop.io.DataInputBuffer nextValue()

nextKey

public org.apache.hadoop.io.DataInputBuffer nextKey()

reinitialize

public void reinitialize()
                  throws IOException

Throws:

IOException

exitResetMode

public void exitResetMode()
                   throws IOException

This function is called the ValuesIterator when a mark is called outside of a reset zone.

Throws:

IOException

getOutputStream

public DataOutputStream getOutputStream(int length)
                                 throws IOException

For writing the first key and value bytes directly from the value iterators, pass the current underlying output stream

Parameters:

length - The length of the impending write

Throws:

IOException

updateCounters

public void updateCounters(int length)

This method is called by the valueIterators after writing the first key and value bytes to the BackupStore

Parameters:

length -

clearMark

public void clearMark()
               throws IOException

Throws:

IOException