org.apache.hadoop.mapreduce.lib.join

Class CompositeRecordReader<K extends org.apache.hadoop.io.WritableComparable<?>,V extends org.apache.hadoop.io.Writable,X extends org.apache.hadoop.io.Writable>

java.lang.Object

org.apache.hadoop.mapreduce.RecordReader<K,V>

org.apache.hadoop.mapreduce.lib.join.ComposableRecordReader<K,X>

org.apache.hadoop.mapreduce.lib.join.CompositeRecordReader<K,V,X>

All Implemented Interfaces:

Closeable, AutoCloseable, Comparable<ComposableRecordReader<K,?>>, org.apache.hadoop.conf.Configurable

Direct Known Subclasses:

JoinRecordReader, MultiFilterRecordReader

@InterfaceAudience.Public
@InterfaceStability.Stable
public abstract class CompositeRecordReader<K extends org.apache.hadoop.io.WritableComparable<?>,V extends org.apache.hadoop.io.Writable,X extends org.apache.hadoop.io.Writable>
extends ComposableRecordReader<K,X>
implements org.apache.hadoop.conf.Configurable

A RecordReader that can effect joins of RecordReaders sharing a common key type and partitioning.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	CompositeRecordReader.JoinCollector Collector for join values.

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.hadoop.conf.Configuration	conf
protected CompositeRecordReader.JoinCollector	jc
protected K	key
protected Class<? extends org.apache.hadoop.io.WritableComparable>	keyclass
protected ComposableRecordReader<K,? extends V>[]	kids
protected X	value

Constructor Summary

Constructors

Constructor and Description
CompositeRecordReader(int id, int capacity, Class<? extends org.apache.hadoop.io.WritableComparator> cmpcl) Create a RecordReader with capacity children to position id in the parent reader.

Method Summary

Methods

Modifier and Type	Method and Description
void	accept(CompositeRecordReader.JoinCollector jc, K key) If key provided matches that of this Composite, give JoinCollector iterator over values it may emit.
void	add(ComposableRecordReader<K,? extends V> rr) Add a RecordReader to this collection.
void	close() Close all child RRs.
protected abstract boolean	combine(Object[] srcs, TupleWritable value)
int	compareTo(ComposableRecordReader<K,?> other) Implement Comparable contract (compare key of join or head of heap with that of another).
protected K	createKey() Create a new key common to all child RRs.
protected TupleWritable	createTupleWritable() Create a value to be used internally for joins.
protected void	fillJoinCollector(K iterkey) For all child RRs offering the key provided, obtain an iterator at that position in the JoinCollector.
protected org.apache.hadoop.io.WritableComparator	getComparator() Return comparator defining the ordering for RecordReaders in this composite.
org.apache.hadoop.conf.Configuration	getConf()
K	getCurrentKey() Get the current key
X	getCurrentValue() Get the current value.
protected abstract ResetableIterator<X>	getDelegate() Obtain an iterator over the child RRs apropos of the value type ultimately emitted from this join.
float	getProgress() Report progress as the minimum of all child RR progress.
protected PriorityQueue<ComposableRecordReader<K,?>>	getRecordReaderQueue() Return sorted list of RecordReaders for this composite.
boolean	hasNext() Return true if it is possible that this could emit more values.
int	id() Return the position in the collector this class occupies.
void	initialize(InputSplit split, TaskAttemptContext context) Called once at initialization.
K	key() Return the key for the current join or the value at the top of the RecordReader heap.
void	key(K key) Clone the key at the top of this RR into the given object.
void	setConf(org.apache.hadoop.conf.Configuration conf)
void	skip(K key) Pass skip key to child RRs.

Methods inherited from class org.apache.hadoop.mapreduce.RecordReader

nextKeyValue

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

conf

protected org.apache.hadoop.conf.Configuration conf

keyclass

protected Class<? extends org.apache.hadoop.io.WritableComparable> keyclass

jc

protected final CompositeRecordReader.JoinCollector jc

kids

protected final ComposableRecordReader<K extends org.apache.hadoop.io.WritableComparable<?>,? extends V extends org.apache.hadoop.io.Writable>[] kids

key

protected K extends org.apache.hadoop.io.WritableComparable<?> key

value

protected X extends org.apache.hadoop.io.Writable value

Constructor Detail

CompositeRecordReader

public CompositeRecordReader(int id,
                     int capacity,
                     Class<? extends org.apache.hadoop.io.WritableComparator> cmpcl)
                      throws IOException

Create a RecordReader with capacity children to position id in the parent reader. The id of a root CompositeRecordReader is -1 by convention, but relying on this is not recommended.

Throws:

IOException

Method Detail

combine

protected abstract boolean combine(Object[] srcs,
              TupleWritable value)

initialize

public void initialize(InputSplit split,
              TaskAttemptContext context)
                throws IOException,
                       InterruptedException

Description copied from class: RecordReader

Called once at initialization.

Specified by:

initialize in class RecordReader<K extends org.apache.hadoop.io.WritableComparable<?>,X extends org.apache.hadoop.io.Writable>

Parameters:

split - the split that defines the range of records to read

context - the information about the task

Throws:

IOException

InterruptedException

id

public int id()

Return the position in the collector this class occupies.

setConf

public void setConf(org.apache.hadoop.conf.Configuration conf)

Specified by:

setConf in interface org.apache.hadoop.conf.Configurable

getConf

public org.apache.hadoop.conf.Configuration getConf()

Specified by:

getConf in interface org.apache.hadoop.conf.Configurable

getRecordReaderQueue

protected PriorityQueue<ComposableRecordReader<K,?>> getRecordReaderQueue()

Return sorted list of RecordReaders for this composite.

getComparator

protected org.apache.hadoop.io.WritableComparator getComparator()

Return comparator defining the ordering for RecordReaders in this composite.

add

public void add(ComposableRecordReader<K,? extends V> rr)
         throws IOException,
                InterruptedException

Add a RecordReader to this collection. The id() of a RecordReader determines where in the Tuple its entry will appear. Adding RecordReaders with the same id has undefined behavior.

Throws:

IOException

InterruptedException

key

public K key()

Return the key for the current join or the value at the top of the RecordReader heap.

key

public void key(K key)
         throws IOException

Clone the key at the top of this RR into the given object.

Throws:

IOException

getCurrentKey

public K getCurrentKey()

Description copied from class: RecordReader

Get the current key

Specified by:

getCurrentKey in class RecordReader<K extends org.apache.hadoop.io.WritableComparable<?>,X extends org.apache.hadoop.io.Writable>

Returns:

the current key or null if there is no current key

hasNext

public boolean hasNext()

Return true if it is possible that this could emit more values.

skip

public void skip(K key)
          throws IOException,
                 InterruptedException

Pass skip key to child RRs.

Throws:

IOException

InterruptedException

getDelegate

protected abstract ResetableIterator<X> getDelegate()

Obtain an iterator over the child RRs apropos of the value type ultimately emitted from this join.

accept

public void accept(CompositeRecordReader.JoinCollector jc,
          K key)
            throws IOException,
                   InterruptedException

If key provided matches that of this Composite, give JoinCollector iterator over values it may emit.

Throws:

IOException

InterruptedException

fillJoinCollector

protected void fillJoinCollector(K iterkey)
                          throws IOException,
                                 InterruptedException

For all child RRs offering the key provided, obtain an iterator at that position in the JoinCollector.

Throws:

IOException

InterruptedException

compareTo

public int compareTo(ComposableRecordReader<K,?> other)

Implement Comparable contract (compare key of join or head of heap with that of another).

Specified by:

compareTo in interface Comparable<ComposableRecordReader<K extends org.apache.hadoop.io.WritableComparable<?>,?>>

createKey

protected K createKey()

Create a new key common to all child RRs.

Throws:

ClassCastException - if key classes differ.

createTupleWritable

protected TupleWritable createTupleWritable()

Create a value to be used internally for joins.

getCurrentValue

public X getCurrentValue()
                                                        throws IOException,
                                                               InterruptedException

Get the current value.

Specified by:

getCurrentValue in class RecordReader<K extends org.apache.hadoop.io.WritableComparable<?>,X extends org.apache.hadoop.io.Writable>

Returns:

the object that was read

Throws:

IOException

InterruptedException

close

public void close()
           throws IOException

Close all child RRs.

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Specified by:

close in class RecordReader<K extends org.apache.hadoop.io.WritableComparable<?>,X extends org.apache.hadoop.io.Writable>

Throws:

IOException

getProgress

public float getProgress()
                  throws IOException,
                         InterruptedException

Report progress as the minimum of all child RR progress.

Specified by:

getProgress in class RecordReader<K extends org.apache.hadoop.io.WritableComparable<?>,X extends org.apache.hadoop.io.Writable>

Returns:

a number between 0.0 and 1.0 that is the fraction of the data read

Throws:

IOException

InterruptedException