Class Combiner
- java.lang.Object
-
- org.apache.accumulo.core.iterators.WrappingIterator
-
- org.apache.accumulo.core.iterators.Combiner
-
- All Implemented Interfaces:
OptionDescriber
,SortedKeyValueIterator<Key,Value>
,YieldingKeyValueIterator<Key,Value>
- Direct Known Subclasses:
TypedValueCombiner
public abstract class Combiner extends WrappingIterator implements OptionDescriber
A SortedKeyValueIterator that combines the Values for different versions (timestamp) of a Key within a row into a single Value. Combiner will replace one or more versions of a Key and their Values with the most recent Key and a Value which is the result of the reduce method. AnIteratorSetting.Column
which only specifies a column family will combine all Keys in that column family individually. Similarly, aIteratorSetting.Column
which specifies a column family and column qualifier will combine all Keys in column family and qualifier individually. Combination is only ever performed on multiple versions and not across column qualifiers or column visibilities.Implementations must provide a reduce method:
public Value reduce(Key key, Iterator<Value> iter)
.This reduce method will be passed the most recent Key and an iterator over the Values for all non-deleted versions of that Key. A combiner will not combine keys that differ by more than the timestamp.
This class and its implementations do not automatically filter out unwanted columns from those being combined, thus it is generally recommended to use a
Combiner
implementation with theScannerBase.fetchColumnFamily(Text)
orScannerBase.fetchColumn(Text, Text)
methods.WARNING : Using deletes with Combiners may not work as intended. See
setReduceOnFullCompactionOnly(IteratorSetting, boolean)
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description static class
Combiner.ValueIterator
A Java Iterator that iterates over the Values for a given Key from a source SortedKeyValueIterator.-
Nested classes/interfaces inherited from interface org.apache.accumulo.core.iterators.OptionDescriber
OptionDescriber.IteratorOptions
-
-
Field Summary
Fields Modifier and Type Field Description protected static String
ALL_OPTION
protected static String
COLUMNS_OPTION
protected static String
REDUCE_ON_FULL_COMPACTION_ONLY_OPTION
-
Constructor Summary
Constructors Constructor Description Combiner()
-
Method Summary
All Methods Static Methods Instance Methods Abstract Methods Concrete Methods Modifier and Type Method Description SortedKeyValueIterator<Key,Value>
deepCopy(IteratorEnvironment env)
Creates a deep copy of this iterator as though seek had not yet been called.OptionDescriber.IteratorOptions
describeOptions()
Gets an iterator options object that contains information needed to configure this iterator.Key
getTopKey()
Returns top key.Value
getTopValue()
Returns top value.boolean
hasTop()
Returns true if the iterator has more elements.void
init(SortedKeyValueIterator<Key,Value> source, Map<String,String> options, IteratorEnvironment env)
Initializes the iterator.void
next()
Advances to the next K,V pair.abstract Value
reduce(Key key, Iterator<Value> iter)
Reduces a list of Values into a single Value.void
seek(Range range, Collection<ByteSequence> columnFamilies, boolean inclusive)
Seeks to the first key in the Range, restricting the resulting K,V pairs to those with the specified columns.static void
setColumns(IteratorSetting is, List<IteratorSetting.Column> columns)
A convenience method to set which columns a combiner should be applied to.static void
setCombineAllColumns(IteratorSetting is, boolean combineAllColumns)
A convenience method to set the "all columns" option on a Combiner.static void
setReduceOnFullCompactionOnly(IteratorSetting is, boolean reduceOnFullCompactionOnly)
Combiners may not work correctly with deletes.boolean
validateOptions(Map<String,String> options)
Check to see if an options map contains all options required by an iterator and that the option values are in the expected formats.-
Methods inherited from class org.apache.accumulo.core.iterators.WrappingIterator
getSource, setSource
-
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
-
Methods inherited from interface org.apache.accumulo.core.iterators.YieldingKeyValueIterator
enableYielding
-
-
-
-
Field Detail
-
COLUMNS_OPTION
protected static final String COLUMNS_OPTION
- See Also:
- Constant Field Values
-
ALL_OPTION
protected static final String ALL_OPTION
- See Also:
- Constant Field Values
-
REDUCE_ON_FULL_COMPACTION_ONLY_OPTION
protected static final String REDUCE_ON_FULL_COMPACTION_ONLY_OPTION
- See Also:
- Constant Field Values
-
-
Method Detail
-
getTopKey
public Key getTopKey()
Description copied from interface:SortedKeyValueIterator
Returns top key. Can be called 0 or more times without affecting behavior of next() or hasTop(). Note that in minor compaction scope and in non-full major compaction scopes the iterator may see deletion entries. These entries should be preserved by all iterators except ones that are strictly scan-time iterators that will never be configured for the minc or majc scopes. Deletion entries are only removed during full major compactions.For performance reasons, iterators reserve the right to reuse objects returned by
getTopKey
whenSortedKeyValueIterator.next()
is called, changing the data that the object references. Iterators that need to save an object returned bygetTopKey
ought to copy the object's data into a new object in order to avoid aliasing bugs.- Specified by:
getTopKey
in interfaceSortedKeyValueIterator<Key,Value>
- Overrides:
getTopKey
in classWrappingIterator
- Returns:
K
-
getTopValue
public Value getTopValue()
Description copied from interface:SortedKeyValueIterator
Returns top value. Can be called 0 or more times without affecting behavior of next() or hasTop().For performance reasons, iterators reserve the right to reuse objects returned by
getTopValue
whenSortedKeyValueIterator.next()
is called, changing the underlying data that the object references. Iterators that need to save an object returned bygetTopValue
ought to copy the object's data into a new object in order to avoid aliasing bugs.- Specified by:
getTopValue
in interfaceSortedKeyValueIterator<Key,Value>
- Overrides:
getTopValue
in classWrappingIterator
- Returns:
V
-
hasTop
public boolean hasTop()
Description copied from interface:SortedKeyValueIterator
Returns true if the iterator has more elements. Note that if this iterator has yielded (@see YieldingKeyValueIterator.enableYielding(YieldCallback)), this this method must return false.- Specified by:
hasTop
in interfaceSortedKeyValueIterator<Key,Value>
- Overrides:
hasTop
in classWrappingIterator
- Returns:
true
if the iterator has more elements.
-
next
public void next() throws IOException
Description copied from interface:SortedKeyValueIterator
Advances to the next K,V pair. Note that in minor compaction scope and in non-full major compaction scopes the iterator may see deletion entries. These entries should be preserved by all iterators except ones that are strictly scan-time iterators that will never be configured for the minc or majc scopes. Deletion entries are only removed during full major compactions.- Specified by:
next
in interfaceSortedKeyValueIterator<Key,Value>
- Overrides:
next
in classWrappingIterator
- Throws:
IOException
- if an I/O error occurs.
-
seek
public void seek(Range range, Collection<ByteSequence> columnFamilies, boolean inclusive) throws IOException
Description copied from interface:SortedKeyValueIterator
Seeks to the first key in the Range, restricting the resulting K,V pairs to those with the specified columns. An iterator does not have to stop at the end of the range. The whole range is provided so that iterators can make optimizations. Seek may be called multiple times with different parameters afterSortedKeyValueIterator.init(org.apache.accumulo.core.iterators.SortedKeyValueIterator<K, V>, java.util.Map<java.lang.String, java.lang.String>, org.apache.accumulo.core.iterators.IteratorEnvironment)
is called. Iterators that examine groups of adjacent key/value pairs (e.g. rows) to determine their top key and value should be sure that they properly handle a seek to a key in the middle of such a group (e.g. the middle of a row). Even if the client always seeks to a range containing an entire group (a,c), the tablet server could send back a batch of entries corresponding to (a,b], then reseek the iterator to range (b,c) when the scan is continued.columnFamilies
is used, at the lowest level, to determine which data blocks inside of an RFile need to be opened for this iterator. This set of data blocks is also the set of locality groups defined for the given table. If no columnFamilies are provided, the data blocks for all locality groups inside of the correct RFile will be opened and seeked in an attempt to find the correct start key, regardless of the startKey in therange
. In an Accumulo instance in which multiple locality groups exist for a table, it is important to ensure thatcolumnFamilies
is properly set to the minimum required column families to ensure that data from separate locality groups is not inadvertently read.- Specified by:
seek
in interfaceSortedKeyValueIterator<Key,Value>
- Overrides:
seek
in classWrappingIterator
- Parameters:
range
-Range
of keys to iterate over.columnFamilies
-Collection
of column families to include or exclude.inclusive
-boolean
that indicates whether to include (true) or exclude (false) column families.- Throws:
IOException
- if an I/O error occurs.
-
reduce
public abstract Value reduce(Key key, Iterator<Value> iter)
Reduces a list of Values into a single Value.- Parameters:
key
- The most recent version of the Key being reduced.iter
- An iterator over the Values for different versions of the key.- Returns:
- The combined Value.
-
init
public void init(SortedKeyValueIterator<Key,Value> source, Map<String,String> options, IteratorEnvironment env) throws IOException
Description copied from interface:SortedKeyValueIterator
Initializes the iterator. Data should not be read from the source in this method.- Specified by:
init
in interfaceSortedKeyValueIterator<Key,Value>
- Overrides:
init
in classWrappingIterator
- Parameters:
source
-SortedKeyValueIterator
source to read data from.options
-Map
map of string option names to option values.env
-IteratorEnvironment
environment in which iterator is being run.- Throws:
IOException
- unused.
-
deepCopy
public SortedKeyValueIterator<Key,Value> deepCopy(IteratorEnvironment env)
Description copied from interface:SortedKeyValueIterator
Creates a deep copy of this iterator as though seek had not yet been called. init should be called on an iterator before deepCopy is called. init should not need to be called on the copy that is returned by deepCopy; that is, when necessary init should be called in the deepCopy method on the iterator it returns. The behavior is unspecified if init is called after deepCopy either on the original or the copy. A proper implementation would call deepCopy on the source.- Specified by:
deepCopy
in interfaceSortedKeyValueIterator<Key,Value>
- Overrides:
deepCopy
in classWrappingIterator
- Parameters:
env
-IteratorEnvironment
environment in which iterator is being run.- Returns:
SortedKeyValueIterator
a copy of this iterator (with the same source and settings).
-
describeOptions
public OptionDescriber.IteratorOptions describeOptions()
Description copied from interface:OptionDescriber
Gets an iterator options object that contains information needed to configure this iterator. This object will be used by the accumulo shell to prompt the user to input the appropriate information.- Specified by:
describeOptions
in interfaceOptionDescriber
- Returns:
- an iterator options object
-
validateOptions
public boolean validateOptions(Map<String,String> options)
Description copied from interface:OptionDescriber
Check to see if an options map contains all options required by an iterator and that the option values are in the expected formats.- Specified by:
validateOptions
in interfaceOptionDescriber
- Parameters:
options
- a map of option names to option values- Returns:
- true if options are valid, false otherwise (IllegalArgumentException preferred)
-
setColumns
public static void setColumns(IteratorSetting is, List<IteratorSetting.Column> columns)
A convenience method to set which columns a combiner should be applied to. For each column specified, all versions of a Key which match that @{link IteratorSetting.Column} will be combined individually in each row. This method is likely to be used in conjunction withScannerBase.fetchColumnFamily(Text)
orScannerBase.fetchColumn(Text,Text)
.- Parameters:
is
- iterator settings object to configurecolumns
- a list of columns to encode as the value for the combiner column configuration
-
setCombineAllColumns
public static void setCombineAllColumns(IteratorSetting is, boolean combineAllColumns)
A convenience method to set the "all columns" option on a Combiner. This will combine all columns individually within each row.- Parameters:
is
- iterator settings object to configurecombineAllColumns
- if true, the columns option is ignored and the Combiner will be applied to all columns
-
setReduceOnFullCompactionOnly
public static void setReduceOnFullCompactionOnly(IteratorSetting is, boolean reduceOnFullCompactionOnly)
Combiners may not work correctly with deletes. Sometimes when Accumulo compacts the files in a tablet, it only compacts a subset of the files. If a delete marker exists in one of the files that is not being compacted, then data that should be deleted may be combined. See ACCUMULO-2232 for more information. For correctness deletes should not be used with columns that are combined OR this option should be set to true.When this method is set to true all data is passed through during partial major compactions and no reducing is done. Reducing is only done during scan and full major compactions, when deletes can be correctly handled. Only reducing on full major compactions may have negative performance implications, leaving lots of work to be done at scan time.
When this method is set to false, combiners will log an error if a delete is seen during any compaction. This can be suppressed by adjusting logging configuration. Errors will not be logged more than once an hour per Combiner, regardless of how many deletes are seen.
This method was added in 1.6.4 and 1.7.1. If you want your code to work in earlier versions of 1.6 and 1.7 then do not call this method. If not set this property defaults to false in order to maintain compatibility.
- Since:
- 1.6.5 1.7.1 1.8.0
-
-