org.apache.accumulo.core.iterators.user

Class TransformingIterator

java.lang.Object

org.apache.accumulo.core.iterators.WrappingIterator

org.apache.accumulo.core.iterators.user.TransformingIterator

All Implemented Interfaces:

OptionDescriber, SortedKeyValueIterator<Key,Value>

public abstract class TransformingIterator
extends WrappingIterator
implements OptionDescriber

The TransformingIterator allows portions of a key (except for the row) to be transformed. This iterator handles the details that come with modifying keys (i.e., that the sort order could change). In order to do so, however, the iterator must put all keys sharing the same prefix in memory. Prefix is defined as the parts of the key that are not modified by this iterator. That is, if the iterator modifies column qualifier and timestamp, then the prefix is row and column family. In that case, the iterator must load all column qualifiers for each row/column family pair into memory. Given this constraint, care must be taken by users of this iterator to ensure it is not run in such a way that will overrun memory in a tablet server.

If the implementing iterator is transforming column families, then it must also override untransformColumnFamilies(Collection) to handle the case when column families are fetched at scan time. The fetched column families will/must be in the transformed space, and the untransformed column families need to be passed to this iterator's source. If it is not possible to write a reverse transformation (e.g., the column family transformation depends on the row value or something like that), then the iterator must not fetch specific column families (or only fetch column families that are known to not transform at all).

If the implementing iterator is transforming column visibilities, then users must be careful NOT to fetch column qualifiers from the scanner. The reason for this is due to ACCUMULO-??? (insert issue number).

If the implementing iterator is transforming column visibilities, then the user should be sure to supply authorizations via the AUTH_OPT iterator option (note that this is only necessary for scan scope iterators). The supplied authorizations should be in the transformed space, but the authorizations supplied to the scanner should be in the untransformed space. That is, if the iterator transforms A to 1, B to 2, C to 3, etc, then the auths supplied when the scanner is constructed should be A,B,C,... and the auths supplied to the iterator should be 1,2,3,... The reason for this is that the scanner performs security filtering before this iterator is called, so the authorizations need to be in the original untransformed space. Since the iterator can transform visibilities, it is possible that it could produce visibilities that the user cannot see, so the transformed keys must be tested to ensure the user is allowed to view them. Note that this test is not necessary when the iterator is not used in the scan scope since no security filtering is performed during major and minor compactions. It should also be noted that this iterator implements the security filtering rather than relying on a follow-on iterator to do it so that we ensure the test is performed.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	TransformingIterator.KVBuffer

Nested classes/interfaces inherited from interface org.apache.accumulo.core.iterators.OptionDescriber

OptionDescriber.IteratorOptions

Field Summary

Fields

Modifier and Type	Field and Description
static String	AUTH_OPT
protected int	keyPos
protected ArrayList<Pair<Key,Value>>	keys
protected org.apache.log4j.Logger	log
static String	MAX_BUFFER_SIZE_OPT
protected boolean	scanning
protected Collection<ByteSequence>	seekColumnFamilies
protected boolean	seekColumnFamiliesInclusive
protected Range	seekRange

Constructor Summary

Constructors

Constructor and Description
TransformingIterator()

Method Summary

Methods

Modifier and Type	Method and Description
protected boolean	canSee(Key key) Indicates whether or not the user is able to see key.
protected boolean	canSeeColumnFamily(Key key) Indicates whether or not key can be seen, according to the fetched column families for this iterator.
protected Range	computeReseekRange(Range range) Possibly expand range to include everything for the key prefix we are working with.
protected Key	copyPartialKey(Key key, PartialKey part) Creates a copy of key, copying only the parts of the key specified in part.
SortedKeyValueIterator<Key,Value>	deepCopy(IteratorEnvironment env) Creates a deep copy of this iterator as though seek had not yet been called.
OptionDescriber.IteratorOptions	describeOptions() Gets an iterator options object that contains information needed to configure this iterator.
protected abstract PartialKey	getKeyPrefix() Indicates the prefix of keys that will be transformed by this iterator.
Key	getTopKey() Returns top key.
Value	getTopValue() Returns top value.
boolean	hasTop() Returns true if the iterator has more elements.
protected boolean	includeTransformedKey(Key transformedKey) Determines whether or not to include transformedKey in the output.
void	init(SortedKeyValueIterator<Key,Value> source, Map<String,String> options, IteratorEnvironment env) Initializes the iterator.
protected boolean	isSetAfterPart(Key key, PartialKey part) Indicates whether or not any part of key excluding part is set.
void	next() Advances to the next K,V pair.
protected Key	replaceColumnFamily(Key originalKey, org.apache.hadoop.io.Text newColFam) Make a new key with all parts (including delete flag) coming from originalKey but use newColFam as the column family.
protected Key	replaceColumnQualifier(Key originalKey, org.apache.hadoop.io.Text newColQual) Make a new key with all parts (including delete flag) coming from originalKey but use newColQual as the column qualifier.
protected Key	replaceColumnVisibility(Key originalKey, org.apache.hadoop.io.Text newColVis) Make a new key with all parts (including delete flag) coming from originalKey but use newColVis as the column visibility.
protected Key	replaceKeyParts(Key originalKey, org.apache.hadoop.io.Text newColQual, org.apache.hadoop.io.Text newColVis) Make a new key with a column qualifier, and column visibility.
protected Key	replaceKeyParts(Key originalKey, org.apache.hadoop.io.Text newColFam, org.apache.hadoop.io.Text newColQual, org.apache.hadoop.io.Text newColVis) Make a new key with a column family, column qualifier, and column visibility.
void	seek(Range range, Collection<ByteSequence> columnFamilies, boolean inclusive) Seeks to the first key in the Range, restricting the resulting K,V pairs to those with the specified columns.
static void	setAuthorizations(IteratorSetting config, Authorizations auths) Configure authorizations used for post transformation filtering.
static void	setMaxBufferSize(IteratorSetting config, long maxBufferSize) Configure the maximum amount of memory that can be used for transformation.
protected void	transformKeys() Reads all keys matching the first key's prefix from the source iterator, transforms them, and sorts the resulting keys.
protected abstract void	transformRange(SortedKeyValueIterator<Key,Value> input, TransformingIterator.KVBuffer output) Transforms input.
protected Collection<ByteSequence>	untransformColumnFamilies(Collection<ByteSequence> columnFamilies) Reverses the transformation applied to column families that are fetched at seek time.
boolean	validateOptions(Map<String,String> options) Check to see if an options map contains all options required by an iterator and that the option values are in the expected formats.

Methods inherited from class org.apache.accumulo.core.iterators.WrappingIterator

getSource, setSource

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

AUTH_OPT

public static final String AUTH_OPT

See Also:

Constant Field Values

MAX_BUFFER_SIZE_OPT

public static final String MAX_BUFFER_SIZE_OPT

See Also:

Constant Field Values

log

protected org.apache.log4j.Logger log

keys

protected ArrayList<Pair<Key,Value>> keys

keyPos

protected int keyPos

scanning

protected boolean scanning

seekRange

protected Range seekRange

seekColumnFamilies

protected Collection<ByteSequence> seekColumnFamilies

seekColumnFamiliesInclusive

protected boolean seekColumnFamiliesInclusive

Constructor Detail

TransformingIterator

public TransformingIterator()

Method Detail

init

public void init(SortedKeyValueIterator<Key,Value> source,
        Map<String,String> options,
        IteratorEnvironment env)
          throws IOException

Description copied from interface: SortedKeyValueIterator

Initializes the iterator. Data should not be read from the source in this method.

Specified by:

init in interface SortedKeyValueIterator<Key,Value>

Overrides:

init in class WrappingIterator

Parameters:

source - SortedKeyValueIterator source to read data from.

options - Map map of string option names to option values.

env - IteratorEnvironment environment in which iterator is being run.

Throws:

IOException - unused.

describeOptions

public OptionDescriber.IteratorOptions describeOptions()

Description copied from interface: OptionDescriber

Gets an iterator options object that contains information needed to configure this iterator. This object will be used by the accumulo shell to prompt the user to input the appropriate information.

Specified by:

describeOptions in interface OptionDescriber

Returns:

an iterator options object

validateOptions

public boolean validateOptions(Map<String,String> options)

Description copied from interface: OptionDescriber

Check to see if an options map contains all options required by an iterator and that the option values are in the expected formats.

Specified by:

validateOptions in interface OptionDescriber

Parameters:

options - a map of option names to option values

Returns:

true if options are valid, false otherwise (IllegalArgumentException preferred)

deepCopy

public SortedKeyValueIterator<Key,Value> deepCopy(IteratorEnvironment env)

Description copied from interface: SortedKeyValueIterator

Creates a deep copy of this iterator as though seek had not yet been called. init should be called on an iterator before deepCopy is called. init should not need to be called on the copy that is returned by deepCopy; that is, when necessary init should be called in the deepCopy method on the iterator it returns. The behavior is unspecified if init is called after deepCopy either on the original or the copy.

Specified by:

deepCopy in interface SortedKeyValueIterator<Key,Value>

Overrides:

deepCopy in class WrappingIterator

Parameters:

env - IteratorEnvironment environment in which iterator is being run.

Returns:

SortedKeyValueIterator a copy of this iterator (with the same source and settings).

hasTop

public boolean hasTop()

Description copied from interface: SortedKeyValueIterator

Returns true if the iterator has more elements.

Specified by:

hasTop in interface SortedKeyValueIterator<Key,Value>

Overrides:

hasTop in class WrappingIterator

Returns:

true if the iterator has more elements.

getTopKey

public Key getTopKey()

Description copied from interface: SortedKeyValueIterator

Returns top key. Can be called 0 or more times without affecting behavior of next() or hasTop(). Note that in minor compaction scope and in non-full major compaction scopes the iterator may see deletion entries. These entries should be preserved by all iterators except ones that are strictly scan-time iterators that will never be configured for the minc or majc scopes. Deletion entries are only removed during full major compactions.

Specified by:

getTopKey in interface SortedKeyValueIterator<Key,Value>

Overrides:

getTopKey in class WrappingIterator

Returns:

K

getTopValue

public Value getTopValue()

Description copied from interface: SortedKeyValueIterator

Returns top value. Can be called 0 or more times without affecting behavior of next() or hasTop().

Specified by:

getTopValue in interface SortedKeyValueIterator<Key,Value>

Overrides:

getTopValue in class WrappingIterator

Returns:

V

next

public void next()
          throws IOException

Description copied from interface: SortedKeyValueIterator

Advances to the next K,V pair. Note that in minor compaction scope and in non-full major compaction scopes the iterator may see deletion entries. These entries should be preserved by all iterators except ones that are strictly scan-time iterators that will never be configured for the minc or majc scopes. Deletion entries are only removed during full major compactions.

Specified by:

next in interface SortedKeyValueIterator<Key,Value>

Overrides:

next in class WrappingIterator

Throws:

IOException - if an I/O error occurs.

seek

public void seek(Range range,
        Collection<ByteSequence> columnFamilies,
        boolean inclusive)
          throws IOException

Description copied from interface: SortedKeyValueIterator

Seeks to the first key in the Range, restricting the resulting K,V pairs to those with the specified columns. An iterator does not have to stop at the end of the range. The whole range is provided so that iterators can make optimizations. Seek may be called multiple times with different parameters after SortedKeyValueIterator.init(org.apache.accumulo.core.iterators.SortedKeyValueIterator, java.util.Map, org.apache.accumulo.core.iterators.IteratorEnvironment) is called. Iterators that examine groups of adjacent key/value pairs (e.g. rows) to determine their top key and value should be sure that they properly handle a seek to a key in the middle of such a group (e.g. the middle of a row). Even if the client always seeks to a range containing an entire group (a,c), the tablet server could send back a batch of entries corresponding to (a,b], then reseek the iterator to range (b,c) when the scan is continued. columnFamilies is used, at the lowest level, to determine which data blocks inside of an RFile need to be opened for this iterator. This set of data blocks is also the set of locality groups defined for the given table. If no columnFamilies are provided, the data blocks for all locality groups inside of the correct RFile will be opened and seeked in an attempt to find the correct start key, regardless of the startKey in the range. In an Accumulo instance in which multiple locality groups exist for a table, it is important to ensure that columnFamilies is properly set to the minimum required column families to ensure that data from separate locality groups is not inadvertently read.

Specified by:

seek in interface SortedKeyValueIterator<Key,Value>

Overrides:

seek in class WrappingIterator

Parameters:

range - Range of keys to iterate over.

columnFamilies - Collection of column families to include or exclude.

inclusive - boolean that indicates whether to include (true) or exclude (false) column families.

Throws:

IOException - if an I/O error occurs.

transformKeys

protected void transformKeys()
                      throws IOException

Reads all keys matching the first key's prefix from the source iterator, transforms them, and sorts the resulting keys. Transformed keys that fall outside of our seek range or can't be seen by the user are excluded.

Throws:

IOException

includeTransformedKey

protected boolean includeTransformedKey(Key transformedKey)

Determines whether or not to include transformedKey in the output. It is possible that transformation could have produced a key that falls outside of the seek range, a key with a visibility the user can't see, a key with a visibility that doesn't parse, or a key with a column family that wasn't fetched. We only do some checks (outside the range, user can see) if we're scanning. The range check is not done for major/minor compaction since seek ranges won't be in our transformed key space and we will never change the row so we can't produce keys that would fall outside the tablet anyway.

Parameters:

transformedKey - the key to check

Returns:

true if the key should be included and false if not

canSee

protected boolean canSee(Key key)

Indicates whether or not the user is able to see key. If the user has not supplied authorizations, or the iterator is not in the scan scope, then this method simply returns true. Otherwise, key's column visibility is tested against the user-supplied authorizations, and the test result is returned. For performance, the test results are cached so that the same visibility is not tested multiple times.

Parameters:

key - the key to test

Returns:

true if the key is visible or iterator is not scanning, and false if not

canSeeColumnFamily

protected boolean canSeeColumnFamily(Key key)

Indicates whether or not key can be seen, according to the fetched column families for this iterator.

Parameters:

key - the key whose column family is to be tested

Returns:

true if key's column family is one of those fetched in the set passed to our seek(Range, Collection, boolean) method

computeReseekRange

protected Range computeReseekRange(Range range)

Possibly expand range to include everything for the key prefix we are working with. That is, if our prefix is ROW_COLFAM, then we need to expand the range so we're sure to include all entries having the same row and column family as the start/end of the range.

Parameters:

range - the range to expand

Returns:

the modified range

isSetAfterPart

protected boolean isSetAfterPart(Key key,
                     PartialKey part)

Indicates whether or not any part of key excluding part is set. For example, if part is ROW_COLFAM_COLQUAL, then this method determines whether or not the column visibility, timestamp, or delete flag is set on key.

Parameters:

key - the key to check

part - the part of the key that doesn't need to be checked (everything after does)

Returns:

true if anything after part is set on key, and false if not

copyPartialKey

protected Key copyPartialKey(Key key,
                 PartialKey part)

Creates a copy of key, copying only the parts of the key specified in part. For example, if part is ROW_COLFAM_COLQUAL, then this method would copy the row, column family, and column qualifier from key into a new key.

Parameters:

key - the key to copy

part - the parts of key to copy

Returns:

the new key containing part of key

replaceColumnFamily

protected Key replaceColumnFamily(Key originalKey,
                      org.apache.hadoop.io.Text newColFam)

Make a new key with all parts (including delete flag) coming from originalKey but use newColFam as the column family.

replaceColumnQualifier

protected Key replaceColumnQualifier(Key originalKey,
                         org.apache.hadoop.io.Text newColQual)

Make a new key with all parts (including delete flag) coming from originalKey but use newColQual as the column qualifier.

replaceColumnVisibility

protected Key replaceColumnVisibility(Key originalKey,
                          org.apache.hadoop.io.Text newColVis)

Make a new key with all parts (including delete flag) coming from originalKey but use newColVis as the column visibility.

replaceKeyParts

protected Key replaceKeyParts(Key originalKey,
                  org.apache.hadoop.io.Text newColFam,
                  org.apache.hadoop.io.Text newColQual,
                  org.apache.hadoop.io.Text newColVis)

Make a new key with a column family, column qualifier, and column visibility. Copy the rest of the parts of the key (including delete flag) from originalKey.

replaceKeyParts

protected Key replaceKeyParts(Key originalKey,
                  org.apache.hadoop.io.Text newColQual,
                  org.apache.hadoop.io.Text newColVis)

Make a new key with a column qualifier, and column visibility. Copy the rest of the parts of the key (including delete flag) from originalKey.

untransformColumnFamilies

protected Collection<ByteSequence> untransformColumnFamilies(Collection<ByteSequence> columnFamilies)

Reverses the transformation applied to column families that are fetched at seek time. If this iterator is transforming column families, then this method should be overridden to reverse the transformation on the supplied collection of column families. This is necessary since the fetch/seek will be performed in the transformed space, but when passing the column family set on to the source, the column families need to be in the untransformed space.

Parameters:

columnFamilies - the column families that have been fetched at seek time

Returns:

the untransformed column families that would transform info columnFamilies

getKeyPrefix

protected abstract PartialKey getKeyPrefix()

Indicates the prefix of keys that will be transformed by this iterator. In other words, this is the part of the key that will not be transformed by this iterator. For example, if this method returns ROW_COLFAM, then transformKeys() may be changing the column qualifier, column visibility, or timestamp, but it won't be changing the row or column family.

Returns:

the part of the key this iterator is not transforming

transformRange

protected abstract void transformRange(SortedKeyValueIterator<Key,Value> input,
                  TransformingIterator.KVBuffer output)
                                throws IOException

Transforms input. This method must not change the row part of the key, and must only change the parts of the key after the return value of getKeyPrefix(). Implementors must also remember to copy the delete flag from originalKey onto the new key. Or, implementors should use one of the helper methods to produce the new key. See any of the replaceKeyParts methods.

Parameters:

input - An iterator over a group of keys with the same prefix. This iterator provides an efficient view, bounded by the prefix, of the underlying iterator and can not be seeked.

output - An output buffer that holds transformed key values. All key values added to the buffer must have the same prefix as the input keys.

Throws:

IOException

See Also:

replaceColumnFamily(Key, Text), replaceColumnQualifier(Key, Text), replaceColumnVisibility(Key, Text), replaceKeyParts(Key, Text, Text), replaceKeyParts(Key, Text, Text, Text)

setAuthorizations

public static void setAuthorizations(IteratorSetting config,
                     Authorizations auths)

Configure authorizations used for post transformation filtering.

setMaxBufferSize

public static void setMaxBufferSize(IteratorSetting config,
                    long maxBufferSize)

Configure the maximum amount of memory that can be used for transformation. If this memory is exceeded an exception will be thrown.

Parameters:

maxBufferSize - size in bytes