Class IndexedDocIterator
- java.lang.Object
-
- org.apache.accumulo.core.iterators.user.IntersectingIterator
-
- org.apache.accumulo.core.iterators.user.IndexedDocIterator
-
- All Implemented Interfaces:
SortedKeyValueIterator<Key,Value>
,YieldingKeyValueIterator<Key,Value>
public class IndexedDocIterator extends IntersectingIterator
This iterator facilitates document-partitioned indexing. It is an example of extending the IntersectingIterator to customize the placement of the term and docID. As with the IntersectingIterator, documents are grouped together and indexed into a single row of an Accumulo table. This allows a tablet server to perform boolean AND operations on terms in the index. This iterator also stores the document contents in a separate column family in the same row so that the full document can be returned with each query. The table structure should have the following form: row: shardID, colfam: docColf\0doctype, colqual: docID, value: doc row: shardID, colfam: indexColf, colqual: term\0doctype\0docID\0info, value: (empty) When you configure this iterator with a set of terms, it will return only the docIDs and docs that appear with all of the specified terms. The result will have the following form: row: shardID, colfam: indexColf, colqual: doctype\0docID\0info, value: doc This iterator is commonly used with BatchScanner or AccumuloInputFormat, to parallelize the search over all shardIDs.
-
-
Nested Class Summary
-
Nested classes/interfaces inherited from class org.apache.accumulo.core.iterators.user.IntersectingIterator
IntersectingIterator.TermSource
-
-
Field Summary
Fields Modifier and Type Field Description static org.apache.hadoop.io.Text
DEFAULT_DOC_COLF
static org.apache.hadoop.io.Text
DEFAULT_INDEX_COLF
SortedKeyValueIterator<Key,Value>
docSource
-
Fields inherited from class org.apache.accumulo.core.iterators.user.IntersectingIterator
currentDocID, currentPartition, nullText, sources, topKey, value
-
-
Constructor Summary
Constructors Constructor Description IndexedDocIterator()
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description protected void
advanceToIntersection()
protected Key
buildDocKey()
protected Key
buildKey(org.apache.hadoop.io.Text partition, org.apache.hadoop.io.Text term)
protected Key
buildKey(org.apache.hadoop.io.Text partition, org.apache.hadoop.io.Text term, org.apache.hadoop.io.Text docID)
SortedKeyValueIterator<Key,Value>
deepCopy(IteratorEnvironment env)
Creates a deep copy of this iterator as though seek had not yet been called.protected org.apache.hadoop.io.Text
getDocID(Key key)
protected org.apache.hadoop.io.Text
getTerm(Key key)
void
init(SortedKeyValueIterator<Key,Value> source, Map<String,String> options, IteratorEnvironment env)
Initializes the iterator.static org.apache.hadoop.io.Text
parseDocID(Key key)
void
seek(Range range, Collection<ByteSequence> seekColumnFamilies, boolean inclusive)
Seeks to the first key in the Range, restricting the resulting K,V pairs to those with the specified columns.static void
setColfs(IteratorSetting is, String indexColf, String docColfPrefix)
A convenience method for setting the index column family and document column family prefix.static void
setDocColfPrefix(IteratorSetting is, String docColfPrefix)
A convenience method for setting the document column family prefix.static void
setIndexColf(IteratorSetting is, String indexColf)
A convenience method for setting the index column family.-
Methods inherited from class org.apache.accumulo.core.iterators.user.IntersectingIterator
buildFollowingPartitionKey, decodeBooleans, decodeColumns, encodeBooleans, encodeColumns, getPartition, getTopKey, getTopValue, hasTop, next, setColumnFamilies, setColumnFamilies, stringTopKey
-
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
-
Methods inherited from interface org.apache.accumulo.core.iterators.YieldingKeyValueIterator
enableYielding
-
-
-
-
Field Detail
-
DEFAULT_INDEX_COLF
public static final org.apache.hadoop.io.Text DEFAULT_INDEX_COLF
-
DEFAULT_DOC_COLF
public static final org.apache.hadoop.io.Text DEFAULT_DOC_COLF
-
docSource
public SortedKeyValueIterator<Key,Value> docSource
-
-
Method Detail
-
buildKey
protected Key buildKey(org.apache.hadoop.io.Text partition, org.apache.hadoop.io.Text term, org.apache.hadoop.io.Text docID)
- Overrides:
buildKey
in classIntersectingIterator
-
buildKey
protected Key buildKey(org.apache.hadoop.io.Text partition, org.apache.hadoop.io.Text term)
- Overrides:
buildKey
in classIntersectingIterator
-
getDocID
protected org.apache.hadoop.io.Text getDocID(Key key)
- Overrides:
getDocID
in classIntersectingIterator
-
parseDocID
public static org.apache.hadoop.io.Text parseDocID(Key key)
-
getTerm
protected org.apache.hadoop.io.Text getTerm(Key key)
- Overrides:
getTerm
in classIntersectingIterator
-
init
public void init(SortedKeyValueIterator<Key,Value> source, Map<String,String> options, IteratorEnvironment env) throws IOException
Description copied from interface:SortedKeyValueIterator
Initializes the iterator. Data should not be read from the source in this method.- Specified by:
init
in interfaceSortedKeyValueIterator<Key,Value>
- Overrides:
init
in classIntersectingIterator
- Parameters:
source
-SortedKeyValueIterator
source to read data from.options
-Map
map of string option names to option values.env
-IteratorEnvironment
environment in which iterator is being run.- Throws:
IOException
- unused.
-
deepCopy
public SortedKeyValueIterator<Key,Value> deepCopy(IteratorEnvironment env)
Description copied from interface:SortedKeyValueIterator
Creates a deep copy of this iterator as though seek had not yet been called. init should be called on an iterator before deepCopy is called. init should not need to be called on the copy that is returned by deepCopy; that is, when necessary init should be called in the deepCopy method on the iterator it returns. The behavior is unspecified if init is called after deepCopy either on the original or the copy. A proper implementation would call deepCopy on the source.- Specified by:
deepCopy
in interfaceSortedKeyValueIterator<Key,Value>
- Overrides:
deepCopy
in classIntersectingIterator
- Parameters:
env
-IteratorEnvironment
environment in which iterator is being run.- Returns:
SortedKeyValueIterator
a copy of this iterator (with the same source and settings).
-
seek
public void seek(Range range, Collection<ByteSequence> seekColumnFamilies, boolean inclusive) throws IOException
Description copied from interface:SortedKeyValueIterator
Seeks to the first key in the Range, restricting the resulting K,V pairs to those with the specified columns. An iterator does not have to stop at the end of the range. The whole range is provided so that iterators can make optimizations. Seek may be called multiple times with different parameters afterSortedKeyValueIterator.init(org.apache.accumulo.core.iterators.SortedKeyValueIterator<K, V>, java.util.Map<java.lang.String, java.lang.String>, org.apache.accumulo.core.iterators.IteratorEnvironment)
is called. Iterators that examine groups of adjacent key/value pairs (e.g. rows) to determine their top key and value should be sure that they properly handle a seek to a key in the middle of such a group (e.g. the middle of a row). Even if the client always seeks to a range containing an entire group (a,c), the tablet server could send back a batch of entries corresponding to (a,b], then reseek the iterator to range (b,c) when the scan is continued.columnFamilies
is used, at the lowest level, to determine which data blocks inside of an RFile need to be opened for this iterator. This set of data blocks is also the set of locality groups defined for the given table. If no columnFamilies are provided, the data blocks for all locality groups inside of the correct RFile will be opened and seeked in an attempt to find the correct start key, regardless of the startKey in therange
. In an Accumulo instance in which multiple locality groups exist for a table, it is important to ensure thatcolumnFamilies
is properly set to the minimum required column families to ensure that data from separate locality groups is not inadvertently read.- Specified by:
seek
in interfaceSortedKeyValueIterator<Key,Value>
- Overrides:
seek
in classIntersectingIterator
- Parameters:
range
-Range
of keys to iterate over.seekColumnFamilies
-Collection
of column families to include or exclude.inclusive
-boolean
that indicates whether to include (true) or exclude (false) column families.- Throws:
IOException
- if an I/O error occurs.
-
advanceToIntersection
protected void advanceToIntersection() throws IOException
- Overrides:
advanceToIntersection
in classIntersectingIterator
- Throws:
IOException
-
buildDocKey
protected Key buildDocKey()
-
setIndexColf
public static void setIndexColf(IteratorSetting is, String indexColf)
A convenience method for setting the index column family.- Parameters:
is
- IteratorSetting object to configure.indexColf
- the index column family
-
setDocColfPrefix
public static void setDocColfPrefix(IteratorSetting is, String docColfPrefix)
A convenience method for setting the document column family prefix.- Parameters:
is
- IteratorSetting object to configure.docColfPrefix
- the prefix of the document column family (colf will be of the form docColfPrefix\0doctype)
-
setColfs
public static void setColfs(IteratorSetting is, String indexColf, String docColfPrefix)
A convenience method for setting the index column family and document column family prefix.- Parameters:
is
- IteratorSetting object to configure.indexColf
- the index column familydocColfPrefix
- the prefix of the document column family (colf will be of the form docColfPrefix\0doctype)
-
-