org.deeplearning4j.text.invertedindex

Class LuceneInvertedIndex<T extends SequenceElement>

java.lang.Object

org.deeplearning4j.text.invertedindex.LuceneInvertedIndex<T>

All Implemented Interfaces:

Serializable, Iterator<List<T>>, org.apache.lucene.index.IndexReader.ReaderClosedListener, InvertedIndex<T>

public class LuceneInvertedIndex<T extends SequenceElement>
extends Object
implements InvertedIndex<T>, org.apache.lucene.index.IndexReader.ReaderClosedListener, Iterator<List<T>>

Lucene based inverted index

Author:

Adam Gibson

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	LuceneInvertedIndex.BatchDocIter
static class	LuceneInvertedIndex.Builder<T extends SequenceElement>
class	LuceneInvertedIndex.DocIter

Field Summary

Fields

Modifier and Type	Field and Description
static String	DEFAULT_INDEX_DIR
static String	INDEX_PATH
static String	LABEL
static String	WORD_FIELD

Constructor Summary

Constructors

Constructor and Description
LuceneInvertedIndex(VocabCache vocabCache)
LuceneInvertedIndex(VocabCache vocabCache, boolean cache)
LuceneInvertedIndex(VocabCache vocabCache, boolean cache, String indexPath)

Method Summary

Modifier and Type	Method and Description
void	addLabelForDoc(int doc, String label) Adds words to the given document
void	addLabelForDoc(int doc, T word) Add word to a document
void	addLabelsForDoc(int doc, Collection<String> label) Adds words to the given document
void	addLabelsForDoc(int doc, List<T> label) Add word to a document
void	addWordsToDoc(int doc, List<T> words) Adds words to the given document
void	addWordsToDoc(int doc, List<T> words, Collection<String> label) Adds words to the given document
void	addWordsToDoc(int doc, List<T> words, String label) Adds words to the given document
void	addWordsToDoc(int doc, List<T> words, T label) Adds words to the given document
void	addWordsToDocVocabWord(int doc, List<T> words, Collection<T> label) Adds words to the given document
void	addWordToDoc(int doc, T word) Add word to a document
int[]	allDocs() Returns a list of all documents
Iterator<List<List<T>>>	batchIter(int batchSize) Iterate over batches
int	batchSize() For word vectors, this is the batch size for which to train on
void	cleanup() Cleanup any resources used
Iterator<List<T>>	docs() Iterate over documents
List<T>	document(int index) Returns a list of words for a document
int[]	documents(T vocabWord) Returns the list of documents a vocab word is in
Pair<List<T>,String>	documentWithLabel(int index) Returns a list of words for a document and the associated label
Pair<List<T>,Collection<String>>	documentWithLabels(int index) Returns a list of words associated with the document and the associated labels
void	eachDoc(com.google.common.base.Function<List<T>,Void> func, ExecutorService exec) Iterate over each document
void	eachDocWithLabel(com.google.common.base.Function<Pair<List<T>,String>,Void> func, ExecutorService exec) Iterate over each document with a label
void	eachDocWithLabels(com.google.common.base.Function<Pair<List<T>,Collection<String>>,Void> func, ExecutorService exec) Iterate over each document with a label
void	finish() Finishes saving data
boolean	hasNext()
Iterator<List<T>>	miniBatches() Iterates over mini batches
List<T>	next()
int	numDocuments() Returns the number of documents
void	onClose(org.apache.lucene.index.IndexReader reader)
void	remove()
double	sample() Sampling for creating mini batches
long	totalWords() Total number of words in the index
void	unlock() Unlock the index

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.util.Iterator

forEachRemaining

Field Detail

WORD_FIELD

public static final String WORD_FIELD

See Also:

Constant Field Values

LABEL

public static final String LABEL

See Also:

Constant Field Values

INDEX_PATH

public static final String INDEX_PATH

See Also:

Constant Field Values

DEFAULT_INDEX_DIR

public static final String DEFAULT_INDEX_DIR

See Also:

Constant Field Values

Constructor Detail

LuceneInvertedIndex

public LuceneInvertedIndex(VocabCache vocabCache,
                           boolean cache)

LuceneInvertedIndex

public LuceneInvertedIndex(VocabCache vocabCache,
                           boolean cache,
                           String indexPath)

LuceneInvertedIndex

public LuceneInvertedIndex(VocabCache vocabCache)

Method Detail

batchIter

public Iterator<List<List<T>>> batchIter(int batchSize)

Description copied from interface: InvertedIndex

Iterate over batches

Specified by:

batchIter in interface InvertedIndex<T extends SequenceElement>

Returns:

the batch size

docs

public Iterator<List<T>> docs()

Description copied from interface: InvertedIndex

Iterate over documents

Specified by:

docs in interface InvertedIndex<T extends SequenceElement>

Returns:

unlock

public void unlock()

Description copied from interface: InvertedIndex

Unlock the index

Specified by:

unlock in interface InvertedIndex<T extends SequenceElement>

cleanup

public void cleanup()

Description copied from interface: InvertedIndex

Cleanup any resources used

Specified by:

cleanup in interface InvertedIndex<T extends SequenceElement>

sample

public double sample()

Description copied from interface: InvertedIndex

Sampling for creating mini batches

Specified by:

sample in interface InvertedIndex<T extends SequenceElement>

Returns:

the sampling for mini batches

miniBatches

public Iterator<List<T>> miniBatches()

Description copied from interface: InvertedIndex

Iterates over mini batches

Specified by:

miniBatches in interface InvertedIndex<T extends SequenceElement>

Returns:

the mini batches created by this vectorizer

document

public List<T> document(int index)

Description copied from interface: InvertedIndex

Returns a list of words for a document

Specified by:

document in interface InvertedIndex<T extends SequenceElement>

Returns:

documents

public int[] documents(T vocabWord)

Description copied from interface: InvertedIndex

Returns the list of documents a vocab word is in

Specified by:

documents in interface InvertedIndex<T extends SequenceElement>

Parameters:

vocabWord - the vocab word to get documents for

Returns:

the documents for a vocab word

numDocuments

public int numDocuments()

Description copied from interface: InvertedIndex

Returns the number of documents

Specified by:

numDocuments in interface InvertedIndex<T extends SequenceElement>

Returns:

allDocs

public int[] allDocs()

Description copied from interface: InvertedIndex

Returns a list of all documents

Specified by:

allDocs in interface InvertedIndex<T extends SequenceElement>

Returns:

the list of all documents

addWordToDoc

public void addWordToDoc(int doc,
                         T word)

Description copied from interface: InvertedIndex

Add word to a document

Specified by:

addWordToDoc in interface InvertedIndex<T extends SequenceElement>

Parameters:

doc - the document to add to

word - the word to add

addWordsToDoc

public void addWordsToDoc(int doc,
                          List<T> words)

Description copied from interface: InvertedIndex

Adds words to the given document

Specified by:

addWordsToDoc in interface InvertedIndex<T extends SequenceElement>

Parameters:

doc - the document to add to

words - the words to add

documentWithLabel

public Pair<List<T>,String> documentWithLabel(int index)

Description copied from interface: InvertedIndex

Returns a list of words for a document and the associated label

Specified by:

documentWithLabel in interface InvertedIndex<T extends SequenceElement>

Returns:

documentWithLabels

public Pair<List<T>,Collection<String>> documentWithLabels(int index)

Description copied from interface: InvertedIndex

Returns a list of words associated with the document and the associated labels

Specified by:

documentWithLabels in interface InvertedIndex<T extends SequenceElement>

Returns:

addLabelForDoc

public void addLabelForDoc(int doc,
                           T word)

Description copied from interface: InvertedIndex

Add word to a document

Specified by:

addLabelForDoc in interface InvertedIndex<T extends SequenceElement>

Parameters:

doc - the document to add to

word - the word to add

addLabelForDoc

public void addLabelForDoc(int doc,
                           String label)

Description copied from interface: InvertedIndex

Adds words to the given document

Specified by:

addLabelForDoc in interface InvertedIndex<T extends SequenceElement>

Parameters:

doc - the document to add to

addWordsToDoc

public void addWordsToDoc(int doc,
                          List<T> words,
                          String label)

Description copied from interface: InvertedIndex

Adds words to the given document

Specified by:

addWordsToDoc in interface InvertedIndex<T extends SequenceElement>

Parameters:

doc - the document to add to

words - the words to add

label - the label for the document

addWordsToDoc

public void addWordsToDoc(int doc,
                          List<T> words,
                          T label)

Description copied from interface: InvertedIndex

Adds words to the given document

Specified by:

addWordsToDoc in interface InvertedIndex<T extends SequenceElement>

Parameters:

doc - the document to add to

words - the words to add

label - the label for the document

addLabelsForDoc

public void addLabelsForDoc(int doc,
                            List<T> label)

Description copied from interface: InvertedIndex

Add word to a document

Specified by:

addLabelsForDoc in interface InvertedIndex<T extends SequenceElement>

Parameters:

doc - the document to add to

label - the word to add

addLabelsForDoc

public void addLabelsForDoc(int doc,
                            Collection<String> label)

Description copied from interface: InvertedIndex

Adds words to the given document

Specified by:

addLabelsForDoc in interface InvertedIndex<T extends SequenceElement>

Parameters:

doc - the document to add to

label - the labels to add

addWordsToDoc

public void addWordsToDoc(int doc,
                          List<T> words,
                          Collection<String> label)

Description copied from interface: InvertedIndex

Adds words to the given document

Specified by:

addWordsToDoc in interface InvertedIndex<T extends SequenceElement>

Parameters:

doc - the document to add to

words - the words to add

label - the label for the document

addWordsToDocVocabWord

public void addWordsToDocVocabWord(int doc,
                                   List<T> words,
                                   Collection<T> label)

Description copied from interface: InvertedIndex

Adds words to the given document

Specified by:

addWordsToDocVocabWord in interface InvertedIndex<T extends SequenceElement>

Parameters:

doc - the document to add to

words - the words to add

label - the label for the document

finish

public void finish()

Description copied from interface: InvertedIndex

Finishes saving data

Specified by:

finish in interface InvertedIndex<T extends SequenceElement>

totalWords

public long totalWords()

Description copied from interface: InvertedIndex

Total number of words in the index

Specified by:

totalWords in interface InvertedIndex<T extends SequenceElement>

Returns:

the total number of words in the index

batchSize

public int batchSize()

Description copied from interface: InvertedIndex

For word vectors, this is the batch size for which to train on

Specified by:

batchSize in interface InvertedIndex<T extends SequenceElement>

Returns:

the batch size for which to train on

eachDocWithLabels

public void eachDocWithLabels(com.google.common.base.Function<Pair<List<T>,Collection<String>>,Void> func,
                              ExecutorService exec)

Description copied from interface: InvertedIndex

Iterate over each document with a label

Specified by:

eachDocWithLabels in interface InvertedIndex<T extends SequenceElement>

Parameters:

func - the function to apply

exec - executor service for execution

eachDocWithLabel

public void eachDocWithLabel(com.google.common.base.Function<Pair<List<T>,String>,Void> func,
                             ExecutorService exec)

Description copied from interface: InvertedIndex

Iterate over each document with a label

Specified by:

eachDocWithLabel in interface InvertedIndex<T extends SequenceElement>

Parameters:

func - the function to apply

exec - executor service for execution

eachDoc

public void eachDoc(com.google.common.base.Function<List<T>,Void> func,
                    ExecutorService exec)

Description copied from interface: InvertedIndex

Iterate over each document

Specified by:

eachDoc in interface InvertedIndex<T extends SequenceElement>

Parameters:

func - the function to apply

exec - executor service for execution

onClose

public void onClose(org.apache.lucene.index.IndexReader reader)

Specified by:

onClose in interface org.apache.lucene.index.IndexReader.ReaderClosedListener

hasNext

public boolean hasNext()

Specified by:

hasNext in interface Iterator<List<T extends SequenceElement>>

next

public List<T> next()

Specified by:

next in interface Iterator<List<T extends SequenceElement>>

remove

public void remove()

Specified by:

remove in interface Iterator<List<T extends SequenceElement>>