org.deeplearning4j.text.invertedindex

Class LuceneInvertedIndex

java.lang.Object

org.deeplearning4j.text.invertedindex.LuceneInvertedIndex

All Implemented Interfaces:

Serializable, Iterator<List<VocabWord>>, org.apache.lucene.index.IndexReader.ReaderClosedListener, InvertedIndex

public class LuceneInvertedIndex
extends Object
implements InvertedIndex, org.apache.lucene.index.IndexReader.ReaderClosedListener, Iterator<List<VocabWord>>

Lucene based inverted index

Author:

Adam Gibson

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	LuceneInvertedIndex.Builder

Field Summary

Fields

Modifier and Type	Field and Description
static String	DEFAULT_INDEX_DIR
static String	INDEX_PATH
static String	WORD_FIELD

Constructor Summary

Constructors

Constructor and Description
LuceneInvertedIndex(VocabCache vocabCache, boolean cache)
LuceneInvertedIndex(VocabCache vocabCache, boolean cache, String indexPath)

Method Summary

Modifier and Type	Method and Description
void	addWordsToDoc(int doc, List<VocabWord> words) Adds words to the given document
void	addWordToDoc(int doc, VocabWord word) Add word to a document
int[]	allDocs() Returns a list of all documents
int	batchSize() For word vectors, this is the batch size for which to train on
List<VocabWord>	document(int index) Returns a list of words for a document
int[]	documents(VocabWord vocabWord) Returns the list of documents a vocab word is in
void	eachDoc(com.google.common.base.Function<List<VocabWord>,Void> func, ExecutorService exec) Iterate over each document
void	finish() Finishes saving data
boolean	hasNext()
Iterator<List<VocabWord>>	miniBatches() Iterates over mini batches
List<VocabWord>	next()
int	numDocuments() Returns the number of documents
void	onClose(org.apache.lucene.index.IndexReader reader)
void	remove()
double	sample() Sampling for creating mini batches
int	totalWords() Total number of words in the index

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.util.Iterator

forEachRemaining

Field Detail

WORD_FIELD

public static final String WORD_FIELD

See Also:

Constant Field Values

INDEX_PATH

public static final String INDEX_PATH

See Also:

Constant Field Values

DEFAULT_INDEX_DIR

public static final String DEFAULT_INDEX_DIR

See Also:

Constant Field Values

Constructor Detail

LuceneInvertedIndex

public LuceneInvertedIndex(VocabCache vocabCache,
                           boolean cache)

LuceneInvertedIndex

public LuceneInvertedIndex(VocabCache vocabCache,
                           boolean cache,
                           String indexPath)

Method Detail

sample

public double sample()

Description copied from interface: InvertedIndex

Sampling for creating mini batches

Specified by:

sample in interface InvertedIndex

Returns:

the sampling for mini batches

miniBatches

public Iterator<List<VocabWord>> miniBatches()

Description copied from interface: InvertedIndex

Iterates over mini batches

Specified by:

miniBatches in interface InvertedIndex

Returns:

the mini batches created by this vectorizer

document

public List<VocabWord> document(int index)

Description copied from interface: InvertedIndex

Returns a list of words for a document

Specified by:

document in interface InvertedIndex

Returns:

documents

public int[] documents(VocabWord vocabWord)

Description copied from interface: InvertedIndex

Returns the list of documents a vocab word is in

Specified by:

documents in interface InvertedIndex

Parameters:

vocabWord - the vocab word to get documents for

Returns:

the documents for a vocab word

numDocuments

public int numDocuments()

Description copied from interface: InvertedIndex

Returns the number of documents

Specified by:

numDocuments in interface InvertedIndex

Returns:

allDocs

public int[] allDocs()

Description copied from interface: InvertedIndex

Returns a list of all documents

Specified by:

allDocs in interface InvertedIndex

Returns:

the list of all documents

addWordToDoc

public void addWordToDoc(int doc,
                         VocabWord word)

Description copied from interface: InvertedIndex

Add word to a document

Specified by:

addWordToDoc in interface InvertedIndex

Parameters:

doc - the document to add to

word - the word to add

addWordsToDoc

public void addWordsToDoc(int doc,
                          List<VocabWord> words)

Description copied from interface: InvertedIndex

Adds words to the given document

Specified by:

addWordsToDoc in interface InvertedIndex

Parameters:

doc - the document to add to

words - the words to add

finish

public void finish()

Description copied from interface: InvertedIndex

Finishes saving data

Specified by:

finish in interface InvertedIndex

totalWords

public int totalWords()

Description copied from interface: InvertedIndex

Total number of words in the index

Specified by:

totalWords in interface InvertedIndex

Returns:

the total number of words in the index

batchSize

public int batchSize()

Description copied from interface: InvertedIndex

For word vectors, this is the batch size for which to train on

Specified by:

batchSize in interface InvertedIndex

Returns:

the batch size for which to train on

eachDoc

public void eachDoc(com.google.common.base.Function<List<VocabWord>,Void> func,
                    ExecutorService exec)

Description copied from interface: InvertedIndex

Iterate over each document

Specified by:

eachDoc in interface InvertedIndex

Parameters:

func - the function to apply

exec - exectuor service for execution

onClose

public void onClose(org.apache.lucene.index.IndexReader reader)

Specified by:

onClose in interface org.apache.lucene.index.IndexReader.ReaderClosedListener

hasNext

public boolean hasNext()

Specified by:

hasNext in interface Iterator<List<VocabWord>>

next

public List<VocabWord> next()

Specified by:

next in interface Iterator<List<VocabWord>>

remove

public void remove()

Specified by:

remove in interface Iterator<List<VocabWord>>