org.deeplearning4j.bagofwords.vectorizer

Class TfidfVectorizer

java.lang.Object

org.deeplearning4j.bagofwords.vectorizer.BaseTextVectorizer

org.deeplearning4j.bagofwords.vectorizer.TfidfVectorizer

All Implemented Interfaces:

Serializable, TextVectorizer, Vectorizer

public class TfidfVectorizer
extends BaseTextVectorizer
implements Serializable

Turns a applyTransformToDestination of documents in to a tfidf bag of words

Author:

Adam Gibson

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	TfidfVectorizer.Builder

Field Summary

Fields inherited from class org.deeplearning4j.bagofwords.vectorizer.BaseTextVectorizer

batchSize, cache, cleanup, docIter, index, labels, labelSentenceIter, minWordFrequency, numWordsEncountered, sample, sentenceIterator, stem, stopWords, tokenizerFactory, trainingSystem

Constructor Summary

Constructors

Modifier	Constructor and Description
	TfidfVectorizer()
protected	TfidfVectorizer(VocabCache cache, TokenizerFactory tokenizerFactory, List<String> stopWords, int minWordFrequency, DocumentIterator docIter, SentenceIterator sentenceIterator, List<String> labels, InvertedIndex index, int batchSize, double sample, boolean stem, boolean cleanup)

Method Summary

Modifier and Type	Method and Description
org.nd4j.linalg.api.ndarray.INDArray	transform(String text) Transforms the matrix
org.nd4j.linalg.dataset.DataSet	vectorize()
org.nd4j.linalg.dataset.DataSet	vectorize(File input, String label)
org.nd4j.linalg.dataset.DataSet	vectorize(InputStream is, String label) Text coming from an input stream considered as one document
org.nd4j.linalg.dataset.DataSet	vectorize(String text, String label) Vectorizes the passed in text treating it as one document

Methods inherited from class org.deeplearning4j.bagofwords.vectorizer.BaseTextVectorizer

batchSize, fit, getCache, getDocIter, getMinWordFrequency, getSentenceIterator, getStopWords, getTokenizerFactory, index, numWordsEncountered, sample, setCache, setDocIter, setMinWordFrequency, setSentenceIterator, setStopWords, setTokenizerFactory, vocab

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TfidfVectorizer

public TfidfVectorizer()

TfidfVectorizer

protected TfidfVectorizer(VocabCache cache,
                          TokenizerFactory tokenizerFactory,
                          List<String> stopWords,
                          int minWordFrequency,
                          DocumentIterator docIter,
                          SentenceIterator sentenceIterator,
                          List<String> labels,
                          InvertedIndex index,
                          int batchSize,
                          double sample,
                          boolean stem,
                          boolean cleanup)

Method Detail

vectorize

public org.nd4j.linalg.dataset.DataSet vectorize(InputStream is,
                                                 String label)

Description copied from interface: TextVectorizer

Text coming from an input stream considered as one document

Specified by:

vectorize in interface TextVectorizer

Parameters:

is - the input stream to read from

label - the label to assign

Returns:

a dataset with a applyTransformToDestination of weights(relative to impl; could be word counts or tfidf scores)

vectorize

public org.nd4j.linalg.dataset.DataSet vectorize(String text,
                                                 String label)

Description copied from interface: TextVectorizer

Vectorizes the passed in text treating it as one document

Specified by:

vectorize in interface TextVectorizer

Parameters:

text - the text to vectorize

label - the label of the text

Returns:

a dataset with a applyTransformToDestination of weights(relative to impl; could be word counts or tfidf scores)

vectorize

public org.nd4j.linalg.dataset.DataSet vectorize(File input,
                                                 String label)

Specified by:

vectorize in interface TextVectorizer

Parameters:

input - the text to vectorize

Returns:

a dataset with a applyTransformToDestination of weights(relative to impl; could be word counts or tfidf scores)

transform

public org.nd4j.linalg.api.ndarray.INDArray transform(String text)

Transforms the matrix

Specified by:

transform in interface TextVectorizer

Parameters:

text -

Returns:

vectorize

public org.nd4j.linalg.dataset.DataSet vectorize()

Specified by:

vectorize in interface Vectorizer