org.deeplearning4j.iterator

Class CnnSentenceDataSetIterator

java.lang.Object

org.deeplearning4j.iterator.CnnSentenceDataSetIterator

All Implemented Interfaces:

Serializable, Iterator<org.nd4j.linalg.dataset.DataSet>, org.nd4j.linalg.dataset.api.iterator.DataSetIterator

public class CnnSentenceDataSetIterator
extends Object
implements org.nd4j.linalg.dataset.api.iterator.DataSetIterator

A DataSetIterator that provides data for training a CNN sentence classification models (though can of course be used for general documents, not just sentences. The iterator handles conversion of sentences to training data for CNNs, where each word is encoded using the word vector from the specified WordVectors (i.e., word2vec etc) model.
Labels are encoded using a one-hot representation and are 2d - i.e., are intended to be used with a model that utilizes global pooling.

Specifically:
- Features have shape [minibatchSize, 1, maxSentenceLength, wordVectorSize] OR [minibatchSize, 1, wordVectorSize, maxSentenceLength] depending on the configuration (for sentencesAlongHeight = true/false respectively)
- Labels are a 2d array with shape [minibatchSize, numLabels].
Sentences and labels are provided by a LabeledSentenceProvider - different implementations of this provide different ways of loading sentences/documents with labels - for example, from files, etc.

Note: With regard to labels to class index assignment, they are sorted alphabetically. To get the assigment/mapping, use getLabels() or getLabelClassMap()

Author:

Alex Black

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	CnnSentenceDataSetIterator.Builder
static class	CnnSentenceDataSetIterator.Format Format of features: CNN1D: For use with 1d convolution layers: Shape [minibatch, vectorSize, sentenceLength] CNN2D: For use with 2d convolution layers: Shape [minibatch, 1, vectorSize, sentenceLength] or [minibatch, 1, sentenceLength, vectorSize], depending on the setting for 'sentencesAlongHeight' configuration.
static class	CnnSentenceDataSetIterator.UnknownWordHandling

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	CnnSentenceDataSetIterator(CnnSentenceDataSetIterator.Builder builder)

Method Summary

Modifier and Type	Method and Description
boolean	asyncSupported()
int	batch()
Map<String,Integer>	getLabelClassMap()
List<String>	getLabels()
org.nd4j.linalg.dataset.api.DataSetPreProcessor	getPreProcessor()
boolean	hasNext()
int	inputColumns()
org.nd4j.linalg.api.ndarray.INDArray	loadSingleSentence(String sentence) Generally used post training time to load a single sentence for predictions
org.nd4j.linalg.dataset.DataSet	next()
org.nd4j.linalg.dataset.DataSet	next(int num)
void	remove()
void	reset()
boolean	resetSupported()
void	setPreProcessor(org.nd4j.linalg.dataset.api.DataSetPreProcessor preProcessor)
int	totalOutcomes()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.util.Iterator

forEachRemaining

Constructor Detail

CnnSentenceDataSetIterator

protected CnnSentenceDataSetIterator(CnnSentenceDataSetIterator.Builder builder)

Method Detail

loadSingleSentence

public org.nd4j.linalg.api.ndarray.INDArray loadSingleSentence(String sentence)

Generally used post training time to load a single sentence for predictions

getLabelClassMap

public Map<String,Integer> getLabelClassMap()

getLabels

public List<String> getLabels()

Specified by:

getLabels in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

hasNext

public boolean hasNext()

Specified by:

hasNext in interface Iterator<org.nd4j.linalg.dataset.DataSet>

next

public org.nd4j.linalg.dataset.DataSet next()

Specified by:

next in interface Iterator<org.nd4j.linalg.dataset.DataSet>

next

public org.nd4j.linalg.dataset.DataSet next(int num)

Specified by:

next in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

inputColumns

public int inputColumns()

Specified by:

inputColumns in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

totalOutcomes

public int totalOutcomes()

Specified by:

totalOutcomes in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

resetSupported

public boolean resetSupported()

Specified by:

resetSupported in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

asyncSupported

public boolean asyncSupported()

Specified by:

asyncSupported in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

reset

public void reset()

Specified by:

reset in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

batch

public int batch()

Specified by:

batch in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

setPreProcessor

public void setPreProcessor(org.nd4j.linalg.dataset.api.DataSetPreProcessor preProcessor)

Specified by:

setPreProcessor in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

getPreProcessor

public org.nd4j.linalg.dataset.api.DataSetPreProcessor getPreProcessor()

Specified by:

getPreProcessor in interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

remove

public void remove()

Specified by:

remove in interface Iterator<org.nd4j.linalg.dataset.DataSet>