Package org.deeplearning4j.iterator

Class BertIterator

java.lang.Object

org.deeplearning4j.iterator.BertIterator

All Implemented Interfaces:

Serializable, Iterator<org.nd4j.linalg.dataset.api.MultiDataSet>, org.nd4j.linalg.dataset.api.iterator.MultiDataSetIterator

public class BertIterator
extends Object
implements org.nd4j.linalg.dataset.api.iterator.MultiDataSetIterator

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	BertIterator.Builder
static class	BertIterator.FeatureArrays
static class	BertIterator.LengthHandling
static class	BertIterator.Task
static class	BertIterator.UnsupervisedLabelFormat

Field Summary

Fields

Modifier and Type	Field	Description
protected String	appendToken
protected BertIterator.FeatureArrays	featureArrays
protected BertIterator.LengthHandling	lengthHandling
protected BertSequenceMasker	masker
protected String	maskToken
protected int	maxTokens
protected int	minibatchSize
protected boolean	padMinibatches
protected String	prependToken
protected org.nd4j.linalg.dataset.api.MultiDataSetPreProcessor	preProcessor
protected LabeledPairSentenceProvider	sentencePairProvider
protected LabeledSentenceProvider	sentenceProvider
protected BertIterator.Task	task
protected TokenizerFactory	tokenizerFactory
protected BertIterator.UnsupervisedLabelFormat	unsupervisedLabelFormat
protected List<String>	vocabKeysAsList
protected Map<String,Integer>	vocabMap

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	BertIterator(BertIterator.Builder b)

Method Summary

Modifier and Type	Method	Description
boolean	asyncSupported()
static BertIterator.Builder	builder()
org.nd4j.common.primitives.Pair<org.nd4j.linalg.api.ndarray.INDArray[],org.nd4j.linalg.api.ndarray.INDArray[]>	featurizeSentencePairs(List<org.nd4j.common.primitives.Pair<String,String>> listOnlySentencePairs)	For use during inference.
org.nd4j.common.primitives.Pair<org.nd4j.linalg.api.ndarray.INDArray[],org.nd4j.linalg.api.ndarray.INDArray[]>	featurizeSentences(List<String> listOnlySentences)	For use during inference.
boolean	hasNext()
org.nd4j.linalg.dataset.api.MultiDataSet	next()
org.nd4j.linalg.dataset.api.MultiDataSet	next(int num)
void	remove()
void	reset()
boolean	resetSupported()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.util.Iterator

forEachRemaining

Methods inherited from interface org.nd4j.linalg.dataset.api.iterator.MultiDataSetIterator

getPreProcessor, setPreProcessor

Field Detail

task

protected BertIterator.Task task

tokenizerFactory

protected TokenizerFactory tokenizerFactory

maxTokens

protected int maxTokens

minibatchSize

protected int minibatchSize

padMinibatches

protected boolean padMinibatches

preProcessor

protected org.nd4j.linalg.dataset.api.MultiDataSetPreProcessor preProcessor

sentenceProvider

protected LabeledSentenceProvider sentenceProvider

sentencePairProvider

protected LabeledPairSentenceProvider sentencePairProvider

lengthHandling

protected BertIterator.LengthHandling lengthHandling

featureArrays

protected BertIterator.FeatureArrays featureArrays

vocabMap

protected Map<String,Integer> vocabMap

masker

protected BertSequenceMasker masker

unsupervisedLabelFormat

protected BertIterator.UnsupervisedLabelFormat unsupervisedLabelFormat

maskToken

protected String maskToken

prependToken

protected String prependToken

appendToken

protected String appendToken

vocabKeysAsList

protected List<String> vocabKeysAsList

Constructor Detail

BertIterator

protected BertIterator(BertIterator.Builder b)

Method Detail

hasNext

public boolean hasNext()

Specified by:

hasNext in interface Iterator<org.nd4j.linalg.dataset.api.MultiDataSet>

next

public org.nd4j.linalg.dataset.api.MultiDataSet next()

Specified by:

next in interface Iterator<org.nd4j.linalg.dataset.api.MultiDataSet>

remove

public void remove()

Specified by:

remove in interface Iterator<org.nd4j.linalg.dataset.api.MultiDataSet>

next

public org.nd4j.linalg.dataset.api.MultiDataSet next(int num)

Specified by:

next in interface org.nd4j.linalg.dataset.api.iterator.MultiDataSetIterator

featurizeSentences

public org.nd4j.common.primitives.Pair<org.nd4j.linalg.api.ndarray.INDArray[],org.nd4j.linalg.api.ndarray.INDArray[]> featurizeSentences(List<String> listOnlySentences)

For use during inference. Will convert a given list of sentences to features and feature masks as appropriate.

Parameters:

listOnlySentences -

Returns:

Pair of INDArrays[], first element is feature arrays and the second is the masks array

featurizeSentencePairs

public org.nd4j.common.primitives.Pair<org.nd4j.linalg.api.ndarray.INDArray[],org.nd4j.linalg.api.ndarray.INDArray[]> featurizeSentencePairs(List<org.nd4j.common.primitives.Pair<String,String>> listOnlySentencePairs)

For use during inference. Will convert a given pair of a list of sentences to features and feature masks as appropriate.

Parameters:

listOnlySentencePairs -

Returns:

Pair of INDArrays[], first element is feature arrays and the second is the masks array

resetSupported

public boolean resetSupported()

Specified by:

resetSupported in interface org.nd4j.linalg.dataset.api.iterator.MultiDataSetIterator

asyncSupported

public boolean asyncSupported()

Specified by:

asyncSupported in interface org.nd4j.linalg.dataset.api.iterator.MultiDataSetIterator

reset

public void reset()

Specified by:

reset in interface org.nd4j.linalg.dataset.api.iterator.MultiDataSetIterator

builder

public static BertIterator.Builder builder()