Uses of Interface
org.deeplearning4j.text.tokenization.tokenizer.TokenPreProcess

Packages that use TokenPreProcess

Package	Description
org.deeplearning4j.text.tokenization.tokenizer
org.deeplearning4j.text.tokenization.tokenizer.preprocessor
org.deeplearning4j.text.tokenization.tokenizerfactory

Uses of TokenPreProcess in org.deeplearning4j.text.tokenization.tokenizer

Methods in org.deeplearning4j.text.tokenization.tokenizer with parameters of type TokenPreProcess

Modifier and Type	Method	Description
void	BertWordPieceTokenizer.setTokenPreProcessor(TokenPreProcess tokenPreProcessor)
void	DefaultStreamTokenizer.setTokenPreProcessor(TokenPreProcess tokenPreProcessor)
void	DefaultTokenizer.setTokenPreProcessor(TokenPreProcess tokenPreProcessor)
void	NGramTokenizer.setTokenPreProcessor(TokenPreProcess tokenPreProcessor)
void	Tokenizer.setTokenPreProcessor(TokenPreProcess tokenPreProcessor)	Set the token pre process

Constructors in org.deeplearning4j.text.tokenization.tokenizer with parameters of type TokenPreProcess

Constructor	Description
BertWordPieceStreamTokenizer(InputStream tokens, Charset encoding, NavigableMap<String,Integer> vocab, TokenPreProcess preTokenizePreProcessor, TokenPreProcess tokenPreProcess)
BertWordPieceTokenizer(String tokens, NavigableMap<String,Integer> vocab, TokenPreProcess preTokenizePreProcessor, TokenPreProcess tokenPreProcess)

Uses of TokenPreProcess in org.deeplearning4j.text.tokenization.tokenizer.preprocessor

Classes in org.deeplearning4j.text.tokenization.tokenizer.preprocessor that implement TokenPreProcess

Modifier and Type	Class	Description
class	BertWordPiecePreProcessor
class	CommonPreprocessor
class	CompositePreProcessor
class	EndingPreProcessor	Gets rid of endings: ed,ing, ly, s, .
class	LowCasePreProcessor

Constructors in org.deeplearning4j.text.tokenization.tokenizer.preprocessor with parameters of type TokenPreProcess

Constructor	Description
CompositePreProcessor(@NonNull TokenPreProcess... preProcessors)

Constructor parameters in org.deeplearning4j.text.tokenization.tokenizer.preprocessor with type arguments of type TokenPreProcess

Constructor	Description
CompositePreProcessor(@NonNull Collection<? extends TokenPreProcess> preProcessors)

Uses of TokenPreProcess in org.deeplearning4j.text.tokenization.tokenizerfactory

Methods in org.deeplearning4j.text.tokenization.tokenizerfactory that return TokenPreProcess

Modifier and Type	Method	Description
TokenPreProcess	DefaultTokenizerFactory.getTokenPreProcessor()	Returns TokenPreProcessor set for this TokenizerFactory instance
TokenPreProcess	NGramTokenizerFactory.getTokenPreProcessor()	Returns TokenPreProcessor set for this TokenizerFactory instance
TokenPreProcess	TokenizerFactory.getTokenPreProcessor()	Returns TokenPreProcessor set for this TokenizerFactory instance

Methods in org.deeplearning4j.text.tokenization.tokenizerfactory with parameters of type TokenPreProcess

Modifier and Type	Method	Description
void	DefaultTokenizerFactory.setTokenPreProcessor(TokenPreProcess preProcessor)
void	NGramTokenizerFactory.setTokenPreProcessor(TokenPreProcess preProcessor)
void	TokenizerFactory.setTokenPreProcessor(TokenPreProcess preProcessor)	Sets a token pre processor to be used with every tokenizer

Constructors in org.deeplearning4j.text.tokenization.tokenizerfactory with parameters of type TokenPreProcess

Constructor	Description
BertWordPieceTokenizerFactory(NavigableMap<String,Integer> vocab, TokenPreProcess preTokenizePreProcessor)