Uses of Interface
org.deeplearning4j.text.tokenization.tokenizer.TokenPreProcess

Packages that use TokenPreProcess

Package	Description
org.deeplearning4j.text.tokenization.tokenizer
org.deeplearning4j.text.tokenization.tokenizer.preprocessor
org.deeplearning4j.text.tokenization.tokenizerfactory

Uses of TokenPreProcess in org.deeplearning4j.text.tokenization.tokenizer

Methods in org.deeplearning4j.text.tokenization.tokenizer with parameters of type TokenPreProcess

Modifier and Type	Method and Description
void	DefaultStreamTokenizer.setTokenPreProcessor(TokenPreProcess tokenPreProcessor)
void	DefaultTokenizer.setTokenPreProcessor(TokenPreProcess tokenPreProcessor)
void	BertWordPieceTokenizer.setTokenPreProcessor(TokenPreProcess tokenPreProcessor)
void	NGramTokenizer.setTokenPreProcessor(TokenPreProcess tokenPreProcessor)
void	Tokenizer.setTokenPreProcessor(TokenPreProcess tokenPreProcessor) Set the token pre process

Constructors in org.deeplearning4j.text.tokenization.tokenizer with parameters of type TokenPreProcess

Constructor and Description
BertWordPieceStreamTokenizer(InputStream tokens, Charset encoding, NavigableMap<String,Integer> vocab, TokenPreProcess preTokenizePreProcessor, TokenPreProcess tokenPreProcess)
BertWordPieceTokenizer(String tokens, NavigableMap<String,Integer> vocab, TokenPreProcess preTokenizePreProcessor, TokenPreProcess tokenPreProcess)

Uses of TokenPreProcess in org.deeplearning4j.text.tokenization.tokenizer.preprocessor

Classes in org.deeplearning4j.text.tokenization.tokenizer.preprocessor that implement TokenPreProcess

Modifier and Type	Class and Description
class	BertWordPiecePreProcessor
class	CommonPreprocessor
class	CompositePreProcessor
class	EndingPreProcessor Gets rid of endings: ed,ing, ly, s, .
class	LowCasePreProcessor

Constructors in org.deeplearning4j.text.tokenization.tokenizer.preprocessor with parameters of type TokenPreProcess

Constructor and Description
CompositePreProcessor(TokenPreProcess... preProcessors)

Uses of TokenPreProcess in org.deeplearning4j.text.tokenization.tokenizerfactory

Methods in org.deeplearning4j.text.tokenization.tokenizerfactory that return TokenPreProcess

Modifier and Type	Method and Description
TokenPreProcess	NGramTokenizerFactory.getTokenPreProcessor() Returns TokenPreProcessor set for this TokenizerFactory instance
TokenPreProcess	DefaultTokenizerFactory.getTokenPreProcessor() Returns TokenPreProcessor set for this TokenizerFactory instance
TokenPreProcess	TokenizerFactory.getTokenPreProcessor() Returns TokenPreProcessor set for this TokenizerFactory instance

Methods in org.deeplearning4j.text.tokenization.tokenizerfactory with parameters of type TokenPreProcess

Modifier and Type	Method and Description
void	NGramTokenizerFactory.setTokenPreProcessor(TokenPreProcess preProcessor)
void	DefaultTokenizerFactory.setTokenPreProcessor(TokenPreProcess preProcessor)
void	TokenizerFactory.setTokenPreProcessor(TokenPreProcess preProcessor) Sets a token pre processor to be used with every tokenizer

Constructors in org.deeplearning4j.text.tokenization.tokenizerfactory with parameters of type TokenPreProcess

Constructor and Description
BertWordPieceTokenizerFactory(NavigableMap<String,Integer> vocab, TokenPreProcess preTokenizePreProcessor)