Uses of Interface
org.deeplearning4j.text.tokenization.tokenizer.TokenPreProcess

Packages that use TokenPreProcess

Package	Description
org.deeplearning4j.text.tokenization.tokenizer
org.deeplearning4j.text.tokenization.tokenizer.preprocessor
org.deeplearning4j.text.tokenization.tokenizerfactory

Uses of TokenPreProcess in org.deeplearning4j.text.tokenization.tokenizer

Methods in org.deeplearning4j.text.tokenization.tokenizer with parameters of type TokenPreProcess

Modifier and Type	Method and Description
void	NGramTokenizer.setTokenPreProcessor(TokenPreProcess tokenPreProcessor)
void	PosUimaTokenizer.setTokenPreProcessor(TokenPreProcess tokenPreProcessor)
void	UimaTokenizer.setTokenPreProcessor(TokenPreProcess tokenPreProcessor)
void	Tokenizer.setTokenPreProcessor(TokenPreProcess tokenPreProcessor) Set the token pre process
void	DefaultStreamTokenizer.setTokenPreProcessor(TokenPreProcess tokenPreProcessor)
void	DefaultTokenizer.setTokenPreProcessor(TokenPreProcess tokenPreProcessor)

Uses of TokenPreProcess in org.deeplearning4j.text.tokenization.tokenizer.preprocessor

Classes in org.deeplearning4j.text.tokenization.tokenizer.preprocessor that implement TokenPreProcess

Modifier and Type	Class and Description
class	CommonPreprocessor
class	CustomStemmingPreprocessor This is StemmingPreprocessor compatible with different StemmingProcessors defined as lucene/tartarus SnowballProgram Like, but not limited to: RussianStemmer, DutchStemmer, FrenchStemmer etc PLEASE NOTE: This preprocessor is NOT thread-safe.
class	EndingPreProcessor Gets rid of endings: ed,ing, ly, s, .
class	LowCasePreProcessor
class	StemmingPreprocessor This tokenizer preprocessor implements basic cleaning inherited from CommonPreprocessor + does english Porter stemming on tokens

Uses of TokenPreProcess in org.deeplearning4j.text.tokenization.tokenizerfactory

Methods in org.deeplearning4j.text.tokenization.tokenizerfactory that return TokenPreProcess

Modifier and Type	Method and Description
TokenPreProcess	UimaTokenizerFactory.getTokenPreProcessor() Returns TokenPreProcessor set for this TokenizerFactory instance
TokenPreProcess	NGramTokenizerFactory.getTokenPreProcessor() Returns TokenPreProcessor set for this TokenizerFactory instance
TokenPreProcess	PosUimaTokenizerFactory.getTokenPreProcessor() Returns TokenPreProcessor set for this TokenizerFactory instance
TokenPreProcess	TokenizerFactory.getTokenPreProcessor() Returns TokenPreProcessor set for this TokenizerFactory instance
TokenPreProcess	DefaultTokenizerFactory.getTokenPreProcessor() Returns TokenPreProcessor set for this TokenizerFactory instance

Methods in org.deeplearning4j.text.tokenization.tokenizerfactory with parameters of type TokenPreProcess

Modifier and Type	Method and Description
void	UimaTokenizerFactory.setTokenPreProcessor(TokenPreProcess preProcessor)
void	NGramTokenizerFactory.setTokenPreProcessor(TokenPreProcess preProcessor)
void	PosUimaTokenizerFactory.setTokenPreProcessor(TokenPreProcess preProcessor)
void	TokenizerFactory.setTokenPreProcessor(TokenPreProcess preProcessor) Sets a token pre processor to be used with every tokenizer
void	DefaultTokenizerFactory.setTokenPreProcessor(TokenPreProcess preProcessor)