Package org.deeplearning4j.text.tokenization.tokenizer

Interface Summary

Interface	Description
Tokenizer	A representation of a tokenizer.
TokenPreProcess	Token preprocessing

Class Summary

Class	Description
DefaultStreamTokenizer	Tokenizer based on the StreamTokenizer
DefaultTokenizer	Default tokenizer
NGramTokenizer