Package org.deeplearning4j.text.tokenization.tokenizer

Interface Summary

Interface	Description
Tokenizer	A representation of a tokenizer.
TokenPreProcess	Token preprocessing

Class Summary

Class	Description
BertWordPieceStreamTokenizer	A tokenizer that works with a vocab from a published bert model and tokenizes a token at a time from a stream
BertWordPieceTokenizer	A tokenizer that works with a vocab from a published bert model
DefaultStreamTokenizer	Tokenizer based on the StreamTokenizer
DefaultTokenizer	Default tokenizer
NGramTokenizer