Uses of Interface
org.deeplearning4j.text.tokenization.tokenizer.Tokenizer

Packages that use Tokenizer

Package	Description
org.deeplearning4j.text.tokenization.tokenizer
org.deeplearning4j.text.tokenization.tokenizerfactory

Uses of Tokenizer in org.deeplearning4j.text.tokenization.tokenizer

Classes in org.deeplearning4j.text.tokenization.tokenizer that implement Tokenizer

Modifier and Type	Class	Description
class	BertWordPieceStreamTokenizer
class	BertWordPieceTokenizer
class	DefaultStreamTokenizer	Tokenizer based on the StreamTokenizer
class	DefaultTokenizer	Default tokenizer
class	NGramTokenizer

Constructors in org.deeplearning4j.text.tokenization.tokenizer with parameters of type Tokenizer

Constructor	Description
NGramTokenizer(Tokenizer tokenizer, Integer minN, Integer maxN)

Uses of Tokenizer in org.deeplearning4j.text.tokenization.tokenizerfactory

Methods in org.deeplearning4j.text.tokenization.tokenizerfactory that return Tokenizer

Modifier and Type	Method	Description
Tokenizer	BertWordPieceTokenizerFactory.create(InputStream toTokenize)
Tokenizer	BertWordPieceTokenizerFactory.create(String toTokenize)
Tokenizer	DefaultTokenizerFactory.create(InputStream toTokenize)
Tokenizer	DefaultTokenizerFactory.create(String toTokenize)
Tokenizer	NGramTokenizerFactory.create(InputStream toTokenize)
Tokenizer	NGramTokenizerFactory.create(String toTokenize)
Tokenizer	TokenizerFactory.create(InputStream toTokenize)	Create a tokenizer based on an input stream
Tokenizer	TokenizerFactory.create(String toTokenize)	The tokenizer to createComplex