Uses of Interface
org.deeplearning4j.text.tokenization.tokenizer.Tokenizer

Packages that use Tokenizer

Package	Description
org.deeplearning4j.text.tokenization.tokenizer
org.deeplearning4j.text.tokenization.tokenizerfactory

Uses of Tokenizer in org.deeplearning4j.text.tokenization.tokenizer

Classes in org.deeplearning4j.text.tokenization.tokenizer that implement Tokenizer

Modifier and Type	Class and Description
class	BertWordPieceStreamTokenizer A tokenizer that works with a vocab from a published bert model and tokenizes a token at a time from a stream
class	BertWordPieceTokenizer A tokenizer that works with a vocab from a published bert model
class	DefaultStreamTokenizer Tokenizer based on the StreamTokenizer
class	DefaultTokenizer Default tokenizer
class	NGramTokenizer

Constructors in org.deeplearning4j.text.tokenization.tokenizer with parameters of type Tokenizer

Constructor and Description
NGramTokenizer(Tokenizer tokenizer, Integer minN, Integer maxN)

Uses of Tokenizer in org.deeplearning4j.text.tokenization.tokenizerfactory

Methods in org.deeplearning4j.text.tokenization.tokenizerfactory that return Tokenizer

Modifier and Type	Method and Description
Tokenizer	BertWordPieceTokenizerFactory.create(InputStream toTokenize)
Tokenizer	DefaultTokenizerFactory.create(InputStream toTokenize)
Tokenizer	NGramTokenizerFactory.create(InputStream toTokenize)
Tokenizer	TokenizerFactory.create(InputStream toTokenize) Create a tokenizer based on an input stream
Tokenizer	BertWordPieceTokenizerFactory.create(String toTokenize)
Tokenizer	DefaultTokenizerFactory.create(String toTokenize)
Tokenizer	NGramTokenizerFactory.create(String toTokenize)
Tokenizer	TokenizerFactory.create(String toTokenize) The tokenizer to createComplex