| Interface | Description |
|---|---|
| Tokenizer | |
| TokenPreProcess |
| Class | Description |
|---|---|
| BertWordPieceStreamTokenizer | |
| BertWordPieceTokenizer | |
| DefaultStreamTokenizer |
Tokenizer based on the
StreamTokenizer |
| DefaultTokenizer |
Default tokenizer
|
| NGramTokenizer |
Copyright © 2022. All rights reserved.