Package

com.johnsnowlabs.nlp.annotators.tokenizer

bpe

Permalink

package bpe

Visibility
  1. Public
  2. All

Type Members

  1. class Gpt2Tokenizer extends BpeTokenizer

    Permalink

  2. class RobertaTokenizer extends Gpt2Tokenizer

    Permalink

  3. case class SpecialToken(content: String, id: Int, singleWord: Boolean = false, lstrip: Boolean = false, rstrip: Boolean = false) extends Product with Serializable

    Permalink

Value Members

  1. object BpeTokenizer

    Permalink

Ungrouped