Class SpecialTokenRegistry


  • public class SpecialTokenRegistry
    extends Object
    Immutable named lists of "special tokens" - strings which should override the normal tokenizer semantics and be tokenized into a single token.
    Author:
    bratseth
    • Constructor Detail

      • SpecialTokenRegistry

        public SpecialTokenRegistry()
        Creates an empty special token registry
      • SpecialTokenRegistry

        public SpecialTokenRegistry​(com.yahoo.vespa.configdefinition.SpecialtokensConfig config)
        Create a special token registry from a configuration object.
      • SpecialTokenRegistry

        public SpecialTokenRegistry​(List<SpecialTokens> specialTokensList)
    • Method Detail

      • getSpecialTokens

        public SpecialTokens getSpecialTokens​(String name)
        Returns the list of special tokens for a given name.
        Parameters:
        name - the name of the special tokens to return null, the empty string or the string "default" returns the default ones
        Returns:
        a read-only list of SpecialToken instances, an empty list if this name has no special tokens