Package com.yahoo.language.process

Class SpecialTokens

  • public class SpecialTokens
extends java.lang.Object
    An immutable list of special tokens - strings which should override the normal tokenizer semantics and be tokenized into a single token. Special tokens are case insensitive.
    Author:
    bratseth

    • Nested Class Summary

      Nested Classes 
      Modifier and Type Class Description
      static class  SpecialTokens.Token
      An immutable special token

    • Method Summary

      All Methods Static Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      java.util.Map<java.lang.String,​java.lang.String> asMap()
      Returns the tokens of this as an immutable map from token to replacement.
      static SpecialTokens empty()  
      java.lang.String name()
      Returns the name of this special tokens list
      SpecialTokens.Token tokenize​(java.lang.String string, boolean substring)
      Returns the special token starting at the start of the given string, or null if no special token starts at this string

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Constructor Detail

      • SpecialTokens

        public SpecialTokens​(java.lang.String name,
                     java.util.List<SpecialTokens.Token> tokens)

    • Method Detail

      • name

        public java.lang.String name()
        Returns the name of this special tokens list

      • asMap

        public java.util.Map<java.lang.String,​java.lang.String> asMap()
        Returns the tokens of this as an immutable map from token to replacement. Tokens which do not have a replacement token maps to themselves.

      • tokenize

        public SpecialTokens.Token tokenize​(java.lang.String string,
                                    boolean substring)
        Returns the special token starting at the start of the given string, or null if no special token starts at this string
        Parameters:
        string - the string to search for a special token at the start position
        substring - true to allow the special token to be followed by a character which does not mark the end of a token