Package com.yahoo.language.process

Class SpecialTokens

java.lang.Object

com.yahoo.language.process.SpecialTokens

public class SpecialTokens
extends Object

An immutable list of special tokens - strings which should override the normal tokenizer semantics and be tokenized into a single token. Special tokens are case insensitive.

Author:

bratseth

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static final class	SpecialTokens.Token	An immutable special token

Constructor Summary

Constructors

Constructor	Description
SpecialTokens(String name, List<SpecialTokens.Token> tokens)

Method Summary

Modifier and Type	Method	Description
Map<String,String>	asMap()	Returns the tokens of this as an immutable map from token to replacement.
static SpecialTokens	empty()
String	name()	Returns the name of this special tokens list
SpecialTokens.Token	tokenize(String string, boolean substring)	Returns the special token starting at the start of the given string, or null if no special token starts at this string

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

SpecialTokens

public SpecialTokens(String name,
 List<SpecialTokens.Token> tokens)

Method Details

name

public String name()

Returns the name of this special tokens list

asMap

public Map<String,String> asMap()

Returns the tokens of this as an immutable map from token to replacement. Tokens which do not have a replacement token maps to themselves.

tokenize

public SpecialTokens.Token tokenize(String string,
 boolean substring)

Returns the special token starting at the start of the given string, or null if no special token starts at this string

Parameters:

string - the string to search for a special token at the start position

substring - true to allow the special token to be followed by a character which does not mark the end of a token

empty

public static SpecialTokens empty()