tokenizer

Type Members

1. class EnglishNormalizer extends Normalizer

   

2. trait Normalizer extends AnyRef

   

   Normalizes individual tokens Author: mihais Date: 3/21/17

3. class OpenDomainEnglishLexer extends TokenizerLexer

   

   Tokenizer using the OpenDomainLexer.g grammar

4. class OpenDomainEnglishTokenizer extends Tokenizer

   

   English open domain tokenizer

5. class OpenDomainLexer extends Lexer

   

6. case class RawToken(text: String, startOffset: Int, endOffset: Int) extends Product with Serializable

   

7. class Tokenizer extends AnyRef

   

   Generic tokenizer Author: mihais Date: 3/15/17

8. trait TokenizerLexer extends AnyRef

   

   Thin wrapper over the Antlr lexer Author: mihais Date: 3/21/17