UniversalTextTokenizer (Nitrite 4.2.2 API)

java.lang.Object
- org.dizitart.no2.index.fulltext.BaseTextTokenizer
- - org.dizitart.no2.index.fulltext.UniversalTextTokenizer

All Implemented Interfaces:

TextTokenizer
```
public class UniversalTextTokenizer
extends BaseTextTokenizer
```
A TextTokenizer implementation that tokenizes text using a universal approach.

Since:

2.1.0

Author:

Anindya Chatterjee

See Also:

Languages

Constructor Summary

Constructors
Constructor Description

UniversalTextTokenizer()

UniversalTextTokenizer(Languages... languages)

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method	Description
`Languages`	`getLanguage()`	Gets the language for the tokenizer.
`Set<String>`	`stopWords()`	Gets all stop-words for a language.

Methods inherited from class org.dizitart.no2.index.fulltext.BaseTextTokenizer
tokenize

Methods inherited from class java.lang.Object
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Constructor Detail
  - UniversalTextTokenizer
```
public UniversalTextTokenizer()
```
  - UniversalTextTokenizer
```
public UniversalTextTokenizer(Languages... languages)
```
- Method Detail
  - getLanguage
```
public Languages getLanguage()
```
    Description copied from interface: TextTokenizer
    
    Gets the language for the tokenizer.
    
    Returns:
    
    the language for this tokenizer.
  - stopWords
```
public Set<String> stopWords()
```
    Description copied from interface: TextTokenizer
    
    Gets all stop-words for a language.
    
    Returns:
    
    the set of all stop-words.