Package com.aspectran.core.context.expr.token

Class Tokenizer

java.lang.Object

com.aspectran.core.context.expr.token.Tokenizer

public class Tokenizer
extends Object

The Class Tokenizer.

Created: 2008. 03. 29 AM 1:55:03

Constructor Summary

Constructors

Constructor	Description
Tokenizer()

Method Summary

Modifier and Type	Method	Description
static Token[]	optimize(Token[] tokens)	Returns an array of tokens that is optimized.
static List<Token>	tokenize(CharSequence input, boolean textTrim)	Returns a list of tokens that contains tokenized string.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

Tokenizer

public Tokenizer()

Method Details

tokenize

public static List<Token> tokenize(CharSequence input,
 boolean textTrim)

Returns a list of tokens that contains tokenized string.

Parameters:

input - the string to tokenize

textTrim - whether to trim text

Returns:

a list of tokens

optimize

public static Token[] optimize(Token[] tokens)

Returns an array of tokens that is optimized. Eliminates unnecessary white spaces for the first and last tokens.

Parameters:

tokens - the tokens before optimizing

Returns:

the optimized tokens