Package com.aspectran.core.context.expr.token

Class Tokenizer

java.lang.Object

com.aspectran.core.context.expr.token.Tokenizer

public class Tokenizer
extends Object

The Class Tokenizer.

Created: 2008. 03. 29 AM 1:55:03

Constructor Summary

Constructors

Constructor	Description
Tokenizer()

Method Summary

Modifier and Type	Method	Description
static Token[]	optimize(Token[] tokens)	Returns an array of tokens that is optimized.
static List<Token>	tokenize(CharSequence chars, boolean textTrim)	Returns a list of Token objects separated by token expression.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

Tokenizer

public Tokenizer()

Method Details

tokenize

@NonNull
public static List<Token> tokenize(CharSequence chars,
 boolean textTrim)

Returns a list of Token objects separated by token expression.

Parameters:

chars - the char values to tokenize

textTrim - whether to trim text

Returns:

a list of tokens

optimize

public static Token[] optimize(Token[] tokens)

Returns an array of tokens that is optimized. Eliminates unnecessary white spaces for the first and last tokens.

Parameters:

tokens - the tokens before optimizing

Returns:

the optimized tokens