Package net.morimekta.util.lexer

Interface Tokenizer<TT,T extends Token<TT>>

All Known Implementing Classes:

TokenizerBase, TokenizerRepeater

public interface Tokenizer<TT,T extends Token<TT>>

Method Summary

Modifier and Type	Method	Description
java.lang.CharSequence	currentLine()
int	currentLineNo()	Get the current line number.
int	currentLinePos()	Get the current line position.
T	parseNextToken()	Continue parsing content and return the next token to be found.
T	readUntil(java.lang.CharSequence terminator, TT type, boolean allowEof)	Read all content until the given terminator string is encountered.

Method Detail

parseNextToken

@Nullable
T parseNextToken()
          throws LexerException,
                 java.io.IOException

Continue parsing content and return the next token to be found.

Returns:

The next token, or null if there is none.

Throws:

LexerException - If parsing token failed.

java.io.IOException - If reading failed.

readUntil

@Nullable
T readUntil(@Nonnull
            java.lang.CharSequence terminator,
            @Nonnull
            TT type,
            boolean allowEof)
     throws LexerException,
            java.io.IOException

Read all content until the given terminator string is encountered. The terminator should not become part of the returned token. The returned sequence may span more than one line.

Parameters:

terminator - The terminator string.

type - The token type for the resulting token.

allowEof - Set to true if EOF is allowed as a replacement for the terminator.

Returns:

The char sequence from the current position until the encountered terminator or the end of the file. Or null if no chars before the terminator.

Throws:

LexerException - On parse errors or validation failure.

java.io.IOException - If unable to parse token.

currentLineNo

int currentLineNo()

Get the current line number. Note that the lines should be 1-indexed, meaning the first line (before the first newline) is line no 1. This should point to the position where it will continue parsing, any previous tokens should be represented by itself.

Returns:

The current line number.

currentLinePos

int currentLinePos()

Get the current line position. This should point to the char position of the current line, meaning it is 0-indexed. This should point to the position where it will continue parsing, any previous tokens should be represented by itself.

Returns:

The current line position.

currentLine

@Nonnull
java.lang.CharSequence currentLine()

Returns:

Content of the current line. The char-sequence must be effectively immutable.