Package net.morimekta.util.lexer

Class TokenizerBase<TT,T extends Token<TT>>

java.lang.Object

java.io.Reader

net.morimekta.util.io.LineBufferedReader

net.morimekta.util.lexer.TokenizerBase<TT,T>

Type Parameters:

TT - TokenType type.

T - Token type.

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable, java.lang.Readable, Tokenizer<TT,T>

public abstract class TokenizerBase<TT,T extends Token<TT>>
extends LineBufferedReader
implements Tokenizer<TT,T>

Base tokenizer used around providence. The base tokenizer supports the minimum of what all the tokenizers require, so that each will mostly expand support, not much reduce it.

Field Summary

Fields

Modifier and Type	Field	Description
static int	DEFAULT_BUFFER_SIZE

Fields inherited from class net.morimekta.util.io.LineBufferedReader

buffer, bufferLimit, bufferLineEnd, bufferOffset, lastChar, lineNo, linePos, preLoaded, reader

Fields inherited from class java.io.Reader

lock

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	TokenizerBase(java.io.Reader in, int bufferSize, boolean preLoadAll)

Method Summary

Modifier and Type	Method	Description
protected boolean	allowIdentifier(int last)
java.lang.CharSequence	currentLine()
int	currentLineNo()	Get the current line number.
int	currentLinePos()	Get the current line position.
protected LexerException	eofFailure(java.lang.String message, java.lang.Object... params)
LexerException	failure(T token, java.lang.String message, java.lang.Object... params)
protected abstract T	genericToken(char[] buffer, int offset, int len, TT type, int lineNo, int linePos)
protected boolean	identifierSeparator(int last)
protected abstract T	identifierToken(char[] buffer, int offset, int len, int lineNo, int linePos)
protected boolean	isWhitespace()
protected T	nextSymbol()
protected abstract T	numberToken(char[] buffer, int offset, int len, int lineNo, int linePos)
T	parseNextToken()	Continue parsing content and return the next token to be found.
T	readUntil(java.lang.CharSequence terminator, TT type, boolean allowEof)	Read all content until the given terminator string is encountered.
protected boolean	startIdentifier()
protected boolean	startNumber()
protected boolean	startString()
protected boolean	startSymbol()
protected abstract T	stringToken(char[] buffer, int offset, int len, int lineNo, int linePos)
protected abstract T	symbolToken(char[] buffer, int offset, int len, int lineNo, int linePos)
java.lang.String	toString()

Methods inherited from class net.morimekta.util.io.LineBufferedReader

close, getLine, getLineNo, getLinePos, getRemainingLines, getRestOfLine, maybeConsolidateBuffer, read, read, readNextChar

Methods inherited from class java.io.Reader

mark, markSupported, nullReader, read, read, ready, reset, skip, transferTo

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

DEFAULT_BUFFER_SIZE

public static final int DEFAULT_BUFFER_SIZE

See Also:

Constant Field Values

Constructor Detail

TokenizerBase

protected TokenizerBase(java.io.Reader in,
                        int bufferSize,
                        boolean preLoadAll)

Method Detail

toString

public java.lang.String toString()

Overrides:

toString in class java.lang.Object

genericToken

protected abstract T genericToken(char[] buffer,
                                  int offset,
                                  int len,
                                  @Nonnull
                                  TT type,
                                  int lineNo,
                                  int linePos)

identifierToken

protected abstract T identifierToken(char[] buffer,
                                     int offset,
                                     int len,
                                     int lineNo,
                                     int linePos)

stringToken

protected abstract T stringToken(char[] buffer,
                                 int offset,
                                 int len,
                                 int lineNo,
                                 int linePos)

numberToken

protected abstract T numberToken(char[] buffer,
                                 int offset,
                                 int len,
                                 int lineNo,
                                 int linePos)

symbolToken

protected abstract T symbolToken(char[] buffer,
                                 int offset,
                                 int len,
                                 int lineNo,
                                 int linePos)

failure

@Nonnull
public LexerException failure(@Nonnull
                              T token,
                              @Nonnull
                              java.lang.String message,
                              @Nonnull
                              java.lang.Object... params)

eofFailure

@Nonnull
protected LexerException eofFailure(@Nonnull
                                    java.lang.String message,
                                    @Nonnull
                                    java.lang.Object... params)

currentLineNo

public int currentLineNo()

Description copied from interface: Tokenizer

Get the current line number. Note that the lines should be 1-indexed, meaning the first line (before the first newline) is line no 1. This should point to the position where it will continue parsing, any previous tokens should be represented by itself.

Specified by:

currentLineNo in interface Tokenizer<TT,T extends Token<TT>>

Returns:

The current line number.

currentLinePos

public int currentLinePos()

Description copied from interface: Tokenizer

Get the current line position. This should point to the char position of the current line, meaning it is 0-indexed. This should point to the position where it will continue parsing, any previous tokens should be represented by itself.

Specified by:

currentLinePos in interface Tokenizer<TT,T extends Token<TT>>

Returns:

The current line position.

currentLine

@Nonnull
public java.lang.CharSequence currentLine()

Specified by:

currentLine in interface Tokenizer<TT,T extends Token<TT>>

Returns:

Content of the current line. The char-sequence must be effectively immutable.

readUntil

@Nullable
public T readUntil(@Nonnull
                   java.lang.CharSequence terminator,
                   @Nonnull
                   TT type,
                   boolean allowEof)
            throws java.io.IOException

Description copied from interface: Tokenizer

Read all content until the given terminator string is encountered. The terminator should not become part of the returned token. The returned sequence may span more than one line.

Specified by:

readUntil in interface Tokenizer<TT,T extends Token<TT>>

Parameters:

terminator - The terminator string.

type - The token type for the resulting token.

allowEof - Set to true if EOF is allowed as a replacement for the terminator.

Returns:

The char sequence from the current position until the encountered terminator or the end of the file. Or null if no chars before the terminator.

Throws:

LexerException - On parse errors or validation failure.

java.io.IOException - If unable to parse token.

parseNextToken

@Nullable
public T parseNextToken()
                 throws java.io.IOException

Description copied from interface: Tokenizer

Continue parsing content and return the next token to be found.

Specified by:

parseNextToken in interface Tokenizer<TT,T extends Token<TT>>

Returns:

The next token, or null if there is none.

Throws:

LexerException - If parsing token failed.

java.io.IOException - If reading failed.

isWhitespace

protected boolean isWhitespace()

startNumber

protected boolean startNumber()

startString

protected boolean startString()

startIdentifier

protected boolean startIdentifier()

allowIdentifier

protected boolean allowIdentifier(int last)

identifierSeparator

protected boolean identifierSeparator(int last)

startSymbol

protected boolean startSymbol()

nextSymbol

@Nonnull
protected T nextSymbol()
                throws java.io.IOException

Throws:

java.io.IOException