net.morimekta.providence.serializer.pretty

Class Tokenizer

java.lang.Object

java.io.InputStream

net.morimekta.providence.serializer.pretty.Tokenizer

All Implemented Interfaces:

Closeable, AutoCloseable

public class Tokenizer
extends InputStream

Simple tokenizer for the pretty serializer that strips away comments based on the "#" (shell) comment character. Each comment lasts until the next newline.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	Tokenizer.TokenValidator

Field Summary

Fields

Modifier and Type	Field and Description
protected int	lineNo
protected int	linePos
protected int	readOffset

Constructor Summary

Constructors

Constructor and Description
Tokenizer(InputStream in, boolean enclosedContent)

Method Summary

Modifier and Type	Method and Description
Token	expect(String expected) Expect at a valid token containing anything.
Token	expect(String expected, Tokenizer.TokenValidator validator)
Token	expectIdentifier(String message)
Token	expectInteger(String message)
Token	expectLiteral(String message)
char	expectSymbol(String expected, char... symbols)
protected TokenizerException	failure(int startLineNo, int startLinePos, int length, String format, Object... params)
protected TokenizerException	failure(String format, Object... params)
protected TokenizerException	failure(Throwable cause, int startLineNo, int startLinePos, int length, String message, Object... params)
TokenizerException	failure(Token token, String message, Object... params)
String	getLine(int theLine) Get the full non-delimited line content of the 1-indexed line.
boolean	hasNext()
Token	next()
protected Token	nextSymbol(int lastByte)
Token	peek()
Token	peek(String expected)
int	read()
String	readBinary(char end) Read the 'content' of encoded binary.
protected Token	token(int off, int len, int linePos)
Token	token(int off, int len, int lineNo, int linePos)

Methods inherited from class java.io.InputStream

available, close, mark, markSupported, read, read, reset, skip

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

readOffset

protected int readOffset

lineNo

protected int lineNo

linePos

protected int linePos

Constructor Detail

Tokenizer

public Tokenizer(InputStream in,
                 boolean enclosedContent)
          throws IOException

Throws:

IOException

Method Detail

read

public int read()

Specified by:

read in class InputStream

expect

public Token expect(@Nonnull
                    String expected)
             throws IOException

Expect at a valid token containing anything.

Parameters:

expected - The expectation description.

Returns:

The token.

Throws:

IOException - If failed to read a token.

expect

public Token expect(@Nonnull
                    String expected,
                    @Nonnull
                    Tokenizer.TokenValidator validator)
             throws IOException

Throws:

IOException

peek

public Token peek(@Nonnull
                  String expected)
           throws IOException

Throws:

IOException

peek

public Token peek()
           throws IOException

Throws:

IOException

expectSymbol

public char expectSymbol(@Nonnull
                         String expected,
                         char... symbols)
                  throws IOException

Throws:

IOException

expectIdentifier

public Token expectIdentifier(@Nonnull
                              String message)
                       throws IOException

Throws:

IOException

expectInteger

@Nonnull
public Token expectInteger(String message)
                             throws IOException

Throws:

IOException

expectLiteral

@Nonnull
public Token expectLiteral(String message)
                             throws IOException

Throws:

IOException

hasNext

public boolean hasNext()
                throws IOException

Throws:

IOException

next

public Token next()
           throws IOException

Throws:

IOException

nextSymbol

protected Token nextSymbol(int lastByte)
                    throws TokenizerException

Throws:

TokenizerException

getLine

@Nonnull
public String getLine(int theLine)

Get the full non-delimited line content of the 1-indexed line.

Parameters:

theLine - The nine number.

Returns:

The line string content.

readBinary

public String readBinary(char end)
                  throws IOException

Read the 'content' of encoded binary. This does not parse the binary, just read out from the buffer the string representing the binary data, as delimited by the requested 'end' char.

Parameters:

end - The char that ends the binary content.

Returns:

The string encoded string representation.

Throws:

TokenizerException - On illegal content.

IOException

failure

@Nonnull
public TokenizerException failure(Token token,
                                           String message,
                                           Object... params)

failure

@Nonnull
protected TokenizerException failure(int startLineNo,
                                              int startLinePos,
                                              int length,
                                              String format,
                                              Object... params)

failure

@Nonnull
protected TokenizerException failure(Throwable cause,
                                              int startLineNo,
                                              int startLinePos,
                                              int length,
                                              String message,
                                              Object... params)

failure

@Nonnull
protected TokenizerException failure(String format,
                                              Object... params)

token

@Nonnull
protected Token token(int off,
                               int len,
                               int linePos)

token

@Nonnull
public Token token(int off,
                            int len,
                            int lineNo,
                            int linePos)