Class Tokenizer
- java.lang.Object
-
- java.io.Reader
-
- net.morimekta.util.io.LineBufferedReader
-
- net.morimekta.providence.serializer.pretty.Tokenizer
-
- All Implemented Interfaces:
java.io.Closeable
,java.lang.AutoCloseable
,java.lang.Readable
public class Tokenizer extends net.morimekta.util.io.LineBufferedReader
Simple tokenizer for the pretty serializer that strips away comments based on the "#" (shell) comment character. Each comment lasts until the next newline.
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description static interface
Tokenizer.TokenValidator
-
Field Summary
Fields Modifier and Type Field Description static int
DEFAULT_BUFFER_SIZE
-
Constructor Summary
Constructors Constructor Description Tokenizer(java.io.InputStream in)
Create a JSON tokenizer that reads from the input steam.Tokenizer(java.io.InputStream in, int bufferSize)
Create a JSON tokenizer that reads from the input steam.Tokenizer(java.io.Reader in, int bufferSize, boolean preLoadAll)
Create a tokenizer that will read everything from the input stream and handle it as a single multi-line buffer.
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description protected TokenizerException
eof(java.lang.String format, java.lang.Object... params)
Token
expect(java.lang.String expected)
Expect a new JSON token on the stream.Token
expect(java.lang.String expected, Tokenizer.TokenValidator validator)
Expect at a valid token containing anything.Token
expectIdentifier(java.lang.String message)
Token
expectInteger(java.lang.String message)
Token
expectLiteral(java.lang.String message)
char
expectSymbol(java.lang.String expected, char... symbols)
protected TokenizerException
failure(int startLineNo, int startLinePos, int length, java.lang.String format, java.lang.Object... params)
protected TokenizerException
failure(java.lang.String format, java.lang.Object... params)
TokenizerException
failure(Token token, java.lang.String message, java.lang.Object... params)
Token
getLastToken()
boolean
hasNext()
Whether there is another token on the stream.Token
next()
Returns the next token on the stream, or null if there are no more JSON tokens on the stream.protected Token
nextSymbol()
Token
peek()
Return the next token or throw an exception.Token
peek(java.lang.String message)
Return the next token or throw an exception.java.lang.String
readBinary(char end)
Read the 'content' of encoded binary.-
Methods inherited from class net.morimekta.util.io.LineBufferedReader
close, getLine, getLineNo, getLinePos, getRemainingLines, getRestOfLine, maybeConsolidateBuffer, read, read, readNextChar
-
-
-
-
Field Detail
-
DEFAULT_BUFFER_SIZE
public static final int DEFAULT_BUFFER_SIZE
- See Also:
- Constant Field Values
-
-
Constructor Detail
-
Tokenizer
public Tokenizer(java.io.InputStream in)
Create a JSON tokenizer that reads from the input steam. It will only read as far as requested, and no bytes further. It has no checking of whether the document follows the JSON standard, but will only accept JSON formatted tokens. Note that the content is assumed to be separated with newlines, which means that if multiple JSON contents are read from the same stream, they MUST have a separating newline. A single JSON object may still have newlines in it's stream.- Parameters:
in
- Input stream to parse from.
-
Tokenizer
public Tokenizer(java.io.InputStream in, int bufferSize)
Create a JSON tokenizer that reads from the input steam. It will only read as far as requested, and no bytes further. It has no checking of whether the document follows the JSON standard, but will only accept JSON formatted tokens. Note that the content is assumed to be separated with newlines, which means that if multiple JSON contents are read from the same stream, they MUST have a separating newline. A single JSON object may still have newlines in it's stream.- Parameters:
in
- Input stream to parse from.bufferSize
- The size of the char buffer. Default is 2048 chars (4096 bytes).
-
Tokenizer
public Tokenizer(java.io.Reader in, int bufferSize, boolean preLoadAll)
Create a tokenizer that will read everything from the input stream and handle it as a single multi-line buffer.- Parameters:
in
- Reader of content to parse.bufferSize
- The size of the char buffer. Default is 2048 chars (4096 bytes).preLoadAll
- Load all content up front. Handy for config and thrift program files.
-
-
Method Detail
-
expect
@Nonnull public Token expect(@Nonnull java.lang.String expected) throws java.io.IOException
Expect a new JSON token on the stream.- Parameters:
expected
- Message to add to exception if there are no more JSON tokens on the stream.- Returns:
- The next token.
- Throws:
java.io.IOException
- If unable to read from stream.
-
expect
public Token expect(@Nonnull java.lang.String expected, @Nonnull Tokenizer.TokenValidator validator) throws java.io.IOException
Expect at a valid token containing anything.- Parameters:
expected
- The expectation description.validator
- Validator callback.- Returns:
- The token.
- Throws:
java.io.IOException
- If failed to read a token.
-
expectSymbol
public char expectSymbol(@Nonnull java.lang.String expected, char... symbols) throws java.io.IOException
- Parameters:
expected
- Message to add to exception if there are no more JSON tokens on the stream.symbols
- List of symbol characters to expect.- Returns:
- The symbol that was encountered.
- Throws:
java.io.IOException
- If unable to read from stream.
-
expectIdentifier
public Token expectIdentifier(@Nonnull java.lang.String message) throws java.io.IOException
- Throws:
java.io.IOException
-
expectInteger
@Nonnull public Token expectInteger(java.lang.String message) throws java.io.IOException
- Throws:
java.io.IOException
-
expectLiteral
@Nonnull public Token expectLiteral(java.lang.String message) throws java.io.IOException
- Throws:
java.io.IOException
-
hasNext
public boolean hasNext() throws java.io.IOException
Whether there is another token on the stream. This will read up until it finds a JSON token, or until the stream ends.- Returns:
- True if (and only if) there is at least one more token on the stream.
- Throws:
java.io.IOException
- If unable to read from stream.
-
getLastToken
public Token getLastToken()
-
readBinary
public java.lang.String readBinary(char end) throws java.io.IOException
Read the 'content' of encoded binary. This does not parse the binary, just read out from the buffer the string representing the binary data, as delimited by the requested 'end' char.- Parameters:
end
- The char that ends the binary content.- Returns:
- The string encoded string representation.
- Throws:
TokenizerException
- On illegal content.java.io.IOException
-
peek
public Token peek() throws java.io.IOException
Return the next token or throw an exception. Though it does not consume that token.- Returns:
- The next token.
- Throws:
java.io.IOException
- If unable to read from stream.
-
peek
@Nonnull public Token peek(java.lang.String message) throws java.io.IOException
Return the next token or throw an exception. Though it does not consume that token.- Parameters:
message
- Message to add to exception if there are no more JSON tokens on the stream.- Returns:
- The next token.
- Throws:
java.io.IOException
- If unable to read from stream.
-
next
@Nullable public Token next() throws java.io.IOException
Returns the next token on the stream, or null if there are no more JSON tokens on the stream.- Returns:
- The next token, or null.
- Throws:
java.io.IOException
- If unable to read from stream.
-
nextSymbol
@Nonnull protected Token nextSymbol() throws java.io.IOException
- Throws:
java.io.IOException
-
failure
@Nonnull public TokenizerException failure(Token token, java.lang.String message, java.lang.Object... params)
-
failure
@Nonnull protected final TokenizerException failure(int startLineNo, int startLinePos, int length, java.lang.String format, java.lang.Object... params)
-
eof
@Nonnull protected final TokenizerException eof(java.lang.String format, java.lang.Object... params)
-
failure
@Nonnull protected TokenizerException failure(java.lang.String format, java.lang.Object... params)
-
-