Package net.morimekta.providence.serializer.pretty

Class PrettyTokenizer

  • All Implemented Interfaces:
    java.io.Closeable, java.lang.AutoCloseable, java.lang.Readable, net.morimekta.util.lexer.Tokenizer<PrettyTokenType,​PrettyToken>
    public class PrettyTokenizer
extends net.morimekta.util.lexer.TokenizerBase<PrettyTokenType,​PrettyToken>
    Simple tokenizer for the pretty serializer that strips away comments based on the "#" (shell) comment character. Each comment lasts until the next newline.

    • Field Summary

      • Fields inherited from class net.morimekta.util.lexer.TokenizerBase

        DEFAULT_BUFFER_SIZE

      • Fields inherited from class net.morimekta.util.io.LineBufferedReader

        buffer, bufferLimit, bufferLineEnd, bufferOffset, lastChar, lineNo, linePos, preLoaded, reader

      • Fields inherited from class java.io.Reader

        lock

    • Constructor Summary

      Constructors 
      Constructor Description
      PrettyTokenizer​(java.io.InputStream in)
      Create a pretty tokenizer that reads from the input steam.
      PrettyTokenizer​(java.io.InputStream in, int bufferSize)
      Create a pretty tokenizer that reads from the input steam.
      PrettyTokenizer​(java.io.Reader in, int bufferSize, boolean preLoadAll)
      Create a pretty tokenizer that will read everything from the input stream and handle it as a single multi-line buffer.

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      protected PrettyToken genericToken​(char[] buffer, int offset, int len, PrettyTokenType type, int lineNo, int linePos)  
      protected PrettyToken identifierToken​(char[] buffer, int offset, int len, int lineNo, int linePos)  
      protected PrettyToken numberToken​(char[] buffer, int offset, int len, int lineNo, int linePos)  
      protected boolean startString()  
      protected PrettyToken stringToken​(char[] buffer, int offset, int len, int lineNo, int linePos)  
      protected PrettyToken symbolToken​(char[] buffer, int offset, int len, int lineNo, int linePos)  

      • Methods inherited from class net.morimekta.util.lexer.TokenizerBase

        allowIdentifier, currentLine, currentLineNo, currentLinePos, eofFailure, failure, identifierSeparator, isWhitespace, nextSymbol, parseNextToken, readUntil, startIdentifier, startNumber, startSymbol, toString

      • Methods inherited from class net.morimekta.util.io.LineBufferedReader

        close, getLine, getLineNo, getLinePos, getRemainingLines, getRestOfLine, maybeConsolidateBuffer, read, read, readNextChar

      • Methods inherited from class java.io.Reader

        mark, markSupported, nullReader, read, read, ready, reset, skip, transferTo

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

    • Constructor Detail

      • PrettyTokenizer

        public PrettyTokenizer​(java.io.InputStream in)
        Create a pretty tokenizer that reads from the input steam. It will only read as far as requested, and no bytes further. It has no checking of whether the document follows the JSON standard, but will only accept JSON formatted tokens. Note that the content is assumed to be separated with newlines, which means that if multiple JSON contents are read from the same stream, they MUST have a separating newline. A single JSON object may still have newlines in it's stream.
        Parameters:
        in - Input stream to parse from.

      • PrettyTokenizer

        public PrettyTokenizer​(java.io.InputStream in,
                       int bufferSize)
        Create a pretty tokenizer that reads from the input steam. It will only read as far as requested, and no bytes further. It has no checking of whether the document follows the JSON standard, but will only accept JSON formatted tokens. Note that the content is assumed to be separated with newlines, which means that if multiple JSON contents are read from the same stream, they MUST have a separating newline. A single JSON object may still have newlines in it's stream.
        Parameters:
        in - Input stream to parse from.
        bufferSize - The size of the char buffer. Default is 2048 chars (4096 bytes).

      • PrettyTokenizer

        public PrettyTokenizer​(java.io.Reader in,
                       int bufferSize,
                       boolean preLoadAll)
        Create a pretty tokenizer that will read everything from the input stream and handle it as a single multi-line buffer.
        Parameters:
        in - Reader of content to parse.
        bufferSize - The size of the char buffer. Default is 2048 chars (4096 bytes).
        preLoadAll - Load all content up front. Handy for config and thrift program files.

    • Method Detail

      • startString

        protected boolean startString()
        Overrides:
        startString in class net.morimekta.util.lexer.TokenizerBase<PrettyTokenType,​PrettyToken>

      • genericToken

        protected PrettyToken genericToken​(char[] buffer,
                                   int offset,
                                   int len,
                                   @Nonnull
                                   PrettyTokenType type,
                                   int lineNo,
                                   int linePos)
        Specified by:
        genericToken in class net.morimekta.util.lexer.TokenizerBase<PrettyTokenType,​PrettyToken>

      • identifierToken

        protected PrettyToken identifierToken​(char[] buffer,
                                      int offset,
                                      int len,
                                      int lineNo,
                                      int linePos)
        Specified by:
        identifierToken in class net.morimekta.util.lexer.TokenizerBase<PrettyTokenType,​PrettyToken>

      • stringToken

        protected PrettyToken stringToken​(char[] buffer,
                                  int offset,
                                  int len,
                                  int lineNo,
                                  int linePos)
        Specified by:
        stringToken in class net.morimekta.util.lexer.TokenizerBase<PrettyTokenType,​PrettyToken>

      • numberToken

        protected PrettyToken numberToken​(char[] buffer,
                                  int offset,
                                  int len,
                                  int lineNo,
                                  int linePos)
        Specified by:
        numberToken in class net.morimekta.util.lexer.TokenizerBase<PrettyTokenType,​PrettyToken>

      • symbolToken

        protected PrettyToken symbolToken​(char[] buffer,
                                  int offset,
                                  int len,
                                  int lineNo,
                                  int linePos)
        Specified by:
        symbolToken in class net.morimekta.util.lexer.TokenizerBase<PrettyTokenType,​PrettyToken>