overridden in UnitScanners: apply brace patch if one exists for this offset return true if subsequent end of line handling should be suppressed.
the base of a number
the base of a number
Should doc comments be built?
A character buffer for literals
the last read character
the last read character
Parse character literal if current character is followed by \', or follow with given op and return a symbol literal token
The offset one past the last read character
The offset one past the last read character
Convert current strVal to char value
buffer for the documentation comment
the last error offset
read next token, filling TokenData fields of Scanner.
read next token, filling TokenData fields of Scanner.
Convert current strVal, base to double value
Return current docBuffer and set docBuffer to null
read fractional part and exponent of floating point number if one is present.
read fractional part and exponent of floating point number if one is present.
read next character in character or string literal:
read next character in character or string literal:
Read a number into strVal and set base
Read a number into strVal and set base
overridden in UnitScanners
Can token start a statement?
Can token end a statement?
signal an error where the input ended in the middle of a token
Initialization method: read first char, then first token
Convert current strVal, base to long value This is tricky because of max negative value.
Is last character a unicode escape \\uxxxx?
Is last character a unicode escape \\uxxxx?
The start offset of the line before the current one
The start offset of the line before the current one
the offset of the character following the token preceding this one
the offset of the character following the token preceding this one
The start offset of the current line
The start offset of the current line
A new reader that takes off at the current character position
A new reader that takes off at the current character position
the name of an identifier
the name of an identifier
we need one token lookahead and one token history
Advance one character; reducing CR;LF pairs to just LF
Advance one character; reducing CR;LF pairs to just LF
Advance one character, leaving CR;LF pairs intact.
Advance one character, leaving CR;LF pairs intact. This is for use in multi-line strings, so there are no "potential line ends" here.
Produce next token, filling TokenData fields of Scanner.
the offset of the first character of the current token
the offset of the first character of the current token
overridden in UnitScanners
append Unicode character to "cbuf" buffer
append Unicode character to "cbuf" buffer
add the given character to the documentation buffer
add the given character to the documentation buffer
a stack of tokens which indicates whether line-ends can be statement separators
read next token and return last offset
the string value of a literal
the string value of a literal
generate an error at the current token offset
generate an error at the given offset
the next token
the next token