ScannerCommon

The token for given name. Either IDENTIFIER or a keyword.

Generate an error at the given offset

Clear buffer and set name and token If target is different from this, don't treat identifiers as end tokens

signal an error where the input ended in the middle of a token

append Unicode character to "litBuf" buffer

Clear buffer and set string

Switch whether unicode should be decoded

Is current token first one after a newline?

Is token a COLON, after having converted COLONEOL to COLON? The conversion means that indentation is not significant after : anymore. So, warning: this is a side-effecting operation.

Is last character a unicode escape \uxxxx?

A new reader that takes off at the current character position

Advance one character; reducing CR;LF pairs to just LF

Advance one character, leaving CR;LF pairs intact. This is for use in multi-line strings, so there are no "potential line ends" here.

the last error offset

A character buffer for literals

the base of a number

the last read character

The offset one past the last read character

The offset before the last read character

the offset of the character following the token preceding this one

the offset of the newline immediately preceding the token, or -1 if token is not preceded by a newline.

The start offset of the current line

the name of an identifier

the offset of the first character of the current token

the string value of a literal

the next token