Scanner

Parse character literal if current character is followed by ', or follow with given op and return a symbol literal token

Return a list of all the comment positions

Returns the closest docstring preceding the position supplied

Handle newlines, possibly inserting an INDENT, OUTDENT, NEWLINE, or NEWLINES token in front of the current token. This depends on whether indentation is significant or not.

Indentation is significant if indentSyntax is set, and we are not inside a {...}, [...], (...), case ... => pair, nor in a if/while condition (i.e. currentRegion is empty).

There are three rules:

1. Insert NEWLINE or NEWLINES if

   • the closest enclosing sepRegion is { ... } or for ... do/yield, or we are on the toplevel, i.e. currentRegion is empty, and
   • the previous token can end a statement, and
   • the current token can start a statement, and
   • the current token is not a leading infix operator, and
   • if indentation is significant then the current token starts at the current indentation width or to the right of it.

   The inserted token is NEWLINES if the current token is preceded by a whitespace line, or NEWLINE otherwise.

2. Insert INDENT if

   • indentation is significant, and
   • the last token can start an indentation region.
   • the indentation of the current token is strictly greater than the previous indentation width, or the two widths are the same and the current token is one of : or match.

   The following tokens can start an indentation region:

   : = => <- if then else while do try catch finally for yield match

   Inserting an INDENT starts a new indentation region with the indentation of the current token as indentation width.

3. Insert OUTDENT if

   • indentation is significant, and
   • the indentation of the current token is strictly less than the previous indentation width,
   • the current token is not a leading infix operator.

   Inserting an OUTDENT closes an indentation region. In this case, issue an error if the indentation of the current token does not match the indentation of some previous line in an enclosing indentation region.

   If a token is inserted and consumed, the original source token is still considered to start a new line, so the process that inserts an OUTDENT might repeat several times.

Indentation widths are strings consisting of spaces and tabs, ordered by the prefix relation. I.e. a <= b iff b.startsWith(a). If indentation is significant it is considered an error if the current indentation width and the indentation of the current token are incomparable.

Is the current token in a position where a modifier is allowed?

The indentation width of the given offset

Insert token at assumed offset in front of current one.

A leading symbolic or backquoted identifier is treated as an infix operator if

• it does not follow a blank line, and
• it is followed by at least one whitespace character and a token that can start an expression. If a leading infix operator is found and the source version is 3.0-migration, emit a change warning.

The next token after this one. The token is computed via fetchToken, so complex two word tokens such as CASECLASS are not recognized. Newlines and indent/unindent tokens are skipped.

Produce next token, filling TokenData fields of Scanner.

Insert an token if next token closes an indentation region. Exception: continue if indentation region belongs to a match and next token is case.

• Join CASE + CLASS => CASECLASS, CASE + OBJECT => CASEOBJECT SEMI + ELSE => ELSE, COLON + => COLONEOL
• Insert missing OUTDENTs at EOF

Skip matching pairs of (...) or [...] parentheses.

read next token and return last offset

Switch whether unicode should be decoded

Generate an error at the given offset

Clear buffer and set name and token If target is different from this, don't treat identifiers as end tokens

signal an error where the input ended in the middle of a token

Is current token first one after a newline?

Is token a COLON, after having converted COLONEOL to COLON? The conversion means that indentation is not significant after : anymore. So, warning: this is a side-effecting operation.

Is last character a unicode escape \uxxxx?

A new reader that takes off at the current character position

Advance one character; reducing CR;LF pairs to just LF

Advance one character, leaving CR;LF pairs intact. This is for use in multi-line strings, so there are no "potential line ends" here.

append Unicode character to "litBuf" buffer

Clear buffer and set string

The current region. This is initially an Indented region with indentation width.

We need one token lookahead and one token history

the base of a number

the last read character

The offset one past the last read character

the last error offset

The offset before the last read character

the offset of the character following the token preceding this one

the offset of the newline immediately preceding the token, or -1 if token is not preceded by a newline.

The start offset of the current line

A character buffer for literals

the name of an identifier

the offset of the first character of the current token

the string value of a literal

the next token