This object is concerned with non-lexemes: these are tokens that
do not give any special treatment to whitespace.
Whilst the functionality in lexeme is strongly recommended for
wider use in a parser, the functionality here may be useful for more
specialised use-cases. In particular, these may for the building blocks
for more complex tokens (where whitespace is not allowed between them, say),
in which case these compound tokens can be turned into lexemes manually.
For example, the lexer does not have configuration for trailing specifiers
on numeric literals (like, 1024L in Scala, say): the desired numeric
literal parser could be extended with this functionality before whitespace
is consumed by using the variant found in this object.
Alternatively, these tokens can be used for lexical extraction, which
can be performed by the ErrorBuilder
typeclass: this can be used to try and extract tokens from the input stream
when an error happens, to provide a more informative error. In this case,
it is desirable to not consume whitespace after the token to keep the
error tight and precise.
This object contains lexing functionality relevant to the parsing
of numbers.
This object contains lexing functionality relevant to the parsing
of numbers. This is sub-divided into different categories:
integers (both signed and unsigned)
reals (signed only)
a combination of the two (signed and unsigned)
These contain relevant functionality for the processing of
decimal, hexadecimal, octal, and binary literals; or some
mixed combination thereof (as specified by desc.numericDesc).
Additionally, it is possible to ensure literals represent known
sizes or precisions.
This object contains lexing functionality relevant to the parsing
of names, which include operators or identifiers.
This object contains lexing functionality relevant to the parsing
of names, which include operators or identifiers.
The parsing of names is mostly concerned with finding the longest
valid name that is not a reserved name, such as a hard keyword or
a special operator.
This object contains lexing functionality relevant to the parsing
of atomic symbols.
This object contains lexing functionality relevant to the parsing
of atomic symbols.
Symbols are characterised by their "unitness", that is, every parser
inside returns Unit. This is because they all parse a specific
known entity, and, as such, the result of the parse is irrelevant.
These can be things such as reserved names, or small symbols like
parentheses. This object also contains a means of creating new symbols
as well as implicit conversions to allow for Scala's string literals to serve
as symbols within a parser.