lexeme
This object is concerned with lexemes: these are tokens that are treated as "words", such that whitespace will be consumed after each has been parsed.
Ideally, a wider parser should not be concerned with handling whitespace, as it is responsible for dealing with a stream of tokens. With parser combinators, however, it is usually not the case that there is a separate distinction between the parsing phase and the lexing phase. That said, it is good practice to establish a logical separation between the two worlds. As such, this object contains parsers that parse tokens, and these are whitespace-aware. This means that whitespace will be consumed after any of these parsers are parsed. It is not, however, required that whitespace be present.
Attributes
- Since
-
4.0.0
- Source
- Lexer.scala
- Graph
-
- Supertypes
- Self type
-
lexeme.type
Members list
Type members
Classlikes
This object contains helper combinators for parsing terms enclosed by common symbols.
This object contains helper combinators for parsing terms enclosed by common symbols.
Attributes
- Since
-
4.0.0
- Source
- Lexer.scala
- Supertypes
- Self type
-
enclosing.type
This object contains lexing functionality relevant to the parsing of numbers.
This object contains lexing functionality relevant to the parsing of numbers. This is sub-divided into different categories:
integers (both signed and unsigned)
reals (signed only)
a combination of the two (signed and unsigned)
These contain relevant functionality for the processing of decimal, hexadecimal, octal, and binary literals; or some mixed combination thereof (as specified by desc.numericDesc
). Additionally, it is possible to ensure literals represent known sizes or precisions.
Attributes
- Since
-
4.0.0
- Source
- Lexer.scala
- Supertypes
- Self type
-
numeric.type
This object contains helper combinators for parsing terms separated by common symbols.
This object contains helper combinators for parsing terms separated by common symbols.
Attributes
- Since
-
4.0.0
- Source
- Lexer.scala
- Supertypes
- Self type
-
separators.type
This object contains lexing functionality relevant to the parsing of text.
This object contains lexing functionality relevant to the parsing of text. This is sub-divided into different categories:
string literals (both with escapes and raw)
multi-line string literals (both with escapes and raw)
character literals
These contain the relevant functionality required to specify the degree of unicode support for the underlying language, from ASCII to full UTF-16.
Attributes
- Since
-
4.0.0
- Source
- Lexer.scala
- Supertypes
- Self type
-
text.type
Value members
Concrete methods
This combinator turns a non-lexeme parser into a lexeme one by ensuring whitespace is consumed after the parser.
This combinator turns a non-lexeme parser into a lexeme one by ensuring whitespace is consumed after the parser.
When using parser combinators, it is important to establish a consistent whitespace consumption scheme: ideally, there is no wasteful parsing, and whitespace consumption should not impact backtracking. This leads to a convention that whitespace must only be consumed after a token, and only once at the very start of the parser (see fully
). When manually constructing tokens that are not supported by this lexer, use this combinator to ensure it also follows the whitespace convention.
Value parameters
- p
-
the token parser to ensure consumes trailing whitespace.
Attributes
- Since
-
4.0.0
- Source
- Lexer.scala
Concrete fields
This object contains lexing functionality relevant to the parsing of names, which include operators or identifiers.
This object contains lexing functionality relevant to the parsing of names, which include operators or identifiers.
The parsing of names is mostly concerned with finding the longest valid name that is not a reserved name, such as a hard keyword or a special operator.
Attributes
- Since
-
4.0.0
- Source
- Lexer.scala
This object contains lexing functionality relevant to the parsing of atomic symbols.
This object contains lexing functionality relevant to the parsing of atomic symbols.
Symbols are characterised by their "unitness", that is, every parser inside returns Unit
. This is because they all parse a specific known entity, and, as such, the result of the parse is irrelevant. These can be things such as reserved names, or small symbols like parentheses. This object also contains a means of creating new symbols as well as implicit conversions to allow for Scala's string literals to serve as symbols within a parser.
Attributes
- Since
-
4.0.0
- Source
- Lexer.scala