This extractor mixin provides an implementation for
ErrorBuilder.unexpectedToken
when mixed into
an error builder: it will try and parse the residual input to identify a valid lexical token
to report.
When parsing a grammar that as a dedicated lexical distinction, it is nice to be able to report problematic tokens relevant to that grammar as opposed to generic input lifted straight from the input stream. The easiest way of doing this would be having a pre-lexing pass and parsing based on tokens, but this is deliberately not how Parsley is designed. Instead, this extractor can try and parse the remaining input to try and identify a token on demand.
If the lexicalError
flag of the unexpectedToken
method is not set, which would indicate a
problem within a token reported by a classical lexer and not the parser, the extractor will
try to parse each of the provided tokens
in turn: whichever is the longest matched of these
tokens will be reported as the problematic one (this can be changed by overriding selectToken
).
For best effect, these tokens should not consume
whitespace (which would otherwise be included at the end of the token!): this means that, if
using the Lexer
class, the functionality in nonlexeme
should be used. If one of the
givens tokens cannot be parsed, the input until the next valid parsable token (or end of input)
is returned as a Token.Raw
.
Currently, if lexicalError
is true, this extractor will just return the next character
as the problematic item (this may be changed by overriding the extractItem
method).
Attributes
- Since:
4.0.0
- Companion:
- object
- Source:
- LexToken.scala
- Graph
- Supertypes
- Self type
- ErrorBuilder[_]