trait LexToken extends AnyRef
This extractor mixin provides an implementation for
ErrorBuilder.unexpectedToken
when mixed into
an error builder: it will try and parse the residual input to identify a valid lexical token
to report.
When parsing a grammar that as a dedicated lexical distinction, it is nice to be able to report problematic tokens relevant to that grammar as opposed to generic input lifted straight from the input stream. The easiest way of doing this would be having a pre-lexing pass and parsing based on tokens, but this is deliberately not how Parsley is designed. Instead, this extractor can try and parse the remaining input to try and identify a token on demand.
If the lexicalError
flag of the unexpectedToken
method is not set, which would indicate a
problem within a token reported by a classical lexer and not the parser, the extractor will
try to parse each of the provided tokens
in turn: whichever is the longest matched of these
tokens will be reported as the problematic one (this can be changed by overriding selectToken
).
For best effect, these tokens should not consume
whitespace (which would otherwise be included at the end of the token!): this means that, if
using the Lexer
class, the functionality in nonlexeme
should be used. If one of the
givens tokens cannot be parsed, the input until the next valid parsable token (or end of input)
is returned as a Token.Raw
.
Currently, if lexicalError
is true, this extractor will just return the next character
as the problematic item (this may be changed by overriding the extractItem
method).
- Self Type
- LexToken with ErrorBuilder[_]
- Source
- LexToken.scala
- Since
4.0.0
- Alphabetic
- By Inheritance
- LexToken
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Abstract Value Members
- abstract def tokens: Seq[Parsley[String]]
The tokens that should be recognised by this extractor: each parser should return the intended name of the token exactly as it should appear in the
Named
token.The tokens that should be recognised by this extractor: each parser should return the intended name of the token exactly as it should appear in the
Named
token.This should include a whitespace parser for "unexpected whitespace".
- Since
4.0.0
- Note
with the exception of the whitespace parser, these tokens should not consume trailing (and certainly not leading) whitespace: if using definitions from
parsley.token.Lexer
functionality, thenonlexeme
versions of the tokens should be used.
Concrete Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @native()
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- def extractItem(cs: Iterable[Char], amountOfInputParserWanted: Int): Token
If the parser failed during the parsing of a token, this function extracts the problematic item from the remaining input.
If the parser failed during the parsing of a token, this function extracts the problematic item from the remaining input.
The default behaviour mimics
SingleChar
.- Since
4.0.0
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable])
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- def selectToken(matchedToks: List[(String, (Int, Int))]): (String, (Int, Int))
If the extractor is successful in identifying tokens that can be parsed from the residual input, this function will select one of them to report back.
If the extractor is successful in identifying tokens that can be parsed from the residual input, this function will select one of them to report back.
The default behaviour is to take the longest matched token (i.e. the one with the largest paired position).
- matchedToks
the list of tokens successfully parsed, along with the position at the end of that parse (careful: this position starts back at
(1, 1)
, not where the original parser left off!)- returns
the chosen token and position pair
- Since
4.0.0
- Note
the
matchedToks
list is guaranteed to be non-empty
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def unexpectedToken(cs: Iterable[Char], amountOfInputParserWanted: Int, lexicalError: Boolean): Token
- See also
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
This is the documentation for Parsley.
Package structure
The parsley package contains the
Parsley
class, as well as theResult
,Success
, andFailure
types. In addition to these, it also contains the following packages and "modules" (a module is defined as being an object which mocks a package):parsley.Parsley
contains the bulk of the core "function-style" combinators.parsley.combinator
contains many helpful combinators that simplify some common parser patterns.parsley.character
contains the combinators needed to read characters and strings, as well as combinators to match specific sub-sets of characters.parsley.debug
contains debugging combinators, helpful for identifying faults in parsers.parsley.extension
contains syntactic sugar combinators exposed as implicit classes.parsley.io
contains extension methods to run parsers with input sourced from IO sources.parsley.expr
contains the following sub modules:parsley.expr.chain
contains combinators used in expression parsingparsley.expr.precedence
is a builder for expression parsers built on a precedence table.parsley.expr.infix
contains combinators used in expression parsing, but with more permissive types than their equivalents inchain
.parsley.expr.mixed
contains combinators that can be used for expression parsing, but where different fixities may be mixed on the same level: this is rare in practice.parsley.implicits
contains several implicits to add syntactic sugar to the combinators. These are sub-categorised into the following sub modules:parsley.implicits.character
contains implicits to allow you to use character and string literals as parsers.parsley.implicits.combinator
contains implicits related to combinators, such as the ability to make any parser into aParsley[Unit]
automatically.parsley.implicits.lift
enables postfix application of the lift combinator onto a function (or value).parsley.implicits.zipped
enables boths a reversed form of lift where the function appears on the right and is applied on a tuple (useful when type inference has failed) as well as a.zipped
method for building tuples out of several combinators.parsley.errors
contains modules to deal with error messages, their refinement and generation.parsley.errors.combinator
provides combinators that can be used to either produce more detailed errors as well as refine existing errors.parsley.errors.tokenextractors
provides mixins for common token extraction strategies during error message generation: these can be used to avoid implementingunexpectedToken
in theErrorBuilder
.parsley.lift
contains functions which lift functions that work on regular types to those which now combine the results of parsers returning those same types. these are ubiquitous.parsley.ap
contains functions which allow for the application of a parser returning a function to several parsers returning each of the argument types.parsley.registers
contains combinators that interact with the context-sensitive functionality in the form of registers.parsley.token
contains theLexer
class that provides a host of helpful lexing combinators when provided with the description of a language.parsley.position
contains parsers for extracting position information.parsley.genericbridges
contains some basic implementations of the Parser Bridge pattern (see Design Patterns for Parser Combinators in Scala, or the parsley wiki): these can be used before more specialised generic bridge traits can be constructed.