parsley/parsley.token

parsley.token

package parsley.token

Type members

Classlikes

object CharSet

This implementation uses a set of valid tokens. It is converted to a high-performance BitSet.

This implementation uses a set of valid tokens. It is converted to a high-performance BitSet.

Since

2.2.0

sealed trait Impl

The Impl trait is used to provide implementation of the parser requirements from LanguageDef

The Impl trait is used to provide implementation of the parser requirements from LanguageDef

Since

2.2.0

case class LanguageDef(commentStart: String, commentEnd: String, commentLine: String, nestedComments: Boolean, identStart: Impl, identLetter: Impl, opStart: Impl, opLetter: Impl, keywords: Set[String], operators: Set[String], caseSensitive: Boolean, space: Impl)

This class is required to construct a TokenParser. It defines the various characteristics of the language to be tokenised. Where a parameter can be either a Set[Char] or a Parsley object, prefer the Set where possible. It will unlock a variety of faster intrinsic versions of the parsers, which will greatly improve tokenisation performance! In addition, the Sets are one time converted to heavily optimised BitSets, though that has up to 8KB memory usage associated but at least doubles the execution speed for that instruction. See parsley.Impl.

This class is required to construct a TokenParser. It defines the various characteristics of the language to be tokenised. Where a parameter can be either a Set[Char] or a Parsley object, prefer the Set where possible. It will unlock a variety of faster intrinsic versions of the parsers, which will greatly improve tokenisation performance! In addition, the Sets are one time converted to heavily optimised BitSets, though that has up to 8KB memory usage associated but at least doubles the execution speed for that instruction. See parsley.Impl.

Value Params
caseSensitive

Is the language case-sensitive. I.e. is IF equivalent to if?

commentEnd

For multi-line comments; how does the comment end? (If this or commentEnd is the empty string, multi-line comments are disabled)

commentLine

For single-line comments; how does the comment start? (This this is the empty string, single-line comments are disabled)

commentStart

For multi-line comments; how does the comment start? (If this or commentEnd is the empty string, multi-line comments are disabled)

identLetter

What characters can an identifier in the language consist of after the starting character?

identStart

What characters can an identifier in the language start with?

keywords

What keywords does the language contain?

nestedComments

Are multi-line comments allowed to be nested inside each other? E.g. If {- and -} are opening and closing comments, is the following valid syntax: {-{-hello -}-}? Note in C this is not the case.

opLetter

What characters can an operator in the language consist of after the starting character?

opStart

What characters can an operator in the language start with?

operators

What operators does the language contain?

space

What characters count as whitespace in the language?

Since

2.2.0

Companion
object
object LanguageDef

This object contains any preconfigured language definitions

This object contains any preconfigured language definitions

Since

2.2.0

Companion
class
class Lexer(lang: LanguageDef)

When provided with a LanguageDef, this class will produce a large variety of parsers that can be used for tokenisation of a language. This includes parsing numbers and strings in their various formats and ensuring that all operations consume whitespace after them (so-called lexeme parsers). These are very useful in parsing programming languages. This class also has a large number of hand-optimised intrinsic parsers to improve performance!

When provided with a LanguageDef, this class will produce a large variety of parsers that can be used for tokenisation of a language. This includes parsing numbers and strings in their various formats and ensuring that all operations consume whitespace after them (so-called lexeme parsers). These are very useful in parsing programming languages. This class also has a large number of hand-optimised intrinsic parsers to improve performance!

Value Params
lang

The rules that govern the language we are tokenising

Since

2.2.0

case object NotRequired extends Impl

This implementation states that the required functionality is not required. If it is used it will raise an error at parse-time

This implementation states that the required functionality is not required. If it is used it will raise an error at parse-time

Since

2.2.0

final case class Parser(p: Parsley[_]) extends Impl

The implementation provided is a parser which parses the required token.

The implementation provided is a parser which parses the required token.

Value Params
p

The parser which will parse the token

Since

2.2.0

final case class Predicate(f: Char => Boolean) extends Impl

The implementation provided is a function which matches on the input streams characters

The implementation provided is a function which matches on the input streams characters

Value Params
f

The predicate that input tokens are tested against

Since

2.2.0

Deprecated classlikes

@deprecated("This will be removed in Parsley 3.0, use Predicate instead", "2.8.4")
object BitGen

This implementation uses a predicate to generate a BitSet. This should be preferred over Predicate when the function in question is expensive to execute and the parser itself is expected to be used many times. If the predicate is cheap, this is unlikely to provide any performance improvements, but will instead incur heavy space costs

This implementation uses a predicate to generate a BitSet. This should be preferred over Predicate when the function in question is expensive to execute and the parser itself is expected to be used many times. If the predicate is cheap, this is unlikely to provide any performance improvements, but will instead incur heavy space costs

Since

2.2.0

Deprecated