parsley
5.0.0-M2
API
Generated with
parsley/parsley/parsley.token/Lexer/lexeme

lexeme

parsley.token.Lexer.lexeme
object lexeme

This object is concerned with lexemes: these are tokens that are treated as "words", such that whitespace will be consumed after each has been parsed.

Ideally, a wider parser should not be concerned with handling whitespace, as it is responsible for dealing with a stream of tokens. With parser combinators, however, it is usually not the case that there is a separate distinction between the parsing phase and the lexing phase. That said, it is good practice to establish a logical separation between the two worlds. As such, this object contains parsers that parse tokens, and these are whitespace-aware. This means that whitespace will be consumed after any of these parsers are parsed. It is not, however, required that whitespace be present.

Attributes

Since

4.0.0

Source
Lexer.scala
Graph
Supertypes
class Object
trait Matchable
class Any
Self type
lexeme.type

Members list

Value members

Concrete methods

def angles[A](p: => Parsley[A]): Parsley[A]

This combinator parses a p enclosed within angle brackets.

This combinator parses a p enclosed within angle brackets.

First parse an open bracket, any whitespace, then parse, p, producing x. Finally, parse a closing bracket and any whitespace. If all three parts succeeded, then return x. If any of them failed, this combinator fails.

Value parameters

p

the parser to parse between parentheses.

Attributes

Returns

a parser that reads an open bracket, then p, then a closing bracket and returns the result of p.

Since

4.5.0

Example 

scala> ...
scala> val p = lexer.nonlexeme.enclosing.brackets(int)
scala> p.parse("< 5>")
val res0 = Success(5)
scala> p.parse("<5")
val res1 = Failure(...)
scala> p.parse("5>")
val res2 = Failure(...)
Source
Lexer.scala
def apply[A](p: Parsley[A]): Parsley[A]

This combinator turns a non-lexeme parser into a lexeme one by ensuring whitespace is consumed after the parser.

This combinator turns a non-lexeme parser into a lexeme one by ensuring whitespace is consumed after the parser.

When using parser combinators, it is important to establish a consistent whitespace consumption scheme: ideally, there is no wasteful parsing, and whitespace consumption should not impact backtracking. This leads to a convention that whitespace must only be consumed after a token, and only once at the very start of the parser (see fully). When manually constructing tokens that are not supported by this lexer, use this combinator to ensure it also follows the whitespace convention.

Value parameters

p

the token parser to ensure consumes trailing whitespace.

Attributes

Since

4.0.0

Source
Lexer.scala
def braces[A](p: => Parsley[A]): Parsley[A]

This combinator parses a p enclosed within braces.

This combinator parses a p enclosed within braces.

First parse an open brace, any whitespace, then parse, p, producing x. Finally, parse a closing brace and any whitespace. If all three parts succeeded, then return x. If any of them failed, this combinator fails.

Value parameters

p

the parser to parse between parentheses.

Attributes

Returns

a parser that reads an open brace, then p, then a closing brace and returns the result of p.

Since

4.5.0

Example 

scala> ...
scala> val p = lexer.nonlexeme.enclosing.braces(int)
scala> p.parse("{ 5}")
val res0 = Success(5)
scala> p.parse("{5")
val res1 = Failure(...)
scala> p.parse("5}")
val res2 = Failure(...)
Source
Lexer.scala
def brackets[A](p: => Parsley[A]): Parsley[A]

This combinator parses a p enclosed within square brackets.

This combinator parses a p enclosed within square brackets.

First parse an open bracket, any whitespace, then parse, p, producing x. Finally, parse a closing bracket and any whitespace. If all three parts succeeded, then return x. If any of them failed, this combinator fails.

Value parameters

p

the parser to parse between parentheses.

Attributes

Returns

a parser that reads an open bracket, then p, then a closing bracket and returns the result of p.

Since

4.5.0

Example 

scala> ...
scala> val p = lexer.nonlexeme.enclosing.brackets(int)
scala> p.parse("[ 5]")
val res0 = Success(5)
scala> p.parse("[5")
val res1 = Failure(...)
scala> p.parse("5]")
val res2 = Failure(...)
Source
Lexer.scala
def character: CharacterParsers

This is a collection of parsers concerned with handling character literals.

This is a collection of parsers concerned with handling character literals.

Character literals are described generally as follows:

  • desc.textDesc.characterLiteralEnd: the character that starts and ends the literal (for example in many languages this is ')

  • desc.textDesc.graphicCharacter: describes the legal characters that may appear in the literal directly. Usually, this excludes control characters and newlines, but permits most other things. Escape sequences can represent non-graphic characters

  • desc.textDesc.escapeSequences: describes the legal escape sequences that that can appear in a character literal (for example \n or \u000a)

Aside from the generic configuration, characters can be parsed in accordance with varying levels of unicode support, from ASCII-only to full UTF-16 characters. Parsers for each of four different vareties are exposed by this object.

Attributes

Since

4.5.0

Source
Lexer.scala
def commaSep[A](p: Parsley[A]): Parsley[List[A]]

This combinator parses zero or more occurrences of p, separated by commas.

This combinator parses zero or more occurrences of p, separated by commas.

Behaves just like commaSep1, except does not require an initial p, returning the empty list instead.

Value parameters

p

the parser whose results are collected into a list.

Attributes

Returns

a parser that parses p delimited by commas, returning the list of p's results.

Since

4.5.0

Example 

scala> ...
scala> val stmts = lexer.lexeme.separators.commaSep(int)
scala> stmts.parse("7, 3,2")
val res0 = Success(List(7, 3, 2))
scala> stmts.parse("")
val res1 = Success(Nil)
scala> stmts.parse("1")
val res2 = Success(List(1))
scala> stmts.parse("1, 2, ")
val res3 = Failure(..) // no trailing comma allowed
Source
Lexer.scala
def commaSep1[A](p: Parsley[A]): Parsley[List[A]]

This combinator parses one or more occurrences of p, separated by commas.

This combinator parses one or more occurrences of p, separated by commas.

First parses a p. Then parses a comma followed by p until there are no more commas. The results of the p's, x1 through xn, are returned as List(x1, .., xn). If p fails having consumed input, the whole parser fails. Requires at least one p to have been parsed.

Value parameters

p

the parser whose results are collected into a list.

Attributes

Returns

a parser that parses p delimited by commas, returning the list of p's results.

Since

4.5.0

Example 

scala> ...
scala> val stmts = lexer.lexeme.separators.commaSep1(int)
scala> stmts.parse("7, 3,2")
val res0 = Success(List(7, 3, 2))
scala> stmts.parse("")
val res1 = Failure(..)
scala> stmts.parse("1")
val res2 = Success(List(1))
scala> stmts.parse("1, 2, ")
val res3 = Failure(..) // no trailing comma allowed
Source
Lexer.scala
def floating: RealParsers

This is a collection of parsers concerned with handling signed real numbers (like floats and doubles).

This is a collection of parsers concerned with handling signed real numbers (like floats and doubles).

These literals consist of a (possibly optional) integer prefix, with at least one of a fractional component (with .) or an exponential component.

Real numbers are an extension of signed integers with the following additional configuration:

  • desc.numericDesc.leadingDotAllowed: determines whether a literal like .0 would be considered legal

  • desc.numericDesc.trailingDotAllowed: determines whether a literal like 0. would be considered legal

  • desc.numericDesc.realNumbersCanBe{Hexadecimal/Octal/Binary}: these flags control what kind of literals can appear within the number parser. Each type of literal may still be individually parsed with its corresponding parser, regardless of the value of the flag

  • desc.numericDesc.{decimal/hexadecimal/octal/binary}ExponentDesc: describes how the exponential syntax works for each kind of base. If the syntax is legal, then this describes: which characters start it (classically, this would be e or E for decimals); whether or not it is compulsory for the literal (in Java and C, hexadecimal floats are only valid when they have an exponent attached); and whether or not a + sign is mandatory, optional, or illegal for positive exponents

Additional to the parsing of decimal, hexadecimal, octal, and binary floating literals, each parser can be given a precision of IEEE 754 float or double. This can either be achieved by rounding to the nearest representable value, or by ensuring that the literal must be precisely representable as one of these numbers (which is defined as being one of binary, decimal or exact float and double values as described by Java)

Attributes

See also

natural and integer for a full description of the configuration for the start of a real number

Since

4.5.0

Note

alias for real

Source
Lexer.scala
def integer: IntegerParsers

This is a collection of parsers concerned with handling signed integer literals.

This is a collection of parsers concerned with handling signed integer literals.

Signed integer literals are an extension of unsigned integer literals with the following extra configuration:

  • desc.numericDesc.positiveSign: describes whether or not literals are allowed to omit + for positive literals, must write a +, or can never write a +.

Attributes

See also

natural for a full description of integer configuration

Since

4.5.0

Source
Lexer.scala
def multiString: StringParsers

This is a collection of parsers concerned with handling multi-line string literals.

This is a collection of parsers concerned with handling multi-line string literals.

String literals are described generally as follows:

  • desc.textDesc.multiStringEnds: the sequence of characters that can begin or end a multi-line string literal. Regardless of which of these is used for a specific literal, the end of the literal must use the same sequence

  • desc.textDesc.graphicCharacter: describes the legal characters that may appear in the literal directly. Usually, this excludes control characters and newlines, but permits most other things. Escape sequences can represent non-graphic characters for non-raw strings

  • desc.textDesc.escapeSequences: describes the legal escape sequences that that can appear in a string literal (for example \n or \u000a)

Attributes

Since

4.5.0

Source
Lexer.scala
def natural: IntegerParsers

This is a collection of parsers concerned with handling unsigned (positive) integer literals.

This is a collection of parsers concerned with handling unsigned (positive) integer literals.

Natural numbers are described generally as follows:

  • desc.numericDesc.literalBreakChar: determines whether or not it is legal to "break up" the digits within a literal, for example: is 1_000_000 allowed? If this is legal, describes what the break character is, and whether it can appear after a hexadecimal/octal/binary prefix

  • desc.numericDesc.leadingZerosAllowed: determines whether or not it is possible to add extraneous zero digits onto the front of a number or not. In some languages, like C, this is disallowed, as numbers starting with 0 are octal numbers.

  • desc.numericDesc.integerNumbersCanBe{Hexadecimal/Octal/Binary}: these flags control what kind of literals can appear within the number parser. Each type of literal can be individually parsed with its corresponding parser, regardless of the value of the flag

  • desc.numericDesc.{hexadecimal/octal/binary}Leads: controls what character must follow a 0 when starting a number to change it from decimal into another base. This set may be empty, in which case the literal is described purely with leading zero (C style octals would set octalLeads to Set.empty)

Additional to the parsing of decimal, hexadecimal, octal, and binary literals, each parser can be given a bit-width from 8- to 64-bit: this will check the parsed literal to ensure it is a legal literal of that size.

Attributes

Since

4.5.0

Source
Lexer.scala
def parens[A](p: => Parsley[A]): Parsley[A]

This combinator parses a p enclosed within parentheses.

This combinator parses a p enclosed within parentheses.

First parse an open parenthesis, any whitespace, then parse, p, producing x. Finally, parse a closing parenthesis and any whitespace. If all three parts succeeded, then return x. If any of them failed, this combinator fails.

Value parameters

p

the parser to parse between parentheses.

Attributes

Returns

a parser that reads an open parenthesis, then p, then a closing parenthesis and returns the result of p.

Since

4.5.0

Example 

scala> ...
scala> val p = lexer.nonlexeme.enclosing.parens(int)
scala> p.parse("( 5)")
val res0 = Success(5)
scala> p.parse("(5")
val res1 = Failure(...)
scala> p.parse("5)")
val res2 = Failure(...)
Source
Lexer.scala
def rawMultiString: StringParsers

This is a collection of parsers concerned with handling multi-line string literals.

This is a collection of parsers concerned with handling multi-line string literals.

String literals are described generally as follows:

  • desc.textDesc.multiStringEnds: the sequence of characters that can begin or end a multi-line string literal. Regardless of which of these is used for a specific literal, the end of the literal must use the same sequence

  • desc.textDesc.graphicCharacter: describes the legal characters that may appear in the literal directly. Usually, this excludes control characters and newlines, but permits most other things. Escape sequences can represent non-graphic characters for non-raw strings

  • desc.textDesc.escapeSequences: describes the legal escape sequences that that can appear in a string literal (for example \n or \u000a)

Attributes

Since

4.5.0

Note

this will be parsed without handling any escape sequences, this includes literal-end characters and the escape prefix (often " and \ respectively)

Source
Lexer.scala
def rawString: StringParsers

This is a collection of parsers concerned with handling single-line string literals.

This is a collection of parsers concerned with handling single-line string literals.

String literals are described generally as follows:

  • desc.textDesc.stringEnds: the sequence of characters that can begin or end a string literal. Regardless of which of these is used for a specific literal, the end of the literal must use the same sequence

  • desc.textDesc.graphicCharacter: describes the legal characters that may appear in the literal directly. Usually, this excludes control characters and newlines, but permits most other things. Escape sequences can represent non-graphic characters for non-raw strings

  • desc.textDesc.escapeSequences: describes the legal escape sequences that that can appear in a string literal (for example \n or \u000a)

Attributes

Since

4.5.0

Note

this will be parsed without handling any escape sequences, this includes literal-end characters and the escape prefix (often " and \ respectively)

Source
Lexer.scala
def real: RealParsers

This is a collection of parsers concerned with handling signed real numbers (like floats and doubles).

This is a collection of parsers concerned with handling signed real numbers (like floats and doubles).

These literals consist of a (possibly optional) integer prefix, with at least one of a fractional component (with .) or an exponential component.

Real numbers are an extension of signed integers with the following additional configuration:

  • desc.numericDesc.leadingDotAllowed: determines whether a literal like .0 would be considered legal

  • desc.numericDesc.trailingDotAllowed: determines whether a literal like 0. would be considered legal

  • desc.numericDesc.realNumbersCanBe{Hexadecimal/Octal/Binary}: these flags control what kind of literals can appear within the number parser. Each type of literal may still be individually parsed with its corresponding parser, regardless of the value of the flag

  • desc.numericDesc.{decimal/hexadecimal/octal/binary}ExponentDesc: describes how the exponential syntax works for each kind of base. If the syntax is legal, then this describes: which characters start it (classically, this would be e or E for decimals); whether or not it is compulsory for the literal (in Java and C, hexadecimal floats are only valid when they have an exponent attached); and whether or not a + sign is mandatory, optional, or illegal for positive exponents

Additional to the parsing of decimal, hexadecimal, octal, and binary floating literals, each parser can be given a precision of IEEE 754 float or double. This can either be achieved by rounding to the nearest representable value, or by ensuring that the literal must be precisely representable as one of these numbers (which is defined as being one of binary, decimal or exact float and double values as described by Java)

Attributes

See also

natural and integer for a full description of the configuration for the start of a real number

Since

4.5.0

Source
Lexer.scala
def semiSep[A](p: Parsley[A]): Parsley[List[A]]

This combinator parses zero or more occurrences of p, separated by semi-colons.

This combinator parses zero or more occurrences of p, separated by semi-colons.

Behaves just like semiSep1, except does not require an initial p, returning the empty list instead.

Value parameters

p

the parser whose results are collected into a list.

Attributes

Returns

a parser that parses p delimited by semi-colons, returning the list of p's results.

Since

4.5.0

Example 

scala> ...
scala> val stmts = lexer.lexeme.separators.semiSep(int)
scala> stmts.parse("7; 3;2")
val res0 = Success(List(7; 3; 2))
scala> stmts.parse("")
val res1 = Success(Nil)
scala> stmts.parse("1")
val res2 = Success(List(1))
scala> stmts.parse("1; 2; ")
val res3 = Failure(..) // no trailing semi-colon allowed
Source
Lexer.scala
def semiSep1[A](p: Parsley[A]): Parsley[List[A]]

This combinator parses one or more occurrences of p, separated by semi-colons.

This combinator parses one or more occurrences of p, separated by semi-colons.

First parses a p. Then parses a semi-colon followed by p until there are no more semi-colons. The results of the p's, x1 through xn, are returned as List(x1, .., xn). If p fails having consumed input, the whole parser fails. Requires at least one p to have been parsed.

Value parameters

p

the parser whose results are collected into a list.

Attributes

Returns

a parser that parses p delimited by semi-colons, returning the list of p's results.

Since

4.5.0

Example 

scala> ...
scala> val stmts = lexer.lexeme.separators.semiSep1(int)
scala> stmts.parse("7; 3;2")
val res0 = Success(List(7; 3; 2))
scala> stmts.parse("")
val res1 = Failure(..)
scala> stmts.parse("1")
val res2 = Success(List(1))
scala> stmts.parse("1; 2; ")
val res3 = Failure(..) // no trailing semi-colon allowed
Source
Lexer.scala
def signed: IntegerParsers

This is a collection of parsers concerned with handling signed integer literals.

This is a collection of parsers concerned with handling signed integer literals.

Signed integer literals are an extension of unsigned integer literals with the following extra configuration:

  • desc.numericDesc.positiveSign: describes whether or not literals are allowed to omit + for positive literals, must write a +, or can never write a +.

Attributes

See also

unsigned for a full description of signed integer configuration

Since

4.5.0

Note

alias for integer

Source
Lexer.scala
def signedCombined: CombinedParsers

This is a collection of parsers concerned with handling numeric literals that may either be signed integers or signed reals.

This is a collection of parsers concerned with handling numeric literals that may either be signed integers or signed reals.

There is no additional configuration offered over that found in integer or real.

the bit-bounds and precision of the integer or real parts of the result can be specified in any pairing.

Attributes

Since

4.5.0

Source
Lexer.scala
def string: StringParsers

This is a collection of parsers concerned with handling single-line string literals.

This is a collection of parsers concerned with handling single-line string literals.

String literals are described generally as follows:

  • desc.textDesc.stringEnds: the sequence of characters that can begin or end a string literal. Regardless of which of these is used for a specific literal, the end of the literal must use the same sequence

  • desc.textDesc.graphicCharacter: describes the legal characters that may appear in the literal directly. Usually, this excludes control characters and newlines, but permits most other things. Escape sequences can represent non-graphic characters for non-raw strings

  • desc.textDesc.escapeSequences: describes the legal escape sequences that that can appear in a string literal (for example \n or \u000a)

Attributes

Since

4.5.0

Source
Lexer.scala
def unsigned: IntegerParsers

This is a collection of parsers concerned with handling unsigned (positive) integer literals.

This is a collection of parsers concerned with handling unsigned (positive) integer literals.

Natural numbers are described generally as follows:

  • desc.numericDesc.literalBreakChar: determines whether or not it is legal to "break up" the digits within a literal, for example: is 1_000_000 allowed? If this is legal, describes what the break character is, and whether it can appear after a hexadecimal/octal/binary prefix

  • desc.numericDesc.leadingZerosAllowed: determines whether or not it is possible to add extraneous zero digits onto the front of a number or not. In some languages, like C, this is disallowed, as numbers starting with 0 are octal numbers.

  • desc.numericDesc.integerNumbersCanBe{Hexadecimal/Octal/Binary}: these flags control what kind of literals can appear within the number parser. Each type of literal can be individually parsed with its corresponding parser, regardless of the value of the flag

  • desc.numericDesc.{hexadecimal/octal/binary}Leads: controls what character must follow a 0 when starting a number to change it from decimal into another base. This set may be empty, in which case the literal is described purely with leading zero (C style octals would set octalLeads to Set.empty)

Additional to the parsing of decimal, hexadecimal, octal, and binary literals, each parser can be given a bit-width from 8- to 64-bit: this will check the parsed literal to ensure it is a legal literal of that size.

Attributes

Since

4.5.0

Note

alias for natural.

Source
Lexer.scala
def unsignedCombined: CombinedParsers

This is a collection of parsers concerned with handling numeric literals that may either be unsigned integers or unsigned reals.

This is a collection of parsers concerned with handling numeric literals that may either be unsigned integers or unsigned reals.

There is no additional configuration offered over that found in natural or real.

the bit-bounds and precision of the integer or real parts of the result can be specified in any pairing.

Attributes

Since

4.5.0

Source
Lexer.scala

Concrete fields

val names: Names

This object contains lexing functionality relevant to the parsing of names, which include operators or identifiers.

This object contains lexing functionality relevant to the parsing of names, which include operators or identifiers.

The parsing of names is mostly concerned with finding the longest valid name that is not a reserved name, such as a hard keyword or a special operator.

Attributes

Since

4.0.0

Source
Lexer.scala
val symbol: Symbol

This object contains lexing functionality relevant to the parsing of atomic symbols.

This object contains lexing functionality relevant to the parsing of atomic symbols.

Symbols are characterised by their "unitness", that is, every parser inside returns Unit. This is because they all parse a specific known entity, and, as such, the result of the parse is irrelevant. These can be things such as reserved names, or small symbols like parentheses. This object also contains a means of creating new symbols as well as implicit conversions to allow for Scala's string literals to serve as symbols within a parser.

Attributes

Since

4.0.0

Source
Lexer.scala
Generated with