Lexer

class Lexer(lang: LanguageDef)

When provided with a LanguageDef, this class will produce a large variety of parsers that can be used for tokenisation of a language. This includes parsing numbers and strings in their various formats and ensuring that all operations consume whitespace after them (so-called lexeme parsers). These are very useful in parsing programming languages. This class also has a large number of hand-optimised intrinsic parsers to improve performance!

When provided with a LanguageDef, this class will produce a large variety of parsers that can be used for tokenisation of a language. This includes parsing numbers and strings in their various formats and ensuring that all operations consume whitespace after them (so-called lexeme parsers). These are very useful in parsing programming languages. This class also has a large number of hand-optimised intrinsic parsers to improve performance!

Value Params
lang

The rules that govern the language we are tokenising

Since

2.2.0

class Object
trait Matchable
class Any

Value members

Concrete methods

def angles[A](p: => Parsley[A]): Parsley[A]

Lexeme parser angles(p) parses p enclosed in angle brackets ('<', '>'), returning the value of p.

Lexeme parser angles(p) parses p enclosed in angle brackets ('<', '>'), returning the value of p.

def braces[A](p: => Parsley[A]): Parsley[A]

Lexeme parser braces(p) parses p enclosed in braces ('{', '}'), returning the value of 'p'

Lexeme parser braces(p) parses p enclosed in braces ('{', '}'), returning the value of 'p'

def brackets[A](p: => Parsley[A]): Parsley[A]

Lexeme parser brackets(p) parses p enclosed in brackets ('[', ']'), returning the value of p.

Lexeme parser brackets(p) parses p enclosed in brackets ('[', ']'), returning the value of p.

def commaSep[A](p: => Parsley[A]): Parsley[List[A]]

Lexeme parser commaSep(p) parses zero or more occurrences of p separated by comma. Returns a list of values returned by p.

Lexeme parser commaSep(p) parses zero or more occurrences of p separated by comma. Returns a list of values returned by p.

def commaSep1[A](p: => Parsley[A]): Parsley[List[A]]

Lexeme parser commaSep1(p) parses one or more occurrences of p separated by comma. Returns a list of values returned by p.

Lexeme parser commaSep1(p) parses one or more occurrences of p separated by comma. Returns a list of values returned by p.

def keyword(name: String): Parsley[Unit]

The lexeme parser keyword(name) parses the symbol name, but it also checks that the name is not a prefix of a valid identifier. A keyword is treated as a single token using attempt.

The lexeme parser keyword(name) parses the symbol name, but it also checks that the name is not a prefix of a valid identifier. A keyword is treated as a single token using attempt.

def lexeme[A](p: => Parsley[A]): Parsley[A]

lexeme(p) first applies parser p and then the whiteSpace parser, returning the value of p. Every lexical token (lexeme) is defined using lexeme, this way every parse starts at a point without white space. The only point where the whiteSpace parser should be called explicitly is the start of the main parser in order to skip any leading white space.

lexeme(p) first applies parser p and then the whiteSpace parser, returning the value of p. Every lexical token (lexeme) is defined using lexeme, this way every parse starts at a point without white space. The only point where the whiteSpace parser should be called explicitly is the start of the main parser in order to skip any leading white space.

def maxOp(name: String): Parsley[Unit]

The lexeme parser maxOp(name) parses the symbol name, but also checks that the name is not part of a larger reserved operator. An operator is treated as a single token using attempt.

The lexeme parser maxOp(name) parses the symbol name, but also checks that the name is not part of a larger reserved operator. An operator is treated as a single token using attempt.

def maxOp_(name: String): Parsley[Unit]

The non-lexeme parser maxOp_(name) parses the symbol name, but also checks that the name is not part of a larger reserved operator. An operator is treated as a single token using attempt.

The non-lexeme parser maxOp_(name) parses the symbol name, but also checks that the name is not part of a larger reserved operator. An operator is treated as a single token using attempt.

def operator(name: String): Parsley[Unit]

The lexeme parser operator(name) parses the symbol name, but also checks that the name is not the prefix of a valid operator. An operator is treated as a single token using attempt.

The lexeme parser operator(name) parses the symbol name, but also checks that the name is not the prefix of a valid operator. An operator is treated as a single token using attempt.

def operator_(name: String): Parsley[Unit]

The non-lexeme parser operator_(name) parses the symbol name, but also checks that the name is not the prefix of a valid operator. An operator is treated as a single token using attempt.

The non-lexeme parser operator_(name) parses the symbol name, but also checks that the name is not the prefix of a valid operator. An operator is treated as a single token using attempt.

def parens[A](p: => Parsley[A]): Parsley[A]

Lexeme parser parens(p) parses p enclosed in parenthesis, returning the value of p.

Lexeme parser parens(p) parses p enclosed in parenthesis, returning the value of p.

def semiSep[A](p: => Parsley[A]): Parsley[List[A]]

Lexeme parser semiSep(p) parses zero or more occurrences of p separated by semi. Returns a list of values returned by p.

Lexeme parser semiSep(p) parses zero or more occurrences of p separated by semi. Returns a list of values returned by p.

def semiSep1[A](p: => Parsley[A]): Parsley[List[A]]

Lexeme parser semiSep1(p) parses one or more occurrences of p separated by semi. Returns a list of values returned by p.

Lexeme parser semiSep1(p) parses one or more occurrences of p separated by semi. Returns a list of values returned by p.

def symbol(name: String): Parsley[String]

Lexeme parser symbol(s) parses string(s) and skips trailing white space.

Lexeme parser symbol(s) parses string(s) and skips trailing white space.

def symbol(name: Char): Parsley[Char]

Lexeme parser symbol(c) parses char(c) and skips trailing white space.

Lexeme parser symbol(c) parses char(c) and skips trailing white space.

def symbol_(name: String): Parsley[String]

Like symbol, but treats it as a single token using attempt. Only useful for strings, since characters are already single token.

Like symbol, but treats it as a single token using attempt. Only useful for strings, since characters are already single token.

Concrete fields

lazy val charLiteral: Parsley[Char]

This lexeme parser parses a single literal character. Returns the literal character value. This parser deals correctly with escape sequences. The literal character is parsed according to the grammar rules defined in the Haskell report (which matches most programming languages quite closely).

This lexeme parser parses a single literal character. Returns the literal character value. This parser deals correctly with escape sequences. The literal character is parsed according to the grammar rules defined in the Haskell report (which matches most programming languages quite closely).

val colon: Parsley[Char]

Lexeme parser colon parses the character ':' and skips any trailing white space. Returns ":"

Lexeme parser colon parses the character ':' and skips any trailing white space. Returns ":"

val comma: Parsley[Char]

Lexeme parser comma parses the character ',' and skips any trailing white space. Returns ","

Lexeme parser comma parses the character ',' and skips any trailing white space. Returns ","

lazy val decimal: Parsley[Int]

Parses a positive whole number in the decimal system. Returns the value of the number.

Parses a positive whole number in the decimal system. Returns the value of the number.

val dot: Parsley[Char]

Lexeme parser dot parses the character '.' and skips any trailing white space. Returns "."

Lexeme parser dot parses the character '.' and skips any trailing white space. Returns "."

lazy val float: Parsley[Double]

This lexeme parser parses a floating point value. Returns the value of the number. The number is parsed according to the grammar rules defined in the Haskell report. Accepts an optional '+' or '-' sign.

This lexeme parser parses a floating point value. Returns the value of the number. The number is parsed according to the grammar rules defined in the Haskell report. Accepts an optional '+' or '-' sign.

lazy val hexadecimal: Parsley[Int]

Parses a positive whole number in the hexadecimal system. The number should be prefixed with "0x" or "0X". Returns the value of the number.

Parses a positive whole number in the hexadecimal system. The number should be prefixed with "0x" or "0X". Returns the value of the number.

lazy val identifier: Parsley[String]

This lexeme parser parses a legal identifier. Returns the identifier string. This parser will fail on identifiers that are reserved words (i.e. keywords). Legal identifier characters and keywords are defined in the LanguageDef provided to the lexer. An identifier is treated as a single token using attempt.

This lexeme parser parses a legal identifier. Returns the identifier string. This parser will fail on identifiers that are reserved words (i.e. keywords). Legal identifier characters and keywords are defined in the LanguageDef provided to the lexer. An identifier is treated as a single token using attempt.

lazy val integer: Parsley[Int]

This lexeme parser parses an integer (a whole number). This parser is like natural except that it can be prefixed with a sign (i.e '-' or '+'). Returns the value of the number. The number can be specified in decimal, hexadecimal or octal. The number is parsed according to the grammar rules in the haskell report.

This lexeme parser parses an integer (a whole number). This parser is like natural except that it can be prefixed with a sign (i.e '-' or '+'). Returns the value of the number. The number can be specified in decimal, hexadecimal or octal. The number is parsed according to the grammar rules in the haskell report.

lazy val natural: Parsley[Int]

This lexeme parser parses a natural number (a positive whole number). Returns the value of the number. The number can specified in decimal, hexadecimal or octal. The number is parsed according to the grammar rules in the Haskell report.

This lexeme parser parses a natural number (a positive whole number). Returns the value of the number. The number can specified in decimal, hexadecimal or octal. The number is parsed according to the grammar rules in the Haskell report.

lazy val naturalOrFloat: Parsley[Either[Int, Double]]

This lexeme parser parses either natural or unsigned float. Returns the value of the number. This parser deals with any overlap in the grammar rules for naturals and floats. The number is parsed according to the grammar rules defined in the Haskell report.

This lexeme parser parses either natural or unsigned float. Returns the value of the number. This parser deals with any overlap in the grammar rules for naturals and floats. The number is parsed according to the grammar rules defined in the Haskell report.

lazy val number: Parsley[Either[Int, Double]]

This lexeme parser parses either integer or float. Returns the value of the number. This parser deals with any overlap in the grammar rules for naturals and floats. The number is parsed according to the grammar rules defined in the Haskell report.

This lexeme parser parses either integer or float. Returns the value of the number. This parser deals with any overlap in the grammar rules for naturals and floats. The number is parsed according to the grammar rules defined in the Haskell report.

lazy val octal: Parsley[Int]

Parses a positive whole number in the octal system. The number should be prefixed with "0o" or "0O". Returns the value of the number.

Parses a positive whole number in the octal system. The number should be prefixed with "0o" or "0O". Returns the value of the number.

lazy val rawStringLiteral: Parsley[String]

This non-lexeme parser parses a string in a raw fashion. The escape characters in the string remain untouched. While escaped quotes do not end the string, they remain as " in the result instead of becoming a quote character. Does not support string gaps.

This non-lexeme parser parses a string in a raw fashion. The escape characters in the string remain untouched. While escaped quotes do not end the string, they remain as " in the result instead of becoming a quote character. Does not support string gaps.

lazy val reservedOp: Parsley[String]

This lexeme parser parses a reserved operator. Returns the name of the operator. Legal operator characters and reserved operators are defined in the LanguageDef provided to the lexer. A reservedOp is treated as a single token using attempt.

This lexeme parser parses a reserved operator. Returns the name of the operator. Legal operator characters and reserved operators are defined in the LanguageDef provided to the lexer. A reservedOp is treated as a single token using attempt.

lazy val reservedOp_: Parsley[String]

This non-lexeme parser parses a reserved operator. Returns the name of the operator. Legal operator characters and reserved operators are defined in the LanguageDef provided to the lexer. A reservedOp_ is treated as a single token using attempt.

This non-lexeme parser parses a reserved operator. Returns the name of the operator. Legal operator characters and reserved operators are defined in the LanguageDef provided to the lexer. A reservedOp_ is treated as a single token using attempt.

val semi: Parsley[Char]

Lexeme parser semi parses the character ';' and skips any trailing white space. Returns ";"

Lexeme parser semi parses the character ';' and skips any trailing white space. Returns ";"

lazy val skipComments: Parsley[Unit]

Parses any comments and skips them, this includes both line comments and block comments.

Parses any comments and skips them, this includes both line comments and block comments.

lazy val stringLiteral: Parsley[String]

This lexeme parser parses a literal string. Returns the literal string value. This parser deals correctly with escape sequences and gaps. The literal string is parsed according to the grammar rules defined in the Haskell report (which matches most programming languages quite closely).

This lexeme parser parses a literal string. Returns the literal string value. This parser deals correctly with escape sequences and gaps. The literal string is parsed according to the grammar rules defined in the Haskell report (which matches most programming languages quite closely).

lazy val stringLiteral_: Parsley[String]

This non-lexeme parser parses a literal string. Returns the literal string value. This parser deals correctly with escape sequences and gaps. The literal string is parsed according to the grammar rules defined in the Haskell report (which matches most programming languages quite closely).

This non-lexeme parser parses a literal string. Returns the literal string value. This parser deals correctly with escape sequences and gaps. The literal string is parsed according to the grammar rules defined in the Haskell report (which matches most programming languages quite closely).

lazy val unsignedFloat: Parsley[Double]

This lexeme parser parses a floating point value. Returns the value of the number. The number is parsed according to the grammar rules defined in the Haskell report.

This lexeme parser parses a floating point value. Returns the value of the number. The number is parsed according to the grammar rules defined in the Haskell report.

lazy val userOp: Parsley[String]

This lexeme parser parses a legal operator. Returns the name of the operator. This parser will fail on any operators that are reserved operators. Legal operator characters and reserved operators are defined in the LanguageDef provided to the lexer. A userOp is treated as a single token using attempt.

This lexeme parser parses a legal operator. Returns the name of the operator. This parser will fail on any operators that are reserved operators. Legal operator characters and reserved operators are defined in the LanguageDef provided to the lexer. A userOp is treated as a single token using attempt.

lazy val whiteSpace: Parsley[Unit]

Parses any white space. White space consists of zero or more occurrences of a space (as provided by the LanguageDef), a line comment or a block (multi-line) comment. Block comments may be nested. How comments are started and ended is defined in the LanguageDef that is provided to the lexer.

Parses any white space. White space consists of zero or more occurrences of a space (as provided by the LanguageDef), a line comment or a block (multi-line) comment. Block comments may be nested. How comments are started and ended is defined in the LanguageDef that is provided to the lexer.

val whiteSpace_: Impl => Parsley[Unit]

Parses any white space. White space consists of zero or more occurrences of a space (as provided by the parameter), a line comment or a block (multi-line) comment. Block comments may be nested. How comments are started and ended is defined in the LanguageDef that is provided to the lexer.

Parses any white space. White space consists of zero or more occurrences of a space (as provided by the parameter), a line comment or a block (multi-line) comment. Block comments may be nested. How comments are started and ended is defined in the LanguageDef that is provided to the lexer.