Lexer

class Lexer(lang: LanguageDef)

When provided with a LanguageDef, this class will produce a large variety of parsers that can be used for tokenisation of a language. This includes parsing numbers and strings in their various formats and ensuring that all operations consume whitespace after them (so-called lexeme parsers). These are very useful in parsing programming languages. This class also has a large number of hand-optimised intrinsic parsers to improve performance!

Value Params
lang

The rules that govern the language we are tokenising

Since

2.2.0

class Object
trait Matchable
class Any

Value members

Concrete methods

def angles[A](p: => Parsley[A]): Parsley[A]

Lexeme parser angles(p) parses p enclosed in angle brackets ('<', '>'), returning the value of p.

Lexeme parser angles(p) parses p enclosed in angle brackets ('<', '>'), returning the value of p.

def braces[A](p: => Parsley[A]): Parsley[A]

Lexeme parser braces(p) parses p enclosed in braces ('{', '}'), returning the value of 'p'

Lexeme parser braces(p) parses p enclosed in braces ('{', '}'), returning the value of 'p'

def brackets[A](p: => Parsley[A]): Parsley[A]

Lexeme parser brackets(p) parses p enclosed in brackets ('[', ']'), returning the value of p.

Lexeme parser brackets(p) parses p enclosed in brackets ('[', ']'), returning the value of p.

def commaSep[A](p: => Parsley[A]): Parsley[List[A]]

Lexeme parser commaSep(p) parses zero or more occurrences of p separated by comma. Returns a list of values returned by p.

Lexeme parser commaSep(p) parses zero or more occurrences of p separated by comma. Returns a list of values returned by p.

def commaSep1[A](p: => Parsley[A]): Parsley[List[A]]

Lexeme parser commaSep1(p) parses one or more occurrences of p separated by comma. Returns a list of values returned by p.

Lexeme parser commaSep1(p) parses one or more occurrences of p separated by comma. Returns a list of values returned by p.

def keyword(name: String): Parsley[Unit]

The lexeme parser keyword(name) parses the symbol name, but it also checks that the name is not a prefix of a valid identifier. A keyword is treated as a single token using attempt.

The lexeme parser keyword(name) parses the symbol name, but it also checks that the name is not a prefix of a valid identifier. A keyword is treated as a single token using attempt.

def lexeme[A](p: => Parsley[A]): Parsley[A]

lexeme(p) first applies parser p and then the whiteSpace parser, returning the value of p. Every lexical token (lexeme) is defined using lexeme, this way every parse starts at a point without white space. The only point where the whiteSpace parser should be called explicitly is the start of the main parser in order to skip any leading white space.

lexeme(p) first applies parser p and then the whiteSpace parser, returning the value of p. Every lexical token (lexeme) is defined using lexeme, this way every parse starts at a point without white space. The only point where the whiteSpace parser should be called explicitly is the start of the main parser in order to skip any leading white space.

def maxOp(name: String): Parsley[Unit]

The lexeme parser maxOp(name) parses the symbol name, but also checks that the name is not part of a larger reserved operator. An operator is treated as a single token using attempt.

The lexeme parser maxOp(name) parses the symbol name, but also checks that the name is not part of a larger reserved operator. An operator is treated as a single token using attempt.

def maxOp_(name: String): Parsley[Unit]

The non-lexeme parser maxOp_(name) parses the symbol name, but also checks that the name is not part of a larger reserved operator. An operator is treated as a single token using attempt.

The non-lexeme parser maxOp_(name) parses the symbol name, but also checks that the name is not part of a larger reserved operator. An operator is treated as a single token using attempt.

def operator(name: String): Parsley[Unit]

The lexeme parser operator(name) parses the symbol name, but also checks that the name is not the prefix of a valid operator. An operator is treated as a single token using attempt.

The lexeme parser operator(name) parses the symbol name, but also checks that the name is not the prefix of a valid operator. An operator is treated as a single token using attempt.

def operator_(name: String): Parsley[Unit]

The non-lexeme parser operator_(name) parses the symbol name, but also checks that the name is not the prefix of a valid operator. An operator is treated as a single token using attempt.

The non-lexeme parser operator_(name) parses the symbol name, but also checks that the name is not the prefix of a valid operator. An operator is treated as a single token using attempt.

def parens[A](p: => Parsley[A]): Parsley[A]

Lexeme parser parens(p) parses p enclosed in parenthesis, returning the value of p.

Lexeme parser parens(p) parses p enclosed in parenthesis, returning the value of p.

def semiSep[A](p: => Parsley[A]): Parsley[List[A]]

Lexeme parser semiSep(p) parses zero or more occurrences of p separated by semi. Returns a list of values returned by p.

Lexeme parser semiSep(p) parses zero or more occurrences of p separated by semi. Returns a list of values returned by p.

def semiSep1[A](p: => Parsley[A]): Parsley[List[A]]

Lexeme parser semiSep1(p) parses one or more occurrences of p separated by semi. Returns a list of values returned by p.

Lexeme parser semiSep1(p) parses one or more occurrences of p separated by semi. Returns a list of values returned by p.

def symbol(name: String): Parsley[String]

Lexeme parser symbol(s) parses string(s) and skips trailing white space.

Lexeme parser symbol(s) parses string(s) and skips trailing white space.

def symbol(name: Char): Parsley[Char]

Lexeme parser symbol(c) parses char(c) and skips trailing white space.

Lexeme parser symbol(c) parses char(c) and skips trailing white space.

def symbol_(name: String): Parsley[String]

Like symbol, but treats it as a single token using attempt. Only useful for strings, since characters are already single token.

Like symbol, but treats it as a single token using attempt. Only useful for strings, since characters are already single token.

Concrete fields

lazy
val charLiteral: Parsley[Char]

This lexeme parser parses a single literal character. Returns the literal character value. This parser deals correctly with escape sequences. The literal character is parsed according to the grammar rules defined in the Haskell report (which matches most programming languages quite closely).

This lexeme parser parses a single literal character. Returns the literal character value. This parser deals correctly with escape sequences. The literal character is parsed according to the grammar rules defined in the Haskell report (which matches most programming languages quite closely).

val colon: Parsley[Char]

Lexeme parser colon parses the character ':' and skips any trailing white space. Returns ":"

Lexeme parser colon parses the character ':' and skips any trailing white space. Returns ":"

val comma: Parsley[Char]

Lexeme parser comma parses the character ',' and skips any trailing white space. Returns ","

Lexeme parser comma parses the character ',' and skips any trailing white space. Returns ","

lazy
val decimal: Parsley[Int]

Parses a positive whole number in the decimal system. Returns the value of the number.

Parses a positive whole number in the decimal system. Returns the value of the number.

val dot: Parsley[Char]

Lexeme parser dot parses the character '.' and skips any trailing white space. Returns "."

Lexeme parser dot parses the character '.' and skips any trailing white space. Returns "."

lazy
val float: Parsley[Double]

This lexeme parser parses a floating point value. Returns the value of the number. The number is parsed according to the grammar rules defined in the Haskell report. Accepts an optional '+' or '-' sign.

This lexeme parser parses a floating point value. Returns the value of the number. The number is parsed according to the grammar rules defined in the Haskell report. Accepts an optional '+' or '-' sign.

lazy

Parses a positive whole number in the hexadecimal system. The number should be prefixed with "0x" or "0X". Returns the value of the number.

Parses a positive whole number in the hexadecimal system. The number should be prefixed with "0x" or "0X". Returns the value of the number.

lazy
val identifier: Parsley[String]

This lexeme parser parses a legal identifier. Returns the identifier string. This parser will fail on identifiers that are reserved words (i.e. keywords). Legal identifier characters and keywords are defined in the LanguageDef provided to the lexer. An identifier is treated as a single token using attempt.

This lexeme parser parses a legal identifier. Returns the identifier string. This parser will fail on identifiers that are reserved words (i.e. keywords). Legal identifier characters and keywords are defined in the LanguageDef provided to the lexer. An identifier is treated as a single token using attempt.

lazy
val integer: Parsley[Int]

This lexeme parser parses an integer (a whole number). This parser is like natural except that it can be prefixed with a sign (i.e '-' or '+'). Returns the value of the number. The number can be specified in decimal, hexadecimal or octal. The number is parsed according to the grammar rules in the haskell report.

This lexeme parser parses an integer (a whole number). This parser is like natural except that it can be prefixed with a sign (i.e '-' or '+'). Returns the value of the number. The number can be specified in decimal, hexadecimal or octal. The number is parsed according to the grammar rules in the haskell report.

lazy
val natural: Parsley[Int]

This lexeme parser parses a natural number (a positive whole number). Returns the value of the number. The number can specified in decimal, hexadecimal or octal. The number is parsed according to the grammar rules in the Haskell report.

This lexeme parser parses a natural number (a positive whole number). Returns the value of the number. The number can specified in decimal, hexadecimal or octal. The number is parsed according to the grammar rules in the Haskell report.

lazy
val naturalOrFloat: Parsley[Either[Int, Double]]

This lexeme parser parses either natural or unsigned float. Returns the value of the number. This parser deals with any overlap in the grammar rules for naturals and floats. The number is parsed according to the grammar rules defined in the Haskell report.

This lexeme parser parses either natural or unsigned float. Returns the value of the number. This parser deals with any overlap in the grammar rules for naturals and floats. The number is parsed according to the grammar rules defined in the Haskell report.

lazy
val number: Parsley[Either[Int, Double]]

This lexeme parser parses either integer or float. Returns the value of the number. This parser deals with any overlap in the grammar rules for naturals and floats. The number is parsed according to the grammar rules defined in the Haskell report.

This lexeme parser parses either integer or float. Returns the value of the number. This parser deals with any overlap in the grammar rules for naturals and floats. The number is parsed according to the grammar rules defined in the Haskell report.

lazy
val octal: Parsley[Int]

Parses a positive whole number in the octal system. The number should be prefixed with "0o" or "0O". Returns the value of the number.

Parses a positive whole number in the octal system. The number should be prefixed with "0o" or "0O". Returns the value of the number.

lazy

This non-lexeme parser parses a string in a raw fashion. The escape characters in the string remain untouched. While escaped quotes do not end the string, they remain as " in the result instead of becoming a quote character. Does not support string gaps.

This non-lexeme parser parses a string in a raw fashion. The escape characters in the string remain untouched. While escaped quotes do not end the string, they remain as " in the result instead of becoming a quote character. Does not support string gaps.

lazy
val reservedOp: Parsley[String]

This lexeme parser parses a reserved operator. Returns the name of the operator. Legal operator characters and reserved operators are defined in the LanguageDef provided to the lexer. A reservedOp is treated as a single token using attempt.

This lexeme parser parses a reserved operator. Returns the name of the operator. Legal operator characters and reserved operators are defined in the LanguageDef provided to the lexer. A reservedOp is treated as a single token using attempt.

lazy
val reservedOp_: Parsley[String]

This non-lexeme parser parses a reserved operator. Returns the name of the operator. Legal operator characters and reserved operators are defined in the LanguageDef provided to the lexer. A reservedOp_ is treated as a single token using attempt.

This non-lexeme parser parses a reserved operator. Returns the name of the operator. Legal operator characters and reserved operators are defined in the LanguageDef provided to the lexer. A reservedOp_ is treated as a single token using attempt.

val semi: Parsley[Char]

Lexeme parser semi parses the character ';' and skips any trailing white space. Returns ";"

Lexeme parser semi parses the character ';' and skips any trailing white space. Returns ";"

lazy

Parses any comments and skips them, this includes both line comments and block comments.

Parses any comments and skips them, this includes both line comments and block comments.

lazy
val stringLiteral: Parsley[String]

This lexeme parser parses a literal string. Returns the literal string value. This parser deals correctly with escape sequences and gaps. The literal string is parsed according to the grammar rules defined in the Haskell report (which matches most programming languages quite closely).

This lexeme parser parses a literal string. Returns the literal string value. This parser deals correctly with escape sequences and gaps. The literal string is parsed according to the grammar rules defined in the Haskell report (which matches most programming languages quite closely).

lazy
val stringLiteral_: Parsley[String]

This non-lexeme parser parses a literal string. Returns the literal string value. This parser deals correctly with escape sequences and gaps. The literal string is parsed according to the grammar rules defined in the Haskell report (which matches most programming languages quite closely).

This non-lexeme parser parses a literal string. Returns the literal string value. This parser deals correctly with escape sequences and gaps. The literal string is parsed according to the grammar rules defined in the Haskell report (which matches most programming languages quite closely).

lazy
val unsignedFloat: Parsley[Double]

This lexeme parser parses a floating point value. Returns the value of the number. The number is parsed according to the grammar rules defined in the Haskell report.

This lexeme parser parses a floating point value. Returns the value of the number. The number is parsed according to the grammar rules defined in the Haskell report.

lazy
val userOp: Parsley[String]

This lexeme parser parses a legal operator. Returns the name of the operator. This parser will fail on any operators that are reserved operators. Legal operator characters and reserved operators are defined in the LanguageDef provided to the lexer. A userOp is treated as a single token using attempt.

This lexeme parser parses a legal operator. Returns the name of the operator. This parser will fail on any operators that are reserved operators. Legal operator characters and reserved operators are defined in the LanguageDef provided to the lexer. A userOp is treated as a single token using attempt.

lazy
val whiteSpace: Parsley[Unit]

Parses any white space. White space consists of zero or more occurrences of a space (as provided by the LanguageDef), a line comment or a block (multi-line) comment. Block comments may be nested. How comments are started and ended is defined in the LanguageDef that is provided to the lexer.

Parses any white space. White space consists of zero or more occurrences of a space (as provided by the LanguageDef), a line comment or a block (multi-line) comment. Block comments may be nested. How comments are started and ended is defined in the LanguageDef that is provided to the lexer.

val whiteSpace_: Impl => Parsley[Unit]

Parses any white space. White space consists of zero or more occurrences of a space (as provided by the parameter), a line comment or a block (multi-line) comment. Block comments may be nested. How comments are started and ended is defined in the LanguageDef that is provided to the lexer.

Parses any white space. White space consists of zero or more occurrences of a space (as provided by the parameter), a line comment or a block (multi-line) comment. Block comments may be nested. How comments are started and ended is defined in the LanguageDef that is provided to the lexer.