Packages

  • package root

    This is the documentation for Parsley.

    This is the documentation for Parsley.

    Package structure

    The parsley package contains the Parsley class, as well as the Result, Success, and Failure types. In addition to these, it also contains the following packages and "modules" (a module is defined as being an object which mocks a package):

    • parsley.Parsley contains the bulk of the core "function-style" combinators, as well as the implicit classes which enable the "method-style" combinators.
    • parsley.combinator contains many helpful combinators that simplify some common parser patterns.
    • parsley.character contains the combinators needed to read characters and strings, as well as combinators to match specific sub-sets of characters.
    • parsley.debug contains debugging combinators, helpful for identifying faults in parsers.
    • parsley.expr contains the following sub modules:
    • parsley.implicits contains several implicits to add syntactic sugar to the combinators, such as being able to use character and string literals directly as parsers, as well as enabling lifting of functions to work on parsers.
    • parsley.lift contains functions which lift functions that work on regular types to those which now combine the results of parsers returning those same types. these are ubiquitous.
    • parsley.registers contains combinators that interact with the context-sensitive functionality in the form of registers.
    • parsley.token contains the Lexer class that provides a host of helpful lexing combinators when provided with the description of a language.
    • parsley.unsafe contains unsafe (and not thread-safe) ways of speeding up the execution of a parser.

    In addition to the modules and packages outlined above, this version of Parsley (up to version 3.0), also includes the so-called old-style API, which is deprecated (see the Parsley wiki for a discussion of these differences). You should use the modules described above, and avoid the following:

    • parsley.BitGen
    • parsley.Char
    • parsley.CharSet
    • parsley.Combinator
    • parsley.ExpressionParser
    • parsley.Impl
    • parsley.Implicits
    • parsley.LanguageDef
    • parsley.NotRequired
    • parsley.Parser
    • parsley.Predicate
    • parsley.Reg
    • parsley.TokenParser
    Definition Classes
    root
  • package parsley
    Definition Classes
    root
  • package expr
    Definition Classes
    parsley
  • package token
    Definition Classes
    parsley
  • BitGen
  • Char
  • CharSet
  • Combinator
  • ExpressionParser
  • Failure
  • Impl
  • Implicits
  • LanguageDef
  • NotRequired
  • Parser
  • Parsley
  • Predicate
  • Reg
  • Result
  • Success
  • TokenParser
  • character
  • combinator
  • debug
  • implicits
  • lift
  • registers
  • unsafe
c

parsley

TokenParser

final class TokenParser extends Lexer

When provided with a LanguageDef, this class will produce a large variety of parsers that can be used for tokenisation of a language. This includes parsing numbers and strings in their various formats and ensuring that all operations consume whitespace after them (so-called lexeme parsers). These are very useful in parsing programming languages. This class also has a large number of hand-optimised intrinsic parsers to improve performance!

Annotations
@deprecated
Deprecated

(Since version v2.2.0) This class will be removed in Parsley 3.0, use parsley.token.Lexer instead

Linear Supertypes
Lexer, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. TokenParser
  2. Lexer
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new TokenParser(lang: LanguageDef)

    lang

    The rules that govern the language we are tokenising

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. def angles[A](p: => Parsley[A]): Parsley[A]

    Lexeme parser angles(p) parses p enclosed in angle brackets ('<', '>'), returning the value of p.

    Lexeme parser angles(p) parses p enclosed in angle brackets ('<', '>'), returning the value of p.

    Definition Classes
    Lexer
  5. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  6. def braces[A](p: => Parsley[A]): Parsley[A]

    Lexeme parser braces(p) parses p enclosed in braces ('{', '}'), returning the value of 'p'

    Lexeme parser braces(p) parses p enclosed in braces ('{', '}'), returning the value of 'p'

    Definition Classes
    Lexer
  7. def brackets[A](p: => Parsley[A]): Parsley[A]

    Lexeme parser brackets(p) parses p enclosed in brackets ('[', ']'), returning the value of p.

    Lexeme parser brackets(p) parses p enclosed in brackets ('[', ']'), returning the value of p.

    Definition Classes
    Lexer
  8. lazy val charLiteral: Parsley[Char]

    This lexeme parser parses a single literal character.

    This lexeme parser parses a single literal character. Returns the literal character value. This parser deals correctly with escape sequences. The literal character is parsed according to the grammar rules defined in the Haskell report (which matches most programming languages quite closely).

    Definition Classes
    Lexer
  9. def clone(): AnyRef
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @native()
  10. val colon: Parsley[Char]

    Lexeme parser colon parses the character ':' and skips any trailing white space.

    Lexeme parser colon parses the character ':' and skips any trailing white space. Returns ":"

    Definition Classes
    Lexer
  11. val comma: Parsley[Char]

    Lexeme parser comma parses the character ',' and skips any trailing white space.

    Lexeme parser comma parses the character ',' and skips any trailing white space. Returns ","

    Definition Classes
    Lexer
  12. def commaSep[A](p: => Parsley[A]): Parsley[List[A]]

    Lexeme parser commaSep(p) parses zero or more occurrences of p separated by comma.

    Lexeme parser commaSep(p) parses zero or more occurrences of p separated by comma. Returns a list of values returned by p.

    Definition Classes
    Lexer
  13. def commaSep1[A](p: => Parsley[A]): Parsley[List[A]]

    Lexeme parser commaSep1(p) parses one or more occurrences of p separated by comma.

    Lexeme parser commaSep1(p) parses one or more occurrences of p separated by comma. Returns a list of values returned by p.

    Definition Classes
    Lexer
  14. lazy val decimal: Parsley[Int]

    Parses a positive whole number in the decimal system.

    Parses a positive whole number in the decimal system. Returns the value of the number.

    Definition Classes
    Lexer
  15. val dot: Parsley[Char]

    Lexeme parser dot parses the character '.' and skips any trailing white space.

    Lexeme parser dot parses the character '.' and skips any trailing white space. Returns "."

    Definition Classes
    Lexer
  16. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  17. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  18. def finalize(): Unit
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable])
  19. lazy val float: Parsley[Double]

    This lexeme parser parses a floating point value.

    This lexeme parser parses a floating point value. Returns the value of the number. The number is parsed according to the grammar rules defined in the Haskell report. Accepts an optional '+' or '-' sign.

    Definition Classes
    Lexer
  20. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  21. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  22. lazy val hexadecimal: Parsley[Int]

    Parses a positive whole number in the hexadecimal system.

    Parses a positive whole number in the hexadecimal system. The number should be prefixed with "0x" or "0X". Returns the value of the number.

    Definition Classes
    Lexer
  23. lazy val identifier: Parsley[String]

    This lexeme parser parses a legal identifier.

    This lexeme parser parses a legal identifier. Returns the identifier string. This parser will fail on identifiers that are reserved words (i.e. keywords). Legal identifier characters and keywords are defined in the LanguageDef provided to the lexer. An identifier is treated as a single token using attempt.

    Definition Classes
    Lexer
  24. lazy val integer: Parsley[Int]

    This lexeme parser parses an integer (a whole number).

    This lexeme parser parses an integer (a whole number). This parser is like natural except that it can be prefixed with a sign (i.e '-' or '+'). Returns the value of the number. The number can be specified in decimal, hexadecimal or octal. The number is parsed according to the grammar rules in the haskell report.

    Definition Classes
    Lexer
  25. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  26. def keyword(name: String): Parsley[Unit]

    The lexeme parser keyword(name) parses the symbol name, but it also checks that the name is not a prefix of a valid identifier.

    The lexeme parser keyword(name) parses the symbol name, but it also checks that the name is not a prefix of a valid identifier. A keyword is treated as a single token using attempt.

    Definition Classes
    Lexer
  27. def lexeme[A](p: => Parsley[A]): Parsley[A]

    lexeme(p) first applies parser p and then the whiteSpace parser, returning the value of p.

    lexeme(p) first applies parser p and then the whiteSpace parser, returning the value of p. Every lexical token (lexeme) is defined using lexeme, this way every parse starts at a point without white space. The only point where the whiteSpace parser should be called explicitly is the start of the main parser in order to skip any leading white space.

    Definition Classes
    Lexer
  28. def maxOp(name: String): Parsley[Unit]

    The lexeme parser maxOp(name) parses the symbol name, but also checks that the name is not part of a larger reserved operator.

    The lexeme parser maxOp(name) parses the symbol name, but also checks that the name is not part of a larger reserved operator. An operator is treated as a single token using attempt.

    Definition Classes
    Lexer
  29. def maxOp_(name: String): Parsley[Unit]

    The non-lexeme parser maxOp_(name) parses the symbol name, but also checks that the name is not part of a larger reserved operator.

    The non-lexeme parser maxOp_(name) parses the symbol name, but also checks that the name is not part of a larger reserved operator. An operator is treated as a single token using attempt.

    Definition Classes
    Lexer
  30. lazy val natural: Parsley[Int]

    This lexeme parser parses a natural number (a positive whole number).

    This lexeme parser parses a natural number (a positive whole number). Returns the value of the number. The number can specified in decimal, hexadecimal or octal. The number is parsed according to the grammar rules in the Haskell report.

    Definition Classes
    Lexer
  31. lazy val naturalOrFloat: Parsley[Either[Int, Double]]

    This lexeme parser parses either natural or unsigned float.

    This lexeme parser parses either natural or unsigned float. Returns the value of the number. This parser deals with any overlap in the grammar rules for naturals and floats. The number is parsed according to the grammar rules defined in the Haskell report.

    Definition Classes
    Lexer
  32. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  33. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  34. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  35. lazy val number: Parsley[Either[Int, Double]]

    This lexeme parser parses either integer or float.

    This lexeme parser parses either integer or float. Returns the value of the number. This parser deals with any overlap in the grammar rules for naturals and floats. The number is parsed according to the grammar rules defined in the Haskell report.

    Definition Classes
    Lexer
  36. lazy val octal: Parsley[Int]

    Parses a positive whole number in the octal system.

    Parses a positive whole number in the octal system. The number should be prefixed with "0o" or "0O". Returns the value of the number.

    Definition Classes
    Lexer
  37. def operator(name: String): Parsley[Unit]

    The lexeme parser operator(name) parses the symbol name, but also checks that the name is not the prefix of a valid operator.

    The lexeme parser operator(name) parses the symbol name, but also checks that the name is not the prefix of a valid operator. An operator is treated as a single token using attempt.

    Definition Classes
    Lexer
  38. def operator_(name: String): Parsley[Unit]

    The non-lexeme parser operator_(name) parses the symbol name, but also checks that the name is not the prefix of a valid operator.

    The non-lexeme parser operator_(name) parses the symbol name, but also checks that the name is not the prefix of a valid operator. An operator is treated as a single token using attempt.

    Definition Classes
    Lexer
  39. def parens[A](p: => Parsley[A]): Parsley[A]

    Lexeme parser parens(p) parses p enclosed in parenthesis, returning the value of p.

    Lexeme parser parens(p) parses p enclosed in parenthesis, returning the value of p.

    Definition Classes
    Lexer
  40. lazy val rawStringLiteral: Parsley[String]

    This non-lexeme parser parses a string in a raw fashion.

    This non-lexeme parser parses a string in a raw fashion. The escape characters in the string remain untouched. While escaped quotes do not end the string, they remain as \" in the result instead of becoming a quote character. Does not support string gaps.

    Definition Classes
    Lexer
  41. lazy val reservedOp: Parsley[String]

    This lexeme parser parses a reserved operator.

    This lexeme parser parses a reserved operator. Returns the name of the operator. Legal operator characters and reserved operators are defined in the LanguageDef provided to the lexer. A reservedOp is treated as a single token using attempt.

    Definition Classes
    Lexer
  42. lazy val reservedOp_: Parsley[String]

    This non-lexeme parser parses a reserved operator.

    This non-lexeme parser parses a reserved operator. Returns the name of the operator. Legal operator characters and reserved operators are defined in the LanguageDef provided to the lexer. A reservedOp_ is treated as a single token using attempt.

    Definition Classes
    Lexer
  43. val semi: Parsley[Char]

    Lexeme parser semi parses the character ';' and skips any trailing white space.

    Lexeme parser semi parses the character ';' and skips any trailing white space. Returns ";"

    Definition Classes
    Lexer
  44. def semiSep[A](p: => Parsley[A]): Parsley[List[A]]

    Lexeme parser semiSep(p) parses zero or more occurrences of p separated by semi.

    Lexeme parser semiSep(p) parses zero or more occurrences of p separated by semi. Returns a list of values returned by p.

    Definition Classes
    Lexer
  45. def semiSep1[A](p: => Parsley[A]): Parsley[List[A]]

    Lexeme parser semiSep1(p) parses one or more occurrences of p separated by semi.

    Lexeme parser semiSep1(p) parses one or more occurrences of p separated by semi. Returns a list of values returned by p.

    Definition Classes
    Lexer
  46. lazy val skipComments: Parsley[Unit]

    Parses any comments and skips them, this includes both line comments and block comments.

    Parses any comments and skips them, this includes both line comments and block comments.

    Definition Classes
    Lexer
  47. lazy val stringLiteral: Parsley[String]

    This lexeme parser parses a literal string.

    This lexeme parser parses a literal string. Returns the literal string value. This parser deals correctly with escape sequences and gaps. The literal string is parsed according to the grammar rules defined in the Haskell report (which matches most programming languages quite closely).

    Definition Classes
    Lexer
  48. lazy val stringLiteral_: Parsley[String]

    This non-lexeme parser parses a literal string.

    This non-lexeme parser parses a literal string. Returns the literal string value. This parser deals correctly with escape sequences and gaps. The literal string is parsed according to the grammar rules defined in the Haskell report (which matches most programming languages quite closely).

    Definition Classes
    Lexer
  49. def symbol(name: Char): Parsley[Char]

    Lexeme parser symbol(c) parses char(c) and skips trailing white space.

    Lexeme parser symbol(c) parses char(c) and skips trailing white space.

    Definition Classes
    Lexer
  50. def symbol(name: String): Parsley[String]

    Lexeme parser symbol(s) parses string(s) and skips trailing white space.

    Lexeme parser symbol(s) parses string(s) and skips trailing white space.

    Definition Classes
    Lexer
  51. def symbol_(name: String): Parsley[String]

    Like symbol, but treats it as a single token using attempt.

    Like symbol, but treats it as a single token using attempt. Only useful for strings, since characters are already single token.

    Definition Classes
    Lexer
  52. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  53. def toString(): String
    Definition Classes
    AnyRef → Any
  54. lazy val unsignedFloat: Parsley[Double]

    This lexeme parser parses a floating point value.

    This lexeme parser parses a floating point value. Returns the value of the number. The number is parsed according to the grammar rules defined in the Haskell report.

    Definition Classes
    Lexer
  55. lazy val userOp: Parsley[String]

    This lexeme parser parses a legal operator.

    This lexeme parser parses a legal operator. Returns the name of the operator. This parser will fail on any operators that are reserved operators. Legal operator characters and reserved operators are defined in the LanguageDef provided to the lexer. A userOp is treated as a single token using attempt.

    Definition Classes
    Lexer
  56. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  57. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  58. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  59. lazy val whiteSpace: Parsley[Unit]

    Parses any white space.

    Parses any white space. White space consists of zero or more occurrences of a space (as provided by the LanguageDef), a line comment or a block (multi-line) comment. Block comments may be nested. How comments are started and ended is defined in the LanguageDef that is provided to the lexer.

    Definition Classes
    Lexer
  60. val whiteSpace_: (token.Impl) => Parsley[Unit]

    Parses any white space.

    Parses any white space. White space consists of zero or more occurrences of a space (as provided by the parameter), a line comment or a block (multi-line) comment. Block comments may be nested. How comments are started and ended is defined in the LanguageDef that is provided to the lexer.

    Definition Classes
    Lexer

Inherited from Lexer

Inherited from AnyRef

Inherited from Any

Ungrouped