AsTokens

Type class that describes the capacity to tokenize something.

A tokenized value is an indexed sequence (ideally an array). This allows the parser to navigate input not by consuming it bit by bit and constructing tons of intermediate representations, but as a pointer in something array-ish.

Strings, for example, can be turned into an array of characters at very little cost.