Get a list of tokens on success or an error message on failure.
Get a list of tokens on success or an error message on failure. Note that the grammar in tokenizer requires that sequence of tokens produced must be like: (Field Index?)+. This implies the sequence:
1. must contain a Field token at the start 1. must contain at least as many Field tokens as Index tokens 1. must NOT contain two adjacent Index tokens
A string to parse
either a list of tokens on success, or an error message on failure.
This tokenizer extracts sequences of Field and Index tokens which are then used in conjunction with the reflection APIs in the Protocol Buffers library to synthesize code for high-performance, type-safe accessor functions.