org.antlr.v4.runtime

Interface TokenSource

All Known Implementing Classes:

Lexer

public interface TokenSource

A source of tokens must provide a sequence of tokens via nextToken() and also must reveal it's source of characters; CommonToken's text is computed from a CharStream; it only store indices into the char stream. Errors from the lexer are never passed to the parser. Either you want to keep going or you do not upon token recognition error. If you do not want to continue lexing then you do not want to continue parsing. Just throw an exception not under RecognitionException and Java will naturally toss you all the way out of the recognizers. If you want to continue lexing then you should not throw an exception to the parser--it has already requested a token. Keep lexing until you get a valid one. Just report errors and keep going, looking for a valid token.

Method Summary

Methods

Modifier and Type	Method and Description
int	getCharPositionInLine()
CharStream	getInputStream() From what character stream was this token created? You don't have to implement but it's nice to know where a Token comes from if you have include files etc...
int	getLine()
String	getSourceName() Where are you getting tokens from? normally the implication will simply ask lexers input stream.
TokenFactory<?>	getTokenFactory() Gets the factory used for constructing tokens.
Token	nextToken() Return a Token object from your input stream (usually a CharStream).
void	setTokenFactory(TokenFactory<?> factory) Optional method that lets users set factory in lexer or other source

Method Detail

nextToken

Token nextToken()

Return a Token object from your input stream (usually a CharStream). Do not fail/return upon lexing error; keep chewing on the characters until you get a good one; errors are not passed through to the parser.

getLine

int getLine()

getCharPositionInLine

int getCharPositionInLine()

getInputStream

CharStream getInputStream()

From what character stream was this token created? You don't have to implement but it's nice to know where a Token comes from if you have include files etc... on the input.

getSourceName

String getSourceName()

Where are you getting tokens from? normally the implication will simply ask lexers input stream.

setTokenFactory

void setTokenFactory(@NotNull
                   TokenFactory<?> factory)

Optional method that lets users set factory in lexer or other source

getTokenFactory

@NotNull
TokenFactory<?> getTokenFactory()

Gets the factory used for constructing tokens.