org.antlr.v4.runtime

Class CommonTokenStream

java.lang.Object

org.antlr.v4.runtime.BufferedTokenStream

org.antlr.v4.runtime.CommonTokenStream

All Implemented Interfaces:

IntStream, TokenStream

public class CommonTokenStream
extends BufferedTokenStream

The most common stream of tokens where every token is buffered up and tokens are filtered for a certain channel (the parser will only see these tokens). Even though it buffers all of the tokens, this token stream pulls tokens from the tokens source on demand. In other words, until you ask for a token using consume(), LT(), etc. the stream does not pull from the lexer. The only difference between this stream and BufferedTokenStream superclass is that this stream knows how to ignore off channel tokens. There may be a performance advantage to using the superclass if you don't pass whitespace and comments etc. to the parser on a hidden channel (i.e., you set $channel instead of calling skip() in lexer rules.)

See Also:

UnbufferedTokenStream, BufferedTokenStream

Field Summary

Fields

Modifier and Type	Field and Description
protected int	channel Skip tokens on any channel but this one; this is how we skip whitespace...

Fields inherited from class org.antlr.v4.runtime.BufferedTokenStream

fetchedEOF, p, tokens, tokenSource

Fields inherited from interface org.antlr.v4.runtime.IntStream

EOF, UNKNOWN_SOURCE_NAME

Constructor Summary

Constructors

Constructor and Description
CommonTokenStream(TokenSource tokenSource)
CommonTokenStream(TokenSource tokenSource, int channel)

Method Summary

Methods

Modifier and Type	Method and Description
protected int	adjustSeekIndex(int i) Allowed derived classes to modify the behavior of operations which change the current stream position by adjusting the target token index of a seek operation.
int	getNumberOfOnChannelTokens() Count EOF just once.
protected Token	LB(int k)
Token	LT(int k) Get the Token instance associated with the value returned by LA(k).

Methods inherited from class org.antlr.v4.runtime.BufferedTokenStream

consume, fetch, fill, filterForChannel, get, get, getHiddenTokensToLeft, getHiddenTokensToLeft, getHiddenTokensToRight, getHiddenTokensToRight, getSourceName, getText, getText, getText, getText, getTokens, getTokens, getTokens, getTokens, getTokenSource, index, LA, lazyInit, mark, nextTokenOnChannel, previousTokenOnChannel, release, reset, seek, setTokenSource, setup, size, sync

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

channel

protected int channel

Skip tokens on any channel but this one; this is how we skip whitespace...

Constructor Detail

CommonTokenStream

public CommonTokenStream(TokenSource tokenSource)

CommonTokenStream

public CommonTokenStream(TokenSource tokenSource,
                 int channel)

Method Detail

adjustSeekIndex

protected int adjustSeekIndex(int i)

Description copied from class: BufferedTokenStream

Allowed derived classes to modify the behavior of operations which change the current stream position by adjusting the target token index of a seek operation. The default implementation simply returns i. If an exception is thrown in this method, the current stream index should not be changed.

For example, CommonTokenStream overrides this method to ensure that the seek target is always an on-channel token.

Overrides:

adjustSeekIndex in class BufferedTokenStream

Parameters:

i - The target token index.

Returns:

The adjusted target token index.

LB

protected Token LB(int k)

Overrides:

LB in class BufferedTokenStream

LT

public Token LT(int k)

Description copied from interface: TokenStream

Get the Token instance associated with the value returned by LA(k). This method has the same pre- and post-conditions as IntStream.LA(int). In addition, when the preconditions of this method are met, the return value is non-null and the value of LT(k).getType()==LA(k).

Specified by:

LT in interface TokenStream

Overrides:

LT in class BufferedTokenStream

See Also:

IntStream.LA(int)

getNumberOfOnChannelTokens

public int getNumberOfOnChannelTokens()

Count EOF just once.