CommonTokenStream

This class extends {@link BufferedTokenStream} with functionality to filter token streams to tokens on a particular channel (tokens where {@link Token#getChannel} returns a particular value). <p> This token stream provides access to all tokens by index or when calling methods like {@link #getText}. The channel filtering is only used for code accessing tokens via the lookahead methods {@link #LA}, {@link #LT}, and {@link #LB}.</p>

<p> By default, tokens are placed on the default channel ({@link Token#DEFAULT_CHANNEL}), but may be reassigned by using the {@code ->channel(HIDDEN)} lexer command, or by using an embedded action to call {@link Lexer#setChannel}. </p>

<p> Note: lexer rules which use the {@code ->skip} lexer command or call {@link Lexer#skip} do not produce tokens at all, so input text matched by such a rule will not be available as part of the token stream, regardless of channel.</p>

Constructors

this
this(TokenSource tokenSource)

Constructs a new {@link CommonTokenStream} using the specified token source and the default token channel ({@link Token#DEFAULT_CHANNEL}).

this
this(TokenSource tokenSource, int channel)

Constructs a new {@link CommonTokenStream} using the specified token source and filtering tokens to the specified channel. Only tokens whose {@link Token#getChannel} matches {@code channel} or have the {@link Token#getType} equal to {@link Token#EOF} will be returned by the token stream lookahead methods.

Members

Functions

LB
Token LB(int k)

@uml @override

LT
Token LT(int k)

@uml @override

adjustSeekIndex
int adjustSeekIndex(int i)

@uml @override

getNumberOfOnChannelTokens
int getNumberOfOnChannelTokens()

Count EOF just once.

Variables

channel
int channel;

Specifies the channel to use for filtering tokens.

Inherited Members

From BufferedTokenStream

tokenSource
TokenSource tokenSource;

The {@link TokenSource} from which tokens for this stream are fetched.

tokens
Token[] tokens;

A collection of all tokens fetched from the token source. The list is considered a complete view of the input once {@link #fetchedEOF} is set to {@code true}.

fetchedEOF
bool fetchedEOF;

Indicates whether the {@link Token#EOF} token has been fetched from {@link #tokenSource} and added to {@link #tokens}. This field improves performance for the following cases:

getTokenSource
TokenSource getTokenSource()

@uml @override

mark
int mark()
Undocumented in source. Be warned that the author may not have intended to support it.
release
void release(int marker)
Undocumented in source. Be warned that the author may not have intended to support it.
reset
void reset()
Undocumented in source. Be warned that the author may not have intended to support it.
seek
void seek(int index)
Undocumented in source. Be warned that the author may not have intended to support it.
size
int size()
Undocumented in source. Be warned that the author may not have intended to support it.
consume
void consume()
Undocumented in source. Be warned that the author may not have intended to support it.
sync
bool sync(int i)

Make sure index {@code i} in tokens has a token.

fetch
int fetch(int n)

Add {@code n} elements to buffer.

get
Token get(int i)

@uml @override

get
Token[] get(int start, int stop)

Get all tokens from start..stop inclusively

LA
int LA(int i)
Undocumented in source. Be warned that the author may not have intended to support it.
LB
Token LB(int k)
Undocumented in source. Be warned that the author may not have intended to support it.
LT
Token LT(int k)

@uml @override

adjustSeekIndex
int adjustSeekIndex(int i)

Allowed derived classes to modify the behavior of operations which change the current stream position by adjusting the target token index of a seek operation. The default implementation simply returns {@code i}. If an exception is thrown in this method, the current stream index should not be changed.

lazyInit
void lazyInit()
Undocumented in source. Be warned that the author may not have intended to support it.
setup
void setup()
Undocumented in source. Be warned that the author may not have intended to support it.
setTokenSource
void setTokenSource(TokenSource tokenSource)

Reset this token stream by setting its token source.

getTokens
Token[] getTokens()
Undocumented in source. Be warned that the author may not have intended to support it.
getTokens
Token[] getTokens(int start, int stop)
Undocumented in source. Be warned that the author may not have intended to support it.
getTokens
Token[] getTokens(int start, int stop, int[] types)

Given a start and stop index, return a List of all tokens in the token type BitSet. Return null if no tokens were found. This method looks at both on and off channel tokens.

getTokens
Token[] getTokens(int start, int stop, int ttype)
Undocumented in source. Be warned that the author may not have intended to support it.
nextTokenOnChannel
int nextTokenOnChannel(int i, int channel)

Given a starting index, return the index of the next token on channel. Return {@code i} if {@code tokensi} is on channel. Return the index of the EOF token if there are no tokens on channel between {@code i} and EOF.

previousTokenOnChannel
int previousTokenOnChannel(int i, int channel)

Given a starting index, return the index of the previous token on channel. Return {@code i} if {@code tokensi} is on channel. Return -1 if there are no tokens on channel between {@code i} and 0.

getHiddenTokensToRight
Token[] getHiddenTokensToRight(int tokenIndex, int channel)

Collect all tokens on specified channel to the right of the current token up until we see a token on DEFAULT_TOKEN_CHANNEL or EOF. If channel is -1, find any non default channel token.

getHiddenTokensToRight
Token[] getHiddenTokensToRight(int tokenIndex)

Collect all hidden tokens (any off-default channel) to the right of the current token up until we see a token on DEFAULT_TOKEN_CHANNEL of EOF.

getHiddenTokensToLeft
Token[] getHiddenTokensToLeft(int tokenIndex, int channel)

Collect all tokens on specified channel to the left of the current token up until we see a token on DEFAULT_TOKEN_CHANNEL. If channel is -1, find any non default channel token.

getHiddenTokensToLeft
Token[] getHiddenTokensToLeft(int tokenIndex)

Collect all hidden tokens (any off-default channel) to the left of the current token up until we see a token on DEFAULT_TOKEN_CHANNEL.

filterForChannel
Token[] filterForChannel(int from, int to, int channel)

Collect all hidden tokens (any off-default channel) to the left of the current token up until we see a token on DEFAULT_TOKEN_CHANNEL.

getSourceName
string getSourceName()
Undocumented in source. Be warned that the author may not have intended to support it.
getText
Variant getText()

@uml @override

getText
Variant getText(Interval interval)

@uml @override

getText
Variant getText(RuleContext ctx)

@uml @override

getText
Variant getText(Token start, Token stop)

@uml @override

fill
void fill()

Get all tokens from lexer until EOF

index
int index()
Undocumented in source. Be warned that the author may not have intended to support it.

Meta