CommonTokenStream

This class extends {@link BufferedTokenStream} with functionality to filter token streams to tokens on a particular channel (tokens where {@link Token#getChannel} returns a particular value). This token stream provides access to all tokens by index or when calling methods like {@link #getText}. The channel filtering is only used for code accessing tokens via the lookahead methods {@link #LA}, {@link #LT}, and {@link #LB}.

By default, tokens are placed on the default channel ({@link Token#DEFAULT_CHANNEL}), but may be reassigned by using the {@code ->channel(HIDDEN)} lexer command, or by using an embedded action to call {@link Lexer#setChannel}.

Note: lexer rules which use the {@code ->skip} lexer command or call {@link Lexer#skip} do not produce tokens at all, so input text matched by such a rule will not be available as part of the token stream, regardless of channel.

class CommonTokenStream : BufferedTokenStream {

protected

int channel;

this(TokenSource tokenSource);

this(TokenSource tokenSource, int channel);

size_t adjustSeekIndex(size_t i);

Token LB(int k);

Token LT(int k);

int getNumberOfOnChannelTokens();

}

Constructors

this this(TokenSource tokenSource): Constructs a new {@link CommonTokenStream} using the specified token source and the default token channel ({@link Token#DEFAULT_CHANNEL}).
this this(TokenSource tokenSource, int channel): Constructs a new {@link CommonTokenStream} using the specified token source and filtering tokens to the specified channel. Only tokens whose {@link Token#getChannel} matches {@code channel} or have the {@link Token#getType} equal to {@link Token#EOF} will be returned by the token stream lookahead methods.

Members

Functions

LB Token LB(int k): @uml @override
LT Token LT(int k): @uml @override
adjustSeekIndex size_t adjustSeekIndex(size_t i): @uml @override
getNumberOfOnChannelTokens int getNumberOfOnChannelTokens(): Count EOF just once.

Variables

channel int channel;: Specifies the channel to use for filtering tokens.

Inherited Members

From BufferedTokenStream

tokenSource TokenSource tokenSource;: The {@link TokenSource} from which tokens for this stream are fetched.
tokens Token[] tokens;: A collection of all tokens fetched from the token source. The list is considered a complete view of the input once {@link #fetchedEOF} is set to {@code true}.
fetchedEOF bool fetchedEOF;: Indicates whether the {@link Token#EOF} token has been fetched from {@link #tokenSource} and added to {@link #tokens}. This field improves performance for the following cases:
getTokenSource TokenSource getTokenSource(): @uml @override
mark int mark(): Undocumented in source. Be warned that the author may not have intended to support it.
release void release(int marker): Undocumented in source. Be warned that the author may not have intended to support it.
reset void reset(): Undocumented in source. Be warned that the author may not have intended to support it.
seek void seek(size_t index): Undocumented in source. Be warned that the author may not have intended to support it.
size size_t size(): Undocumented in source. Be warned that the author may not have intended to support it.
consume void consume(): Undocumented in source. Be warned that the author may not have intended to support it.
sync bool sync(size_t i): Make sure index {@code i} in tokens has a token.
fetch size_t fetch(size_t n): Add {@code n} elements to buffer.
get Token get(size_t i): @uml @override
get Token[] get(size_t start, size_t stop): Get all tokens from start..stop inclusively
LA dchar LA(int i): Undocumented in source. Be warned that the author may not have intended to support it.
LB Token LB(int k): Undocumented in source. Be warned that the author may not have intended to support it.
LT Token LT(int k): @uml @override
adjustSeekIndex size_t adjustSeekIndex(size_t i): Allowed derived classes to modify the behavior of operations which change the current stream position by adjusting the target token index of a seek operation. The default implementation simply returns {@code i}. If an exception is thrown in this method, the current stream index should not be changed.
lazyInit void lazyInit(): Undocumented in source. Be warned that the author may not have intended to support it.
setup void setup(): Undocumented in source. Be warned that the author may not have intended to support it.
setTokenSource void setTokenSource(TokenSource tokenSource): Reset this token stream by setting its token source.
getTokens Token[] getTokens(): Undocumented in source. Be warned that the author may not have intended to support it.
getTokens Token[] getTokens(size_t start, size_t stop): Undocumented in source. Be warned that the author may not have intended to support it.
getTokens Token[] getTokens(size_t start, size_t stop, int[] types): Given a start and stop index, return a List of all tokens in the token type BitSet. Return null if no tokens were found. This method looks at both on and off channel tokens.
getTokens Token[] getTokens(size_t start, size_t stop, int ttype): Undocumented in source. Be warned that the author may not have intended to support it.
nextTokenOnChannel size_t nextTokenOnChannel(size_t i, int channel): Given a starting index, return the index of the next token on channel. Return {@code i} if {@code tokensi} is on channel. Return the index of the EOF token if there are no tokens on channel between {@code i} and EOF.
previousTokenOnChannel size_t previousTokenOnChannel(size_t i, int channel): Given a starting index, return the index of the previous token on channel. Return {@code i} if {@code tokensi} is on channel. Return -1 if there are no tokens on channel between {@code i} and 0.
getHiddenTokensToRight Token[] getHiddenTokensToRight(size_t tokenIndex, int channel): Collect all tokens on specified channel to the right of the current token up until we see a token on DEFAULT_TOKEN_CHANNEL or EOF. If channel is -1, find any non default channel token.
getHiddenTokensToRight Token[] getHiddenTokensToRight(size_t tokenIndex): Collect all hidden tokens (any off-default channel) to the right of the current token up until we see a token on DEFAULT_TOKEN_CHANNEL of EOF.
getHiddenTokensToLeft Token[] getHiddenTokensToLeft(size_t tokenIndex, int channel): Collect all tokens on specified channel to the left of the current token up until we see a token on DEFAULT_TOKEN_CHANNEL. If channel is -1, find any non default channel token.
getHiddenTokensToLeft Token[] getHiddenTokensToLeft(size_t tokenIndex): Collect all hidden tokens (any off-default channel) to the left of the current token up until we see a token on DEFAULT_TOKEN_CHANNEL.
filterForChannel Token[] filterForChannel(size_t from, size_t to, int channel): Collect all hidden tokens (any off-default channel) to the left of the current token up until we see a token on DEFAULT_TOKEN_CHANNEL.
getSourceName string getSourceName(): Undocumented in source. Be warned that the author may not have intended to support it.
getText Variant getText(): @uml @override
getText Variant getText(Interval interval): @uml @override
getText Variant getText(RuleContext ctx): @uml @override
getText Variant getText(Token start, Token stop): @uml @override
fill void fill(): Get all tokens from lexer until EOF
index size_t index(): Undocumented in source. Be warned that the author may not have intended to support it.