BufferedTokenStream

This implementation of {@link TokenStream} loads tokens from a {@link TokenSource} on-demand, and places the tokens in a buffer to provide access to any previous token by index.

<p> This token stream ignores the value of {@link Token#getChannel}. If your parser requires the token stream filter tokens to only those on a particular channel, such as {@link Token#DEFAULT_CHANNEL} or {@link Token#HIDDEN_CHANNEL}, use a filtering token stream such a {@link CommonTokenStream}.</p>

Constructors

this
this(TokenSource tokenSource)
Undocumented in source.

Members

Functions

LA
int LA(int i)
Undocumented in source. Be warned that the author may not have intended to support it.
LB
Token LB(int k)
Undocumented in source. Be warned that the author may not have intended to support it.
LT
Token LT(int k)

@uml @override

adjustSeekIndex
int adjustSeekIndex(int i)

Allowed derived classes to modify the behavior of operations which change the current stream position by adjusting the target token index of a seek operation. The default implementation simply returns {@code i}. If an exception is thrown in this method, the current stream index should not be changed.

consume
void consume()
Undocumented in source. Be warned that the author may not have intended to support it.
fetch
int fetch(int n)

Add {@code n} elements to buffer.

fill
void fill()

Get all tokens from lexer until EOF

filterForChannel
Token[] filterForChannel(int from, int to, int channel)

Collect all hidden tokens (any off-default channel) to the left of the current token up until we see a token on DEFAULT_TOKEN_CHANNEL.

get
Token get(int i)

@uml @override

get
Token[] get(int start, int stop)

Get all tokens from start..stop inclusively

getHiddenTokensToLeft
Token[] getHiddenTokensToLeft(int tokenIndex, int channel)

Collect all tokens on specified channel to the left of the current token up until we see a token on DEFAULT_TOKEN_CHANNEL. If channel is -1, find any non default channel token.

getHiddenTokensToLeft
Token[] getHiddenTokensToLeft(int tokenIndex)

Collect all hidden tokens (any off-default channel) to the left of the current token up until we see a token on DEFAULT_TOKEN_CHANNEL.

getHiddenTokensToRight
Token[] getHiddenTokensToRight(int tokenIndex, int channel)

Collect all tokens on specified channel to the right of the current token up until we see a token on DEFAULT_TOKEN_CHANNEL or EOF. If channel is -1, find any non default channel token.

getHiddenTokensToRight
Token[] getHiddenTokensToRight(int tokenIndex)

Collect all hidden tokens (any off-default channel) to the right of the current token up until we see a token on DEFAULT_TOKEN_CHANNEL of EOF.

getSourceName
string getSourceName()
Undocumented in source. Be warned that the author may not have intended to support it.
getText
Variant getText()

@uml @override

getText
Variant getText(Interval interval)

@uml @override

getText
Variant getText(RuleContext ctx)

@uml @override

getText
Variant getText(Token start, Token stop)

@uml @override

getTokenSource
TokenSource getTokenSource()

@uml @override

getTokens
Token[] getTokens()
Undocumented in source. Be warned that the author may not have intended to support it.
getTokens
Token[] getTokens(int start, int stop)
Undocumented in source. Be warned that the author may not have intended to support it.
getTokens
Token[] getTokens(int start, int stop, int[] types)

Given a start and stop index, return a List of all tokens in the token type BitSet. Return null if no tokens were found. This method looks at both on and off channel tokens.

getTokens
Token[] getTokens(int start, int stop, int ttype)
Undocumented in source. Be warned that the author may not have intended to support it.
index
int index()
Undocumented in source. Be warned that the author may not have intended to support it.
lazyInit
void lazyInit()
Undocumented in source. Be warned that the author may not have intended to support it.
mark
int mark()
Undocumented in source. Be warned that the author may not have intended to support it.
nextTokenOnChannel
int nextTokenOnChannel(int i, int channel)

Given a starting index, return the index of the next token on channel. Return {@code i} if {@code tokensi} is on channel. Return the index of the EOF token if there are no tokens on channel between {@code i} and EOF.

previousTokenOnChannel
int previousTokenOnChannel(int i, int channel)

Given a starting index, return the index of the previous token on channel. Return {@code i} if {@code tokensi} is on channel. Return -1 if there are no tokens on channel between {@code i} and 0.

release
void release(int marker)
Undocumented in source. Be warned that the author may not have intended to support it.
reset
void reset()
Undocumented in source. Be warned that the author may not have intended to support it.
seek
void seek(int index)
Undocumented in source. Be warned that the author may not have intended to support it.
setTokenSource
void setTokenSource(TokenSource tokenSource)

Reset this token stream by setting its token source.

setup
void setup()
Undocumented in source. Be warned that the author may not have intended to support it.
size
int size()
Undocumented in source. Be warned that the author may not have intended to support it.
sync
bool sync(int i)

Make sure index {@code i} in tokens has a token.

Variables

fetchedEOF
bool fetchedEOF;

Indicates whether the {@link Token#EOF} token has been fetched from {@link #tokenSource} and added to {@link #tokens}. This field improves performance for the following cases:

tokenSource
TokenSource tokenSource;

The {@link TokenSource} from which tokens for this stream are fetched.

tokens
Token[] tokens;

A collection of all tokens fetched from the token source. The list is considered a complete view of the input once {@link #fetchedEOF} is set to {@code true}.

Inherited Members

From TokenStream

LT
Token LT(int k)

Get the {@link Token} instance associated with the value returned by {@link #LA LA(k)}. This method has the same pre- and post-conditions as {@link IntStream#LA}. In addition, when the preconditions of this method are met, the return value is non-null and the value of {@code LT(k).getType()==LA(k)}.

get
Token get(int index)

Gets the {@link Token} at the specified {@code index} in the stream. When the preconditions of this method are met, the return value is non-null.

getTokenSource
TokenSource getTokenSource()

Gets the underlying {@link TokenSource} which provides tokens for this stream.

getText
Variant getText(Interval interval)

Return the text of all tokens within the specified {@code interval}. This method behaves like the following code (including potential exceptions) for violating preconditions of {@link #get}, but may be optimized by the specific implementation.

getText
Variant getText()

Return the text of all tokens in the stream. This method behaves like the following code, including potential exceptions from the calls to {@link IntStream#size} and {@link #getText(Interval)}, but may be optimized by the specific implementation.

getText
Variant getText(RuleContext ctx)

Return the text of all tokens in the source interval of the specified context. This method behaves like the following code, including potential exceptions from the call to {@link #getText(Interval)}, but may be optimized by the specific implementation.

getText
Variant getText(Token start, Token stop)

Return the text of all tokens in this stream between {@code start} and {@code stop} (inclusive).

Meta