BufferedTokenStream

This implementation of {@link TokenStream} loads tokens from a {@link TokenSource} on-demand, and places the tokens in a buffer to provide access to any previous token by index.

<p> This token stream ignores the value of {@link Token#getChannel}. If your parser requires the token stream filter tokens to only those on a particular channel, such as {@link Token#DEFAULT_CHANNEL} or {@link Token#HIDDEN_CHANNEL}, use a filtering token stream such a {@link CommonTokenStream}.</p>

class BufferedTokenStream : TokenStream {

protected

TokenSource tokenSource;

this(TokenSource tokenSource);

TokenSource getTokenSource();

int mark();

void release(int marker);

void reset();

void seek(size_t index);

size_t size();

void consume();

bool sync(size_t i);

size_t fetch(size_t n);

Token get(size_t i);

Token[] get(size_t start, size_t stop);

dchar LA(int i);

Token LB(int k);

Token LT(int k);

size_t adjustSeekIndex(size_t i);

void lazyInit();

void setup();

void setTokenSource(TokenSource tokenSource);

Token[] getTokens();

Token[] getTokens(size_t start, size_t stop);

Token[] getTokens(size_t start, size_t stop, int[] types);

Token[] getTokens(size_t start, size_t stop, int ttype);

size_t nextTokenOnChannel(size_t i, int channel);

size_t previousTokenOnChannel(size_t i, int channel);

Token[] getHiddenTokensToRight(size_t tokenIndex, int channel);

Token[] getHiddenTokensToRight(size_t tokenIndex);

Token[] getHiddenTokensToLeft(size_t tokenIndex, int channel);

Token[] getHiddenTokensToLeft(size_t tokenIndex);

Token[] filterForChannel(size_t from, size_t to, int channel);

string getSourceName();

Variant getText();

Variant getText(Interval interval);

Variant getText(RuleContext ctx);

Variant getText(Token start, Token stop);

void fill();

size_t index();

}

Constructors

this this(TokenSource tokenSource): Undocumented in source.

Members

Functions

LA dchar LA(int i): Undocumented in source. Be warned that the author may not have intended to support it.
LB Token LB(int k): Undocumented in source. Be warned that the author may not have intended to support it.
LT Token LT(int k): @uml @override
adjustSeekIndex size_t adjustSeekIndex(size_t i): Allowed derived classes to modify the behavior of operations which change the current stream position by adjusting the target token index of a seek operation. The default implementation simply returns {@code i}. If an exception is thrown in this method, the current stream index should not be changed.
consume void consume(): Undocumented in source. Be warned that the author may not have intended to support it.
fetch size_t fetch(size_t n): Add {@code n} elements to buffer.
fill void fill(): Get all tokens from lexer until EOF
filterForChannel Token[] filterForChannel(size_t from, size_t to, int channel): Collect all hidden tokens (any off-default channel) to the left of the current token up until we see a token on DEFAULT_TOKEN_CHANNEL.
get Token get(size_t i): @uml @override
get Token[] get(size_t start, size_t stop): Get all tokens from start..stop inclusively
getHiddenTokensToLeft Token[] getHiddenTokensToLeft(size_t tokenIndex, int channel): Collect all tokens on specified channel to the left of the current token up until we see a token on DEFAULT_TOKEN_CHANNEL. If channel is -1, find any non default channel token.
getHiddenTokensToLeft Token[] getHiddenTokensToLeft(size_t tokenIndex): Collect all hidden tokens (any off-default channel) to the left of the current token up until we see a token on DEFAULT_TOKEN_CHANNEL.
getHiddenTokensToRight Token[] getHiddenTokensToRight(size_t tokenIndex, int channel): Collect all tokens on specified channel to the right of the current token up until we see a token on DEFAULT_TOKEN_CHANNEL or EOF. If channel is -1, find any non default channel token.
getHiddenTokensToRight Token[] getHiddenTokensToRight(size_t tokenIndex): Collect all hidden tokens (any off-default channel) to the right of the current token up until we see a token on DEFAULT_TOKEN_CHANNEL of EOF.
getSourceName string getSourceName(): Undocumented in source. Be warned that the author may not have intended to support it.
getText Variant getText(): @uml @override
getText Variant getText(Interval interval): @uml @override
getText Variant getText(RuleContext ctx): @uml @override
getText Variant getText(Token start, Token stop): @uml @override
getTokenSource TokenSource getTokenSource(): @uml @override
getTokens Token[] getTokens(): Undocumented in source. Be warned that the author may not have intended to support it.
getTokens Token[] getTokens(size_t start, size_t stop): Undocumented in source. Be warned that the author may not have intended to support it.
getTokens Token[] getTokens(size_t start, size_t stop, int[] types): Given a start and stop index, return a List of all tokens in the token type BitSet. Return null if no tokens were found. This method looks at both on and off channel tokens.
getTokens Token[] getTokens(size_t start, size_t stop, int ttype): Undocumented in source. Be warned that the author may not have intended to support it.
index size_t index(): Undocumented in source. Be warned that the author may not have intended to support it.
lazyInit void lazyInit(): Undocumented in source. Be warned that the author may not have intended to support it.
mark int mark(): Undocumented in source. Be warned that the author may not have intended to support it.
nextTokenOnChannel size_t nextTokenOnChannel(size_t i, int channel): Given a starting index, return the index of the next token on channel. Return {@code i} if {@code tokensi} is on channel. Return the index of the EOF token if there are no tokens on channel between {@code i} and EOF.
previousTokenOnChannel size_t previousTokenOnChannel(size_t i, int channel): Given a starting index, return the index of the previous token on channel. Return {@code i} if {@code tokensi} is on channel. Return -1 if there are no tokens on channel between {@code i} and 0.
release void release(int marker): Undocumented in source. Be warned that the author may not have intended to support it.
reset void reset(): Undocumented in source. Be warned that the author may not have intended to support it.
seek void seek(size_t index): Undocumented in source. Be warned that the author may not have intended to support it.
setTokenSource void setTokenSource(TokenSource tokenSource): Reset this token stream by setting its token source.
setup void setup(): Undocumented in source. Be warned that the author may not have intended to support it.
size size_t size(): Undocumented in source. Be warned that the author may not have intended to support it.
sync bool sync(size_t i): Make sure index {@code i} in tokens has a token.

Variables

fetchedEOF bool fetchedEOF;: Indicates whether the {@link Token#EOF} token has been fetched from {@link #tokenSource} and added to {@link #tokens}. This field improves performance for the following cases:
tokenSource TokenSource tokenSource;: The {@link TokenSource} from which tokens for this stream are fetched.
tokens Token[] tokens;: A collection of all tokens fetched from the token source. The list is considered a complete view of the input once {@link #fetchedEOF} is set to {@code true}.

Inherited Members

From TokenStream

LT Token LT(int k): Get the {@link Token} instance associated with the value returned by {@link #LA LA(k)}. This method has the same pre- and post-conditions as {@link IntStream#LA}. In addition, when the preconditions of this method are met, the return value is non-null and the value of {@code LT(k).getType()==LA(k)}.
get Token get(size_t index): Gets the {@link Token} at the specified {@code index} in the stream. When the preconditions of this method are met, the return value is non-null.
getTokenSource TokenSource getTokenSource(): Gets the underlying {@link TokenSource} which provides tokens for this stream.
getText Variant getText(Interval interval): Return the text of all tokens within the specified {@code interval}. This method behaves like the following code (including potential exceptions) for violating preconditions of {@link #get}, but may be optimized by the specific implementation.
getText Variant getText(): Return the text of all tokens in the stream. This method behaves like the following code, including potential exceptions from the calls to {@link IntStream#size} and {@link #getText(Interval)}, but may be optimized by the specific implementation.
getText Variant getText(RuleContext ctx): Return the text of all tokens in the source interval of the specified context. This method behaves like the following code, including potential exceptions from the call to {@link #getText(Interval)}, but may be optimized by the specific implementation.
getText Variant getText(Token start, Token stop): Return the text of all tokens in this stream between {@code start} and {@code stop} (inclusive).

BufferedTokenStream

Constructors

Members

Functions

Variables

Inherited Members

From TokenStream

Meta

Source