TokenStream

Members

Functions

LT

Token LT(int k)

Get the {@link Token} instance associated with the value returned by {@link #LA LA(k)}. This method has the same pre- and post-conditions as {@link IntStream#LA}. In addition, when the preconditions of this method are met, the return value is non-null and the value of {@code LT(k).getType()==LA(k)}.

get

Token get(size_t index)

Gets the {@link Token} at the specified {@code index} in the stream. When the preconditions of this method are met, the return value is non-null.

getText

Variant getText()

Return the text of all tokens in the stream. This method behaves like the following code, including potential exceptions from the calls to {@link IntStream#size} and {@link #getText(Interval)}, but may be optimized by the specific implementation.

getText

Variant getText(RuleContext ctx)

Return the text of all tokens in the source interval of the specified context. This method behaves like the following code, including potential exceptions from the call to {@link #getText(Interval)}, but may be optimized by the specific implementation.

getText

Variant getText(Interval interval)

Return the text of all tokens within the specified {@code interval}. This method behaves like the following code (including potential exceptions) for violating preconditions of {@link #get}, but may be optimized by the specific implementation.

getText

Variant getText(Token start, Token stop)

Return the text of all tokens in this stream between {@code start} and {@code stop} (inclusive).

getTokenSource

TokenSource getTokenSource()

Gets the underlying {@link TokenSource} which provides tokens for this stream.

Inherited Members

From IntStream

consume

void consume()

Consumes the current symbol in the stream. This method has the following effects:

LA

dchar LA(int i)

Gets the value of the symbol at offset {@code i} from the current position. When {@code i==1}, this method returns the value of the current symbol in the stream (which is the next symbol to be consumed). When {@code i==-1}, this method returns the value of the previously read symbol in the stream. It is not valid to call this method with {@code i==0}, but the specific behavior is unspecified because this method is frequently called from performance-critical code.

mark

int mark()

A mark provides a guarantee that {@link #seek seek()} operations will be valid over a "marked range" extending from the index where {@code mark()} was called to the current {@link #index index()}. This allows the use of streaming input sources by specifying the minimum buffering requirements to support arbitrary lookahead during prediction.

release

void release(int marker)

This method releases a marked range created by a call to {@link #mark mark()}. Calls to {@code release()} must appear in the reverse order of the corresponding calls to {@code mark()}. If a mark is released twice, or if marks are not released in reverse order of the corresponding calls to {@code mark()}, the behavior is unspecified.

index

size_t index()

Return the index into the stream of the input symbol referred to by {@code LA(1)}.

seek

void seek(size_t index)

Set the input cursor to the position indicated by {@code index}. If the specified index lies past the end of the stream, the operation behaves as though {@code index} was the index of the EOF symbol. After this method returns without throwing an exception, then at least one of the following will be true.

size

size_t size()

Returns the total number of symbols in the stream, including a single EOF symbol.

getSourceName

string getSourceName()

Gets the name of the underlying symbol source. This method returns a non-null, non-empty string. If such a name is not known, this method returns {@link #UNKNOWN_SOURCE_NAME}.

• TokenStream