TokenStream

This TokenStream uses the data type Variant for storing text.

Therefore tabulator based syntax can be used more complex text elements, i.e. text string in combination with number of tabs.

Members

Functions

LT
Token LT(int k)

Get the {@link Token} instance associated with the value returned by {@link #LA LA(k)}. This method has the same pre- and post-conditions as {@link IntStream#LA}. In addition, when the preconditions of this method are met, the return value is non-null and the value of {@code LT(k).getType()==LA(k)}.

get
Token get(size_t index)

Gets the {@link Token} at the specified {@code index} in the stream. When the preconditions of this method are met, the return value is non-null.

getText
Variant getText()

Return the text of all tokens in the stream. This method behaves like the following code, including potential exceptions from the calls to {@link IntStream#size} and {@link #getText(Interval)}, but may be optimized by the specific implementation.

getText
Variant getText(RuleContext ctx)

Return the text of all tokens in the source interval of the specified context. This method behaves like the following code, including potential exceptions from the call to {@link #getText(Interval)}, but may be optimized by the specific implementation.

getText
Variant getText(Interval interval)

Return the text of all tokens within the specified {@code interval}. This method behaves like the following code (including potential exceptions) for violating preconditions of {@link #get}, but may be optimized by the specific implementation.

getText
Variant getText(Token start, Token stop)

Return the text of all tokens in this stream between {@code start} and {@code stop} (inclusive).

getTokenSource
TokenSource getTokenSource()

Gets the underlying {@link TokenSource} which provides tokens for this stream.

Inherited Members

From IntStream

consume
void consume()

Consumes the current symbol in the stream. This method has the following effects:

LA
dchar LA(int i)

Gets the value of the symbol at offset {@code i} from the current position. When {@code i==1}, this method returns the value of the current symbol in the stream (which is the next symbol to be consumed). When {@code i==-1}, this method returns the value of the previously read symbol in the stream. It is not valid to call this method with {@code i==0}, but the specific behavior is unspecified because this method is frequently called from performance-critical code.

mark
int mark()

A mark provides a guarantee that {@link #seek seek()} operations will be valid over a "marked range" extending from the index where {@code mark()} was called to the current {@link #index index()}. This allows the use of streaming input sources by specifying the minimum buffering requirements to support arbitrary lookahead during prediction.

release
void release(int marker)

This method releases a marked range created by a call to {@link #mark mark()}. Calls to {@code release()} must appear in the reverse order of the corresponding calls to {@code mark()}. If a mark is released twice, or if marks are not released in reverse order of the corresponding calls to {@code mark()}, the behavior is unspecified.

index
size_t index()

Return the index into the stream of the input symbol referred to by {@code LA(1)}.

seek
void seek(size_t index)

Set the input cursor to the position indicated by {@code index}. If the specified index lies past the end of the stream, the operation behaves as though {@code index} was the index of the EOF symbol. After this method returns without throwing an exception, then at least one of the following will be true.

size
size_t size()

Returns the total number of symbols in the stream, including a single EOF symbol.

getSourceName
string getSourceName()

Gets the name of the underlying symbol source. This method returns a non-null, non-empty string. If such a name is not known, this method returns {@link #UNKNOWN_SOURCE_NAME}.

Meta