Get the {@link Token} instance associated with the value returned by {@link #LA LA(k)}. This method has the same pre- and post-conditions as {@link IntStream#LA}. In addition, when the preconditions of this method are met, the return value is non-null and the value of {@code LT(k).getType()==LA(k)}.
Gets the {@link Token} at the specified {@code index} in the stream. When the preconditions of this method are met, the return value is non-null.
Return the text of all tokens in the stream. This method behaves like the following code, including potential exceptions from the calls to {@link IntStream#size} and {@link #getText(Interval)}, but may be optimized by the specific implementation.
Return the text of all tokens in the source interval of the specified context. This method behaves like the following code, including potential exceptions from the call to {@link #getText(Interval)}, but may be optimized by the specific implementation.
Return the text of all tokens within the specified {@code interval}. This method behaves like the following code (including potential exceptions) for violating preconditions of {@link #get}, but may be optimized by the specific implementation.
Return the text of all tokens in this stream between {@code start} and {@code stop} (inclusive).
Gets the underlying {@link TokenSource} which provides tokens for this stream.
Consumes the current symbol in the stream. This method has the following effects:
Gets the value of the symbol at offset {@code i} from the current position. When {@code i==1}, this method returns the value of the current symbol in the stream (which is the next symbol to be consumed). When {@code i==-1}, this method returns the value of the previously read symbol in the stream. It is not valid to call this method with {@code i==0}, but the specific behavior is unspecified because this method is frequently called from performance-critical code.
A mark provides a guarantee that {@link #seek seek()} operations will be valid over a "marked range" extending from the index where {@code mark()} was called to the current {@link #index index()}. This allows the use of streaming input sources by specifying the minimum buffering requirements to support arbitrary lookahead during prediction.
This method releases a marked range created by a call to {@link #mark mark()}. Calls to {@code release()} must appear in the reverse order of the corresponding calls to {@code mark()}. If a mark is released twice, or if marks are not released in reverse order of the corresponding calls to {@code mark()}, the behavior is unspecified.
Return the index into the stream of the input symbol referred to by {@code LA(1)}.
Set the input cursor to the position indicated by {@code index}. If the specified index lies past the end of the stream, the operation behaves as though {@code index} was the index of the EOF symbol. After this method returns without throwing an exception, then at least one of the following will be true.
Returns the total number of symbols in the stream, including a single EOF symbol.
Gets the name of the underlying symbol source. This method returns a non-null, non-empty string. If such a name is not known, this method returns {@link #UNKNOWN_SOURCE_NAME}.
This TokenStream uses the data type Variant for storing text.
Therefore tabulator based syntax can be used more complex text elements, i.e. text string in combination with number of tabs.