UnbufferedTokenStream

TODO add class description

Constructors

this this(TokenSource tokenSource): Undocumented in source.
this this(TokenSource tokenSource, int bufferSize): Undocumented in source.

Members

Functions

LA dchar LA(int i): Undocumented in source. Be warned that the author may not have intended to support it.
LT Token LT(int i): Undocumented in source. Be warned that the author may not have intended to support it.
add void add(Token t): Undocumented in source. Be warned that the author may not have intended to support it.
consume void consume(): Undocumented in source. Be warned that the author may not have intended to support it.
fill int fill(int n): Add {@code n} elements to the buffer. Returns the number of tokens actually added to the buffer. If the return value is less than {@code n}, then EOF was reached before {@code n} tokens could be added.
get Token get(size_t i): Undocumented in source. Be warned that the author may not have intended to support it.
getBufferStartIndex size_t getBufferStartIndex(): Undocumented in source. Be warned that the author may not have intended to support it.
getSourceName string getSourceName(): Undocumented in source. Be warned that the author may not have intended to support it.
getText Variant getText(): Undocumented in source. Be warned that the author may not have intended to support it.
getText Variant getText(RuleContext ctx): Undocumented in source. Be warned that the author may not have intended to support it.
getText Variant getText(Token start, Token stop): Undocumented in source. Be warned that the author may not have intended to support it.
getText Variant getText(Interval interval): Undocumented in source. Be warned that the author may not have intended to support it.
getTokenSource TokenSource getTokenSource(): Undocumented in source. Be warned that the author may not have intended to support it.
index size_t index(): @uml @override
mark int mark(): Return a marker that we can release later.
release void release(int marker): Undocumented in source. Be warned that the author may not have intended to support it.
seek void seek(size_t index): Undocumented in source. Be warned that the author may not have intended to support it.
size size_t size(): Undocumented in source. Be warned that the author may not have intended to support it.
sync void sync(int want): Make sure we have 'need' elements from current position {@link #p p}. Last valid {@code p} index is {@code tokens.length-1}. {@code p+need-1} is the tokens index 'need' elements ahead. If we need 1 element, {@code (p+1-1)==p} must be less than {@code tokens.length}.

Variables

currentTokenIndex size_t currentTokenIndex;: Undocumented in source.
lastToken Token lastToken;: Undocumented in source.
lastTokenBufferStart Token lastTokenBufferStart;: Undocumented in source.
n size_t n;: @uml The number of tokens currently in {@link #tokens tokens}.
numMarkers int numMarkers;: @uml Count up with {@link #mark mark()} and down with {@link #release release()}. When we {@code release()} the last mark, {@code numMarkers} reaches 0 and we reset the buffer. Copy {@code tokensp..tokens[n-1]} to {@code tokens[0]..tokens[(n-1)-p]}.
p size_t p;: @uml 0..n-1 index into {@link #tokens tokens} of next token.
tokenSource TokenSource tokenSource;: Undocumented in source.
tokens Token[] tokens;: @uml A moving window buffer of the data being scanned. While there's a marker, we keep adding to buffer. Otherwise, {@link #consume consume()} resets so we start filling at index 0 again.

Inherited Members

From TokenStream

LT Token LT(int k): Get the {@link Token} instance associated with the value returned by {@link #LA LA(k)}. This method has the same pre- and post-conditions as {@link IntStream#LA}. In addition, when the preconditions of this method are met, the return value is non-null and the value of {@code LT(k).getType()==LA(k)}.
get Token get(size_t index): Gets the {@link Token} at the specified {@code index} in the stream. When the preconditions of this method are met, the return value is non-null.
getTokenSource TokenSource getTokenSource(): Gets the underlying {@link TokenSource} which provides tokens for this stream.
getText Variant getText(Interval interval): Return the text of all tokens within the specified {@code interval}. This method behaves like the following code (including potential exceptions) for violating preconditions of {@link #get}, but may be optimized by the specific implementation.
getText Variant getText(): Return the text of all tokens in the stream. This method behaves like the following code, including potential exceptions from the calls to {@link IntStream#size} and {@link #getText(Interval)}, but may be optimized by the specific implementation.
getText Variant getText(RuleContext ctx): Return the text of all tokens in the source interval of the specified context. This method behaves like the following code, including potential exceptions from the call to {@link #getText(Interval)}, but may be optimized by the specific implementation.
getText Variant getText(Token start, Token stop): Return the text of all tokens in this stream between {@code start} and {@code stop} (inclusive).