Lexer

A lexer is recognizer that draws input symbols from a character stream. lexer grammars result in a subclass of this object. A Lexer object uses simplified match() and error recovery mechanisms in the interest of speed.

abstract
class Lexer : Recognizer!(int, LexerATNSimulator), TokenSource , InterfaceLexer {}

Constructors

this
this()
Undocumented in source.
this
this(CharStream input)
Undocumented in source.

Members

Functions

action
void action(InterfaceRuleContext interfaceRuleContext, int ruleIndex, int actionIndex)

@uml @override

emit
void emit(Token token)

By default does not support multiple emits per nextToken invocation for efficiency reasons. Subclass and override this method, nextToken, and getToken (to push tokens into a list and pull from that list rather than a single variable as this implementation does).

emit
Token emit()

The standard method called to automatically emit a token at the outermost lexical rule. The token object should point into the char buffer start..stop. If there is a text override in 'text', use that to set the token's text. Override this method to emit custom Token objects or provide a new factory.

emitEOF
Token emitEOF()
Undocumented in source. Be warned that the author may not have intended to support it.
getAllTokens
Token[] getAllTokens()

Return a list of all Token objects in input char stream. Forces load of all tokens. Does not include EOF token.

getChannel
int getChannel()
Undocumented in source. Be warned that the author may not have intended to support it.
getChannelNames
string[] getChannelNames()
Undocumented in source. Be warned that the author may not have intended to support it.
getCharErrorDisplay
string getCharErrorDisplay(int c)
Undocumented in source. Be warned that the author may not have intended to support it.
getCharIndex
int getCharIndex()

What is the index of the current character of lookahead?

getCharPositionInLine
int getCharPositionInLine()
Undocumented in source. Be warned that the author may not have intended to support it.
getErrorDisplay
string getErrorDisplay(string s)
Undocumented in source. Be warned that the author may not have intended to support it.
getErrorDisplay
string getErrorDisplay(int c)
Undocumented in source. Be warned that the author may not have intended to support it.
getInputStream
CharStream getInputStream()

@uml @override

getLine
int getLine()
Undocumented in source. Be warned that the author may not have intended to support it.
getModeNames
string[] getModeNames()
Undocumented in source. Be warned that the author may not have intended to support it.
getSourceName
string getSourceName()
Undocumented in source. Be warned that the author may not have intended to support it.
getText
string getText()

Return the text matched so far for the current token or any text override.

getToken
Token getToken()

Override if emitting multiple tokens.

getTokenNames
string[] getTokenNames()

Used to print out token names like ID during debugging and error reporting. The generated parsers implement a method that overrides this to point to their String[] tokenNames @uml @override

getType
int getType()
Undocumented in source. Be warned that the author may not have intended to support it.
mode
void mode(int m)
Undocumented in source. Be warned that the author may not have intended to support it.
more
void more()
Undocumented in source. Be warned that the author may not have intended to support it.
nextToken
Token nextToken()

Return a token from this source; i.e., match a token on the char stream.

notifyListeners
void notifyListeners(LexerNoViableAltException e)
Undocumented in source. Be warned that the author may not have intended to support it.
popMode
int popMode()
Undocumented in source. Be warned that the author may not have intended to support it.
pushMode
void pushMode(int m)
Undocumented in source. Be warned that the author may not have intended to support it.
recover
void recover(LexerNoViableAltException e)
Undocumented in source. Be warned that the author may not have intended to support it.
recover
void recover(RecognitionException re)

Lexers can normally match any char in it's vocabulary after matching a token, so do the easy thing and just kill a character and hope it all works out. You can instead use the rule invocation stack to do sophisticated error recovery if you are in a fragment rule.

reset
void reset()
Undocumented in source. Be warned that the author may not have intended to support it.
setChannel
void setChannel(int channel)
Undocumented in source. Be warned that the author may not have intended to support it.
setCharPositionInLine
void setCharPositionInLine(int charPositionInLine)
Undocumented in source. Be warned that the author may not have intended to support it.
setInputStream
void setInputStream(IntStream input)

Set the char stream and reset the lexer @uml @override

setLine
void setLine(int line)
Undocumented in source. Be warned that the author may not have intended to support it.
setText
void setText(string text)

Set the complete text of this token; it wipes any previous changes to the text.

setToken
void setToken(Token token)
Undocumented in source. Be warned that the author may not have intended to support it.
setType
void setType(int ttype)
Undocumented in source. Be warned that the author may not have intended to support it.
skip
void skip()

Instruct the lexer to skip creating a token for current lexer rule and look for another token. nextToken() knows to keep looking when a lexer rule finishes with token set to SKIP_TOKEN. Recall that if token==null at end of any token rule, it creates one for you and emits it.

tokenFactory
TokenFactory!CommonToken tokenFactory()
Undocumented in source. Be warned that the author may not have intended to support it.
tokenFactory
void tokenFactory(TokenFactory!CommonToken tokenFactory)
Undocumented in source. Be warned that the author may not have intended to support it.

Static variables

DEFAULT_MODE
int DEFAULT_MODE;
Undocumented in source.
DEFAULT_TOKEN_CHANNEL
int DEFAULT_TOKEN_CHANNEL;
Undocumented in source.
HIDDEN
int HIDDEN;
Undocumented in source.
MAX_CHAR_VALUE
int MAX_CHAR_VALUE;
Undocumented in source.
MIN_CHAR_VALUE
int MIN_CHAR_VALUE;
Undocumented in source.
MORE
int MORE;
Undocumented in source.
SKIP
int SKIP;
Undocumented in source.

Variables

_channel
int _channel;

The channel number for the current token

_hitEOF
bool _hitEOF;
Undocumented in source.
_input
CharStream _input;
Undocumented in source.
_mode
int _mode;
Undocumented in source.
_modeStack
IntegerStack _modeStack;
Undocumented in source.
_text
string _text;

You can set the text for the current token to override what is in the input char buffer. Use setText() or can set this instance var.

_token
Token _token;

The goal of all lexer rules/methods is to create a token object. This is an instance variable as multiple rules may collaborate to create a single token. nextToken will return this object after matching lexer rule(s). If you subclass to allow multiple token emissions, then set this to the last token to be matched or something nonnull so that the auto token emit mechanism will not emit another token.

_tokenFactorySourcePair
TokenFactorySourcePair _tokenFactorySourcePair;
Undocumented in source.
_tokenStartCharIndex
int _tokenStartCharIndex;

What character index in the stream did the current token start at? Needed, for example, to get the text for current token. Set at the start of nextToken.

_tokenStartCharPositionInLine
int _tokenStartCharPositionInLine;

The character position of first character within the line

_tokenStartLine
int _tokenStartLine;

The line on which the first character of the token resides

_type
int _type;

The token type for the current token

tokenFactory_
TokenFactory!CommonToken tokenFactory_;

How to create token objects @uml @read @write @override

Inherited Members

From TokenSource

getCharPositionInLine
int getCharPositionInLine()

@uml Get the index into the current line for the current position in the input stream. The first character on a line has position 0.

nextToken
Token nextToken()

@uml Return a {@link Token} object from your input stream (usually a {@link CharStream}). Do not fail/return upon lexing error; keep chewing on the characters until you get a good one; errors are not passed through to the parser.

getLine
int getLine()

@uml Get the line number for the current position in the input stream. The first line in the input is line 1.

getInputStream
CharStream getInputStream()

@uml Get the {@link CharStream} from which this token source is currently providing tokens.

getSourceName
string getSourceName()

@uml Gets the name of the underlying input source. This method returns a non-null, non-empty string. If such a name is not known, this method returns {@link IntStream#UNKNOWN_SOURCE_NAME}.

tokenFactory
void tokenFactory(TokenFactory!CommonToken factory)

@uml Set the {@link TokenFactory} this token source should use for creating {@link Token} objects from the input.

tokenFactory
TokenFactory!CommonToken tokenFactory()

@uml Gets the {@link TokenFactory} this token source is currently using for creating {@link Token} objects from the input.

From InterfaceLexer

more
void more()
Undocumented in source.
skip
void skip()
Undocumented in source.
popMode
int popMode()
Undocumented in source.
setType
void setType(int )
Undocumented in source.
pushMode
void pushMode(int )
Undocumented in source.
action
void action(InterfaceRuleContext , int , int )
Undocumented in source.
setChannel
void setChannel(int )
Undocumented in source.
mode
void mode(int )
Undocumented in source.

Meta