Why do lexical rules for keywords conflict with identifiers even if I specify the special case (keywords) before the more general identifier rule?

Created Sep 3, 1999

Terence Parr

What is wrong with the following? Doesn't the lexer know by the order in which I specify the two tokens that I want BEGIN to be a special case of an ID?

class TestLexer extends Lexer;
...
BEGIN	:	"begin"
	;

ID	:	('a'..'z')+
	;

There is no order dependency between the rules in ANTLR. "begin" predicts both BEGIN and ID, unfortuantely.

Use the token {...} section:

class TestLexer extends Lexer;
...
tokens {
        BEGIN="begin";
}

ID	:	('a'..'z')+
	;

The lexer will automatically return BEGIN from the lexer if it sees "begin". The lexer maintains a literals hashtable with all the keywords and other strings and then tests for literals before returning to the parser.