Home / FAQ / Java Tools / ANTLR

Terence Parr

Imagine input:

class P extends Parser;

a : "a" A ;

class L extends Lexer;

A : "a";

ANTLR will think "a" and A are two different tokens as it doesn't know that in the Lexer you will define A and "a":

public interface PTokenTypes {       
        int EOF = 1;     
        int NULL_TREE_LOOKAHEAD = 3;
        int LITERAL_a = 4;
        int A = 5;
}

You could be building your own lexer, for example. However, if you want to have the parser know they are the same, use tokens {...}:

class P extends Parser;

tokens {
        A="a";    
}

a : "a" A ;

Then ANTLR sees them as the same.

public interface PTokenTypes {
        int EOF = 1;
        int NULL_TREE_LOOKAHEAD = 3;
        int A = 4;
}

Recall that all string literals referenced in the parser get tested for automatically in the lexer. When a complete token is found, ANTLR does a simple string compare to see if one of the literals has been found.