Why do literals specified in a lexer cause lexical non-determinism rules with an IDENTIFIER?
0 posts in topic
Flat View  Flat View
TOPIC ACTIONS:
 

Posted By:   Lance_Walton
Posted On:   Tuesday, September 30, 2003 02:13 PM

Hi. I'm curious why literals expressed in a lexer cause lexical non-determinism errors because of clashes with a more general IDENTIFIER rule, even when the IDENTIFIER has testLiterals=true. For example: class MyLexer extends Lexer; options { exportVocab=MyVocab; testLiterals=false; k=3; charVocabulary='u0003'..'uFFFF'; caseSensitive=false; caseSensitiveLiterals=false; } IDENTIFIER options { testLiterals=true; } : ('a'..'z')+; BOOLEAN_LITERAL : "true" | "false"; The above causes lexical non-determinism errors. However, when the BOOLEAN_LITERAL values are defined in the 'tokens' section o   More>>

Hi.

I'm curious why literals expressed in a lexer cause lexical non-determinism errors because of clashes with a more general IDENTIFIER rule, even when the IDENTIFIER has testLiterals=true. For example:

			
class MyLexer extends Lexer;
options {
exportVocab=MyVocab;
testLiterals=false;
k=3;
charVocabulary='u0003'..'uFFFF';
caseSensitive=false;
caseSensitiveLiterals=false;
}

IDENTIFIER options { testLiterals=true; } : ('a'..'z')+;

BOOLEAN_LITERAL : "true" | "false";



The above causes lexical non-determinism errors. However, when the BOOLEAN_LITERAL values are defined in the 'tokens' section of a grammar, it works fine.

Is there some principle (or is it best practice) to define all literals in the grammar rather than in the lexer? If so, it feels upside down to me (although, I'm sure that someone can explain it :-)

Regards,

Lance

   <<Less
About | Sitemap | Contact