dcsimg
Disambiguating lexer rules
1 posts in topic
Flat View  Flat View
TOPIC ACTIONS:
 

Posted By:   Lance_Walton
Posted On:   Tuesday, January 6, 2004 09:55 AM

Hi I have a lexer that contains an IDENTIFIER rule and a parser that makes use of it. I wanted to add an EMAIL rule to the lexer but, alas, the EMAIL production turns out to be a 'super-production' of my IDENTIFIER (i.e. the set of IDENTIFIERs is a subset of the set of EMAILs). This rightly causes ANTLR to hurl when trying to generate the code for the lexer. How do I modify my lexer/parser to avoid the problem? Something to do with dismabiguating syntactic/semantic predicates maybe? One solution I though of was this: since it is the parser that knows whether it's expecting an EMAIL or an IDENTIFIER, everything is unambiguous to the parser. So, I could move the rules out of the lexer and into the parser   More>>

Hi

I have a lexer that contains an IDENTIFIER rule and a parser that makes use of it. I wanted to add an EMAIL rule to the lexer but, alas, the EMAIL production turns out to be a 'super-production' of my IDENTIFIER (i.e. the set of IDENTIFIERs is a subset of the set of EMAILs). This rightly causes ANTLR to hurl when trying to generate the code for the lexer.

How do I modify my lexer/parser to avoid the problem? Something to do with dismabiguating syntactic/semantic predicates maybe?

One solution I though of was this: since it is the parser that knows whether it's expecting an EMAIL or an IDENTIFIER, everything is unambiguous to the parser. So, I could move the rules out of the lexer and into the parser, and set the token type to an imaginary token type so that my tree walker can subsequently use it. Is this a good way?

Regards,

Lance

   <<Less

Re: Disambiguating lexer rules

Posted By:   Monty_Zukowski  
Posted On:   Wednesday, January 7, 2004 08:33 AM

Yes, that sounds good. The main thing to avoid is feedback from parser to lexer. Either do it in the parser completely or lexer completely. Meaning the other alternative would be to make your lexer smarter by doing some parsing there, keeping track of state with "lexer states" (see docs.)
About | Sitemap | Contact