Context sensitive lexer
1 posts in topic
Flat View  Flat View
TOPIC ACTIONS:
 

Posted By:   Patrick_Vachon
Posted On:   Tuesday, January 6, 2004 08:16 AM

Hi, I would like to make a context sensitive lexer. Here is what I mean by that: class MyLexer extends Lexer; { public char separator = ' '; } TAG : "TAG" . { this.separator = $getText.charAt(0); }; CHAR : ('a'..'z' | '0'..'9'); SEPARATOR : ????????; ID : (CHAR)* SEPARATOR!; How should I define SEPARATOR to match the value of this.separator? A sample input would look like: TAG*name1*name2*TAG;name3;TAG-name4-name5-name6- Thanks, Patrick    More>>

Hi,



I would like to make a context sensitive lexer.

Here is what I mean by that:

			
class MyLexer extends Lexer;
{ public char separator = ' '; }
TAG : "TAG" . { this.separator = $getText.charAt(0); };
CHAR : ('a'..'z' | '0'..'9');
SEPARATOR : ????????;
ID : (CHAR)* SEPARATOR!;

How should I define SEPARATOR to match the value of this.separator?



A sample input would look like:

			
TAG*name1*name2*TAG;name3;TAG-name4-name5-name6-

Thanks,

Patrick
   <<Less

Re: Context sensitive lexer

Posted By:   Monty_Zukowski  
Posted On:   Wednesday, January 7, 2004 08:30 AM

That's something you can't do with ANTLR, it's not that dynamic. You will have to do that by hand, either by lexing by hand or by replacing your separator characters in the input stream with a static separator like '*'.


Introducing a dynamic element like that separator has consequences for lookahead calculations. The only way you could do what you want with antlr is to generate the .g file with the right separator and then run antlr.Tool on it, compile the result and run that. Not fun, because the separator may conflict with other characters that are already in use.

About | Sitemap | Contact