Problem with lexer grammar v3.2
0 posts in topic
Flat View  Flat View
TOPIC ACTIONS:
 

Posted By:   Thomas_Radke
Posted On:   Wednesday, December 16, 2009 09:01 AM

Why does the lexer shown below try to match PP_AFTER ? I use ANTLRv3.2 with following grammar: lexer grammar Language; options { language=Java; } PP_END : ':END'; PP_BEFORE : ':BEFORE'; PP_AFTER : ':AFTER'; NEW_TAB : 'NEW_TAB'; DROP_TAB : 'DROP_TAB'; WS : (' '|' ')* { $channel=HIDDEN; }; LINE_END : (' '|' '|'u000C'); IDENTIFIER : (('a'..'z' | 'A'..'Z')) ((('a'..'z' | 'A'..'Z')) | ('0'..'9') | '_')* ; ANY_CHAR : (~(' '|' '|'u000C')); The test input is IF (:BEU.esthhj   More>>

Why does the lexer shown below try to match PP_AFTER ?


I use ANTLRv3.2 with following grammar:


			
lexer grammar Language;

options {
language=Java;
}

PP_END : ':END';

PP_BEFORE : ':BEFORE';

PP_AFTER : ':AFTER';

NEW_TAB : 'NEW_TAB';

DROP_TAB : 'DROP_TAB';

WS : (' '|' ')* { $channel=HIDDEN; };

LINE_END : ('
'|'
'|'u000C');

IDENTIFIER : (('a'..'z' | 'A'..'Z')) ((('a'..'z' | 'A'..'Z')) | ('0'..'9') | '_')* ;

ANY_CHAR : (~('
'|'
'|'u000C'));



The test input is


			
IF (:BEU.esthhj <> :ALT.esthhj)
THEN
raise_application_error(-20509,'Das Feld darf nicht geƤndert werden!');
END IF;



When I run the lexer on this input, I get


			
line 1:7 mismatched character 'U' expecting 'F'
line 1:21 mismatched character 'L' expecting 'F'



because the generated DFA predicts PP_BEFORE on :BEU and PP_AFTER on :ALT . I expected the lexer to take the decision to match PP_BEFORE after it had verified that the whole token could be matched.


If I'm wrong, how do I have to define the tokens to get the desired behaviour?


Thanks in advance.


Thomas

   <<Less
About | Sitemap | Contact