Multiplexing lexers
0 posts in topic
Flat View  Flat View
TOPIC ACTIONS:
 

Posted By:   Ethan_Eade
Posted On:   Thursday, October 30, 2003 07:26 PM

I want to do lexer states (like flex does) using multiplexed lexers, as suggested by the documentation. However, the parser should trigger the lexer state change (when I get the keyword "routines", I want to return the whole next '{'...'}' block as one token). The problem is that with k=3 in the parser, lookahead has already read the beginning of the '{'..'}' block, which screws up the second lexer state (which would return that whole block.) Is there no way to address this problem because of lookahead buffering? There's no way, as far as I can tell, to push the lookahead tokens back on the stream. This means the second lexer state will always start further down the stream than it should. Does this mean I have to put the lexer state transition in th   More>>

I want to do lexer states (like flex does) using multiplexed lexers, as suggested by the documentation. However, the parser should trigger the lexer state change (when I get the keyword "routines", I want to return the whole next '{'...'}' block as one token). The problem is that with k=3 in the parser, lookahead has already read the beginning of the '{'..'}' block, which screws up the second lexer state (which would return that whole block.) Is there no way to address this problem because of lookahead buffering? There's no way, as far as I can tell, to push the lookahead tokens back on the stream. This means the second lexer state will always start further down the stream than it should. Does this mean I have to put the lexer state transition in the first lexer, and check for "routines" as a trigger for the state change?

Here's an example:


			
some {
stuff that;
gets read by the first lexer;
}

routines //everything below should be returned as one token
{
this stuff should;
all be read as one token by the second parser;
}


I'd like to get a "routines" token followed by the whole subsequent code block as one token. Instead, because k=3 in the parser, I get "routines" then '{' then 'this', and then the code block (which can't deal with the extra '}' on the end).


Thanks,

Ethan Eade

   <<Less
About | Sitemap | Contact