using the same lexer more than once
1 posts in topic
Flat View  Flat View
TOPIC ACTIONS:
 

Posted By:   merom_klein
Posted On:   Thursday, May 31, 2001 02:54 AM

hi attached is my code which seems to have a bug (the using the same lexer twice is taken from the example in the yhaoo groups emails): #include #include #include "l.hpp" #include "p.hpp" void InitParser(P& parser) { parser.startRule(); } void main () { std::istringstream x("C=-{SC=ROOT},C=*{SC=$},C=*{ER=100{}} "); static L lexer(x); static P parser(lexer); parser.startRule(); std::istringstream y("C=-{SC=ROOT},C=*{SC=$},C=*{ER=100{}} "); antlr::LexerInputState *l_inputstate = new antlr::LexerInputState(y); antlr::LexerSharedIn   More>>

hi
attached is my code which seems to have
a bug (the using the same lexer twice is taken from the example in the yhaoo groups emails):
#include
#include
#include "l.hpp"
#include "p.hpp"

void InitParser(P& parser)
{

parser.startRule();
}


void main ()
{

std::istringstream x("C=-{SC=ROOT},C=*{SC=$},C=*{ER=100{}}
");
static L lexer(x);
static P parser(lexer);
parser.startRule();

std::istringstream y("C=-{SC=ROOT},C=*{SC=$},C=*{ER=100{}}
");
antlr::LexerInputState *l_inputstate = new antlr::LexerInputState(y);
antlr::LexerSharedInputState nstate (l_inputstate);
lexer.setInputState(nstate);
parser.startRule();

}

the first call to parser.startRule
returns with no error which means
the input is O.K

the second call returns with unexcpected
token although the input is exactly the
same.

removing the first call to parser.startRule()causes the call after it to return with no error
this means i think that something in the
parser.startRule() function harms the
input
can you point on the bug (if it is a bug) and how to fix it?

   <<Less

Re: using the same lexer more than once

Posted By:   Ric_Klaren  
Posted On:   Thursday, May 31, 2001 07:47 AM

The TokenBuffer probably needs a reset as well. I figure now. I'm
working on a patch for this based on some stuff Bogdan Mitu send me. Give
these snippets a spin if you want:


Add this method to CircularQueu.hpp
/// Clear the queue
inline void clear( void )
{
m_offset = 0;
storage.clear();
}

Add this one to InputBuffer.hpp and TokenBuffer.hpp

/// Reset the input buffer to empty state
virtual inline void reset( void )
{
nMarkers = 0;
markerOffset = 0;
numToConsume = 0;
queue.clear();
}

Add this one to LexerSharedInputState:

virtual void initialize( ANTLR_USE_NAMESPACE(std)istream& in, const char* file = "" )
{
column = 1;
line = 1;
tokenStartColumn = 1;
tokenStartLine = 1;
guessing = 0;
filename = file;
if( input && inputResponsible )
delete input;

input = new CharBuffer(in);
inputResponsible = true;
}

And to ParserSharedInputState:

// Reset the ParserInputState and the underlying TokenBuffer
void reset( void )
{
input->reset();
guessing = 0;
}

-------
And change the last bit of your program to:

std::istringstream y("C=-{SC=ROOT},C=*{SC=$},C=*{ER=100{}}
");

L.getInputState().initialize(y);
P.getInputState().reset();

parser.startRule();



This is pretty much untested so beware =) It might be that
P.getInputState is declared const in that case create the
ParserSharedINputstate outside the parser and initalize the parser with it
on creation.

I guess I'll put this in 2.7.2.

About | Sitemap | Contact