Why can't I get the exportVocab/importVocab directives to work with string literals across multiple files?

Ric Klaren

Christopher writes:
The literal ".accept" matches using this set of import/export rules:

class TestingParser extends Parser;
options {
  exportVocab=TestingParser; 
}

command : ".accept"
          {cout << "matched accept" << endl; }
        ;

class TestingLexer extends Lexer;
options {
  charVocabulary='3'..'377';
  exportVocab=TestingParser;
  caseSensitive=false;
}

ID : ( 'a' .. 'z' | '0' .. '9' | '.' )+ ;
WS : ( ' ' | '	' | '
' { newline(); } 
       | '
' | 'b' )+ { $setType(Token::SKIP); } ;

but ".accept" does not match using this set of import/export rules (regardless of whether or not the parser and lexer definitions are in the same file):

class TestingParser extends Parser;
options {
  importVocab=TestingLexer;
  exportVocab=TestingParser;
}

...

class TestingLexer extends Lexer;
options {
  ...
  exportVocab=TestingLexer;
  ...
}
Ric answers:
As a rule of thumb use this scheme when using importvocab/exportvocab with grammars in different files (the parser/lexer in one file scheme is subtly different):
  1. First in the lexer you export the current vocabulary (say L) e.g. exportVocab = L.
  2. Then in the parser you have to get the definitions from the lexer e.g. importVocab = L, in the parser you can extend the vocabulary with for example imaginary tokens. These you might need in a treeparser so you do in the parser a exportVocab = P (don't use the same name as for L!!!).
  3. Last but not least in the treeparser you do a importVocab = P. If you transform the tree and add new nodetypes and want to use those in subsequent treewalkers, then you need to extend the chain of imports/exports similar to the way that is done with parser and lexer.

Also make sure you have your build dependencies set up right. It can be very frustrating to debug something that comes from parser/lexer/walker differing in opinions on tokensets because of a incomplete build... (e.g. build in the order lexer, parser, treewalker)

You can check for errors in this stuff by looking at the xxxTokenTypes files and looking for discrepancies.

To come back to your question: Your last scheme is the right one.. So my guess is that you run your parser through antlr before you run the lexer through it. Best guess is to remove all antlr generated stuff then build the lexer and the parser in that order, then it should work.

Comment and Contribute

 

 

 

 

 


(Maximum characters: 1200). You have 1200 characters left.

 

 

About | Sitemap | Contact