VocabTokenTypes vs. ParserTokenTypes
0 posts in topic
Flat View  Flat View
TOPIC ACTIONS:
 

Posted By:   Tanya_Widen
Posted On:   Thursday, June 10, 2004 04:52 AM

Hi! I am pretty new to antlr, so I hope I have not missed other documentation about this phenomenon but I have stumbled upon some strange behavior I want to ask about. In defining my simple grammar it seems that I need to ensure that the order of tokens in my VocabTokenTypes.java file and ParserTokenTypes.java is the same or my tokens have the wrong type in the parser. I noticed this by using the ParseView tree where my tokens had the wrong type. When I put the lexer rules in the same order as the tokens appear first in the parser then the input files are parsed correctly. If I put all the symbols in the end then it does not work. any comments about this?    More>>

Hi! I am pretty new to antlr, so I hope I have not missed other documentation about this phenomenon but I have stumbled upon some strange behavior I want to ask about.


In defining my simple grammar it seems that I need to ensure that the order of tokens in my VocabTokenTypes.java file and ParserTokenTypes.java is the same or my tokens have the wrong type in the parser.


I noticed this by using the ParseView tree where my tokens had the wrong type.


When I put the lexer rules in the same order as the tokens appear first in the parser then the input files are parsed correctly. If I put all the symbols in the end then it does not work.


any comments about this?


Simple grammar below





class MakefileParser extends Parser;

options {

k = 1;

}



file

: (block)* EOF;



block

: include;


include

: INCLUDE (DOLLAR LPAREN IDENTIFIER RPAREN | IDENTIFIER );



class MakefileLexer extends Lexer;


options {

charVocabulary='3'..'377';

k=2;

}


tokens {

INCLUDE="include";

}


DOLLAR: '$';

LPAREN: '(';



IDENTIFIER: ('a'..'z'|'A'..'Z'|'0'..'9'|'.') ('a'..'z'|'A'..'Z'|'0'..'9'|'.'|'_'|'-')*;


RPAREN: ')';


WS : (' '

| ' ')

{ _ttype = Token.SKIP; }

;


NEWLINE: "
" { _ttype = Token.SKIP; newline();

// System.out.println("Found newline");

}

| ( '
' | '
' ) { _ttype = Token.SKIP; newline();

// System.out.println("Found newline");

}

;

   <<Less
About | Sitemap | Contact