Can I modify hidden tokens?
2 posts in topic
Flat View  Flat View

Posted By:   wang_ning
Posted On:   Tuesday, February 26, 2002 09:43 AM

I want to translate "//COMMENT" into "/*COMMENT*/".
But since the hidden token stream is not visible to
parser, I do not know how to manipulate it.



Re[2]: Can I modify hidden tokens?

Posted By:   wang_ning  
Posted On:   Thursday, February 28, 2002 03:04 AM

Thank you for your kind reply. The way I solve it
is performing $setText(""/*"") after recognizing ""//""
in my lexer. It works. Also like you said hidden token
stream mechanism is not good enough. And I wondering
if hidden token can be appended to the nearest
existing token instead of exactly one before it.


Re: Can I modify hidden tokens?

Posted By:   Monty_Zukowski  
Posted On:   Wednesday, February 27, 2002 06:15 AM

You will have to figure out for yourself how to preserve comments. If this is the only thing you want to do, you won't even need a parser, just use a lexer and make the COMMENTs real tokens. Have another class that reads every token and writes it out, changing the format of the COMMENT if it needs to.

If this is in conjunction with a parser and/or tree parser then you need to figure out how to preserve the comments because they won't be in your tree since they are skipped.

Some people attach comments to the following token, but that doesn't work so well if you do things like discard semicolons in your tree, which is a common practice.

You could also remember where the comments were by absolute character positions and also keep such information in your tokens. When you print out tokens, always check to see if there was a comment between those two tokens, if so then translate it and print it.

There are lots of ways to do this. It really depends on the scope of your project.

About | Sitemap | Contact