Is there an easy way to include comments and white space in a Java AST Tree (for, say, a code reformatter)?

Created Sep 12, 2000

Terence Parr

Please see the examples/java/preserveWhiteSpace example that uses the CommonASTWithHiddenTokens object and the CommonHiddenStreamToken to track whitespace on a hidden channel that is sent from the lexer to the parser and from there into the trees.

How do you pass the hidden stream info to the translator without mucking up the tree grammar? Easy: use AST nodes that save the hidden stream tokens. ANTLR defines CommonASTWithHiddenTokens for you that hooks the hidden stream tokens onto the tree nodes automatically; methods are available to access the hidden tokens associated with a tree node. All you have to do is tell the parser to create nodes of this node type rather than the default CommonAST.

parser.setASTNodeClass("antlr.CommonASTWithHiddenTokens");

Tree nodes are created as functions of Token objects. The initialize() method of the tree node is called with a Token object when the ASTFactory creates the tree node. Tree nodes created from tokens with hidden tokens before or after will have the same hidden tokens.

You should see the doc on token streams.