dcsimg

How can I get an exception thrown in the lexer to escape out to the parser's invoker?

Created May 4, 2012

Terence Parr

[Version 2.7.0 or greater; earlier versions didn't have the TokenStreamException hierarchy]

Use a filter rule to trap invalid sequences and throw a TokenStreamRecognitionException, which is a TokenStream exception indicating that an invalid token was found. The parser or method that invokes the parser must catch this TokenStreamException.

Remember that the filter option changes the behavior of your lexer. When you set a filter rule, there is no such thing as a syntax error. Anything short of a completely recognized (non-protected) lexer rule will result in the filter rule being called. In other words, the lexer will enter the filter rule with input state rewound to the way it was when the lexer was asked for a token.

The following parser/lexer combo illustrates the strategy. The language is just a set of IDs separated by white space. Upon input "ab xyz 34d", however, no lexer rules will match. The lexer will reset everything and invoke BLASTOUT as though that were the only rule in the lexer. Here it merely consumes the first offensive char, '3', and generates an exception that the lexer and parser do not trap automatically. You can think of TokenStreamException as a kind of IO exception if you want. 
class P extends Parser;

a : ( id:ID {System.out.println("found "+id.getText());} )+ ;

class L extends Lexer;
options {
    filter = BLASTOUT;
}

ID : ('a'..'z')+ ;

WS : (' '|'
')+ {$setType(Token.SKIP);} ;

protected
BLASTOUT
    :    c:.
	// must test for EOF_CHAR; plus, Java complains about deadcode
	// if we don't put the "throw" in an if-statement ;)
        {if ( c!=EOF_CHAR )
             throw new TokenStreamRecognitionException(
                new NoViableAltForCharException(c,this)
         );
        }
    ;

The main program simply invokes the parser rule. Any lexical error will throw an exception caught by the System.err.println() block below. 

class Main {
    public static void main(String[] args) {
        try {
            L lexer = new L(System.in);
            P parser = new P(lexer);
            // Parse the input expression
            parser.b();
        }
        catch(Exception e) {
            System.err.println("exception: "+e.getMessage());
        }
    }
}