Thursday, May 24, 2001 06:34 AM
The only way for antlr to lex X number of characters to make a token is for you to keep your own counter and use a semantic predicate. You might find it easier to do the initial lexing by hand or use another lexer toolkit and then hand that off to antlr.
If I were doing it myself based only on the example you give, and I would make my own table of lengths and token types and make my own object implementing TokenStream (which is just one method--nextToken()). Then I would just read the specified number of characters, create a token object with the correct type, and bump my table pointer to the next position for the next token.