How can I track the character position within an input file? I want know at what index does a token start.

Sriram Srinivasan

One way is to implement your own input stream and to override all the read() methods (much like FilterInputStream).

You can supply an instance of this stream to the lexer's constructor.

Note that overriding consume() in the lexer is not good because the lexer can backtrack and the same character position may be visited many times.