Why can't the parser tell the lexer exactly what to go get? (why can't the parser give context information to the lexer to decide what kind of token will be coming down the input stream?)

Terence Parr

Why does ANTLR flag a warning about nondeterminism between BEGIN and ID since BEGIN and ID always follow in that order? The reason is that the parser applies the grammatical structure (sequence "BEGIN ID" in this case) and it pulls tokens from the lexer one at a time. The lexer has no contextual or grammatical information.

class TextParser extends Parser;

start   :       BEGIN ID

class TestLexer extends Lexer;
BEGIN	:	"begin"

ID	:	('a'..'z')+

The parser asks the lexer for what it found--you do not ask it to go get a particular token. Most of the time, the parser would have a choice about what is coming down the token stream and could not tell the lexer what to go get (think about the start of an expr rule. What would the parser tell the lexer to go get? An ID? An integer literal?) It is for this reason that all nontrivial lexers simply go looking for what is there--the parser does not dictate what to find.