Why can't the parser tell the lexer exactly what to go get? (why can't the parser give context information to the lexer to decide what kind of token will be coming down the input stream?)
Why does ANTLR flag a warning about nondeterminism between BEGIN and
ID since BEGIN and ID always follow in that order? The reason is that
the parser applies the grammatical structure (sequence "BEGIN ID" in
this case) and it pulls tokens from the lexer one at a time. The
lexer has no contextual or grammatical information.
class TextParser extends Parser;
start : BEGIN ID
class TestLexer extends Lexer;
BEGIN : "begin"
ID : ('a'..'z')+
The parser asks the lexer for what it found--you do not ask it to go get a particular token. Most of the time, the parser would have a choice about what is coming down the token stream and could not tell the lexer what to go get (think about the start of an expr rule. What would the parser tell the lexer to go get? An ID? An integer literal?) It is for this reason that all nontrivial lexers simply go looking for what is there--the parser does not dictate what to find.