Tokens plus token for 'everything else'
1 posts in topic
Flat View  Flat View

Posted By:   Keith_Whittingham
Posted On:   Tuesday, August 3, 2004 05:09 AM


I'm slowly getting the hang of this ANTLR thing! But, to avoid my dirty workaround...

I'm trying to build a parser that picks up certain elements of human readable text. I think that using filters is the right solution but I like to get back a token for the non-matching stuff.

For example, in the 'filter with rule' examples, how would I get the IGNORE rule to generate a token for the stuff that doesn't match P and BR rules when I call the getToken() method on the lexer?



Re: Tokens plus token for 'everything else'

Posted By:   Keith_Whittingham  
Posted On:   Thursday, August 5, 2004 07:00 AM

Here's a workaround that does alright although it screws up the line/column number. It uses the row/column from the time where the lexer ran into problems rather than the start of token - which is sort of OK... (By the way having to put br's and p's all over the place when you're posting to this forum is a pain in the r's (but I guess you've heard that before).

You need to wrap the lexer (to keep it nice and warm). When stuff is ignored then you need to create a token, then, the overloaded nextToken() method needs to juggle with and queue the tokens. Feed the following with "a1:zzz" and watch in amazement as it returns an "IGNORE" token and then an EOF token.

;-------- Ali.g ------------------------------
class TestLexer extends Lexer;
options {
charVocabulary = '3'..'377';
k = 1;
filter = IGNORE;
} {
protected Token ignored;


protected URL:
( 'a'..'z' )+
( options {greedy=true;}: ~( ' ' | ' ' | '
' | '
' ) )*

protected WORD: ( . )+;

protected IGNORE: ( . )+ {
ignored = makeToken(IGNORE);

;-------- Wrapper.java -----------------------
public class Wrapper extends TestLexer {
public Wrapper(Reader in) {
public Wrapper(InputStream in) {
private Token queued;
public Token nextToken() throws TokenStreamException {
Token t;
if(queued != null) {
t = queued;
queued = null;
} else {
t = super.nextToken();
if(ignored != null) {
queued = t;
t = ignored;
ignored = null;
return t;
About | Sitemap | Contact