I am trying to take text and break it up into individual tokens. The following code gives me the desired output.



import java.util.StringTokenizer;

import java.io.*;


public class Tokens {

public static void main(String args[]) {

try {

FileReader fr = new FileReader("fruit.txt");

BufferedReader br = new BufferedReader(fr);

String s;

while((s = br.readLine()) != null) {

StringTokenizer st = new StringTokenizer(s);

while (st.hasMoreTokens()) {

System.out.println(st.nextToken());

}

System.out.println(s);

}

fr.close();

}

catch(IOException ioe) {

System.out.println(ioe.getMessage());

}

}

}



This assumes that the file in the current directory called fruit.txt that looks like this:



Apples := Bannana + 4371 - Pears DIV Apples

Bananna := Pears + 1734 - Apples DIV Bananna

Pears := Apples + 3471 - Bananna DIV Pears



The output from this is exactly what I wanted. The only problem is that StringTokenizer only separates the tokens by detecting the whitespace between them. I was hoping to use StramTokenizer instead because it is supposed to take care of this issue. I have tried it and all I can get out of it are integer values. Can someone tell me what I am doing wrong or show me how I can get the same output using StreamTokenizer instead of StringTokenizer? Thanks in advance.