Using StringTokenizer and StreamTokenizer
1 posts in topic
Flat View  Flat View
TOPIC ACTIONS:
 

Posted By:   Kevin_Szerszen
Posted On:   Thursday, March 22, 2001 05:58 AM

I am trying to take text and break it up into individual tokens. The following code gives me the desired output. import java.util.StringTokenizer; import java.io.*; public class Tokens { public static void main(String args[]) { try { FileReader fr = new FileReader("fruit.txt"); BufferedReader br = new BufferedReader(fr); String s; while((s = br.readLine()) != null) { StringTokenizer st = new StringTokenizer(s); while (st.hasMoreTokens()) { System.out.println(st.nextToken()); } System.out.println(s); } fr.close(); }    More>>

I am trying to take text and break it up into individual tokens. The following code gives me the desired output.



import java.util.StringTokenizer;

import java.io.*;


public class Tokens {

public static void main(String args[]) {

try {

FileReader fr = new FileReader("fruit.txt");

BufferedReader br = new BufferedReader(fr);

String s;

while((s = br.readLine()) != null) {

StringTokenizer st = new StringTokenizer(s);

while (st.hasMoreTokens()) {

System.out.println(st.nextToken());

}

System.out.println(s);

}

fr.close();

}

catch(IOException ioe) {

System.out.println(ioe.getMessage());

}

}

}



This assumes that the file in the current directory called fruit.txt that looks like this:



Apples := Bannana + 4371 - Pears DIV Apples

Bananna := Pears + 1734 - Apples DIV Bananna

Pears := Apples + 3471 - Bananna DIV Pears



The output from this is exactly what I wanted. The only problem is that StringTokenizer only separates the tokens by detecting the whitespace between them. I was hoping to use StramTokenizer instead because it is supposed to take care of this issue. I have tried it and all I can get out of it are integer values. Can someone tell me what I am doing wrong or show me how I can get the same output using StreamTokenizer instead of StringTokenizer? Thanks in advance.

   <<Less

Re: Using StringTokenizer and StreamTokenizer

Posted By:   Tim_Rohaly  
Posted On:   Thursday, April 19, 2001 02:47 PM

StringTokenizer doesn't just use whitespace
as token delimiters - that is merely the default. To
specify your own set of delimiters, use one of the other
two StringTokenizer constructors. The delimiters
can be only one-character each, but you can have as many
as you want. For example, to scan your input using commas
as delimiters, you would use the following line when
creating your StringTokenizer:

StringTokenizer st = new StringTolenizer(s, ",");

For information on how to properly use StreamTokenizer
(which you probably have to do if you require multi-character
delimiters like ":=") you should use the search feature and
search the IO FAQ for "StreamTokenizer". You will find many
examples this way.
About | Sitemap | Contact