What is the difference between the stream tokenizer and string tokenizer?
The StringTokenizer parses a string into a series of tokens. It is then your responsibility to figure out what each token returned is, beyond the definition of a token separated by delimiters. Usually, you just treat things as words when using StringTokenizer.
On the other hand, a StreamTokenizer allows you to ask is the next token a number, word, quoted string, end-of-file, end-of-line, comment, or whitespace. In this case, the StreamTokenizer is smarter, though can be considerably harder to use.