What is the difference between the stream tokenizer and string tokenizer?

John Zukowski

You can think of the StringTokenizer as a specialized form of StreamTokenizer, where each token that is returned is a String. Depending upon what you want to do with the resultant tokens and/or how complex of a parse job you want to deal with lets you decide which to use.

The StringTokenizer parses a string into a series of tokens. It is then your responsibility to figure out what each token returned is, beyond the definition of a token separated by delimiters. Usually, you just treat things as words when using StringTokenizer.

On the other hand, a StreamTokenizer allows you to ask is the next token a number, word, quoted string, end-of-file, end-of-line, comment, or whitespace. In this case, the StreamTokenizer is smarter, though can be considerably harder to use.

See How do I work with the StreamTokenizer to get number and word tokens from a file? and How do I read input from a stream "one word at a time"? for how they might do similar tasks.