Text file writing performance issue - JDK 1.3
1 posts in topic
Flat View  Flat View
TOPIC ACTIONS:
 

Posted By:   Andrew_Wall
Posted On:   Tuesday, August 27, 2002 07:07 AM

I'm writing an application that needs to create several large text files (~ 20 - 50 megs apiece). I am currently using StringBuffer objects to hold each line of the file and creating each file with a PrintWriter object constructed as: PrintWriter pw = new PrintWriter(new BufferedWriter(new FileWriter(fileName))); Every time I want to write a line to a file, I make a new StringBuffer, append all the values, then call the println() method on the PrintWriter object (the PrintWriter is only created once per file. I was expecting better performance than am currently experiencing and wasn't sure if I am using the best method of creating text files. A   More>>

I'm writing an application that needs to create several
large text files (~ 20 - 50 megs apiece). I am
currently using StringBuffer objects to hold each line
of the file and creating each file with a PrintWriter
object constructed as:

			
PrintWriter pw = new PrintWriter(new BufferedWriter(new FileWriter(fileName)));

Every time I want to write a line to a file, I make a
new StringBuffer, append all the values, then call the
println() method on the PrintWriter object (the PrintWriter
is only created once per file. I was expecting better
performance than am currently experiencing and
wasn't sure if I am using the best method of creating
text files. Any help or advice would be greatly appreciated.

   <<Less

Re: Text file writing performance issue - JDK 1.3

Posted By:   Stephen_Ostermiller  
Posted On:   Tuesday, August 27, 2002 10:50 AM

Using a BufferedWriter is good, if you hadn't done that you might expect performance to be 10 times worse.


My tests show that you would be able to increase performance by 50-100% by eliminating the StringBuffer. If you are not getting times something in this range, you might want to run a profiler because you may have some other problems.


results:

5715 ms: 22957562 bytes written, buffered writer, no string buffer.

9236 ms: 22957562 bytes written, buffered writer, reuse stringbuffer for each line.

12359 ms: 22957562 bytes written, no buffered writer, no string buffer.

9442 ms: 22957562 bytes written, buffered writer, new stringbuffer each line.


import java.io.*;
public class Test {
private static final int lines = 0x40000;
private static final String data = "1234567890";
public static void main(String args[]) throws IOException {
System.out.println("Starting tests...");
PrintWriter out;
long startTime;
StringBuffer sb;
File f = File.createTempFile("test", "txt");

startTime = System.currentTimeMillis();
out = new PrintWriter(new BufferedWriter(new FileWriter(f)));
for (int i=0; i out.print(i);
out.print(" ");
for (int j=0; j<8; j++){
out.print(data);
}
out.println();
}
out.flush();
out.close();
System.out.println((System.currentTimeMillis() - startTime) + " ms: " + f.length() + " bytes written, buffered writer, no string buffer.");

startTime = System.currentTimeMillis();
out = new PrintWriter(new BufferedWriter(new FileWriter(f)));
sb = new StringBuffer();
for (int i=0; i sb.setLength(0);
sb.append(i);
sb.append(" ");
for (int j=0; j<8; j++){
sb.append(data);
}
out.println(sb.toString());
}
out.flush();
out.close();
System.out.println((System.currentTimeMillis() - startTime) + " ms: " + f.length() + " bytes written, buffered writer, reuse stringbuffer for each line.");

startTime = System.currentTimeMillis();
out = new PrintWriter(new FileWriter(f));
for (int i=0; i out.print(i);
out.print(" ");
for (int j=0; j<8; j++){
out.print(data);
}
out.println();
}
out.flush();
out.close();
System.out.println((System.currentTimeMillis() - startTime) + " ms: " + f.length() + " bytes written, no buffered writer, no string buffer.");

startTime = System.currentTimeMillis();
out = new PrintWriter(new BufferedWriter(new FileWriter(f)));
for (int i=0; i sb = new StringBuffer();
sb.append(i);
sb.append(" ");
for (int j=0; j<8; j++){
sb.append(data);
}
out.println(sb.toString());
}
out.flush();
out.close();
System.out.println((System.currentTimeMillis() - startTime) + " ms: " + f.length() + " bytes written, buffered writer, new stringbuffer each line.");

f.delete();
}
}
About | Sitemap | Contact