How do I apply an operation to all the files in a directory, without going through an intermediate File[]?
2 posts in topic
Flat View  Flat View
TOPIC ACTIONS:
 

Posted By:   Alex_Chaffee
Posted On:   Thursday, March 6, 2003 12:28 PM

I happen to have a directory with 10 million files in it, and the normal list() is causing an out-of-memory error. I have a hack solution. Make a FileFilter that does the operation, then returns false (so the list ends up being empty). file.list(new FileFilter() { public boolean accept(File file) { index(file); return false; } }); writer will have to be an instance var, of course. Is there a cleaner way? See also How do you filter the result of the list() method of the File object? How do I get a listing of the files in a    More>>

I happen to have a directory
with 10 million files in it, and the normal list() is causing an
out-of-memory error.



I have a hack solution. Make a FileFilter that does the operation,
then returns false (so the list ends up being empty).

			
file.list(new FileFilter() {
public boolean accept(File file) {
index(file);
return false;
}
});


writer will have to be an instance var, of course.



Is there a cleaner way?



See also

How do you filter the result of the list() method of the File object?

How do I get a listing of the files in a directory?

   <<Less

Re: How do I apply an operation to all the files in a directory, without going through an intermediate File[]?

Posted By:   David_Bates  
Posted On:   Friday, March 14, 2003 07:16 AM

I'm not even going to ask why you have 10 million files in a directory! However, I've knocked up a little script that I think you could probably alter to do what you are trying to achieve. Basically, I'm creating my own FileFilter that filters ranges. This will allow you to get the first 500 files, then the next 500 and so on. This allows you to work piece-wise through the directory listing. I haven't tried this out on a million files (I love my harddrive too much), but I'd be really interested to find out if this works.



Cheers,



David.



PS - ...or just use Perl... ;-)

import java.io.*;

public class Filtering {
public static void main(String[] args) {
Filtering f = new Filtering();
System.out.print(f.getListing());
}

public String getListing() {
String returnString = "";
RangeFilter rf = new RangeFilter(2, 3);
File f = new File("..");
File listing[] = f.listFiles(rf);
for (int i=0; i returnString += listing[i].getName() + "
";
}
return returnString;
}

public class RangeFilter implements FileFilter {
private int startRange;
private int endRange;
private int currentIndex;

public RangeFilter(int startRange, int endRange) {
this.startRange = startRange;
this.endRange = endRange;
currentIndex = 0;
}

public boolean accept(File f) {
currentIndex++;
if (currentIndex >= startRange && currentIndex <= endRange) {
return true;
}
else {
return false;
}
}
}
}

Re: How do I apply an operation to all the files in a directory, without going through an intermediate File[]?

Posted By:   Christopher_Koenigsberg  
Posted On:   Thursday, March 6, 2003 02:05 PM

If you know you are on a Unix system, you could do a Runtime.exec of the wonderful but tricky "find" command.

About | Sitemap | Contact