Search result with 100.000 objects
2 posts in topic
Flat View  Flat View
TOPIC ACTIONS:
 

Posted By:   Brian_Grey
Posted On:   Sunday, February 6, 2005 10:26 AM

Here is my problem: In my web application deployed on Tomcat, there is a search page that can retrieve all “objects” from an Oracle database. These “objects” are phone cards objects that have properties like phone number, profile, etc… The client can have 100.000, 200.000 cards or more in the system. Actually when the client wants to see the result of his search, the system displays it in pages with 15 “objects” (or “cards”) per page. On the top of the page there is a direct link of the other pages. So the client can navigate trough all pages using these links or using the previous/next buttons. The problem is: when the client doesn’t put any criteria into the search page (meaning that he’s going to get all   More>>

Here is my problem:



In my web application deployed on Tomcat, there is a search page that can retrieve all “objects” from an Oracle database. These “objects” are phone cards objects that have properties like phone number, profile, etc…



The client can have 100.000, 200.000 cards or more in the system. Actually when the client wants to see the result of his search, the system displays it in pages with 15 “objects” (or “cards”) per page. On the top of the page there is a direct link of the other pages. So the client can navigate trough all pages using these links or using the previous/next buttons.

The problem is: when the client doesn’t put any criteria into the search page (meaning that he’s going to get all “objects” (= can be 10.000, 100.000 or more)), the system returns “out of memory” after a few long minutes.

The search functionality returns a collection of objects. A “page iterator” splits it into several pages. I think that the collection that contains the result cannot handle it.
By the way, I’m using Kodo as a “persistence manager”.


Here is the method that retrieves all objects from a search (with an empty filter in this case)



protected Collection execute(Class type, String filter) {



final Extent extent = getPersistenceManager().getExtent(type, true);

query = pm.newQuery(extent, filter);

query.setOrdering(orderBy + " " + (orderAscending ? "ascending" : "decending"));



final Collection result = (Collection) query.execute();

return result;

}



Thanks

   <<Less

Re: Search result with 100.000 objects

Posted By:   Anonymous  
Posted On:   Friday, April 8, 2005 01:29 AM

Limit the size of the returned result by whatever Kodo gives you to do this (there is some method for this!! - e.g. setMaxResultSize() or the like). A reasonable limit is something like 1000 or so in many applications. If the size is actually 1000, you should also display a warning "showing only first 1000 hits - enter more search criteria to limit result size" or the like (maybe more tuned to your users' language).


Regards

Harald M.

Re: Search result with 100.000 objects

Posted By:   Anonymous  
Posted On:   Monday, February 7, 2005 01:31 AM

I'm not familiar with Kodo, but using scrollable cursors in JDBC will avoid the need to store a large map of objects:



http://java.sun.com/developer/Books/JDBCTutorial/index.html



Also, if you have enough memory on the machine, you can extend the JVM heap size with the -Xms and -Xmx options. Try keeping the object size down to a bare minimum, by removing data which is not part of the results shown to the client.



Francis

www.shillitoe.com
About | Sitemap | Contact