Saving memory on large queries
1 posts in topic
Flat View  Flat View
Older Post First |   Reply to Topic
TOPIC ACTIONS:
 

Posted By:   Anonymous
Posted On:   Thursday, March 15, 2007 08:17 AM

Hi, I have a high level design problem. We have created a framework which uses xml files and based on these xml files run queries on database and map the results on to files. Now our problem is that when we query on very large databases the amount of data being handled is so large that we are fast running out of memory. The present database we are testing on only has a quater of the expected load and yet the process takes upto 1 GB of RAM footprint. The problem is if we donot take the entire resultset in a single go grouping operations cannot be performed as records belonging to the same group may be in different sets that we are obtaining. Can you ple   More>>

Hi,



I have a high level design problem.



We have created a framework which uses xml files and based on these xml files run queries on database and map the results on to files.



Now our problem is that when we query on very large databases the amount of data being handled is so large that we are fast running out of memory. The present database we are testing on only has a quater of the expected load and yet the process takes upto 1 GB of RAM footprint.



The problem is if we donot take the entire resultset in a single go grouping operations cannot be performed as records belonging to the same group may be in different sets that we are obtaining.



Can you please suggest some design ideas

   <<Less
Report | Quote This | Send private message | Reply | Print

Re: Saving memory on large queries

Posted By:   Tim_Frith  
Posted On:   Thursday, March 15, 2007 02:37 PM

Can you shift the grouping work to the database?


Write some code to examine the xml files, then call a stored proc and pass it the results of the examination. The stored proc could handle the data retrieval, filtering, grouping, etc. and return one nice result set.


It's not a database-neutral way to go, but may make it possible to do all the work on sets (more efficient for the db) rather than iteratively outside the db.


You may also want to see if views can be written to combine all the various result sets into a single one from the database.

Report | Quote This | Send private message | Reply | Print
About | Sitemap | Contact