Browsing performance between File Store and JDBC Store
1 posts in topic
Flat View  Flat View
TOPIC ACTIONS:
 

Posted By:   Karl_Cassaigne
Posted On:   Monday, May 1, 2006 07:01 AM

Hi there, We have a message routing application currently in production whose goal is to get messages, process them and route them to their final destination. Each message received is finally stored in the Archive queue if it has been fully processed and delivered or in the Trash queue otherwise. This application is currently running on a single server with one instance of SwiftMQ router. An upgrade will be done soon so that the application runs on two faster servers with SwiftMQ HA to provide High Availability. The only problem we are facing now is about browsing the Archive queue (which may contains thousands of messages) with a selector to find and select message to be "replayed"(rese   More>>

Hi there,



We have a message routing application currently in production whose goal is to get messages, process them and route them to their final destination. Each message received is finally stored in the Archive queue if it has been fully processed and delivered or in the Trash queue otherwise.



This application is currently running on a single server with one instance of SwiftMQ router. An upgrade will be done soon so that the application runs on two faster servers with SwiftMQ HA to provide High Availability.



The only problem we are facing now is about browsing the Archive queue (which may contains thousands of messages) with a selector to find and select message to be "replayed"(resent into the process queue). On our current server (using File Store), browsing into 6000 messages takes around 30 seconds which is not acceptable since there is a user waiting behind the GUI.



I see 3 solutions to solve the problem :

1) take the Archive messages out of SwiftMQ and store them into a dedicated database (MySQL); search would then be executed in this database but we need to develop some code to recreate message into SwiftMQ from the database.

2) same as 1 but with placing only a copy of each message into the database so that we do not need the code that recreate the message inside SwiftMQ.

3) using another instance of SwiftMQ + JDBC Store swiftlet to host the Archive Queue.



Solution 3 seems the most straight-forward if and only if browsing a queue using JDBC Store is really faster than doing it using the File Store.



My question (as you already got it) is : have you some numbers about peformance of the JDBC Store vs File Store as far as browsing is concerned ? Have you another solution to adress my pb ? (any idea is welcome).



Thanks

   <<Less

Re: Browsing performance between File Store and JDBC Store

Posted By:   Andreas_Mueller  
Posted On:   Tuesday, May 2, 2006 02:23 AM

The standard file store is optimized for FIFO processing. For this use case it is clearly faster than anything else. Direct access of a message somewhere at the end of a large queue is where a JDBC store is faster because of the DBMS' query optimizer etc.

However, the file store can be tuned to hold as much pages in its memory cache. A message requires at least one 2KB page in the store. Large messages require more pages. Say your average message size is 3 KB then it requires 2 pages. For 6000 messages this is 12000 pages. Add 10% for the index so you'd need 13200 pages for this queue. 13200 pages x 2 KB = 26400 KB = ~26 MB. That's not much. So you might try to configure your store cache with ~ twice of this size:



That way you won't have much disk access but your checkpoint duration would take a bit longer. Please give it a try.


Have a look here how to tune the file store.
About | Sitemap | Contact