Router network slow & SwiftMQ Explorer throws java.lang.OutOfMemoryException after production issue.
4 posts in topic
Flat View  Flat View
TOPIC ACTIONS:
 

Posted By:   james_hood
Posted On:   Monday, February 16, 2004 07:35 AM

I manage a 12-node production SwiftMQ v4.7.0 router network. Last week we had a problem with one of the routers where it would connect to the network, but it seemed that the topic messages were not connected - our app could comsume messages on topics that were produced on that same router, but messages on the topic that originated from another router were not getting to this router. We have since taken this box out of service, and now whenever we try connecting to the network via SwiftMQ Explorer, the explorer loads a few routers and then we see a java.lang.OutOfMemory exception in the command window. At this point, the router network seems to hang or become incredibly slow, and doesn't clear up until we kill the Explorer. Oddly, we can connect   More>>

I manage a 12-node production SwiftMQ v4.7.0 router network. Last week we had a problem with one of the routers where it would connect to the network, but it seemed that the topic messages were not connected - our app could comsume messages on topics that were produced on that same router, but messages on the topic that originated from another router were not getting to this router.


We have since taken this box out of service, and now whenever we try connecting to the network via SwiftMQ Explorer, the explorer loads a few routers and then we see a java.lang.OutOfMemory exception in the command window. At this point, the router network seems to hang or become incredibly slow, and doesn't clear up until we kill the Explorer. Oddly, we can connect to the network without problems using CLI.


I am wondering if it is because of the topology of our router network, but the setup has worked fine for over two months without any problems.


Has anyone else seen this behavior? My team and I are stumped on what is causing the problem.


Any help you can provide is greatly appreciated.


Thanks,

Jim

   <<Less

Re: Router network slow & SwiftMQ Explorer throws java.lang.OutOfMemoryException after production issue.

Posted By:   Andreas_Mueller  
Posted On:   Monday, February 16, 2004 11:17 AM

We have since taken this box out of service


Just as a hint: If you take a router out of service which was connected to another router, the other router still sends messages to it and buffers it into the resp. routing queue (rt$...). If you don't use message expiration, a cleanup job, slow subscriber conditions and the like, you'll run into a memory problem at that router. The messages are delivered once the out-of-service-router reconnects.

Re: Router network slow & SwiftMQ Explorer throws java.lang.OutOfMemoryException after production issue.

Posted By:   Andreas_Mueller  
Posted On:   Monday, February 16, 2004 09:12 AM

The OutOfMemoryException of the Explorer is caused by the amount of management messages from 12 routers. To reduce the amount, the smart management tree should be used on each router. You must NOT use


use-smart-tree="false"

in the "router" node!


You should further increase the collect intervals of Threadpool Swiftlet, Queue Manager Swiftlet, Network Swiftlet to 10000 each. This reduces the updates of the resp. Entity Tables.


A good idea is to enable access protection of the Management Swiftlets of the routers. Look here. Big advantage is that you don't get all 12 management trees of the routers when you connect with the Explorer but you get only that one which you have authenticated.


Of course, increase the -Xmx of the Explorer to 1024M.


All this will reduce the amount of management messages between the routers and the Explorer instance. However, you can also use CLI which doesn't need management messages at all but you can do everything.

Re: Router network slow & SwiftMQ Router throws java.lang.OutOfMeoryException after production issue.

Posted By:   james_hood  
Posted On:   Monday, February 16, 2004 07:51 AM

Arrgg! And looking at the subject, I see I made a mistake:
It is the SwiftMQ Explorer that is throwing the java.lang.OutOfMemoryException.

sorry for the error

jim

Re: Router network slow & SwiftMQ Router throws java.lang.OutOfMeoryException after production issue.

Posted By:   james_hood  
Posted On:   Monday, February 16, 2004 07:49 AM

Also - after looking around some more I should note that we are running java v1.4.2_03-b02 to run the explorer and java v1.4.2_02 to run the routers

thx,

jim
About | Sitemap | Contact