Every time my remote method is executed, it creates a number of threads and an equal number of other objects used to save data. However when the remote method returns, the memory allocated for these objects and threads is not freed. As a result, after executing the same method for a number of times, I get a java.lang.OutOfMemoryError.
If the threads you created have a reference to the objects used to save data, then those object will never be garbage collected.
If the threads you created explicitly set their reference to the objects to null and the RMI-Connection thread sets its references to the objects to null, then there are no more references to the object and it is subject to garbage collection.
The RMI Runtime reuses the RMI-Connection thread on subsequent remote invocations so any objects the thread creates remain alive as well as any threads created.
As far as close the present server and open a new one -- NO.
You must solve the storage problem. You have a design flaw somewhere and this is where Java tools come in handy. There are products on the market to display memory and object usage (like JProbe). Look here on JGuru or go over to JavaWorld for a list of tools.