How do java.lang.Runtime.totalMemory() and freeMemory() relate to values reported by top and other tools? In particular, why do Java processes use so much memory (according to top) above the limit set by the Xmx option and above the value reported by totalMemory()? Also, why does the memory use reported by top never decrease?
The correspondence between total process memory usage and the numbers reported by the heap memory manager is not very close. There are a couple of reasons for this:
The process uses memory allocated through mechanisms other than the Java heap manager - for example, the native code in the JVM or any JNI libraries might allocate memory using libc's malloc() mechanism: this takes process space, but not space from the Java heap. For large applications, this "stealth" memory isn't very significant, but the normal overhead of simply running the JVM can swamp the heap usage for small apps.
This doesn't mean that the garbage-collected memory is lost! The JVM keeps track of it and reuses it endlessly. But it cannot be returned to the operating system, which is why the process memory usage reported by top never decreases.
Note that there are a couple of columns of interest in top's output: the memory size, and the resident set size. The latter number indicates how much memory is actually taking up space in RAM. The other (possibly larger) number shows memory allocated to the process: the difference between the two represents memory paged out to disk. It's possible for a large application to settle down and regularly touch only a small part of its allocated memory, thus taking up a much smaller footprint in the memory that really counts: that which is resident in RAM.