Is there a maximum size of a serialized object tree graph?
There are limits, but it's difficult to quantify most of them. For example, the JVM stack size affects the graph serialization, since the tree walking is done depth-first, recursively. (This is called post-order traversal.) Too deep a recursion causes stack overflow. This limit can kick in for data structures on the order of 1000 deep, regardless of the size of the objects.
Other, less restrictive limits are that the object handle assigned to each object in a graph is an int, so there is a (rather large) limit on the number of objects in a graph. Likewise, the JVM places a limit on the number of members in an class, so objects have a limit on their size.
I think the only practical limit you are likely to encounter is the stack size. This can be avoided by customizing serialization with your own structure-specific readObject() and writeObject() methods.