I originally thought that I could control access to an object by returning a reference to it from another object. For instance, I have a FooFactory that returns a reference to a Foo that is unique for each client.

For very short lived objects, this may be a practical way of controlling access to the object. But indeed, a malicious client could put together a stringified IOR and call string_to_object(). While this would be a somewhat difficult task, it is not impossible. Therefore, you should not use this as a substitution for security.