What are the ramifications of mixing JTA and non-JTA transactions where the same resources are involved?
It depends on the context/resource. In general, you shouldn't create this type of heterogeneous transaction scenario, especially when working through a Java application server that shares objects such as database connections. In any situation where the possibility arises for both, design the application to use JTA transactions so that all transactions fall under the control of the transaction manager.
If mixed transactions are necessary (for whatever reason), the application should manipulate the transactions in serial order, that is, start and finish the JTA transaction before creating the non-JTA transaction (or vice versa). If, for example, a JMS client creates a transaction via a UserTransaction and this transaction uses a database resource via the transaction manager, there could be problems if the same application directly accesses the database using a shared database connection.
One of the advantages of using a J2EE-compliant environment is the resource coordination provided by the J2EE implementation via its JTS implementation. If, say, a servlet, accesses multiple resources, for example, via multiple enterprise beans, the J2EE implementation is responsible for ensuring that all related resource usage participates in the global, distributed transaction.