Is it possible for server-side applications, for example, a JMS application, to play the role of a resource in a distributed transaction?

Under certain conditions, yes. In this particular case, the JMS server would have to implement the appropriate resource-related interfaces in javax.transaction.xa so that the JTS transaction manager can enlist the JMS server as an intermediary, playing the role of a resource manager with respect to the JMS application.