Prior to establishing your database as a JTA resource manager, you must enable support of JTA transactions. The following command enables your database:

proutil db-name-C enablejta

For complete syntax details, see PROUTIL ENABLEJTA qualifier.