My MPI job appears to run but never produces output. Is it actually running?

It is possible that there is a problem (software or hardware) on one of the nodes that your jobs was assigned. If that is the case, please open a support ticket so that an administrator can mark the node offline.

One way to diagnose if your job started on all the nodes it was assigned is to run the following command (assuming job ID 12345):

rjobstat 12345

Maybe the job didn't spawn on all nodes correctly. Maybe some of them are deadlocked. Please ask for help if you need it.