The result of a “data scatter” code test compiled with OpenMPI v3 and PMIx-3 and submitted to several nodes on our new cluster. The transport layer is native infiniband and a transmission rate of 8000MB/s is achieved (yes, we can’t get enough of that capital B) giving 64Gb/s.
PMI essentially allows sharing of messages between nodes without all the messy ssh key sharing of traditional clusters. PMIx is the exascale version of PMI because we have an EDR switch and that’s how we roll these days.