Using HPE MPI

Prefixing the mpirun Command

There are several helper binaries, such as omplace, dplace, perfboost, that are part of the HPE MPI suite. Those binaries must be placed right before the executed binary in the mpirun command. These helper binaries cannot be passed via the -mpiflags command line option. It is however possible to execute those commands by setting them via the MPI_PREFIX_CMD environment variable, which places its content right before the executed binary. Please refer to your HPE cluster documentation for details.

Manual Process Pinning

For manual process pinning the dplace command can be used on HPE systems. In addition to setting the MPI_PREFIX_CMD environment variable, Simcenter STAR-CCM+ must be launched with the options -cpubind off -ldpreload /usr/lib64/libdplace.so, for example:

export MPI_PREFIX_CMD="dplace -s2"
starccm+ [...] -mpi hpe -cpubind off -ldpreload /usr/lib64/libdplace.so

Specifying Environment Variables

To export environment variables to the spawned processes, use the following options with the starccm+ command:

-mpiflags "-genv VARIABLE1 value1 -genv VARIABLE2 value2"

Refer to your HPE cluster documentation for details.

Using Hierarchical Collectives (HCOLL)

HPE MPI supports the Hierarchical Collectives (HCOLL) library on Mellanox InfiniBand systems. It is deactivated by default and can be activated with the -hcoll command line flag. Additionally the hpcx libraries must be loaded, usually via module load hpcx. Refer to your HPE cluster documentation for details.