Using Fujitsu Open MPI

Simcenter STAR-CCM+ supports the Fujitsu Open MPI variant for A64FX clusters with Tofu interconnects.

To leverage the capabilities of the Tofu interconnect on Fujitsu A64FX systems:

  1. Compile the MPIwrapper library against Fujitsu Open MPI on the Fujitsu system and set MPITRAMPOLINE_LIB accordingly (see Using Custom MPI Implementations for further information).
  2. Set the OPENMPI_DIR environment variable to point to the Fujitsu Open MPI installation directory.
  3. Select Fujitsu Open MPI by adding -mpi fujitsu to the starccm+ command.

Fujitsu Open MPI is usually installed along Fujitsu's Technical Computing Suite (TCS).

An example for running Simcenter STAR-CCM+ on a Fujitsu A64FX system could be:

export MPITRAMPOLINE_LIB=/path/to/libmpiwrapper.so
export OPENMPI_DIR=/opt/FJSVxtclanga/tcsds-1.2.35/
/path/to/star/bin/starccm+ -mpi fujitsu -np 48 -noconnect <options> example.sim

The regular Open MPI distributed with Simcenter STAR-CCM+ does not support the Tofu interconnect and cannot be used in a multi-node environment.

Please refer to Using Open MPI, the MPI user's guide on the Fujitsu A64FX system, and the Open MPI documentation for further information.

Console Exclusive Session

Fujitsu Open MPI redirects standard output/error messages in a non-standard way, leading to startup issues of Simcenter STAR-CCM+ in a multi-node environment. If hangs during startup are observed, adding the -noconnect option to the command line flags specifies a console-exclusive session and remediates the startup issue. See Command-Line Reference for further information.

Remote Shell

If the remote shell is not correctly set by the system or user (usually pjrsh), it may be necessary to set it manually by adding -rsh /path/to/pjrsh or -mpiflags "-mca plm_rsh_agent /path/to/pjrsh" to the command line flags.

Machinefile and Host Selection

Fujitsu Open MPI does not support manual host selection via -on or -machinefile. Please select the correct number of nodes and processes via the job submission system and solely specify the total number of processes using the -np option.

Fabric Selection

Fujitsu Open MPI does not support fabric selection or fabric verbosity. It is tuned to optimally leverage the special Tofu network interconnect.

Running on a Fast File System

For optimal performance, the Simcenter STAR-CCM+ installation, simulation file, batch macros, and other elements should be located on a fast parallel file system. The location of this file system varies from cluster to cluster. Please make sure to check the local documentation or contact the cluster administrator to find out which file system to use.