Running Simulations with GPGPU

Running Simcenter STAR-CCM+ simulations with GPGPU requires that you activate this option when launching the server. In general, you assign at least the same number of CPU cores as the number of GPGPU cards you intend to use.

Ensure that the GPGPU hardware is supported for use by Simcenter STAR-CCM+.
Some general principles to observe are:
  • The number of CPU processes must be at least equal to the number of GPGPU cards selected for use on the host machine. Otherwise, the number of GPGPUs utilized is restricted to the number of CPU processes assigned.
  • Some GPGPU cards consist of multiple logical GPGPUs. For purposes of Simcenter STAR-CCM+, each logical GPGPU is treated as an independent GPGPU card.
  • If all operations in the simulation are GPGPU enabled, assign the same number of CPUs as the number of GPGPU cards. This helps to avoid the performance penalties associated with GPGPU oversubscription.
  • If some operations present run on the CPU, consider assigning a greater number of CPUs than the number GPGPU cards to avoid poor perfromance when running those operations. In this case, you are advised to set the number of CPU processes to a multiple of the number of GPGPU cards being used. Otherwise, performance issues can occur due to GPGPU load imbalance. When this occurs, Simcenter STAR-CCM+ issues a warning on the command line.
  • If you are using NVIDIA GPUs and the number of CPU processes specified exceeds the number of GPGPU cards being used on a host, you must employ the CUDA Multi-Process Service (MPS). This service is invoked by default. For more information, see Using MPS.
  • For optimal performance, it is important to allocate a reasonable amount of work to the GPGPU in order to achieve high occupancy without exceeding its memory limits. A typical estimate for a suitable problem size is 1.3 GB of GPGPU memory per 1,000,000 cells for trimmed single precision runs. Meaningful error messages are not always generated when exceeding the GPGPU memory, hence caution is advised.
  • To use GPGPU in simulations with algebraic multigrid (AMG), you are advised to make sure that the Cycle Type is V Cycle. For details about the AMG cycle types, see AMG Linear Solver Reference.
  • To monitor the GPGPU memory usage during the course of a simulation run, you can create a GPGPU Resident Memory Report. For more information, see Memory Report.

The following table shows the memory usage for a 5-million-cell, constant-density pipe flow simulation with a two-equation turbulence model on a single GPU card:

Type of Simulation Precision Polyhedral Mesh (GB/million cells) Trimmed Mesh (GB/million cells)
Segregated Flow Mixed 1.5 1.25
Segregated Flow & Energy Mixed 1.75 1.5
Coupled Flow Mixed 3 2.5
Coupled Flow & Energy Mixed 3.75 3
Segregated Flow Double 2.5 2
Segregated Flow & Energy Double 2.75 2.25
Coupled Flow Double 5.25 4
Coupled Flow & Energy Double 6.5 5.25

You can define the GPGPU usage either from the workspace or from the command line.

To start a GPGPU-enabled simulation from the workspace:
  1. Launch the workspace.
    • If you are starting a new simulation, select File > New... to activate the Create a File dialog.
    • If you are opening an existing simulation, select File > Load... to activate the Load a File dialog.
  2. Specify the number of CPU processes for the simulation.
    • If you are running the simulation on one CPU, select the Serial process option (default).
    • If you are running the simulation on multiple CPUs in parallel, select Parallel on Local Host and enter the number of processes in the Compute Processes box.
  3. Enable GPGPU Usage.
    By default the GPGPU Usage option is set to Automatic and the No MPS option is disabled.

  4. Select the GPGPUs from those available on the local machine using one of the following GPGPU usage options:
    • Automatic (default)
    • Count
    • Specific
    • File

    For more information on each of these options, see GPGPU Usage.

    After you have selected the GPGPUs, the equivalent command line which is used to start the server is displayed in the Command text box. For information on starting a GPGPU-enabled simulation from the command line, see GPGPU Options.
  5. Complete the dialog according to the properties described in Interactive Startup Reference.
  6. When setting up the simulation physics, make sure you only select GPGPU compatible solvers and models. For a table of compatible solvers and models, see Supported Solvers and Models.