Using Batch Management Systems

Batch systems manage the allocation of computational jobs to a single large computer or a cluster of smaller computers.

You typically submit a job to a batch system requesting computational resources through the attributes of that particular batch system. The batch system then manages the job, allocating it specific machines when they become available based on the requested resources. In this way, resources are delegated to computational jobs in a fair and efficient manner that is based on rules that are configured by your system administrator.

Simcenter STAR-CCM+ can be run in batch mode under the control of a batch system. If you wish to run Simcenter STAR-CCM+ in parallel under the control of a batch system, certain resource attributes must be obtained from the batch system and used to construct the correct command-line arguments for Simcenter STAR-CCM+. A common way of doing this is to examine attributes of the batch system in the batch system’s submission script and pass these attributes to the Simcenter STAR-CCM+ command line. For example, if you requested two machines from the batch system when you submitted the job, the batch system should provide a way of finding out which machines have been allocated. These machine names can then be used to construct a machine file for the parallel execution of Simcenter STAR-CCM+.

Automatic transfer from the batch system to Simcenter STAR-CCM+ of a limited number of attributes is supported for the following batch systems:

  • Unix-Based Batch Systems:
    • SLURM
    • PBS: Open PBS, PBS Pro, Torque
    • LSF
    • Grid Engine: Sun/Open Grid Engine and variants
  • Windows-Based Batch Systems:
    • Microsoft HPC Pack for Windows Server

License Management

Your batch management system uses a scheduler to implement the work-load management. To avoid situations where Simcenter STAR-CCM+ jobs start to execute on your cluster but then fail due to insufficient licenses being available, you can allow the scheduler to treat your finite set of licenses as a consumable resource.

On your cluster, the administrator must first define the total available capacity of licenses. Then, the internal bookkeeping of your batch management system monitors the consumption of the licenses for all Simcenter STAR-CCM+ jobs. A job only starts to execute if the internal bookkeeping indicates that sufficient licenses are available.

For more information on how to set up consumable resources on your cluster, see the User Guide of your batch management system.

Process Distribution with Batch Systems

Simcenter STAR-CCM+ distributes processes according to the total number of processes and order in which they were requested from the batch system. This factor is especially important to consider when the computing resource is under-subscribed. For example, in GPGPU accelerated simulations, the number of processes on each compute node should match the number of available GPGPUs.

To make sure the number of processes and number of GPGPUs match, submit the job with the correct number of processes per node, rather than reducing the total number of processes using the -np option. To do this:
  • On SLURM systems, set the --ntasks-per-node option to the correct value.
  • On PBS systems, set the ncpus and mpiprocs options as required.

When running with -gpgpu auto, Simcenter STAR-CCM+ automatically distributes the processes evenly across all hosts to improve GPGPU utilization. To return to the default process distribution, use the -gpgpu auto:oversubscribe command (see Command-Line Reference).

Consult your cluster and batch system documentation on how to allocate the correct amount of resources for your job. If unsure, confirm that the expected process distribution is applied using the -cpubind v command line option (see -cpubind <policy>).