Working in Unix-Based Batch Systems

For fine-grained control over the execution of Simcenter STAR-CCM+ a custom job script can be used. This approach is useful when a more complex or custom parallel configuration is used or additional programs must be run before or after Simcenter STAR-CCM+. Due to the multitude of batch systems and possible configurations for them, the details of the particular submission script that you wish to construct are beyond the scope of this section. Refer to the documentation of your cluster, the batch system and, if necessary, consult with your system administrator to identify the attributes to forward to the Simcenter STAR-CCM+ command line.

General Guidelines

The notes given below are general guidelines for a simple implementation of each cluster scheduling system. Please also refer to the documentation of the cluster and batch system.

Resource Allocation

You are advised to always allocate full nodes and use all allocated resources. Some clusters allow multiple jobs to share one node, but this is not advised when running Simcenter STAR-CCM+. Creating imbalances by uneven distribution of processes leads to inefficient use of the cluster resources and can cause a job to run factors of time longer than necessary. Allocating the right amount of resources when submitting a job and using the -batchsystem <system> command line parameter of Simcenter STAR-CCM+ takes care of distribution. No manual control of the number of processes or process distribution (hostlist/machinefile) is necessary.

Under-Subscription

In some cases it can be beneficial to run with less processes on each node than available. This can be the case when the simulation requires a large amount of memory per process or when the simulation is run with GPGPU support (see GPGPU Computation). In this case the number of tasks/processes per node can be reduced when submitting a job to the batch system.

Using Command Line and Submission Script Options

A job can usually be submitted with parameters defined in a submission script or directly on the command line. The submission settings given within the submission script can be overridden when redefined in the command line. Another preference for submission scripts is the repeatability in cluster submission.

A Note on Submission Scripts

A submission script is generally broken up into two parts:

1. Submission directives for the job. These lines are generally indicated by a hash ('#') followed by the Job submission prefix (for example SBATCH, BSUB, PDS, $). This information is parsed when the submission is requested. Job related environment variables are however not defined during this pass.
2. When the job becomes active the file is executed as a script starting the application on the cluster. At this point the job related environment variables are defined. The submission directives are seen as comments and are not used.

The submission directives precede any executable commands in the script. Batch file submission stops processing further submission directives once the first non-comment non-whitespace line has been reached in the script. This has the following implications:

1. All submission directives should be collected into a single block without white space.
2. Between the submission directives and execution script there must be a blank line. This is particularly important for any hash mark-preceded comments which otherwise may be interpreted as submission directives. One way to ensure that this does not occur is to precede each commented-out line with double hash marks ('##').

Wall Time

Estimating the runtime of a simulation can be tricky, especially when running a new model or with different solvers. Make sure to allocate enough time for the simulation to complete but keep in mind that jobs with longer requested runtime might also take longer to be scheduled.