Using PBS

By specifying -batchsystem pbs on the command line, Simcenter STAR-CCM+ automatically distributes processes to all CPUs/cores allocated to the job.

General information about job submission with batch systems can be found in Working in Unix-Based Batch Systems. Please also refer to the specific cluster documentation and the batch system documentation for additional and required parameters or settings.

Different flavors of the PBS batch submission system exist. Below you can find examples for the two most frequent flavors, which differ as follows in terms of submission:

PBS Professional
```
#PBS -l select=2:ncpus=32
```
Open PBS / Torque
```
#PBS -l nodes=2:ppn=32
```

PBS Professional

PBS Professional is a purchased product that has been preferred by customers using PBS.

Documentation Reference

Useful Commands for Job Submission and Monitoring

Display PBS Release Version

$ qstat --version
pbs_version = 18.1.2

Submit a Job

$ qsub <submission options> <script>

Show Job Status

$ qstat

Terminate Job

$ qdel <jobid>

Job Submission

A minimal, example jobscript to launch Simcenter STAR-CCM+ and automatically extract the information about allocated resources:

#!/bin/bash
 
#PBS -N starsim
#PBS -l select=2:ncpus=32
#PBS -l walltime=1:00:00
 
STAR_PATH=/path/to/star/bin
$STAR_PATH/starccm+ -bs pbs -batch step my.sim

This allocates a job named starsim on two nodes with 32 processes per node for one hour. The standard and error output is redirected to separate files.

Job File Submission Prefix

Anything that starts with #PBS in the PBS script (job submission script) is a PBS directive. Writing PBS directives in the PBS Script saves you from writing a long line of options for qsub when submitting a job.

More info on PBS directives can be found by looking at the options in the qsub manual either by running man qsub or by visiting the online manuals.

Submission Options


Parameter	Description
`-N <job_name>`	job name
`-l select=<nodes>:ncpus=<numcpus>[:mpiprocs=<numprocs>]`	delegate `<nodes>` over `<numcpus>` cores and `<numprocs>` processes
`-l walltime hh:mm:ss`	wall time
`-q <queue_name>`	name of queue to submit
`-o output_file`	standard output file
`-e error_file`	error output file
`-j oe`	output file and error file are the same

Submission Environment Variables


Environment Variable	Description
`PBS_JOBDIR`	directory of the job
`PBS_JOBID`	job ID
`PBS_JOBNAME`	name of the job
`PBS_O_QUEUE`	name of the queue
`PBS_O_SHELL`	shell used for the job
`PBS_O_WORKDIR`	working directory of the job
`PBS_QUEUE`	queue name

Using Tight Integration of PBSPro in Open MPI

Torque and PBSPro are compatible libraries as far as Open MPI is concerned, but Open MPI only permits compilation against one of them. The Simcenter STAR-CCM+ Open MPI distribution has an optional dependency on libtorque.so for tight integration of PBS in Open MPI. If there is no libtorque.so installed on your system, but only libpbs.so, this optional dependency cannot be satisfied. In order to enable tight integration in this case, please either install libtorque.so on your system or follow these steps:

Determine the exact libtorque.so dependency of Open MPI's PBS module:

$ ldd `[INSTALLDIR]/star/bin/map_mpi -type openmpi -version $([INSTALLDIR]/star/bin/map_mpi -type openmpi -printversion) -libpath`/openmpi/mca_plm_tm.so | grep libtorque.so

This generates output such as:

libtorque.so.2 => not found

Note the exact name (libtorque.so.2).

Determine the location of the system-installed libpbs.so:
```
$ ldconfig -p | grep -m 1 libpbs.so | tr ' ' '\n' | grep /
```
If this command does not return the path to libpbs.so, please contact your system administrator.
Create a symbolic link from the system-installed libpbs.so to libtorque.so.* as required by Open MPI (for example, for libtorque.so.2 as shown above):
```
$ ln -s /path/to/libpbs.so $PWD/libtorque.so.2
```
Pass the directory of the created symbolic link as a library path to Simcenter STAR-CCM+:
```
$ starccm+ -ldlibpath $PWD <...>
```

OpenPBS / Torque

OpenPBS (previously known as the PBS Professional Open Source Project) where PBS Professional refers to the commercial version.

Documentation References