> This page is for version 10. > For other versions, use one of these documentation indexes: > - 11 (default): https://docs.nvidia.com/base-command-manager/product-manuals/11/llms.txt > - 10: https://docs.nvidia.com/base-command-manager/product-manuals/10/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/base-command-manager/product-manuals/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/base-command-manager/product-manuals/_mcp/server. # PBS Professional And OpenPBS NVIDIA Base Command Manager works with PBS Professional and OpenPBS, which are the modern editions of the original Portable Batch System (PBS) software. The original PBS software was a workload management and job scheduling system to manage computing resources, and was originally developed at NASA in the 1990s. In this manual, PBS is used as a shorter form to refer to both the modern editions. PBS *job scripts* are used to submit and execute jobs. The user puts values into a job script for the resources being requested, such as the number of processors to be used, the memory to be used, or number of nodes required. Other values are also set for the runtime parameters and application-specific variables. The steps for running a job through a PBS job script are: * Creating an application to be run via the job script * Creating the job script, adding directives, applications, runtime parameters, and application-specific variables to the script * Submitting the script to the workload management system This chapter covers the using the workload managers and job scripts with the PBS variants so that users can get a basic understanding of how they are used, and can get started with typical cluster usage. In this chapter: * section [7.1](#71-components-of-a-job-script) covers the components of a job script and job script examples * section [7.2.1](#721-preliminaries-loading-the-modules-environment) covers submitting, monitoring, and deleting a job with a job script More on using PBS is to be found in the PBS Professional Guides, which can be accessed from: [https://community.altair.com/community?id=altair_product_documentation](https://community.altair.com/community?id=altair_product_documentation). ## 7.1 Components Of A Job Script To use PBS, a batch job script is created by the user. The job script is a shell script containing the set of commands that the user wants to run. It also contains the resource requirement directives and other specifications for the job. After preparation, the job script is submitted to the workload manager using the `qsub` command. The workload manager then tries to make the job run according to the job script specifications. A job script can be resubmitted with different parameters (e.g. different sets of data or variables). ### 7.1.1 Sample Script Structure A job script in PBS has a structure illustrated by the following basic example: **Example** ```bash #!/bin/bash # #PBS -l walltime=1:00:00 #PBS -l select=2:ncpus=1:mem=500mb #PBS -j oe cd ${HOME}/myprogs mpirun myprog a b c ``` The first line is the standard “shebang” line used for scripts. The lines that start with `#PBS` are PBS directive lines, described shortly in section [7.1.2](#712-directives). The last two lines are an example of setting any remaining options or configuration settings up, so that the script can run. In this case, a change to the directory `myprogs` is made, and then the executable `myprog` is run, with arguments `a b c`. The line that runs the program is called the executable line (section [7.1.3](#713-the-executable-line)). To run the executable file in the executable line in parallel, the job launcher `mpirun` is placed immediately before the executable file. The number of nodes the parallel job is to run on is assumed to have been specified in the PBS directives. ### 7.1.2 Directives #### 7.1.2.1 Job Script Directives And `qsub` Options A job script typically has several configurable values called job script directives, set with job script directive lines. These are lines that start with a “`#PBS`”. Any directive lines beyond the first executable line are ignored. The lines are comments as far as the shell is concerned because they start with a “`#`”. However, at the same time the lines are special commands when the job script is processed by the `qsub` command. The difference is illustrated by the following: * The following shell comment is only a comment for a job script processed by `qsub`: ```bash # PBS ``` * The following shell comment is also a job script directive when processed by `qsub`: ```bash #PBS ``` Job script directive lines with the `#PBS ` part removed are the same as options applied to the `qsub` command, so a look at the `man` pages of `qsub` describes the possible directives and how they are used. If there is both a job script directive and a `qsub` command option set for the same item, then the `qsub` option takes precedence. Since the job script file is a shell script, the shell interpreter used can be changed to another shell interpreter by modifying the first line (the “`#!`” line) to the preferred shell. Any shell specified by the first line can also be overridden by using the “`#PBS -S`” directive to set the shell path. #### 7.1.2.2 Walltime Directive The workload manager typically has default *walltime* limits per queue with a value limit set by the administrator. The user can set a walltime limit for the job by setting the ”`#PBS -l walltime`” directive to a specific time. The time specified is the maximum time that the user expects the job should run for, and it allows the workload manager to work out an optimum time to run the job. The job can then run sooner than it would by default. If the walltime limit is exceeded by a job, then the job is stopped, and an error message in the following format is displayed: ```bash =>> PBS: job killed: walltime exceeded limit ``` Here, \<*running time*> is the time that the job actually took to run after it ran, while \<*set time*> is the time that the user set as the walltime resource limit. If there is no default walltime set by the cluster administrator, and if the user submits a job with no walltime set too, then the job only runs when all jobs with a walltime have made room. So to avoid waiting a very long time, and to experience a reasonably consistent behavior on different clusters, the user really should set a walltime. #### 7.1.2.3 Resource List Directives Resource list directives specify arguments to the `-l` directive of the job script, and allow users to specify values to use instead of the system defaults. For example, in the sample script structure earlier, a job walltime of one hour and a memory space of at least 500MB are requested (the script requires the size of the space be spelled in lower case, so “`500mb`” is used). If a requested resource list value exceeds what is available, the job is queued until resources become available. For example, if nodes only have 2000MB to spare and 4000MB is requested, then the job is queued indefinitely, and it is up to the user to fix the problem. Resource list directives also allow, for example, the number of nodes (`-l select=2:`) and the number of processor cores for each node, `ncpus`, to be specified. If no value is specified, then the default is 1 node, and 1 core per node (`ncpus=1`). As an aside: To avoid specifying `ncpus` with a directive, the default value for `ncpu` can be configured by the cluster administrator with the PBS `qmgr` configuration setting ([section 7.11.1 of the *Administrator Manual*](/admin/workloadmanagement#7111-other-pbs-professional-customizations-examples)): **Example** ```bash qmgr -c "set server resources_default.ncpus = 4" ``` A directive for a resource such as the wall time applies to the entire job. In PBS, such a resource is called a *job-wide* resource. A collection of resources allocated to a job, but with the resources being requested from the same physical compute node, is called a *chunk*. A directive can be applied at chunk level with the `select=`*chunk name* option. Typically, an MPI job has one chunk per MPI process (rank). Thus, requested resources can be job-wide only (for example, `walltime`), or chunk only (for example, `ncpus`). In a job, a resource cannot be used as chunk as well as job-wide—it can only be used as chunk (one or multiple chunks), or job-wide. So, to run a job on 8 cores, the job-wide specification could be done with: ```bash #PBS -l select=8:ncpus=1 ``` The preceding specification requests 8 CPU cores, and the cores can be anywhere on the cluster. To run a job on one chunk, the chunk specification could be done with: ```bash #PBS -l select=1:ncpus=8 ``` The preceding specification requests 8 CPU cores, and the cores must be on the same physical node. Further examples of node resource specification are given in a table [here](#7127-resource-request-examples). #### 7.1.2.4 Job Directives: Job Name, Logs, And IDs If the name of the job script file is `jobname`, then by default the output and error streams are logged to `jobname.o` \<*`number`*> and `jobname.e` \<*`number`*> respectively, where \<*number*> indicates the associated job number. The default paths for the logs can be changed by using the `-o` and `-e` directives respectively, while the base name (`jobname` here) can be changed using the `-N` directive. Often, a user may simply merge both logs together into one of the two streams using the -j directive. Thus, in the preceding example, “`-j oe`” merges the logs to the output log path, while “`-j eo`” would merge it to error log path. The job ID is an identifier based on the job number and the FQDN of the login node. For a login node called `basecm10.cm.cluster`, the job ID for a job number with the associated value \<*number*> from earlier, would by default be \<*number*>.basecm10.cm.cluster, but it can also simply be abbreviated to \<*number*>. #### 7.1.2.5 Job Queues Sending a job to a particular job queue is sometimes appropriate. An administrator may have set queues up so that some queues are for very long term jobs, or some queues are for users that require GPUs. Submitting a job to a particular queue \<*destination*> is done by using the directive “`#PBS -q` \<*`destination`*>”. #### 7.1.2.6 Directives Summary Some useful job directives are illustrated in the following table:
**Directive Description** **Specified As**
Name the job < *jobname* \> `#PBS -N <` *`jobname`* `>`
Run the job for a maximum runtime of < *walltime* \> `#PBS -l <` *`walltime`* `>`
Run the job for a maximum runtime of 3 hours 10 minutes and 30 seconds `#PBS -l walltime=03:10:30`
Run the job at < *time* \> `#PBS -a <` *`time`* `>`
Set error log name to < *jobname.err* \> `#PBS -e <` *`jobname.err`* `>`
Set output log name to < *jobname.log* \> `#PBS -o <` *`jobname.log`* `>`
Join standard error and standard output to standard error `#PBS -j eo`
Join standard error and standard output to standard output `#PBS -j oe`
Mail to < *user@address* \> `#PBS -M <` *`user@address`* `>`
Mail on < *event* \> `#PBS -m <` *`event`* `>`
where < *event* \> takes the `(a)bort`
value of the letter in `(b)egin`
the parentheses `(e)nd`
`(n) do not send email`
Queue is < *destination* \> `#PBS -q <` *`destination`* `>`
Login shell path is < *shellpath* \> `#PBS -S <` *`shellpath`* `>`
Almost every `qsub` sets a job attribute, and has a corresponding PBS directive with the same syntax as the option. The man page for `qsub` and the man page for `pbs_job_attributes` explain this in detail. #### 7.1.2.7 Resource Request Examples As PBS evolved, the specification for requesting nodes has changed. The form: ```bash #PBS -l nodes=3 ``` is deprecated, and automatically converted to: ```bash #PBS -l select=3 ``` The deprecated changes are described in detail the man page for `pbs_resources`, in the section on `BACKWARD COMPATIBILITY`. Examples of requests for `select=` options are shown in the following table: | **Resource Request Example Description** | **`#PBS -l` Specification** | | :---------------------------------------------------------------- | :---------------------------- | | 8 nodes, anywhere on the cluster | `select=8` | | 2 nodes, 1 processor per node | `select=2:ncpus=1` | | 3 nodes, 8 processors per node | `select=3:ncpus=8` | | 5 nodes, 2 processors per node, and 1 GPU per node | `select=5:ncpus=2:ngpus=1` | | 5 nodes, 2 processors per node, 3 virtual processors for MPI code | `select=5:ncpus=2:mpiprocs=3` | | 5 nodes, 2 processors per node, using any GPU on the nodes | `select=5:ncpus=2:ngpus=1` | | 5 nodes, 2 processors per node, using a GPU with ID 0 from nodes | `select=5:ncpus=2:gpu_id=0` | Some of the examples illustrate requests for GPU resource usage. GPUs and the CUDA utilities for NVIDIA are introduced in Chapter [8](/user/gpu). GPU usage is treated by the workload manager like the attributes of a resource which the cluster administrator will have pre-configured according to local requirements. For further details on resources, the man page for `pbs_resources` can be checked. ### 7.1.3 The Executable Line In the job script structure (section [7.1.1](#711-sample-script-structure)), the executable line is launched with the job launcher command after the directives lines have been dealt with, and after any other shell commands have been carried out to set up the execution environment. #### 7.1.3.1 Using `mpirun` In The Executable Line The `mpirun` command is used for executables compiled with MPI libraries. Executables that have not been compiled with MPI libraries, or which are launched without any specified number of nodes, run on a single free node chosen by the workload manager. The executable line to run a program `myprog` that has been compiled with MPI libraries is run by placing the job-launcher command `mpirun` before it as follows: ```bash mpirun myprog ``` ### 7.1.4 Example Batch Submission Scripts #### 7.1.4.1 Node Availability The following job script tests which out of 4 nodes requested with “`-l nodes`” are made available to the job in the workload manager: **Example** ```bash #!/bin/bash #PBS -l walltime=1:00 # #PBS -l nodes=4 <--- legacy #PBS -l select=4 echo -n "I am on: " hostname; echo finding ssh-accessible nodes: for node in $(cat ${PBS_NODEFILE}) ; do echo -n "running on: " /usr/bin/ssh $node hostname done ``` The directive specifying `walltime` means the script runs at most for 1 minute. The `${PBS_NODEFILE}` array used by the script is created and appended with hosts by the queuing system. The script illustrates how the workload manager generates a `${PBS_NODEFILE}` array based on the requested number of nodes, and which can be used in a job script to spawn child processes. When the script is submitted, the output from the log will look like: ```bash I am on: node001 finding ssh-accessible nodes: running on: node001 running on: node002 running on: node003 running on: node004 ``` This illustrates that the job starts up on a node, and that no more than the number of nodes that were asked for in the resource specification are provided. The list of all nodes for a cluster can be found using the `pbsnodes` command (section [7.2.6](#726-nodes-according-to-pbs)). #### 7.1.4.2 Using InfiniBand A sample PBS script for InfiniBand is: ```bash #!/bin/bash #! #! Sample PBS file #! #! Name of job #PBS -N MPI #! Number of nodes (in this case 8 nodes with 4 CPUs each) #! The total number of nodes passed to mpirun will be nodes*ppn #! Second entry: Total amount of wall-clock time (true time). #! 02:00:00 indicates 02 hours #PBS -l walltime=02:00:00 #PBS -l select=8:ncpus=4 #! Mail to user when job terminates or aborts #PBS -m ae # If modules are needed by the script, then source modules environment: . /etc/profile.d/modules.sh # Add any modules you might require: module add shared mvapich/gcc pbspro #! Full path to application + application name application="" #! Run options for the application options="" #! Work directory workdir="" ############################################################### ### You should not have to change anything below this line #### ############################################################### #! change the working directory (default is home directory) cd $workdir echo Running on host $(hostname) echo Time is $(date) echo Directory is $(pwd) echo PBS job ID is $PBS_JOBID echo This job runs on the following machines: echo $(cat $PBS_NODEFILE | uniq) mpirun_command="mpirun $application $options" #! Run the parallel MPI executable (nodes*ppn) echo Running $mpirun_command eval $mpirun_command ``` In the preceding script, no machine file is needed, since it is automatically built by the workload manager and passed on to the `mpirun` parallel job launcher utility. The job is given a unique ID and run in parallel on the nodes based on the resource specification. ### 7.1.5 Links To PBS Resources A number of useful links are: * The PBS Professional documentation pages, accessible from [https://community.altair.com/community?id=altair_product_documentation](https://community.altair.com/community?id=altair_product_documentation). * The man pages for `qsub`, `pbs_resources`, and `pbs_job_attributes`, for setting up job scripts. * The community support pages for PBS Professional, accessible at [https://community.altair.com/community?id=altair_community_home](https://community.altair.com/community?id=altair_community_home). Commercial support can be arranged for PBS Professional using [https://www.altair.com/tailored-solutions](https://www.altair.com/tailored-solutions). * The OpenPBS community pages, accessible from [http://community.openpbs.org/](http://community.openpbs.org/). ## 7.2 Submitting A Job ### 7.2.1 Preliminaries: Loading The Modules Environment To submit a job to the workload management system, the user must ensure that the following environment modules are loaded: ```bash $ module add shared pbspro ``` Users can pre-load particular environment modules as their default using the “`module init*`” commands (section [2.3.3](/user/clusterusage#233-changing-the-default-environment-modules)). ### 7.2.2 Using `qsub` The command `qsub` is used to submit jobs to the workload manager system. The command returns a unique job identifier, which is used to query and control the job and to identify output. The usage format of `qsub` and some useful options are listed here: ```bash USAGE: qsub [] Option Hint Description ------ ---- ----------- -a at run the job at a certain time -l list request certain resource(s) -q queue job is run in this queue -N name name of job -S shell shell to run job under -j join join output and error files ``` For example, a job script called `mpirun.job` with all the relevant directives set inside the script, may be submitted as follows: **Example** ```bash $ qsub mpirun.job ``` A job may be submitted to a specific queue `testq` as follows: **Example** ```bash $ qsub -q testq mpirun.job ``` The `man` page for `qsub` describes these and other options. The options correspond to PBS directives in job scripts (section [7.1.1](#711-sample-script-structure)). If a particular item is specified by a `qsub` option as well as by a PBS directive, then the `qsub` option takes precedence. ### 7.2.3 Job Output By default, the output from the job script \<*scriptname*> goes into the current working directory for PBS. By default, error output is written to \<*scriptname*>`.e`\<*jobid*> and the application output is written to \<*scriptname*>`.o`\<*jobid*>, where \<*jobid*> is a unique number that the workload manager allocates. Specific output and error files can be set using the `-o` and `-e` options respectively. The error and output files can usefully be concatenated into one file with the `-j oe` or `-j eo` options. More details on this can be found in the `qsub` man page. ### 7.2.4 Monitoring The Status Of A Job To use the commands in this section, the appropriate workload manager module must be loaded: ```bash $ module add pbspro ``` #### 7.2.4.1 `qstat` Basics The main component is `qstat`, which has several options. In this example, the most frequently used options are discussed. In PBS, the command “`qstat -an`” shows what jobs are currently submitted or running on the queuing system. An example output is: ```bash [fred@basecm10 ~]$ qstat -an basecm10: Req'd Req'd Elap Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time --------------- -------- -------- ---------- ------ --- --- ------ ----- - ----- 121.basecm10 noah workq pbjob -- 3 6 -- 01:00 Q -- -- 125.basecm10 noah workq pbjob 26615 3 3 -- 01:00 R -- node001/0+node002/0+node003/0 ``` The output shows the Job ID, the user who owns the job, the queue, the job name, the session ID for a running job, the number of nodes requested, the number of CPUs or tasks requested, the time requested (`-l walltime`), the job state (S) and the elapsed time. In this example, one job is seen to be running (R), and one is still queued (Q). The `-n` parameter causes nodes that are in use by a running job to display at the end of that line. Possible job states include:
**Job States** **Description**
`E` Job is exiting after having run
`F` Job is finished
`H` Job is held
`Q` job is queued, eligible to run or routed
`R` job is running
`S` job is suspended
`T` job is being moved to new location
`W` job is waiting for its execution time
The command “`qstat -q`” shows what queues are available. In the following example, there is one job running in the `testq` queue and 4 are queued. ```bash $ qstat -q server: master.cm.cluster Queue Memory CPU Time Walltime Node Run Queue Lm State ---------------- ------ -------- -------- ---- --- ----- -- ----- testq -- -- 23:59:59 -- 1 4 -- E R default -- -- 23:59:59 -- 0 0 -- E R ---- ----- 1 4 ``` #### 7.2.4.2 Viewing Job Details With `qstat` With `qstat -f` the full output of the job is displayed. The output shows what the jobname is, where the error and output files are stored, and various other settings and variables. ```bash $ qstat -f Job Id: 137.pj-cruncher Job_Name = pbjob Job_Owner = noah@pj-cruncher.cm.cluster resources_used.cpupercent = 0 resources_used.cput = 00:00:00 resources_used.mem = 5456kb resources_used.ncpus = 3 resources_used.vmem = 384164kb resources_used.walltime = 00:00:11 job_state = R queue = workq server = pj-cruncher Checkpoint = u ctime = Fri Mar 20 18:07:06 2020 Error_Path = pj-cruncher.cm.cluster:/home/noah/pbjob.e137 exec_host = node001/0+node002/0+node003/0 exec_vnode = (node001:ncpus=1)+(node002:ncpus=1)+(node003:ncpus=1) Hold_Types = n Join_Path = oe Keep_Files = n Mail_Points = a mtime = Fri Mar 20 18:08:52 2020 Output_Path = pj-cruncher.cm.cluster:/home/noah/pbjob.o137 Priority = 0 qtime = Fri Mar 20 18:07:06 2020 Rerunable = True Resource_List.mpiprocs = 24 Resource_List.ncpus = 3 Resource_List.nodect = 3 Resource_List.place = free Resource_List.select = 3:ncpus=1:mpiprocs=8 Resource_List.walltime = 01:00:00 stime = Fri Mar 20 18:08:42 2020 session_id = 28107 jobdir = /home/noah substate = 42 Variable_List = PBS_O_HOME=/home/noah,PBS_O_LANG=en_US.UTF-8, PBS_O_LOGNAME=noah, PBS_O_PATH=/cm/shared/apps/openpbs/20.0.1/unsupported/fw/bin:/cm/sha red/apps/openpbs/20.0.1/unsupported:/cm/shared/apps/openpbs/20.0.1/ sbin:/cm/shared/apps/openpbs/20.0.1/bin:/cm/shared/apps/mpich/ge/gcc/ 64/3.3.2/bin:/cm/local/apps/gcc/9.2.0/bin:/cm/local/apps/environment-mo dules/4.4.0//bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbi n:/usr/sbin:/cm/local/apps/environment-modules/4.4.0/bin:/home/noah/.lo cal/bin:/home/noah/bin,PBS_O_MAIL=/var/spool/mail/noah, PBS_O_SHELL=/bin/bash,PBS_O_WORKDIR=/home/noah,PBS_O_SYSTEM=Linux, PBS_O_QUEUE=workq,PBS_O_HOST=pj-cruncher.cm.cluster comment = Job run at Fri Mar 20 at 18:08 on (node001:ncpus=1)+(node002:ncpu s=1)+(node003:ncpus=1) etime = Fri Mar 20 18:07:06 2020 run_count = 1 Submit_arguments = pbjob project = _pbs_project_default ``` ### 7.2.5 Deleting A Job An already submitted job can be deleted using the `qdel` command: `$ qdel` \<*`jobid`*> Multiple space-separated job IDs can be specified. ### 7.2.6 Nodes According To PBS The nodes that the workload manager knows about can be viewed using the `pbsnodes` command. The following output is from a cluster made up of nodes with 4 physical cores, as indicated by the value of 4 for `pcpus`. If the node is available to run scripts, then its state is `free` or `time-shared`. When a node is used exclusively (section [8.5.2](/user/gpu#852-the-nvidia-smi-utility)) by one script, the state is `job-exclusive`. For PBS Professional the display resembles (some output elided): ```bash [noah@pj-cruncher ~]$ pbsnodes -a node001 Mom = node001.cm.cluster ntype = PBS state = free pcpus = 4 resources_available.arch = linux resources_available.host = node001 resources_available.mem = 4044784kb ... queue = workq resv_enable = True sharing = default_shared ... node002 Mom = node002.cm.cluster ntype = PBS state = free pcpus = 4 resources_available.arch = linux resources_available.host = node002 resources_available.mem = 4044784kb ... queue = workq resv_enable = True sharing = default_shared last_state_change_time = Fri Mar 20 18:09:47 2020 last_used_time = Fri Mar 20 18:09:47 2020 ... ```