PBS Professional And OpenPBS

NVIDIA Base Command Manager works with PBS Professional and OpenPBS, which are the modern editions of the original Portable Batch System (PBS) software. The original PBS software was a workload management and job scheduling system to manage computing resources, and was originally developed at NASA in the 1990s.

In this manual, PBS is used as a shorter form to refer to both the modern editions.

PBS job scripts are used to submit and execute jobs. The user puts values into a job script for the resources being requested, such as the number of processors to be used, the memory to be used, or number of nodes required. Other values are also set for the runtime parameters and application-specific variables.

The steps for running a job through a PBS job script are:

• Creating an application to be run via the job script

• Creating the job script, adding directives, applications, runtime parameters, and application-specific variables to the script

• Submitting the script to the workload management system

This chapter covers the using the workload managers and job scripts with the PBS variants so that users can get a basic understanding of how they are used, and can get started with typical cluster usage.

In this chapter:

• section 7.1 covers the components of a job script and job script examples

• section 7.2.1 covers submitting, monitoring, and deleting a job with a job script

More on using PBS is to be found in the PBS Professional Guides, which can be accessed from: https://community.altair.com/community?id=altair_product_documentation.

7.1 Components Of A Job Script

To use PBS, a batch job script is created by the user. The job script is a shell script containing the set of commands that the user wants to run. It also contains the resource requirement directives and other specifications for the job. After preparation, the job script is submitted to the workload manager using the qsub command. The workload manager then tries to make the job run according to the job script specifications.

A job script can be resubmitted with different parameters (e.g. different sets of data or variables).

7.1.1 Sample Script Structure

A job script in PBS has a structure illustrated by the following basic example:

Example

#!/bin/bash
#
#PBS -l walltime=1:00:00
#PBS -l select=2:ncpus=1:mem=500mb
#PBS -j oe
cd ${HOME}/myprogs
mpirun myprog a b c

The first line is the standard “shebang” line used for scripts.

The lines that start with #PBS are PBS directive lines, described shortly in section 7.1.2.

The last two lines are an example of setting any remaining options or configuration settings up, so that the script can run. In this case, a change to the directory myprogs is made, and then the executable myprog is run, with arguments a b c. The line that runs the program is called the executable line (section 7.1.3).

To run the executable file in the executable line in parallel, the job launcher mpirun is placed immediately before the executable file. The number of nodes the parallel job is to run on is assumed to have been specified in the PBS directives.

7.1.2 Directives

7.1.2.1 Job Script Directives And qsub Options

A job script typically has several configurable values called job script directives, set with job script directive lines. These are lines that start with a “#PBS”. Any directive lines beyond the first executable line are ignored.

The lines are comments as far as the shell is concerned because they start with a “#”. However, at the same time the lines are special commands when the job script is processed by the qsub command. The difference is illustrated by the following:

• The following shell comment is only a comment for a job script processed by qsub:

  # PBS

• The following shell comment is also a job script directive when processed by qsub:

  #PBS

Job script directive lines with the #PBS part removed are the same as options applied to the qsub command, so a look at the man pages of qsub describes the possible directives and how they are used. If there is both a job script directive and a qsub command option set for the same item, then the qsub option takes precedence.

Since the job script file is a shell script, the shell interpreter used can be changed to another shell interpreter by modifying the first line (the “#!” line) to the preferred shell. Any shell specified by the first line can also be overridden by using the “#PBS -S” directive to set the shell path.

7.1.2.2 Walltime Directive

The workload manager typically has default walltime limits per queue with a value limit set by the administrator. The user can set a walltime limit for the job by setting the ”#PBS -l walltime” directive to a specific time. The time specified is the maximum time that the user expects the job should run for, and it allows the workload manager to work out an optimum time to run the job. The job can then run sooner than it would by default.

If the walltime limit is exceeded by a job, then the job is stopped, and an error message in the following format is displayed:

=>> PBS: job killed: walltime <running time> exceeded limit <set time>

Here, <running time> is the time that the job actually took to run after it ran, while <set time> is the time that the user set as the walltime resource limit.

If there is no default walltime set by the cluster administrator, and if the user submits a job with no walltime set too, then the job only runs when all jobs with a walltime have made room. So to avoid waiting a very long time, and to experience a reasonably consistent behavior on different clusters, the user really should set a walltime.

7.1.2.3 Resource List Directives

Resource list directives specify arguments to the -l directive of the job script, and allow users to specify values to use instead of the system defaults.

For example, in the sample script structure earlier, a job walltime of one hour and a memory space of at least 500MB are requested (the script requires the size of the space be spelled in lower case, so “500mb” is used).

If a requested resource list value exceeds what is available, the job is queued until resources become available.

For example, if nodes only have 2000MB to spare and 4000MB is requested, then the job is queued indefinitely, and it is up to the user to fix the problem.

Resource list directives also allow, for example, the number of nodes (-l select=2:) and the number of processor cores for each node, ncpus, to be specified. If no value is specified, then the default is 1 node, and 1 core per node (ncpus=1).

As an aside: To avoid specifying ncpus with a directive, the default value for ncpu can be configured by the cluster administrator with the PBS qmgr configuration setting (section 7.11.1 of the Administrator Manual):

Example

    qmgr -c "set server resources_default.ncpus = 4"

A directive for a resource such as the wall time applies to the entire job. In PBS, such a resource is called a job-wide resource.

A collection of resources allocated to a job, but with the resources being requested from the same physical compute node, is called a chunk. A directive can be applied at chunk level with the select=chunk name option. Typically, an MPI job has one chunk per MPI process (rank).

Thus, requested resources can be job-wide only (for example, walltime), or chunk only (for example, ncpus). In a job, a resource cannot be used as chunk as well as job-wide—it can only be used as chunk (one or multiple chunks), or job-wide.

So, to run a job on 8 cores, the job-wide specification could be done with:

#PBS -l select=8:ncpus=1

The preceding specification requests 8 CPU cores, and the cores can be anywhere on the cluster.

To run a job on one chunk, the chunk specification could be done with:

#PBS -l select=1:ncpus=8

The preceding specification requests 8 CPU cores, and the cores must be on the same physical node.

Further examples of node resource specification are given in a table here.

7.1.2.4 Job Directives: Job Name, Logs, And IDs

If the name of the job script file is jobname, then by default the output and error streams are logged to jobname.o <number> and jobname.e <number> respectively, where <number> indicates the associated job number. The default paths for the logs can be changed by using the -o and -e directives respectively, while the base name (jobname here) can be changed using the -N directive.

Often, a user may simply merge both logs together into one of the two streams using the -j directive. Thus, in the preceding example, “-j oe” merges the logs to the output log path, while “-j eo” would merge it to error log path.

The job ID is an identifier based on the job number and the FQDN of the login node. For a login node called basecm10.cm.cluster, the job ID for a job number with the associated value <number> from earlier, would by default be <number>.basecm10.cm.cluster, but it can also simply be abbreviated to <number>.

7.1.2.5 Job Queues

Sending a job to a particular job queue is sometimes appropriate. An administrator may have set queues up so that some queues are for very long term jobs, or some queues are for users that require GPUs. Submitting a job to a particular queue <destination> is done by using the directive “#PBS -q <destination>”.

7.1.2.6 Directives Summary

Some useful job directives are illustrated in the following table:

Almost every qsub sets a job attribute, and has a corresponding PBS directive with the same syntax as the option. The man page for qsub and the man page for pbs_job_attributes explain this in detail.

7.1.2.7 Resource Request Examples

As PBS evolved, the specification for requesting nodes has changed. The form:

#PBS -l nodes=3

is deprecated, and automatically converted to:

#PBS -l select=3

The deprecated changes are described in detail the man page for pbs_resources, in the section on BACKWARD COMPATIBILITY.

Examples of requests for select= options are shown in the following table:

Some of the examples illustrate requests for GPU resource usage. GPUs and the CUDA utilities for NVIDIA are introduced in Chapter 8. GPU usage is treated by the workload manager like the attributes of a resource which the cluster administrator will have pre-configured according to local requirements.

For further details on resources, the man page for pbs_resources can be checked.

7.1.3 The Executable Line

In the job script structure (section 7.1.1), the executable line is launched with the job launcher command after the directives lines have been dealt with, and after any other shell commands have been carried out to set up the execution environment.

7.1.3.1 Using mpirun In The Executable Line

The mpirun command is used for executables compiled with MPI libraries. Executables that have not been compiled with MPI libraries, or which are launched without any specified number of nodes, run on a single free node chosen by the workload manager.

The executable line to run a program myprog that has been compiled with MPI libraries is run by placing the job-launcher command mpirun before it as follows:

mpirun myprog

7.1.4 Example Batch Submission Scripts

7.1.4.1 Node Availability

The following job script tests which out of 4 nodes requested with “-l nodes” are made available to the job in the workload manager:

Example

#!/bin/bash
#PBS -l walltime=1:00
#   #PBS -l nodes=4     <--- legacy
#PBS -l select=4
echo -n "I am on: "
hostname;
echo finding ssh-accessible nodes:
for node in $(cat ${PBS_NODEFILE}) ; do
    echo -n "running on: "
    /usr/bin/ssh $node hostname
done

The directive specifying walltime means the script runs at most for 1 minute. The ${PBS_NODEFILE} array used by the script is created and appended with hosts by the queuing system. The script illustrates how the workload manager generates a ${PBS_NODEFILE} array based on the requested number of nodes, and which can be used in a job script to spawn child processes. When the script is submitted, the output from the log will look like:

I am on: node001
finding ssh-accessible nodes:
running on: node001
running on: node002
running on: node003
running on: node004

This illustrates that the job starts up on a node, and that no more than the number of nodes that were asked for in the resource specification are provided.

The list of all nodes for a cluster can be found using the pbsnodes command (section 7.2.6).

7.1.4.2 Using InfiniBand

A sample PBS script for InfiniBand is:

#!/bin/bash
#!
#! Sample PBS file
#!
#! Name of job
#PBS -N MPI
#! Number of nodes (in this case 8 nodes with 4 CPUs each)
#! The total number of nodes passed to mpirun will be nodes*ppn
#! Second entry: Total amount of wall-clock time (true time).
#! 02:00:00 indicates 02 hours
#PBS -l walltime=02:00:00
#PBS -l select=8:ncpus=4
#! Mail to user when job terminates or aborts
#PBS -m ae
# If modules are needed by the script, then source modules environment:
. /etc/profile.d/modules.sh
# Add any modules you might require:
module add shared mvapich/gcc pbspro
#! Full path to application + application name
application="<application>"
#! Run options for the application
options="<options>"
#! Work directory
workdir="<work dir>"
###############################################################
### You should not have to change anything below this line ####
###############################################################
#! change the working directory (default is home directory)
cd $workdir
echo Running on host $(hostname)
echo Time is $(date)
echo Directory is $(pwd)
echo PBS job ID is $PBS_JOBID
echo This job runs on the following machines:
echo $(cat $PBS_NODEFILE | uniq)
mpirun_command="mpirun $application $options"
#! Run the parallel MPI executable (nodes*ppn)
echo Running $mpirun_command
eval $mpirun_command

In the preceding script, no machine file is needed, since it is automatically built by the workload manager and passed on to the mpirun parallel job launcher utility. The job is given a unique ID and run in parallel on the nodes based on the resource specification.

7.1.5 Links To PBS Resources

A number of useful links are:

• The PBS Professional documentation pages, accessible from https://community.altair.com/community?id=altair_product_documentation.

• The man pages for qsub, pbs_resources, and pbs_job_attributes, for setting up job scripts.

• The community support pages for PBS Professional, accessible at https://community.altair.com/community?id=altair_community_home. Commercial support can be arranged for PBS Professional using https://www.altair.com/tailored-solutions.

• The OpenPBS community pages, accessible from http://community.openpbs.org/.

7.2 Submitting A Job

7.2.1 Preliminaries: Loading The Modules Environment

To submit a job to the workload management system, the user must ensure that the following environment modules are loaded:

$ module add shared pbspro

Users can pre-load particular environment modules as their default using the “module init*” commands (section 2.3.3).

7.2.2 Using qsub

The command qsub is used to submit jobs to the workload manager system. The command returns a unique job identifier, which is used to query and control the job and to identify output. The usage format of qsub and some useful options are listed here:

USAGE: qsub [<options>] <job script>
Option  Hint   Description
------  ----   -----------
-a      at     run the job at a certain time
-l      list   request certain resource(s)
-q      queue  job is run in this queue
-N      name   name of job
-S      shell  shell to run job under
-j      join   join output and error files

For example, a job script called mpirun.job with all the relevant directives set inside the script, may be submitted as follows:

Example

$ qsub mpirun.job

A job may be submitted to a specific queue testq as follows:

Example

$ qsub -q testq mpirun.job

The man page for qsub describes these and other options. The options correspond to PBS directives in job scripts (section 7.1.1). If a particular item is specified by a qsub option as well as by a PBS directive, then the qsub option takes precedence.

7.2.3 Job Output

By default, the output from the job script <scriptname> goes into the current working directory for PBS.

By default, error output is written to <scriptname>.e<jobid> and the application output is written to <scriptname>.o<jobid>, where <jobid> is a unique number that the workload manager allocates. Specific output and error files can be set using the -o and -e options respectively. The error and output files can usefully be concatenated into one file with the -j oe or -j eo options. More details on this can be found in the qsub man page.

7.2.4 Monitoring The Status Of A Job

To use the commands in this section, the appropriate workload manager module must be loaded:

$ module add pbspro

7.2.4.1 qstat Basics

The main component is qstat, which has several options. In this example, the most frequently used options are discussed.

In PBS, the command “qstat -an” shows what jobs are currently submitted or running on the queuing system. An example output is:

[fred@basecm10 ~]$ qstat -an
basecm10:                                                   Req'd  Req'd   Elap
Job ID          Username Queue    Jobname    SessID NDS TSK Memory Time  S Time
--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----
121.basecm10    noah     workq    pbjob         --    3   6    --  01:00 Q   --
    --
125.basecm10    noah     workq    pbjob       26615   3   3    --  01:00 R   --
   node001/0+node002/0+node003/0

The output shows the Job ID, the user who owns the job, the queue, the job name, the session ID for a running job, the number of nodes requested, the number of CPUs or tasks requested, the time requested (-l walltime), the job state (S) and the elapsed time. In this example, one job is seen to be running (R), and one is still queued (Q). The -n parameter causes nodes that are in use by a running job to display at the end of that line.

Possible job states include:

The command “qstat -q” shows what queues are available. In the following example, there is one job running in the testq queue and 4 are queued.

$ qstat -q
server: master.cm.cluster
Queue            Memory CPU Time Walltime Node  Run Queue Lm  State
---------------- ------ -------- -------- ----  --- ----- --  -----
testq              --      --    23:59:59   --    1     4 --   E R
default            --      --    23:59:59   --    0     0 --   E R
                                               ---- -----
                                                  1     4

7.2.4.2 Viewing Job Details With qstat

With qstat -f the full output of the job is displayed. The output shows what the jobname is, where the error and output files are stored, and various other settings and variables.

$ qstat -f
Job Id: 137.pj-cruncher
    Job_Name = pbjob
    Job_Owner = noah@pj-cruncher.cm.cluster
    resources_used.cpupercent = 0
    resources_used.cput = 00:00:00
    resources_used.mem = 5456kb
    resources_used.ncpus = 3
    resources_used.vmem = 384164kb
    resources_used.walltime = 00:00:11
    job_state = R
    queue = workq
    server = pj-cruncher
    Checkpoint = u
    ctime = Fri Mar 20 18:07:06 2020
    Error_Path = pj-cruncher.cm.cluster:/home/noah/pbjob.e137
    exec_host = node001/0+node002/0+node003/0
    exec_vnode = (node001:ncpus=1)+(node002:ncpus=1)+(node003:ncpus=1)
    Hold_Types = n
    Join_Path = oe
    Keep_Files = n
    Mail_Points = a
    mtime = Fri Mar 20 18:08:52 2020
    Output_Path = pj-cruncher.cm.cluster:/home/noah/pbjob.o137
    Priority = 0
    qtime = Fri Mar 20 18:07:06 2020
    Rerunable = True
    Resource_List.mpiprocs = 24
    Resource_List.ncpus = 3
    Resource_List.nodect = 3
    Resource_List.place = free
    Resource_List.select = 3:ncpus=1:mpiprocs=8
    Resource_List.walltime = 01:00:00
    stime = Fri Mar 20 18:08:42 2020
    session_id = 28107
    jobdir = /home/noah
    substate = 42
    Variable_List = PBS_O_HOME=/home/noah,PBS_O_LANG=en_US.UTF-8,
        PBS_O_LOGNAME=noah,
        PBS_O_PATH=/cm/shared/apps/openpbs/20.0.1/unsupported/fw/bin:/cm/sha
        red/apps/openpbs/20.0.1/unsupported:/cm/shared/apps/openpbs/20.0.1/
        sbin:/cm/shared/apps/openpbs/20.0.1/bin:/cm/shared/apps/mpich/ge/gcc/
        64/3.3.2/bin:/cm/local/apps/gcc/9.2.0/bin:/cm/local/apps/environment-mo
        dules/4.4.0//bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbi
        n:/usr/sbin:/cm/local/apps/environment-modules/4.4.0/bin:/home/noah/.lo
        cal/bin:/home/noah/bin,PBS_O_MAIL=/var/spool/mail/noah,
        PBS_O_SHELL=/bin/bash,PBS_O_WORKDIR=/home/noah,PBS_O_SYSTEM=Linux,
        PBS_O_QUEUE=workq,PBS_O_HOST=pj-cruncher.cm.cluster
    comment = Job run at Fri Mar 20 at 18:08 on (node001:ncpus=1)+(node002:ncpu
        s=1)+(node003:ncpus=1)
    etime = Fri Mar 20 18:07:06 2020
    run_count = 1
    Submit_arguments = pbjob
    project = _pbs_project_default

7.2.5 Deleting A Job

An already submitted job can be deleted using the qdel command:

$ qdel <jobid>

Multiple space-separated job IDs can be specified.

7.2.6 Nodes According To PBS

The nodes that the workload manager knows about can be viewed using the pbsnodes command.

The following output is from a cluster made up of nodes with 4 physical cores, as indicated by the value of 4 for pcpus. If the node is available to run scripts, then its state is free or time-shared. When a node is used exclusively (section 8.5.2) by one script, the state is job-exclusive.

For PBS Professional the display resembles (some output elided):

[noah@pj-cruncher ~]$ pbsnodes -a
node001
     Mom = node001.cm.cluster
     ntype = PBS
     state = free
     pcpus = 4
     resources_available.arch = linux
     resources_available.host = node001
     resources_available.mem = 4044784kb
     ...
     queue = workq
     resv_enable = True
     sharing = default_shared
     ...
node002
     Mom = node002.cm.cluster
     ntype = PBS
     state = free
     pcpus = 4
     resources_available.arch = linux
     resources_available.host = node002
     resources_available.mem = 4044784kb
     ...
     queue = workq
     resv_enable = True
     sharing = default_shared
     last_state_change_time = Fri Mar 20 18:09:47 2020
     last_used_time = Fri Mar 20 18:09:47 2020
     ...