> This page is for version 10. > For other versions, use one of these documentation indexes: > - 11 (default): https://docs.nvidia.com/base-command-manager/product-manuals/11/llms.txt > - 10: https://docs.nvidia.com/base-command-manager/product-manuals/10/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/base-command-manager/product-manuals/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/base-command-manager/product-manuals/_mcp/server. # Workload Managers Quick Reference ## F.1 Slurm Slurm is a GPL-licensed workload management system and developed largely at Lawrence Livermore National Laboratory. The name was originally an acronym for Simple Linux Utility for Resource Management, but the acronym is deprecated because it no longer does justice to the advanced capabilities of Slurm. The Slurm service and outputs are normally handled using the Base View or `cmsh` front end tools for CMDaemon (section [7.4](/admin/workloadmanagement#74-enabling-disabling-and-monitoring-workload-managers)). From the command line, direct Slurm commands that may sometimes come in useful include the following: * `sacct`: used to report job or job step accounting information about active or completed jobs. **Example** ```bash # sacct -j 43 -o jobid,AllocCPUs,NCPUS,NNodes,NTasks,ReqCPUs JobID AllocCPUS NCPUS NNodes NTasks ReqCPUS ------------ ---------- ---------- -------- -------- -------- 43 1 1 1 1 ``` * `salloc`: used to allocate resources for a job in real time. Typically this is used to allocate resources and spawn a shell. The shell is then used to execute srun commands to launch parallel tasks. * `sattach` used to attach standard input, output, and error plus signal capabilities to a currently running job or job step. One can attach to and detach from jobs multiple times. * `sbatch`: used to submit a job script for later execution. The script typically contains one or more srun commands to launch parallel tasks. * `sbcast`: used to transfer a file from local disk to local disk on the nodes allocated to a job. This can be used to effectively use diskless compute nodes or provide improved performance relative to a shared filesystem. * `scancel`: used to cancel a pending or running job or job step. It can also be used to send an arbitrary signal to all processes associated with a running job or job step. * `scontrol`: the administrative tool used to view and/or modify Slurm state. Note that many scontrol commands can only be executed as user root. **Example** ```bash [fred@basecm10 ~]$ scontrol show nodes NodeName=basecm10 Arch=x86_64 CoresPerSocket=1 CPUAlloc=0 CPUErr=0 CPUTot=1 CPULoad=0.05 Features=(null) ... ``` If a node, for example `node001`, is stuck in a `CG` state (“completing”), and rebooting it is not feasible, then the following may clear it in some cases: **Example** ```bash [fred@basecm10 ~]$ scontrol update nodename=node001 state=down reason=hung [fred@basecm10 ~]$ scontrol update nodename=node001 state=resume ``` * `sinfo`: reports the state of partitions and nodes managed by Slurm. It has a wide variety of filtering, sorting, and formatting options. **Example** ```bash basecm10:~ # sinfo -o "%9P %.5a %.10l %.6D %.6t %C %N" PARTITION AVAIL TIMELIMIT NODES STATE CPUS(A/I/O/T) NODELIST defq* up infinite 1 alloc 1/0/0/1 basecm10 ``` * `smap`: reports state information for jobs, partitions, and nodes managed by Slurm, but graphically displays the information to reflect network topology. * `squeue`: reports the state of jobs or job steps. It has a wide variety of filtering, sorting, and formatting options. By default, it reports the running jobs in priority order and then the pending jobs in priority order. **Example** ```bash basecm10:~ # squeue -o "%.18i %.9P %.8j %.8u %.2t %.10M %.6D %C %R" JOBID PARTITION NAME USER ST TIME NODES CPUS NODELIST(REASON) 43 defq bash fred R 16:22 1 1 basecm10 ... ``` * `srun`: used to submit a job for execution or initiate job steps in real time. srun has a wide variety of options to specify resource requirements, including: minimum and maximum node count, processor count, specific nodes to use or not use, and specific node characteristics (so much memory, disk space, certain required features, etc.). A job can contain multiple job steps executing sequentially or in parallel on independent or shared nodes within the job’s node allocation. **Auto Scaler dynamic node issues with `srun`:** [A concern about `srun` (`https://bugs.schedmd.com/show_bug.cgi?id=1333`)](https://bugs.schedmd.com/show_bug.cgi?id=1333) at the time of writing (January 2023) is the following: After an `srun` job has been queued, and a new node is added, the `slurm.conf` file is not read again. This means that the new node resource is not seen by jobs using `srun`. Thus, with `srun` jobs, nodes launched dynamically by Auto Scaler remain unused, and can fail. Workarounds are to use `salloc` or `sbatch` instead of `srun`. * `smap`: reports state information for jobs, partitions, and nodes managed by Slurm, but graphically displays the information to reflect network topology. * `strigger`: used to set, get or view event triggers. Event triggers include things such as nodes going down or jobs approaching their time limit. * `sview`: a graphical user interface to get and update state information for jobs, partitions, and nodes managed by Slurm. There are man pages for these commands. Full documentation on Slurm is available online at: [http://slurm.schedmd.com/documentation.html](http://slurm.schedmd.com/documentation.html). ## F.2 Altair Grid Engine Altair Grid Engine (GE) is a workload management system that was originally derived from Sun Grid Engine. GE services should be handled using CMDaemon, as explained in section [7.4](/admin/workloadmanagement#74-enabling-disabling-and-monitoring-workload-managers). Managing queues and jobs can be done via the `cmsh` or Base View front ends, but can also be carried out via the GE tools. * The `sge_qmaster` daemon on the head node can be started, stopped, or viewed using `service sgemaster` `start`|`stop`|`status`, or alternatively handled at a lower, more fine-grained, level using the `-k` (kill), `-s` (show) and `-a` (activate) options of `qconf`. Further details on these options can be found in the `qconf` help text, or `man qconf(1)`. For example, help on the `-k` options can be found by running: `qconf -help | grep '\-k'` or running: `man qconf(1)` and scrolling down to the interesting part by searching for `shutdown`). * The `sge_execd` execution daemon running on each compute node accepts, manages, and returns the results of the jobs on the compute nodes. The daemon can be managed via `service sgeexecd` `start`|`stop`|`status`, or alternatively (de)registered from `qmaster` via `qconf` options. * To see why a queue is in an error state, `qstat -explain E` shows the reason. Further details on these options can be found in the `qconf` help text, or `man qstat(1)`. Queues in an error state can be cleared with a `qmod -cq `. GE can be configured and managed generally with the command line utility `qconf`, which is what most administrators become familiar with. GE also provides a GUI alternative, `qmon`. GE commands are listed below. The details of these are in the `man` page of the command and the GE documentation. * `qalter`: modify existing batch jobs * `qacct`: show usage information from accounting data * `qconf`: configure GE * `qdel`: delete batch jobs * `qhold`: place hold on batch jobs * `qhost`: display compute node queues, states, jobs * `qlogin`: start `login`-based interactive session with a node * `qmake`: distributed, parallel make utility * `qmod`: suspend/enable queues and jobs * `qmon`: configure GE with an X11 GUI interface * `qping`: check `sge_qmaster` and `sge_execd` status * `qquota`: list resource quotas * `qresub`: create new jobs by copying existing jobs * `qrdel`: cancel advance reservations * `qrls`: release batch jobs from a held state * `qrsh`: start `rsh`-based interactive session with node * `qrstat`: show status of advance reservations * `qrsub`: submit advanced reservation * `qselect`: select queues based on argument values * `qsh`: start `sh` interactive session with a node * `qstat`: show status of batch jobs and queues * `qsub`: submit new jobs (related: qalter, qresub) ## F.3 PBS Professional The following commands can be used in PBS Professional to view queues, jobs, and server status: ```bash qstat query queue status qstat -a show only queued or running jobs for a destination, or all states for a job ID qstat -r show only running or suspended jobs for a destination, or all states for a job ID qstat -q show queue status for destinations qstat -rn only running or suspended jobs, with list of allocated nodes (`exec_host` string) qstat -i information on queued, held, waiting jobs is given for specified destination. Inform- ation about the job is given if a job ID is specified, regardless of the job status qstat -B display server status for the specified servers qstat -u show jobs for a user for the specified destination. Status information for the job is displayed for a specified job ID. ``` Other useful commands are: ```bash tracejob show what happened today to tracejob -n search last days for qmgr administrator interface to batch system (man qmgr.8B for more details) qterm terminates PBS server (but BCM starts pbs_server again) pbsnodes query status of compute node pbsnodes -a query status of all compute nodes ``` The commands of PBS Professional are documented in the man pages, and also in the extensive documentation available via [https://community.altair.com/community?id=altair_product_documentation](https://community.altair.com/community?id=altair_product_documentation).