> This page is for version 10. > For other versions, use one of these documentation indexes: > - 11 (default): https://docs.nvidia.com/base-command-manager/product-manuals/11/llms.txt > - 10: https://docs.nvidia.com/base-command-manager/product-manuals/10/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/base-command-manager/product-manuals/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/base-command-manager/product-manuals/_mcp/server. # Altair Grid Engine Altair Grid Engine (GE) is a workload management and job scheduling system. It is a further development by Univa of Sun Grid Engine (SGE). SGE was developed by Sun Microsystems before 2011, and was at one time open source. Univa Grid Engine (UGE) was a proprietary fork of it. After Univa was acquired by Altair, UGE engine is now called Altair Grid Engine (GE). GE has both a graphical interface and command line tools for submitting, monitoring, modifying and deleting jobs. GE uses *job scripts* to submit and execute jobs. Various settings can be put in the job script, such as number of processors, resource usage and application-specific variables. The steps for running a job through GE are to: * Create a job script * Select the directives to use * Add the scripts and applications and runtime parameters * Submit it to the workload management system ## 6.1 Writing A Job Script A binary cannot be submitted directly to GE—a job script is needed for that. A job script can contain various settings and variables to go with the application. A job script format looks like: ```bash #!/bin/bash #$ Script options # Optional script directives shell commands # Optional shell commands application # Application itself ``` ### 6.1.1 Directives It is possible to specify options (‘directives’) to GE by using “`#$`” in the script. The difference in the meaning of lines that start with the “`#`” character in the job script file should be noted:
**Line Starts With** **Treated As**
`#` Comment in shell and GE
`#$` Comment in shell, directive in GE
`# $` Comment in shell and GE
### 6.1.2 GE Environment Variables Available environment variables for GE can be seen in the man page for `qsub`. Some of these are called pseudo environment variables: ```bash $HOME - Home directory on execution machine $USER - User ID of job owner $JOB_ID - Current job ID $JOB_NAME - Current job name; (like the -N option in qsub, qsh, qrsh, qlogin and qalter) $HOSTNAME - Name of the execution host $TASK_ID - Array job task index number ``` There are also about 30 environment variables begining with the prefix `SGE_`. ### 6.1.3 Job Script Options Options can be set in the job script file using this line format for each option: ```bash #$ {option} {parameter} ``` Available options and their descriptions can be seen with the output of `qsub -help`:
*Table [6.1.3](#613-job-script-options): GE Job Script Options*
**Option and parameter** **Description**
`-a date_time` request a start time
`-adds list_attr_name attr_key attr_value` add additional sublist parameter
`-ac context_list` add context variables
`-ar ar_id` bind job to advance reservation
`-A account_string` account string in accounting record
`-b y[es]|n[o]` handle command as binary
`-binding [env|pe|set] exp|lin|str` binds job to processor cores
`-c ckpt_selector` define type of checkpointing for job
`-ckpt ckpt-name` request checkpoint method
`-clear` skip previous definitions for job
`-clearp attr_name` clear/delete a parameter
`-clears list_attr_name attr_key` clear/delete sublist parameter
`-cwd` use current working directory
`-C directive_prefix` define command prefix for job script
`-dc simple_context_list` delete context variable(s)
`-dl date_time` request a deadline initiation time
`-e path_list` specify standard error stream path(s)
`-h` place user hold on job
`-hard` consider following requests "hard"
`-help` print this help
`-hold_jid job_identifier_list` define jobnet interdependencies
`-hold_jid_ad job_identifier_list` define jobnet array interdependencies
`-i file_list` specify standard input stream file(s)
`-j y[es]|n[o]` merge stdout and stderr stream of job
`-jc jc_name` derive job from the specified job class
`-js job_share` share tree or functional job share
`-jsv jsv_url` job submission verification script to be used
`-l resource_list` request the given resources
`-m mail_options` define mail notification events
`-mods list_attr_name attr_key attr_value` modify sublist parameter
`-masterl resource_list` request the given resources for the master task of a parallel job
`-masterq wc_queue_list` bind master task to queue(s)
`-mbind mbind_string` specifies a memory binding strategy (lx-amd64 only)
`-notify` notify job before killing/suspending it
`-now y[es]|n[o]` start job immediately or not at all
`-M mail_list` notify these e-mail addresses
`-N name` specify job name
`-o path_list` specify standard output stream path(s)
`-P project_name` set job’s project
`-p priority` define job’s relative priority
`-par pe-allocation-rule` request overwriting pe allocation rule
`-pe pe-name slot_range` request slot range for parallel jobs
`-pty y[es]|n[o]` start job in a pty
`-q wc_queue_list` bind job to queue(s)
`-R y[es]|n[o]` reservation desired
`-r y[es]|n[o]` define job as (not) restartable
`-rou reporting variable list` write online usage of job to the reporting database
`-rdi y[es]|n[o]` request dispatching information
`-sc context_list` set job context (replaces old context)
`-shell y[es]|n[o]` start command with or without wrapping -c
`-si session_id` execute GE requests as part of the specified session
`-soft` consider following requests as soft
`-sync sync_options` define sync notification events
`-S path_list` command interpreter to be used
`-t task_id_range` create a job-array with these tasks
`-tc max_running_tasks` throttle the number of concurrent tasks
`-tcon y[es]|n[o]` run all tasks concurrently
`-terse` tersed output, print only the job-id
`-umask mask` set umask of the job
`-v variable_list` export these environment variables
`-verify` do not submit, just verify
`-V` export all environment variables
`-w e|w|n|v|p` verify mode (error\
`-wd working_directory` use working_directory
`-xdv docker_volumes` specify docker volume(s)
`-xd docker_options` specify docker run options
`-xd --help` show a list of available docker run options
`-xd_run_as_image_user y[es]|n[o]` start autostart Docker job as the user defined in the Docker image
`-@ file` read commandline input from file
More detail on these options and their use is found in the man page for `qsub`. ### 6.1.4 The Executable Line In a job script, the executable line is launched with the job launcher command after the directives lines have been dealt with, and after any other shell commands have been carried out to set up the execution environment. #### 6.1.4.1 Using `mpirun` In The Executable Line The `mpirun` job-launcher command is used for executables compiled with MPI libraries. Executables that have not been compiled with MPI libraries, or which are launched without any specified number of nodes, run on a single free node chosen by the workload manager. The executable line to run a program `myprog` that has been compiled with MPI libraries is run by placing the job-launcher command `mpirun` before it as follows: ```bash [fred@basecm10 ~]$ mpirun myprog ``` ### 6.1.5 Job Script Examples Some job script examples are given in this section. Each job script can use a number of variables and directives. #### 6.1.5.1 Single Node Example Script An example script for GE: ```bash [fred@basecm10 ~]$ cat application #!/bin/sh # Give the job a name #$ -N sleep # Use this interpreter #$ -S /bin/sh # Make sure that the .e and .o file arrive in the # working directory #$ -cwd #Merge the standard out and standard error to one file #$ -j y sleep 60 echo Now it is: `date` ``` #### 6.1.5.2 Parallel Example Script For parallel jobs the `-pe` (parallel environment) option must be assigned to the script. Depending on the interconnect, there may be a choice between a number of parallel environments such as MPICH (Ethernet) or MVAPICH2 (InfiniBand). ```bash #!/bin/sh # # Your job name #$ -N My_Job # # Use current working directory #$ -cwd # # Join stdout and stderr #$ -j y # # pe (Parallel environment) request. Set your number of requested slots here. # qconf -spl shows current PE list #$ -pe mpich2 2 # # Run job through bash shell #$ -S /bin/bash # If modules are needed, source modules environment: . /etc/profile.d/modules.sh # Add any modules you might require: module add gcc openmpi/gcc/64/1.10.7 # The following output will show in the output file. Used for debugging. echo "Got $NSLOTS processors." echo "Machines:" cat machines # Use MPIRUN to run the application. application must be compiled with right MPI library. # That is openmpi here, compiled with: # mpicc hello.c -o hello mpirun ./hello ``` The number of available slots can be set by the administrator to an arbitrary value. However, it is typically set so that it matches the number of cores in the cluster, for efficiency reasons. More slots can be set than cores, but most administrators prefer not to do that. In a job script, the user can request slots from the available slots. Requesting multiple slots therefore typically means requesting multiple cores. In the case of an environment that is not set up as a parallel environment, the request for slots is done with the `-np` option. For jobs that run in a parallel environment, the `-pe` option is used. Mixed jobs, running in both non-MPI and parallel environments are also possible if the administrator has allowed it in the complex attribute slots settings. Whether the request is granted is decided by the workload manager policy set by the administrator. If the request exceeds the number of available slots, then the request is not granted. If the administrator has configured the cluster to use cloud computing with Auto Scale (`cm-scale`) ([Chapter 8 of the *Administrator Manual*)](/admin/cm-scale), then the total number of slots available to a cluster changes over time automatically, as nodes are started up and stopped dynamically. ## 6.2 Submitting A Job The GE module `uge` must be loaded first so that UGE commands can be accessed: ```bash fred@basecm10 ~]$ module add shared uge ``` With GE a job can be submitted with `qsub`. The `qsub` command has the following syntax: ```bash qsub [ options ] [ jobscript | -- [ jobscript args ]] ``` After completion (either successful or not), output is put in the user’s current directory, appended with the job number which is assigned by GE to the job nmae. For example, for a jobscript with the job option `#$ -N myapp`, the job name is set to `myapp`. When the job completes, then by default there is an error and there is an output file. The names they take are of the form: ```bash myapp.e myapp.o ``` If the `-j y` job script option is set, then the error file is combined with the output file into a single `.o` \<*`job ID`*> file. ### 6.2.1 Submitting To A Specific Queue Some clusters have specific queues for jobs which run are configured to house a certain type of job: long and short duration jobs, high resource jobs, or a queue for a specific type of node. To see which queues are available on the cluster the `qstat` command can be used: ```bash fred@basecm10 ~]$ qstat -g c CLUSTER QUEUE CQLOAD USED RES AVAIL TOTAL aoACDS cdsuE ----------------------------------------------------------------- long.q 0.01 0 0 144 288 0 144 default.q 0.01 0 0 144 288 0 144 ``` The job is then submitted, for example to the `long.q` queue: ```bash fred@basecm10 ~]$ qsub -q long.q sleeper.sh ``` ### 6.2.2 Queue Assignment Required For Auto Scale (`cm-scale`) If `cm-scale` is used with GE, then jobs must be assigned to a queue by default, or `cm-scale` ignores the job. If the cluster administrator has not configured GE to assign a queue to jobs by default, then the user can assign a queue to the job with the `-q` option in order to have `cm-scale` consider the job. ## 6.3 Monitoring A Job The job status can be viewed with `qstat`. In this example the `sleeper.sh` script has been submitted. Using `qstat` without options will only display a list of jobs, with no queue status options: ```bash fred@basecm10 ~]$ qstat job-ID prior name user state submit/start at queue jclass slots ja-task-ID --------------------------------------------------------------------------------------- 80 0.55500 myapp fred r 03/12/2020 23:02:20 all.q@node003 2 81 0.55500 myapp fred qw 03/12/2020 23:02:06 2 82 0.55500 myapp fred qw 03/12/2020 23:02:07 2 83 0.55500 myapp fred qw 03/12/2020 23:02:07 2 ``` More details are visible when using the `-f` (for full) option. Some of the details that can be seen are: * The queue-type `qtype`, which can take a state of Batch (B), Interactive (I), or Parallel (P). * The `resv/used/tot` column is the count of reserved/used/total free slots in the queue. * The `states` column is the state of the queue. ```bash fred@basecm10 ~]$ qstat -f queuename qtype resv/used/tot. np_load arch states --------------------------------------------------------------------------------- all.q@node001.cm.cluster BIP 0/1/1 0.01 lx-amd64 88 0.55500 myapp fred r 03/12/2020 23:11:35 1 --------------------------------------------------------------------------------- all.q@node002.cm.cluster BIP 0/0/1 0.01 lx-amd64 --------------------------------------------------------------------------------- all.q@node003.cm.cluster BIP 0/1/1 0.00 lx-amd64 88 0.55500 myapp fred r 03/12/2020 23:11:35 1 ############################################################################### - PENDING JOBS - PENDING JOBS - PENDING JOBS - PENDING JOBS - PENDING JOBS ############################################################################### 89 0.55500 myapp fred qw 03/12/2020 23:10:56 2 90 0.55500 myapp fred qw 03/12/2020 23:10:56 2 91 0.55500 myapp fred qw 03/12/2020 23:10:56 2 ``` The man page for `qstat` gives details on the job states and the queue states. By default the `qstat` command shows only jobs belonging to the current user, i.e. the command is executed with the option `-u $user`. To see jobs from other users too, the following format is used: ```bash fred@basecm10 ~]$ qstat -u "*" ``` ## 6.4 Deleting A Job A job can be deleted in GE with the following command ```bash fred@basecm10 ~]$ qdel ``` The job-id is the number assigned by GE when the job is submitted using `qsub`. Only jobs belonging to the logged-in user can be deleted. Using `qdel` will delete a user’s job regardless of whether the job is running or in the queue.