Altair Grid Engine | NVIDIA Base Command Manager

Altair Grid Engine (GE) is a workload management and job scheduling system. It is a further development by Univa of Sun Grid Engine (SGE). SGE was developed by Sun Microsystems before 2011, and was at one time open source. Univa Grid Engine (UGE) was a proprietary fork of it. After Univa was acquired by Altair, UGE engine is now called Altair Grid Engine (GE).

GE has both a graphical interface and command line tools for submitting, monitoring, modifying and deleting jobs.

GE uses job scripts to submit and execute jobs. Various settings can be put in the job script, such as number of processors, resource usage and application-specific variables.

The steps for running a job through GE are to:

Create a job script
Select the directives to use
Add the scripts and applications and runtime parameters
Submit it to the workload management system

6.1 Writing A Job Script

A binary cannot be submitted directly to GE—a job script is needed for that. A job script can contain various settings and variables to go with the application. A job script format looks like:

#!/bin/bash
#$ Script options  # Optional script directives
shell commands     # Optional shell commands
application        # Application itself

6.1.1 Directives

It is possible to specify options (‘directives’) to GE by using “#$” in the script. The difference in the meaning of lines that start with the “#” character in the job script file should be noted:

Line Starts With	Treated As
`#`	Comment in shell and GE
`#$`	Comment in shell, directive in GE
`# $`	Comment in shell and GE

6.1.2 GE Environment Variables

Available environment variables for GE can be seen in the man page for qsub. Some of these are called pseudo environment variables:

$HOME - Home directory on execution machine
$USER - User ID of job owner
$JOB_ID - Current job ID
$JOB_NAME - Current job name; (like the -N option in qsub, qsh, qrsh, qlogin and qalter)
$HOSTNAME - Name of the execution host
$TASK_ID - Array job task index number

There are also about 30 environment variables begining with the prefix SGE_.

6.1.3 Job Script Options

Options can be set in the job script file using this line format for each option:

#$ {option} {parameter}

Available options and their descriptions can be seen with the output of qsub -help:

Table 6.1.3: GE Job Script Options
Option and parameter	Description
`-a date_time`	request a start time
`-adds list_attr_name attr_key attr_value`	add additional sublist parameter
`-ac context_list`	add context variables
`-ar ar_id`	bind job to advance reservation
`-A account_string`	account string in accounting record
`-b y[es]\|n[o]`	handle command as binary
`-binding [env\|pe\|set] exp\|lin\|str`	binds job to processor cores
`-c ckpt_selector`	define type of checkpointing for job
`-ckpt ckpt-name`	request checkpoint method
`-clear`	skip previous definitions for job
`-clearp attr_name`	clear/delete a parameter
`-clears list_attr_name attr_key`	clear/delete sublist parameter
`-cwd`	use current working directory
`-C directive_prefix`	define command prefix for job script
`-dc simple_context_list`	delete context variable(s)
`-dl date_time`	request a deadline initiation time
`-e path_list`	specify standard error stream path(s)
`-h`	place user hold on job
`-hard`	consider following requests “hard”
`-help`	print this help
`-hold_jid job_identifier_list`	define jobnet interdependencies
`-hold_jid_ad job_identifier_list`	define jobnet array interdependencies
`-i file_list`	specify standard input stream file(s)
`-j y[es]\|n[o]`	merge stdout and stderr stream of job
`-jc jc_name`	derive job from the specified job class
`-js job_share`	share tree or functional job share
`-jsv jsv_url`	job submission verification script to be used
`-l resource_list`	request the given resources
`-m mail_options`	define mail notification events
`-mods list_attr_name attr_key attr_value`	modify sublist parameter
`-masterl resource_list`	request the given resources for the master task of a parallel job
`-masterq wc_queue_list`	bind master task to queue(s)
`-mbind mbind_string`	specifies a memory binding strategy (lx-amd64 only)
`-notify`	notify job before killing/suspending it
`-now y[es]\|n[o]`	start job immediately or not at all
`-M mail_list`	notify these e-mail addresses
`-N name`	specify job name
`-o path_list`	specify standard output stream path(s)
`-P project_name`	set job’s project
`-p priority`	define job’s relative priority
`-par pe-allocation-rule`	request overwriting pe allocation rule
`-pe pe-name slot_range`	request slot range for parallel jobs
`-pty y[es]\|n[o]`	start job in a pty
`-q wc_queue_list`	bind job to queue(s)
`-R y[es]\|n[o]`	reservation desired
`-r y[es]\|n[o]`	define job as (not) restartable
`-rou reporting variable list`	write online usage of job to the reporting database
`-rdi y[es]\|n[o]`	request dispatching information
`-sc context_list`	set job context (replaces old context)
`-shell y[es]\|n[o]`	start command with or without wrapping <loginshell> -c
`-si session_id`	execute GE requests as part of the specified session
`-soft`	consider following requests as soft
`-sync sync_options`	define sync notification events
`-S path_list`	command interpreter to be used
`-t task_id_range`	create a job-array with these tasks
`-tc max_running_tasks`	throttle the number of concurrent tasks
`-tcon y[es]\|n[o]`	run all tasks concurrently
`-terse`	tersed output, print only the job-id
`-umask mask`	set umask of the job
`-v variable_list`	export these environment variables
`-verify`	do not submit, just verify
`-V`	export all environment variables
`-w e\|w\|n\|v\|p`	verify mode (error\
`-wd working_directory`	use working_directory
`-xdv docker_volumes`	specify docker volume(s)
`-xd docker_options`	specify docker run options
`-xd —help`	show a list of available docker run options
`-xd_run_as_image_user y[es]\|n[o]`	start autostart Docker job as the user defined in the Docker image
`-@ file`	read commandline input from file

More detail on these options and their use is found in the man page for qsub.

6.1.4 The Executable Line

In a job script, the executable line is launched with the job launcher command after the directives lines have been dealt with, and after any other shell commands have been carried out to set up the execution environment.

6.1.4.1 Using `mpirun` In The Executable Line

The mpirun job-launcher command is used for executables compiled with MPI libraries. Executables that have not been compiled with MPI libraries, or which are launched without any specified number of nodes, run on a single free node chosen by the workload manager.

The executable line to run a program myprog that has been compiled with MPI libraries is run by placing the job-launcher command mpirun before it as follows:

[fred@basecm10 ~]$ mpirun myprog

6.1.5 Job Script Examples

Some job script examples are given in this section. Each job script can use a number of variables and directives.

6.1.5.1 Single Node Example Script

An example script for GE:

[fred@basecm10 ~]$ cat application
#!/bin/sh
# Give the job a name
#$ -N sleep
# Use this interpreter
#$ -S /bin/sh
# Make sure that the .e and .o file arrive in the
# working directory
#$ -cwd
#Merge the standard out and standard error to one file
#$ -j y
sleep 60
echo Now it is: `date`

6.1.5.2 Parallel Example Script

For parallel jobs the -pe (parallel environment) option must be assigned to the script. Depending on the interconnect, there may be a choice between a number of parallel environments such as MPICH (Ethernet) or MVAPICH2 (InfiniBand).

#!/bin/sh
#
# Your job name
#$ -N My_Job
#
# Use current working directory
#$ -cwd
#
# Join stdout and stderr
#$ -j y
#
# pe (Parallel environment) request. Set your number of requested slots here.
# qconf -spl shows current PE list
#$ -pe mpich2 2
#
# Run job through bash shell
#$ -S /bin/bash
# If modules are needed, source modules environment:
. /etc/profile.d/modules.sh
# Add any modules you might require:
module add gcc openmpi/gcc/64/1.10.7
# The following output will show in the output file. Used for debugging.
echo "Got $NSLOTS processors."
echo "Machines:"
cat machines
# Use MPIRUN to run the application. application must be compiled with right MPI library.
# That is openmpi here, compiled with:
# mpicc hello.c -o hello
mpirun ./hello

The number of available slots can be set by the administrator to an arbitrary value. However, it is typically set so that it matches the number of cores in the cluster, for efficiency reasons. More slots can be set than cores, but most administrators prefer not to do that.

In a job script, the user can request slots from the available slots. Requesting multiple slots therefore typically means requesting multiple cores. In the case of an environment that is not set up as a parallel environment, the request for slots is done with the -np option. For jobs that run in a parallel environment, the -pe option is used. Mixed jobs, running in both non-MPI and parallel environments are also possible if the administrator has allowed it in the complex attribute slots settings.

Whether the request is granted is decided by the workload manager policy set by the administrator. If the request exceeds the number of available slots, then the request is not granted.

If the administrator has configured the cluster to use cloud computing with Auto Scale (cm-scale) (Chapter 8 of the Administrator Manual), then the total number of slots available to a cluster changes over time automatically, as nodes are started up and stopped dynamically.

6.2 Submitting A Job

The GE module uge must be loaded first so that UGE commands can be accessed:

fred@basecm10 ~]$ module add shared uge

With GE a job can be submitted with qsub. The qsub command has the following syntax:

  qsub [ options ] [ jobscript | -- [ jobscript args ]]

After completion (either successful or not), output is put in the user’s current directory, appended with the job number which is assigned by GE to the job nmae.

For example, for a jobscript with the job option #$ -N myapp, the job name is set to myapp. When the job completes, then by default there is an error and there is an output file. The names they take are of the form:

myapp.e<job ID>
myapp.o<job ID>

If the -j y job script option is set, then the error file is combined with the output file into a single .o <job ID> file.

6.2.1 Submitting To A Specific Queue

Some clusters have specific queues for jobs which run are configured to house a certain type of job: long and short duration jobs, high resource jobs, or a queue for a specific type of node.

To see which queues are available on the cluster the qstat command can be used:

    fred@basecm10 ~]$ qstat -g c
    CLUSTER QUEUE   CQLOAD   USED    RES  AVAIL  TOTAL aoACDS  cdsuE
    -----------------------------------------------------------------
    long.q          0.01      0      0    144    288      0    144
    default.q       0.01      0      0    144    288      0    144

The job is then submitted, for example to the long.q queue:

    fred@basecm10 ~]$ qsub -q long.q sleeper.sh

6.2.2 Queue Assignment Required For Auto Scale (`cm-scale`)

If cm-scale is used with GE, then jobs must be assigned to a queue by default, or cm-scale ignores the job. If the cluster administrator has not configured GE to assign a queue to jobs by default, then the user can assign a queue to the job with the -q option in order to have cm-scale consider the job.

6.3 Monitoring A Job

The job status can be viewed with qstat. In this example the sleeper.sh script has been submitted. Using qstat without options will only display a list of jobs, with no queue status options:

    fred@basecm10 ~]$ qstat
    job-ID  prior   name   user  state submit/start at      queue jclass   slots ja-task-ID
    ---------------------------------------------------------------------------------------
        80  0.55500 myapp  fred  r     03/12/2020 23:02:20  all.q@node003  2
        81  0.55500 myapp  fred  qw    03/12/2020 23:02:06  2
        82  0.55500 myapp  fred  qw    03/12/2020 23:02:07  2
        83  0.55500 myapp  fred  qw    03/12/2020 23:02:07  2

More details are visible when using the -f (for full) option. Some of the details that can be seen are:

The queue-type qtype, which can take a state of Batch (B), Interactive (I), or Parallel (P).
The resv/used/tot column is the count of reserved/used/total free slots in the queue.
The states column is the state of the queue.

fred@basecm10 ~]$ qstat -f
queuename                      qtype resv/used/tot. np_load  arch          states
---------------------------------------------------------------------------------
all.q@node001.cm.cluster       BIP   0/1/1          0.01     lx-amd64
        88 0.55500 myapp      fred         r     03/12/2020 23:11:35     1
---------------------------------------------------------------------------------
all.q@node002.cm.cluster       BIP   0/0/1          0.01     lx-amd64
---------------------------------------------------------------------------------
all.q@node003.cm.cluster       BIP   0/1/1          0.00     lx-amd64
        88 0.55500 myapp      fred         r     03/12/2020 23:11:35     1
###############################################################################
 - PENDING JOBS - PENDING JOBS - PENDING JOBS - PENDING JOBS - PENDING JOBS
###############################################################################
        89 0.55500 myapp      fred         qw    03/12/2020 23:10:56     2
        90 0.55500 myapp      fred         qw    03/12/2020 23:10:56     2
        91 0.55500 myapp      fred         qw    03/12/2020 23:10:56     2

The man page for qstat gives details on the job states and the queue states.

By default the qstat command shows only jobs belonging to the current user, i.e. the command is executed with the option -u $user. To see jobs from other users too, the following format is used:

fred@basecm10 ~]$ qstat -u "*"

6.4 Deleting A Job

A job can be deleted in GE with the following command

fred@basecm10 ~]$ qdel <jobid>

The job-id is the number assigned by GE when the job is submitted using qsub. Only jobs belonging to the logged-in user can be deleted. Using qdel will delete a user’s job regardless of whether the job is running or in the queue.