Workload Management

4.1 What Is A Workload Manager?

A workload management system (also known as a queuing system, job scheduler or batch submission system) manages the available resources such as CPUs, GPUs, and memory for jobs submitted to the system by users.

Jobs are submitted by the users using job scripts. Job scripts are constructed by users and include requests for resources. How resources are allocated depends upon policies that the system administrator sets up for the workload manager.

4.2 Why Use A Workload Manager?

Workload managers are used so that users do not manually have to keep track of node usage in a cluster in order to plan efficient and fair use of cluster resources.

Users may still perhaps run jobs on the compute nodes outside of the workload manager, if that is administratively permitted. However, running jobs outside a workload manager tends to eventually lead to an abuse of the cluster resources as more people use the cluster. This leads to inefficient use of available resources. It is therefore usually forbidden as a policy by the system administrator on production clusters.

4.3 How Does A Workload Manager Function?

A workload manager uses policies to ensure that the resources of a cluster are used efficiently, and must therefore track cluster resources and jobs. A workload manager is therefore generally able to:

• Monitor:

  • the node status (up, down)

  • all available resources (available cores, memory on the nodes)

  • the state of jobs (queued, on hold, deleted, failed, completed)

• Modify:

  • the status of jobs (freeze/hold the job, resume the job, delete the job)

  • the priority and execution order for jobs

  • the run status of a job. For example, by adding checkpoints to freeze a job.

  • (optional) how related tasks in a job are handled according to their resource requirements. For example, a job with two tasks may have a greater need for disk I/O resources for the first task, and a greater need for CPU resources during the second task.

Some workload managers can adapt to external triggers such as hardware failure, and send alerts or attempt automatic recovery.

4.4 Job Submission Process

Whenever a job is submitted, the workload management system checks on the resources requested by the job script. It assigns cores, accelerators, local disk space, and memory to the job, and sends the job to the nodes for computation. If the required number of cores or memory are not yet available, it queues the job until these resources become available. If the job requests resources that are always going to exceed those that can become available, then the job accordingly remains queued indefinitely.

The workload management system keeps track of the status of the job and returns the resources to the available pool when a job has finished (that is, been deleted, has crashed or successfully completed).

4.5 What Do Job Scripts Look Like?

A job script looks very much like an ordinary shell script, and certain commands and variables can be put in there that are needed for the job. The exact composition of a job script depends on the workload manager used, but normally includes:

• commands to load relevant modules or set environment variables for the job to run

• directives for the workload manager for items associated with the job. These items can be a request for resources, output control, or setting the email addresses for messages to go to

• an execution (job submission) line

When running a job script, the workload manager is normally responsible for generating a machine file based on the requested number of processor cores (np), as well as being responsible for the allocation any other requested resources.

The executable submission line in a job script is the line where the job is submitted to the workload manager. This can take various forms.

Example

For the Slurm workload manager, the line might look like:

srun a.out

Example

For PBS Professional it may simply be:

mpirun ./a.out

Example

For GE it may look like:

mpirun -np 4 -machinefile $TMP/machines ./a.out

4.6 Running Jobs On A Workload Manager

The details of running jobs through the following workload managers is discussed later on, for:

• Slurm (Chapter 5)

• GE (Chapter 6)

• PBS Professional (Chapter 7)

4.7 Running Jobs In Cluster Extension Cloud Nodes Using cmjob

Extra computational power from cloud service providers such as the Amazon Elastic Compute Cloud (EC2) can be used by an appropriately configured cluster managed by NVIDIA Base Command Manager.

If the head node is running outside a cloud services provider, and at least some of the compute nodes are in the cloud, then this “hybrid” cluster configuration is called a Cluster Extension cluster, with the compute nodes in the cloud being the cloud extension of the cluster.

For a Cluster Extension cluster, job scripts to a workload manager should be submitted using NVIDIA Base Command Manager’s cmjob utility. The cmjob utility should be configured beforehand by the administrator (section 4.3 of the Cloudbursting Manual). The cmjob utility allows the job to be considered for running on the extension (the cloud nodes). Jobs that are to run on the local regular nodes (not in a cloud) are not dealt with by cmjob.

The cluster administrator must have created a certificate for cmjob use for the user that is using the cmjob job submission utility:

Example

[basecm10->user[fred*]]% set createcmjobcertificate yes

In addition, the environment module (section 2.3) cmjob is typically configured by the system administrator to load by default on the head node. It must be loaded for the cmjob utility to work.

The default settings for cmjob are configured by the cluster administrator. By default, the storage provided by Amazon’s S3 has an expiry date of 30 days (section 4.3 of the Cloudbursting Manual). Users planning to that keep data in the cloud may wish to check with the administrator what the S3 expiry setting is for them, if they have not already been informed about it.

4.7.1 Introduction To Options For cmjob

The syntax for cmjob has the following form:

cmjob [options] [arguments]

The submit option is the most important option for users. A basic example for job submission takes the following form with the submit option:

cmjob submit <script>

Details for cmjob are given by the help option (-h|--help). The help option gives details for deeper level options, as well as at the top level. At the top level, the help text that is displayed is:

[fred@head ~]$ cmjob -h
usage: cmjob [-h]
             {submit,status,cancel,list-labels,list-files-for-label,create-label,delete-label,\
upload,download,list-fsx-instances,create-fsx-instance,share-fsx-instance,delete-fsx-instance,\
list-anf-volumes,create-anf-volume,share-anf-volume,delete-anf-volume}
             ...
Submit a job.
positional arguments:
  {submit,status,cancel,list-labels,list-files-for-label,create-label,delete-label,upload,\
download,list-fsx-instances,create-fsx-instance,share-fsx-instance,delete-fsx-instance,\
list-anf-volumes,create-anf-volume,share-anf-volume,delete-anf-volume}
    submit              Submit a job to execute on the compute nodes.
    status              Obtain the status of a certain job.
    cancel              Cancel job with a specified job id.
    list-labels         List all known labels
    list-files-for-label
                        List all files stored under a certain label or list
                        all files for all labels.
    create-label        Create a label with the given name
    delete-label        Delete the label with the given name
    upload              Upload input data under some label.
    download            Download labeled output data from previous jobs.
    list-fsx-instances  List available FSx instances that can be used by this
                        user.
    create-fsx-instance
                        Create a new FSx instance.
    share-fsx-instance  Share an owned FSx instance with other users. These
                        users are allowed to use the instance, but will not be
                        able to delete it.
    delete-fsx-instance
                        Delete an existing FSx instance.
    list-anf-volumes    List available ANF volumes that can be used by this
                        user.
    create-anf-volume   Create a new ANF volume.
    share-anf-volume    Share an owned ANF volume with other users. These
                        users are allowed to use the volume, but will not be
                        able to delete it.
    delete-anf-volume   Delete an existing ANF volume.
optional arguments:
  -h, --help            show this help message and exit
[fred@head ~]$

4.7.2 Running cmjob submit

Users that are used to running jobs as root should note that the root user cannot usefully run a job with cmjob submit.

Regular users can submit job scripts to the cloud via the command line, using the basic submit option:

Example

$ cat myscript1
#!/bin/sh
hostname
$ cmjob submit myscript1
Submitting job: myscript1(slurm-2) [slurm:2] ... OK

Users can also run cmjob submit in more sophisticated ways. The submission can be done with some cloud-related options set. This can be done in two styles with cmjob:

1. command line style (here)

2. job directive style (here)

4.7.2.1 Command Line Style Options For cmjob submit

The main command line options for cmjob submit can be viewed via the help text:

[root@basecm10 ~]# cmjob submit -h
usage: cmjob submit [-h] [--remote-output-list <file>]
                    [-D <output file>[:label]] [-n <number>]
                    [--job-name-prefix <prefix>] [-X EXTRA_FLAGS]
                    [-w WLMANAGER]
                    [--expected-output-size EXPECTED_OUTPUT_SIZE]
                    [-i <input file>[@<new_name>][:<label>]]
                    [--input-list <file>]
                    [-o <output file>[@<new_name>][:<label>]]
                    [--output-list <file>] [--fsx-instance <name | fsx_id>]
                    [--on-demand-fsx] [--anf-volume <name | id>]
                    [--on-demand-anf] [-c <cert>] [-k <key>]
                    [-u <head node url>] [--dry-run] [--debug] [-v]
                    script
positional arguments:
  script                The script that is to be executed on the compute
                        nodes. Must start with a shebang line.
optional arguments:
  -h, --help            show this help message and exit
  --remote-output-list <file>
                        One remote file (located on the cloud nodes)
                        containing a list of output files. This remote file is
                        usually generated by the job itself.
  -D <output file>[:label], --no-download <output file>[:label]
                        Output files to not download from the object store.
                        This is useful when you want them to be consumed by a
                        subsequent job directly from the object store, or if
                        the output files are large. E.g. `cmjob -o file1:label
                        --no-download file1:product produce.sh; cmjob -i
                        file1:product consume.sh`
  -n <number>, --num-threads <number>
                        Number of parallel file transfers to/from S3.
  --job-name-prefix <prefix>
                        String that will be used as prefix in cloud job name.
  -X EXTRA_FLAGS, --extra-flags EXTRA_FLAGS
                        Extra flags for the workload manager during
                        submission. (e.g: -X "-p partition"
  -w WLMANAGER, --wlmanager WLMANAGER
                        Workload manager used for job submission. If no name
                        is specified, then a workload manager is worked out
                        from the modules currently loaded.
  --expected-output-size EXPECTED_OUTPUT_SIZE
                        In gigabytes. Together with the size of the input
                        files, this controls the size of the allocated storage
                        volume.
  -i <input file>[@<new_name>][:<label>], --input <input file>[@<new_name>][:<label>]
                        One or more input files for the job. Can be specified
                        multiple times.
  --input-list <file>   One file containing a list of input files.
  -o <output file>[@<new_name>][:<label>], --output <output file>[@<new_name>][:<label>]
                        One or more output files produced by a cmjob job that
                        will be stored in the object store (and downloaded to
                        the head node unless --no-download is used). Can be
                        specified multiple times.
  --output-list <file>  One file containing a list of output files.
  --fsx-instance <name | fsx_id>
                        Use a previously created FSx device instead of a
                        storage node to read and write data from.
  --on-demand-fsx       Do not use storage nodes, but create and use an on-
                        demand FSx instance instead. The volume will be
                        deleted upon job completion. Use --output flag to copy
                        the output data out of the volume before it's deleted
  --anf-volume <name | id>
                        Use a previously created ANF volume instead of a
                        storage node to read and write data from.
  --on-demand-anf       Do not use storage nodes, but create and use an on-
                        demand ANF volume instead. The volume will be deleted
                        upon job completion. Use --output flag to copy the
                        output data out of the volume before it's deleted
  -c <cert>, --cert <cert>
                        The cert file to use when connection to the cmdaemon
                        api.
  -k <key>, --key <key>
                        The key to use when connecting to the cmdaemon api.
  -u <head node url>, --url <head node url>
                        The url of the cmdaemon api to connect to.
  --dry-run             Don't submit job description to cmdaemon.
  --debug               Enable debug output.
  -v, --verbose         Enhance verbosity of cmjob commands.
Note: All of these options (except for -h | --help) can be used on the command
line, or in the <script>

4.7.2.2 Job-directive Style #CMJOB Directives For cmjob submit

All cmjob submit command line options can also be specified in a job-directive style format in the job script itself, using the “#CMJOB” tag to indicate an option.

Example

$ cat myscript2
#!/bin/sh
#CMJOB --input-list=/home/user/myjob.in
#CMJOB --output-list=/home/user/myjob.out
#CMJOB --remote-output-list=/home/user/file-which-will-be-created
#CMJOB --input=/home/user/onemoreinput.dat
#CMJOB --input=/home/user/myexec
myexec
$ cmjob submit myscript2
Submitting job: myscript2(slurm-2) [slurm:2] ... OK

4.7.2.3 Workload Manager Job-directives And Workload Manager Flag Options For cmjob

Beside the two cmjob submit styles using options and directives, there is another way, independent of those, to pass on job handling instructions. This is using by workload manager job specifications.

Workload manager batch job tags are tags associated with the workload manager, such as #SBATCH for Slurm (Chapter 5), or #PBS for PBS Professional (Chapter 7). These tags can also be used by the user to specify flags to be sent to the workload manager when submitting the job via cmjob submit. This can be done by

• using the tag itself (e.g: #SBATCH for Slurm, #PBS for PBS Professional …) These can be used along with the #CMJOB tags.

  Example

  $ cat myscript3
  #!/bin/sh
  #SBATCH -p partition
  hostname
  $ cmjob submit myscript3
  Submitting job: myscript3(slurm-3) [slurm:3] ... OK

• or by using the cmjob flag option --extra-flags to specify the flag associated with the tag.

  Example

  $ cat myscript4
  #!/bin/sh
  hostname
  $ cmjob submit --extra-flags="-p partition" myscript4
  Submitting job: myscript4(slurm-4) [slurm:4] ... OK

4.7.3 An Example Session Of Running cmjob With Cloud Storage

In this section a job is submitted with cmjob to a cloud compute node. For this to work, the cluster administrator should have already arranged for the cloud director and cloud compute nodes to be up.

If cloud storage is needed for a job, then cmjob causes it to be powered up. If it is not needed, then it is powered down. If AWS is used, then the cloud storage gets input from cmjob via an S3 bucket. By default the S3 bucket storage expires after 30 days.

The job submitted via cmjob could be something like the slurmhello.sh script from section 5.3.1, but with appropriate cmjob options.

Cluster administrators can use cmsh to list cloud jobs that are running via cmjob. The jobs are listed under the cloudjobs submode.

In the following example, where a job has been submitted, the cloud storage is not yet up:

Example

[root@basecm10 ~]# cmsh
[basecm10]% cmjob
[basecm10->cmjob[cmjob]]% cloudjobs
[basecm10->cmjob[cmjob]->cloudjobs]% list
Name (key) jobStatus                                       storageNode           region    user
---------- ----------------------------------------------- --------------------- --------- ----
slurm-39   Waiting for available storage node...                                 us-east-1 fred

The cmjob utility takes care of the storage requirements and directives specified by the user. In the preceding example it brings up cloud storage automatically.

4.7.3.1 Tracking Job Status With cmsh

The job status can be viewed using cmsh by the cluster administrator, and also by users with the appropriate profile privileges. Once the storage node is up, a job typically displays a sequence similar to the following in the jobStatus column:

Example

Name (key) jobStatus                                       storageNode           region    user
---------- ----------------------------------------------- --------------------- --------- ----
slurm-39   Attaching EBS volume...                         us-east-1-cstorage001 us-east-1 fred

followed by:

slurm-39   Transferring input data from S3 to the storage+ us-east-1-cstorage001 us-east-1 fred

followed by:

slurm-39   Running...                                      us-east-1-cstorage001 us-east-1 fred

followed by:

slurm-39   Transferring result from the storage node to S3 us-east-1-cstorage001 us-east-1 fred

followed by (for jobs with a data label):

slurm-39   Deregistering job as producer for labels        us-east-1-cstorage001 us-east-1 fred

followed by:

slurm-39   Detaching EBS volume...                         us-east-1-cstorage001 us-east-1 fred

followed by:

slurm-39   Finished                                        us-east-1-cstorage001 us-east-1 fred

4.7.3.2 Tracking Job Status With cmjob

Regular users without cmsh profile privileges can track the same job status sequence via the cmjob status <job name> command:

Example

[fred@basecm10 ~]$ cmjob status slurm-39
+---------------+--------------------------------------------------------------------+
| Property      | Value                                                              |
+---------------+--------------------------------------------------------------------+
| Job name      | slurm-39                                                           |
| Job ID        | 39                                                                 |
| Job status    | Transferring input data from S3 to the storage node... (0m 2s ago) |
| Total runtime | 3m 31s                                                             |
| Storage type  | NFSViaStorageNode                                                  |
| Job script    | slurmhello.sh                                                      |
| Region        | us-east-1                                                          |
| Job Queue     | us-east-1                                                          |
+---------------+--------------------------------------------------------------------+
[fred@basecm10 ~]$

4.7.3.3 Running cmjob With Labels For Data

Labels can be assigned to data placed in the cloud by a job—job A for example. Data that that is left in the cloud for the next job—job B, for example—-can then be accessed by job B using that label, and then be kept in the cloud under a new label. This can be a more efficient way to run chained jobs. The data can also be taken out from the cloud and stored locally, even if this is not efficient.

In other words, a label allows users to avoid having to move data in and out of the cloud when jobs use the same data, which can save time and money.

Scripts for the example session: The following example session is based on the following job scripts, which use basic Unix utilities such as seq, paste, bc to do some simple arithmetic:

[fred@basecm10 ~]$ cat generate.sh
#!/bin/bash
#SBATCH -n 1
#SBATCH -p us-east-1
## make a column of numbers from 1..10
## save to file 1to10
seq 10 > 1to10

[fred@basecm10 ~]$ cat basicsum.sh
#!/bin/bash
#SBATCH -n 1
#SBATCH -p us-east-1
## sum up the column of numbers in 1to10 file
## save to file basicsum
paste -s -d+ 1to10 | bc > basicsum

[fred@basecm10 ~]$ cat square.sh
#!/bin/bash
#SBATCH -n 1
#SBATCH -p us-east-1
## Take 1to10 column and make a column of squares
## save to file squares
{
for i in $(cat 1to10)
do
  echo "$i*$i" | bc
done
} > squares

[fred@basecm10 ~]$ cat squaressum.sh
#!/bin/bash
#SBATCH -n 1
#SBATCH -p us-east-1
## sum up the column of numbers in file squares
## send output to file squaredsum
paste -s -d+ squares | bc > squaredsum

Example session run for user fred: The Slurm workload manager and cmjob modules should already be loaded (module load slurm cmjob).

The cloud director and compute nodes should also be up and running.

• Submission, labels, and no downloading:
  The session run starts with a user fred running:

  [fred@basecm10 ~]$ cmjob submit --output 1to10:mylabel --no-download 1to10:mylabel generate.sh
  Submitting workload job...
  Submitted workload job slurm-74

  which submits the generate.sh script. The script is run in the cloud, and it produces the numbers 1 to 10 on the compute node, which the user can pretend is a huge amount of data that is best kept in the cloud. The output is placed in a file 1to10. The --output option means that the output file is given the label mylabel. The --no-download option means the output—accessed using the label mylabel—is prevented from being downloaded back to the directory of fred on premises, which may be sensible for a huge amount of data that may be used by another job.

  The job status can be checked with cmjob status or cmjob status slurm-74:

  [fred@basecm10 ~]$ cmjob status slurm-74
  +---------------+-------------------+
  | Property      | Value             |
  +---------------+-------------------+
  | Job name      | slurm-74          |
  | Job ID        | 74                |
  | Job status    | Finished          |
  | Total runtime | 3m 11s            |
  | Storage type  | NFSViaStorageNode |
  | Job script    | generate.sh       |
  | Region        | us-east-1         |
  | Job Queue     | us-east-1         |
  +---------------+-------------------+

  The label status can be checked with cmjob list-labels:

  [fred@basecm10 ~]$ cmjob list-labels
  +---------+-----------------------------------+---------------------------------------+
  | Label   | Created at                        | Expires at                            |
  +---------+-----------------------------------+---------------------------------------+
  | mylabel | 2020-04-20 00:26:27 - 15m 54s ago | 2020-05-20 02:00:00 - 30d 1h from now |
  +---------+-----------------------------------+---------------------------------------+

  The files stored in the object store under a label can be checked using cmjob list-files-for-label:

  [fred@basecm10 ~]$ cmjob list-files-for-label .
  +------------------+---------+-----------------------------------+------+
  | Name             | Label   | Last modified                     | Size |
  +------------------+---------+-----------------------------------+------+
  | /home/fred/1to10 | mylabel | 2020-04-20 00:30:17 - 12m 11s ago | 21   |
  +------------------+---------+-----------------------------------+------+

  If needed, the 1to10 file can be checked to see if it exists, and how it looks. It can be brought down from the cloud with the download option of cmjob:

  [fred@basecm10 ~]$ cmjob download --output 1to10:mylabel
  Submitting download job...
  Submitted download job download-54
  [fred@basecm10 ~]$ cat 1to10
  1
  2
  3
  4
  5
  6
  7
  8
  9
  10

• Using a large object already in the cloud and bringing out a result locally:
  The next job that is to be run, can pick up the labeled object in the cloud and do something with it. In this case, it picks up the numbers as input from the labeled object file in the cloud, and sums the numbers up with the script in the cloud compute node. The output of the script, which the user can pretend is considerably smaller than the input, can then be placed as output in the local directory of fred using the --output option:

  [fred@basecm10 ~]$ cmjob submit --input 1to10:mylabel --output basicsum basicsum.sh
  Submitting workload job...
  Submitted workload job slurm-75

  After the job is finished (according to the report from cmjob status slurm-75), the file basicsum shows up locally:

  [fred@basecm10 ~]$ cat basicsum
  55

• Processing large data in the cloud and leaving the data there:
  In the next case considered, the square.sh file can be run so that it uses the “large” file labeled with mylabel, 1to10, and uses it as input data. It keeps the output file, squares, which is also a “large” file, in the object store in the cloud using the label numbers2:

  [fred@basecm10 ~]$ cmjob submit --input 1to10:mylabel --output squares:numbers2 \
  --no-download *:numbers2 square.sh
  Submitting workload job...
  Submitted workload job slurm-76

  After that job is run, its output file, squares, can be picked up and checked locally with:

  [fred@basecm10 ~]$ cmjob download --output squares:numbers2
  Submitting download job...
  Submitted download job download-57
  [fred@basecm10 ~]$ cat squares
  1
  4
  9
  16
  25
  36
  49
  64
  81
  100

  If the download is repeated, or otherwise risks overwriting a local file, then the user is prompted with a warning.

  However, in any case, the idea is to avoid unnecessary downloads of the “large” file, so typically a download is not done for large data.

• Processing the new large data in the cloud and downloading a reduced end result:
  Continuing on, this new “large” data can be processed with the script squaressum.sh, to sum the squares file in the object store in the cloud. The squares file is accessed in the cloud with the label numbers2, and reduced to a single number. The shrunk result is brought down from the cloud and kept locally as the file squaredsum:

  [fred@basecm10 ~]$ cmjob submit --input squares:numbers2 --output squaredsum squaressum.sh

  The job status can be checked with:

  cmjob status <job name>

  When the job is in the finished state, the user can run:

  [fred@basecm10 ~]$ cat squaredsum
  385

• Processing Using High-Performance AWS FSx For Lustre Storage:

  The user Fred may wish to use a high-performance filesystem in the cloud. Lustre is a high-performance parallel filesystem aimed at HPC needs. Amazon offers AWS FSx with Lustre in some regions of AWS, as documented at https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/.

  Region restrictions: Changes may need to be carried out to the region used, to make use of FSx.

  For example, the scripts generate.sh and so on from here onward used a region, us-east-1, that at the time of writing (May 2020) did not have FSx. The region us-west-1 on the other hand did have FSx.

  So, if the user would like to have jobs using FSx, and the cluster administrator has set up the FSx-enabled region, such as us-west-1, with a cloud director, cloud nodes, and cloud storage, then the user should also change the Slurm queue option (the partition option, -p) in the scripts generate.sh and so on from here.

  [An aside for the cluster administrator: The cluster administrator can manage the Slurm queues for the region via the jobqueue submode (cmsh→wlm→jobqueue, section 7.7.2 of the Administrator Manual). A Slurm queue for cloud storage can also be created as an option when the cluster administrator runs cm-cloud-storage-setup when setting up a region.]

  Permission restrictions: The high performance of Lustre is expensive, so a cluster administrator typically restricts its use. Some permissions may therefore be needed before a user can use Lustre.

  [Aside for the administrator: Granting users FSx-related permissions is described here.]

  Three ways to launch FSx: Once the appropriate tokens are set, the user can launch FSx volumes for their jobs in three ways—on-demand, user-managed, or admin-managed. The user should check with the cluster administrator which of these is available:

  1. On-demand: With this way, the option --on-demand-fsx can be added to cmjob. This creates an FSx volume on demand, instead of a storage node, and the FSx volume is deleted after the job ends.

     For example, a session can be run, where the user fred submits a job in a region that supports FSx, with

     [fred@basecm10 ~]$ cmjob submit --on-demand-fsx --output 1to10:mylabel2  generate.sh

     When the job is completed, the FSx volume that came up on demand, is deleted.

  2. User-managed: A more hands-on way to manage an FSx instance is with a create-use-delete sequence that is run by the user.

     1. Creation can be carried out in this format:

        cmjob create-fsx-instance <FSx instance name>

        The command creates an FSx instance with a name <FSx instance name> that the user specifies. On execution, the FSx instance ID is displayed. FSx instance IDs can also be seen if the command cmjob list-fsx-instances is run.

     2. Jobs can then be submitted with the --fsx-instance option, in a format similar to:

        cmjob submit --fsx-instance <FSx instance name, or FSx instance ID>

        and the jobs then use the pre-created instance.

     3. Unlike the --on-demand-fsx option, the FSx volume is not deleted when the job ends. So, after use, the instance must be deleted explicitly with a command in a form such as:

        cmjob delete-fsx-instance <FSx instance name, or FSx instance ID>

     Thus a session can be run where a user fred creates an FSx instance, fsxa, with:

     Example

     [fred@basecm10 ~]$ cmjob create-fsx-instance fsxa
     FSx instance 'fsxa' is being created. The default capacity will be used.
     The instance is being created with id fs-07f668884b4eda47c in region 'us-west-1' and will be made
     available in VPC 'vpc-071f3150b0bd5a87e'. Capacity: 1200GiB.
     This might take several minutes. Use 'list-fsx-instances' to see the progress.
     [fred@basecm10 ~]$ cmjob list-fsx-instances
     +------+----------------------+----------+----------+------+
     | Name | ID                   | Status   | Capacity | Jobs |
     +------+----------------------+----------+----------+------+
     | fsxa | fs-07f668884b4eda47c | CREATING | 1200     |      |
     +------+----------------------+----------+----------+------+
     [fred@basecm10 ~]$ cmjob list-fsx-instances
     +------+----------------------+-----------+----------+------+
     | Name | ID                   | Status    | Capacity | Jobs |
     +------+----------------------+-----------+----------+------+
     | fsxa | fs-07f668884b4eda47c | AVAILABLE | 1200     |      |
     +------+----------------------+-----------+----------+------+

     Once the FSx storage is available, it can be used merely by adding the --fsx-instance option, along with the FSx storage identifier, to cmjob:

     Example

     [fred@basecm10 ~]$ cmjob submit --fsx-instance fsxa --output 1to10:mylabel3  generate.sh
     Submitting workload job...
     Submitted workload job slurm-4

     Once the job is finished, and fred no longer requires the data, the instance should be deleted:

     Example

     [fred@basecm10 ~]$ cmjob delete-fsx-instance fsxa
     About to delete this FSX instance
     fsxa (fs-07f668884b4eda47c)
       owner: fred
       status: AVAILABLE
       capacity: 1200GB
       created at: 2020-05-01T21:55:52.162000+02:00
       managed: UserManaged
       shared with: no one
     Enter 'yes' to continue> yes
     Deleting FSx instance fs-07f668884b4eda47c
     [fred@basecm10 ~]$

  3. Admin-managed: With the admin-managed option, it is the cluster administrator who creates an FSx instance, and who shares it with one or more other regular users. Users are then able to use the instance just as in user-managed instances, but they cannot delete the instance or view the files of other users.

• Processing Using High-Performance ANF Volumes:

  ANF (Azure NetApp Files) is a high-performance filesystem available in Azure clouds. The procedures to use it are very similar to that for FSx volumes in AWS clouds.

  The cluster administrator can set up a cloud director, cloud nodes, and cloud storage, in an ANF-enabled region, such as westeurope. If a regular user would like to have jobs using ANF, and the user is using the scripts generate.sh and so on from here, then the user should set the Slurm queue option (the partition option, -p) so that that ANF-enabled region is used.

  [An aside for the cluster administrator: The cluster administrator can manage the Slurm queues for the region via the jobqueue submode (cmsh→wlm→jobqueue, section 7.7.2 of the Administrator Manual). A Slurm queue for cloud storage can also be created as an option when the cluster administrator runs cm-cloud-storage-setup when setting up a region.]

  Permission restrictions: ANF is expensive to use, so a cluster administrator typically restricts its use. Some permissions may therefore be needed before a user can use ANF.

  [Aside for the administrator: Granting users ANF-related permissions is described here.]

  Three ways to launch ANF: Once the appropriate tokens are set, the user can launch ANF volumes for their jobs in three ways—on-demand, user-managed, or admin-managed. The user should check with the cluster administrator which of these is available:

  1. On-demand: With this way, the option --on-demand-anf can be added to cmjob. This creates an ANF volume on demand, instead of a storage node, and the ANF volume is deleted after the job ends.

     For example, a session can be run, where the user fred submits a job in a region that supports ANF, with

     [fred@basecm10 ~]$ cmjob submit --on-demand-anf --output 1to10:mylabel2  generate.sh

     When the job is completed, the ANF volume that came up on demand, is deleted.

  2. User-managed: A more hands-on way to manage an ANF instance is with a create-use-delete sequence that is run by the user.

     1. Creation can be carried out in this format:

        cmjob create-anf-volume <ANF volume name>

        The command creates an ANF volume with a name <ANF volume name> that the user specifies. On execution, the ANF instance ID is displayed. ANF volume IDs can also be seen if the command cmjob list-anf-volumes is run.

     2. Jobs can then be submitted with the --anf-volume option, in a format similar to:

        cmjob submit --anf-volume <ANF volume name, or ANF volume ID>

        and the jobs then use the pre-created instance.

     3. Unlike the --on-demand-anf option, the ANF volume is not deleted when the job ends. So, after use, the instance must be deleted explicitly with a command in a form such as:

        cmjob delete-anf-volume <ANF volume name, or ANF volume ID>

     Thus a session can be run where a user fred creates an ANF instance, anfa, with:

     Example

     [fred@basecm10 ~]$ cmjob create-anf-volume anfa
     ANF volume 'anfa' is being created. The default size will be used.
     The volume named anfa (anfa-jrcjlSJXgP) is created in location 'westeurope' and will be made
     available in resource group 'pj-mojo-westeurope-bcm'. Capacity: 4TiB.
     This might take several minutes. Use 'list-anf-volumes' to see the progress.
     [fred@basecm10 ~]$ cmjob list-anf-volumes
     +---------+--------------------+----------+----------+------+
     | Name    | ID                 | Status   | Capacity | Jobs |
     +---------+--------------------+----------+----------+------+
     | anfa    | anfa-jrcjlSJXgP    | CREATING | 4TiB     |      |
     +---------+--------------------+----------+----------+------+
     [fred@basecm10 ~]$ cmjob list-anf-volumes
     +---------+--------------------+-----------+----------+------+
     | Name    | ID                 | Status    | Capacity | Jobs |
     +---------+--------------------+-----------+----------+------+
     | anfa    | anfa-jrcjlSJXgP    | AVAILABLE | 4TiB     |      |                                  |
     +---------+--------------------+-----------+----------+------+

     Once the ANF volume is available, it can be used merely by adding the --anf-volume option, along with the ANF volume identifier, to cmjob:

     Example

     [fred@basecm10 ~]$ cmjob submit --anf-volume anfa --output 1to10:mylabel3  generate.sh
     Submitting workload job...
     Submitted workload job slurm-4

     Once the job is finished, and fred no longer requires the data, the instance should be deleted:

     Example

     [fred@basecm10 ~]$ cmjob delete-anf-volume anfa
     About to delete this ANF volume
     anfa (anfa-jrcjlSJXgP)
       owner: fred
       status: AVAILABLE
       size: 4TB
       created at: 2020-05-10T23:16:07.871298Z
       managed: UserManaged
       shared with: no one
     Enter 'yes' to continue> yes
     Deleting ANF volume anfa
     [fred@basecm10 ~]$

  3. Admin-managed: With the admin-managed option, it is the cluster administrator who creates an ANF volume, and who shares it with one or more other regular users. Users are then able to use the instance just as in user-managed instances, but they cannot delete the instance or view the files of other users.