6. Workload Examples

In this section, we give instructions for running several Workloads on your Run:ai on DGX Cloud cluster. The examples are not exhaustive, but can be adapted for your own workloads.

The following images are used in the workload examples and have been tested and validated as functional. Any other images are un-tested and may require adjustments to environment variables or additional debugging to function optimally.

nvcr.io/nvidia/nemo:24.05.01

nvcr.io/nvidia/nemo:24.05

nvcr.io/nvidia/nemo:24.07

nvcr.io/nvidia/pytorch:24.06-py3

nvcr.io/nvidia/pytorch:24.07-py3

nvcr.io/nvidia/pytorch:24.08-py3

nvcr.io/nvidia/rapidsai/notebooks:24.08-cuda11.8-py3.9

6.1. Interactive NeMo Workload Job

In this example, we step through the process of creating an interactive workload using the NeMo container from NGC. Interactive workloads in Run:ai are called Workspaces. In this particular example, we will run a Jupyter notebook to fine-tune an LLM (Llama3-8B Instruct) using LoRA against the PubMedQA dataset.

6.1.1. Prerequisites and Requirements

The following are required before running the interactive NeMo job:

You must have accepted an invitation to your NGC org and added your NGC credentials to the Run:ai. See details here.
You must have the user role L1 researcher, ML Engineer, or System Administrator to be able to run through all sections of this tutorial.
Your user must be able to access a project and department.
At least one A100 or newer GPU with 80 GB of memory.
You must have access to a compute resource with at least one GPU created in your scope that you can use.
You must create a Hugging Face account and agree to the Meta Llama 3 Community License Agreement while signed in to your Hugging Face account. You must then generate a Hugging Face read access token in your account settings. This token is required to access the Llama3 model in the Jupyter Notebook.

6.1.2. Creating the Data Source

To make it easier to reuse code and/or checkpoints in future jobs, a persistent volume chain (PVC) is created as a data source. The PVC can be mounted in jobs and will persist after the job completes, allowing any generated data to be reused.

To create a new PVC, go to the Data Sources page. Click New Data Source and then PVC to open the PVC creation form.
On the new form, set the desired scope. Depending on the sensitivity of the data, the PVC can be limited in scope down to the project level or be shared more broadly with the whole org or department(s).
Give the PVC a memorable name like nemo-lora-checkpoints and add a description if desired.
For the data options, select a new PVC storage class that suits your needs according to the PVC recommendations here. In this example, zonal-rwx is sufficient. To allow all nodes to read and write from/to the PVC, select Read-write by many nodes for the access mode. Enter 10 TB for the size to ensure we have plenty of capacity for future jobs. Select Filesystem as the volume mode. Lastly, set the Container path to /checkpoints, where the PVC will be mounted inside containers. This completed section should look as shown below.
Once the form is complete, create the data source using the button at the bottom of the page.

Note

The first time a job is launched using a new PVC, it will take longer to start as the storage gets provisioned only once the first claim to the PVC is made.

6.1.3. Creating the Workspace

With the PVC created, we can configure the workspace to tell Run:ai how the job should be launched. To do so, go to the Workloads overview page and click the + New Workload button in the top left. A drop-down menu will appear. From the drop-down menu, select Workspace. You will be taken to the New workspace creation page.
Select the desired project to run your job in.
Leave the Template pane of the form set to Start from scratch.
Enter a descriptive name for your Workspace, such as nemo-pubmedqa-lora. Click Continue. After a few seconds the Environment pane of the creation process will appear.
The environment is a template with common settings that can be reused for similar jobs. For example, an environment might be based on a PyTorch container and have Jupyter exposed as a tool enabling anyone who needs to run interactive PyTorch jobs inside a Jupyter notebook to use the same template instead of launching unique jobs and re-configuring the settings each time they want to experiment. In this example, a new environment will be created based on the NeMo container and expose Jupyter as a tool to make it easier to reuse the template in the future.
To create a new environment for the NeMo job, click the + New Environment button in the top right of the Environment pane. The Environment creation form will open.
In the Environment creation form, enter a name for the environment, such as “nemo-jupyter” and optionally add a description.
Under Image URL, put nvcr.io/nvidia/nemo:24.07. This pulls the latest NeMo container with the dev tag from NGC.
If desired, the image pull policy can be changed. The default value of “if not already present” is recommended for most cases, but if you are pushing new containers to the NGC org with the same tag, “always pull the image from the registry” should be used to check if there are updates to the image. Given that the dev tag for NeMo is updated frequently, “always pull the image from the registry” is likely a better option to get the latest updates.
Under the Workload architecture & type pane, select Standard and Workspace if they are not selected already. These options are likely already selected and grayed out.
Click the Tools pane to open the tools menu, and click +Tool. This list allows you to add additional tools to the workspace to make it easier to connect to their resources, such as Jupyter for connecting to external Jupyter lab sessions, VS Code for connecting to VS Code servers inside the job, and more.
Since we want to run an interactive Jupyter notebook, select Jupyter in the Select tool list. The Connection type drop-down should show External URL, Auto generate and Container port “8888”.
Click the Runtime settings pane, then click to expand the commands and arguments pane. This section allows us to override the default container run settings as needed. Since we want to launch a Jupyter lab session, enter jupyter-lab as the command, and --NotebookApp.base_url=/${RUNAI_PROJECT}/${RUNAI_JOB_NAME} --NotebookApp.token='' --ServerApp.allow_remote_access=true --allow-root for the arguments. This tells Run:ai how to connect to the Jupyter lab session in the UI.
For the directory, specify /opt/NeMo as the directory to launch the job in. This will launch the Jupyter lab session inside the /opt/NeMo directory inside the container.
Click Create Environment in the bottom right of the page. You should see a pop-up saying that your environment has been created. The page will refresh, and you will be taken back to the top of the Environment pane of the New workspace creation page. The environment you just created should now be selected.
Scroll down to the Compute resource pane and select a single-GPU compute resource for your environment. While it is possible to use more GPUs for LoRA, most tasks can be completed with just a single Ampere or newer GPU with 80GBs of memory in a relatively short period.
Scroll to the bottom of the page and click Create Workspace. This will add your job to the queue and it will automatically get scheduled once cluster resources are available.
After creating the workspace, you will be taken to the workload overview page, where you can view the status of your workload. Your interactive workload is ready once the status reads “Running”. This may take a while if the container you chose has not previously been pulled to the cluster and/or the PVC needs to be provisioned for the first time.
Once the status is “Running”, you can launch the Jupyter lab session by selecting the checkbox next to your workload and clicking CONNECT → Jupyter on the top menu bar. Jupyter lab will open in a new window.

6.1.4. Running the Notebook

Once connected to the Jupyter lab session, navigate to the example notebook by opening to the tutorials/llm/llama-3 directory inside the /opt/NeMo folder the container automatically opens to. Inside this directory are a couple of notebooks, including llama3-lora-nemofw.ipynb, which is the one we will be following. Double-click that file to open the notebook on the main screen.

The notebook walks through what each step does, but at a high level, it does the following:

Downloads the Llama3-8B Instruct model from Hugging Face

Converts the model to the .nemo format to be easily ingested by the NeMo toolkit

Downloads and preprocesses the PubMedQA dataset

Fine-tunes the Llama3-8B Instruct model using the LoRA PEFT technique against the PubMedQA dataset

Deploys the model for local inference

Evaluates the model against the test dataset to determine the overall accuracy of the fine-tuned model

The notebook can be stepped through cell-by-cell or run all at once, and the results viewed later. The only important note is that cell 2 requires you to add your Hugging Face access token, as listed in the requirements section in the cell text, in order to authenticate with Hugging Face to download the model.

As the notebook runs, you can navigate back to the Run:ai workloads page, where your job now shows metrics based on your resource usage. The metrics tab shows the GPU and CPU compute and memory usage during the job. This is a great resource for determining how your job is performing.

6.1.5. Persisting the Model to Storage

As the PVC was added to the workspace and is mounted at /checkpoints inside the container, any files or directories that are saved to /checkpoints inside the container will be saved in the PVC and available in future jobs.

To save the generated model to the PVC, open a terminal tab inside the Jupyter lab session and navigate to the notebook directory with:
cd /opt/NeMo/tutorials/llm/llama-3

The model is saved at results/Meta-Llama-3-8B-Instruct/checkpoints/megatron_gpt_peft_lora_tuning.nemo. It can be verified with:

  $ du -sh results/Meta-Llama-3-8B-Instruct/checkpoints/megatron_gpt_peft_lora_tuning.nemo
  21M     results/Meta-Llama-3-8B-Instruct/checkpoints/megatron_gpt_peft_lora_tuning.nemo

To copy the file to the PVC, run:

cp results/Meta-Llama-3-8B-Instruct/checkpoints/megatron_gpt_peft_lora_tuning.nemo /checkpoints

Once the checkpoint is copied, it will be available in all future workloads that use that same PVC unless the file is intentionally removed.

6.1.6. Cleaning up the Environment

After the job finishes, it can be deleted to free up resources for other jobs. Note that certain jobs do not self-terminate and should be monitored to ensure resources are not idle for completed jobs.

6.2. RAPIDS and Polars Workspace

In this example, we step through the process of creating another Workspace, which uses the RAPIDS container from NGC. We will launch a Jupyter environment to experiment with the new integration between RAPIDS and Polars.

6.2.1. Prerequisites and Requirements

The following are required before running the RAPIDS workspace:

You must have the user role L1 researcher, ML Engineer, or System Administrator to be able to run through all sections of this tutorial.
Your user must be able to access a project and department.
At least one idle GPU within your DGX Cloud cluster.
You must have access to a compute resource with at least one GPU created in your scope that you can use.

6.2.2. Creating the Workspace

Go to the Workloads overview page and click the + New Workload button in the top left. A drop-down menu will appear. From the drop-down menu, select Workspace. You will be taken to the New workspace creation page.
Select the desired project to run your job in.
Leave the Template pane of the form set to Start from scratch.
Enter a descriptive name for your Workspace, such as rapids-polars. Click Continue. After a few seconds the Environment pane of the creation process will appear.
To create a new environment for the RAPIDS workspace, click the + New Environment button in the top right of the Environment pane. The Environment creation form will open.
In the Environment creation form, enter a name for the environment, such as “rapids-jupyter” and optionally add a description.
Under Image URL, put nvcr.io/nvidia/rapidsai/notebooks:24.08-cuda11.8-py3.9.
If desired, the image pull policy can be changed. The default value of “if not already present” is recommended for most cases but if you are pushing new containers to the NGC org with the same tag “always pull the image from the registry” should be used to check if there are updates to the image.
Under the Workload architecture & type pane, select Standard and Workspace if they are not selected already. These options are likely already selected and grayed out.
Click the Tools pane to open the tools menu, and click +Tool. This list allows you to add additional tools to the workspace to make it easier to connect to their resources, such as Jupyter for connecting to external Jupyter lab sessions, VS Code for connecting to VS Code servers inside the workspace, and more.
Since we want to run a Jupyter notebook, select Jupyter in the Select tool list. The Connection type drop-down should show External URL, Auto generate and Container port “8888”.
Click the Runtime settings pane, then click to expand the commands and arguments pane. This section allows us to override the default container run settings as needed. Since we want to launch a Jupyter lab session, enter jupyter-lab as the command, and --NotebookApp.base_url=/${RUNAI_PROJECT}/${RUNAI_JOB_NAME} --NotebookApp.token='' --allow-root for the arguments. This tells Run:ai how to connect to the Jupyter lab session in the UI.
Do not modify any other settings.
Click Create Environment in the bottom right of the page. You should see a pop-up saying that your environment has been created. The page will refresh, and you will be taken back to the top of the Environment pane of the New workspace creation page. The environment you just created should now be selected.
Scroll down to the Compute resource pane and select a single GPU compute resource for your environment.
Scroll to the bottom of the page and click Create Workspace. This will add your job to the queue and it will automatically get scheduled once cluster resources are available.
After creating the workspace, you will be taken to the workload overview page, where you can view the status of your workload. Your workspace is ready once the status reads “Running”. This may take a while if the container you chose has not previously been pulled to the cluster.
Once the status is “Running”, you can launch the Jupyter lab session by selecting the checkbox next to your workload, and clicking CONNECT → Jupyter on the top menu bar. Jupyter lab will open in a new window.

6.2.3. Downloading the Polars Notebook

Once connected to the Jupyter lab session, you can navigate to the integrated RAPIDS notebook samples within the notebooks directory and experiment within your workspace.

To get started downloading the GPU-accelerated Polars example notebook, open a terminal interface by clicking the Terminal box.

In the resulting terminal, run the following command to download the new notebook.

wget https://raw.githubusercontent.com/rapidsai-community/showcase/refs/heads/main/accelerated_data_processing_examples/polars_gpu_engine_demo.ipynb

6.2.4. Running the Polars Notebook

The notebook provides an introduction to what Polars is and where it is best used, along with code to:

Install Polars with GPU Engine, along with other notebook dependencies

Download a sample Kaggle dataset

Initialize Polars to use the GPU in DGX Cloud

Run various sample analyses to provide performance comparisons between Polars with CPU only and Polars with GPU

To open the notebook, double-click on the file named polars_gpu_engine_demo.ipynb when it becomes visible in the Jupyter file navigation pane.

The notebook can be stepped through cell-by-cell or run all at once, and the results can be viewed later.

As the notebook is running, you can navigate back to the Run:ai workloads page, where your job now shows metrics based on your resource usage. The metrics tab shows the GPU and CPU compute and memory usage during the job. This is a great resource for determining how your job is performing. Within the Jupyter notebook page you are using, you can also use an integrated GPU performance visualization plugin from the RAPIDS NGC container (it is the tab on the left side of the page with a GPU icon on it).

6.2.5. Cleaning up the Environment

After you are finished with the workload, it can be stopped or deleted to free up resources for other workloads. Note that certain workloads do not self-terminate and workloads should be monitored to ensure resources are not sitting idle for completed workloads.

6.3. Download Data From S3 in a Training Workload

In this example, we will create a new Workload to download data from a private S3 bucket. We will use a Training workload due to the noninteractive nature of this example. We will configure the workload with private information to access the S3 bucket and execute the commands necessary to install and run the s5cmd tool to download the data from that bucket to a PVC.

6.3.1. Prerequisites and Requirements

The following are required before running the data download Training workload:

You must have the user role L1 researcher, ML Engineer, or System Administrator to run through all sections of this tutorial.

Your user must be able to access a project and department.

You must have access to a compute resource in your scope that you can use—it can be CPU-only, which is ideal to avoid using GPU resources when it is unnecessary.

You must have a private S3 bucket with data in it.

You must have the necessary permissions to download that data from S3 via access key ID and secret access key.

6.3.2. Creating the Credentials

To securely insert the necessary keys into the workload, we will create a Credential for the access key ID and the secret access key, allowing access to our S3 bucket.

To create a new Credential, go to the Credentials page. Click + NEW CREDENTIALS and then Access key to open the access key creation form.
On the form, set the desired scope. The scope should be as narrow as possible—in this case, set the scope to match the targeted project for your workload.
Give the access key a descriptive name, such as aws-access-keys, and add a description if desired.
In the Secret pane, select New secret. Set the Access key field to your AWS access key ID, and the Access secret field to your AWS secret access key.
Click CREATE CREDENTIALS to save the new Credential.

6.3.3. Creating the Data Source

We will create a PVC Data Source to allow the resulting dataset to be shared with future workloads.

To create a new PVC, go to the Data Sources page. Click New Data Source and then PVC to open the PVC creation form.
On the new form, set the desired scope. Depending on the sensitivity of the data, the PVC can be limited in scope down to the project level or be shared more broadly with the whole org or department(s).
Give the PVC a memorable name like s3-dataset and add a description if desired.
For the data options, select a new PVC storage class that suits your needs according to the PVC recommended storage classes. In this example, zonal-rwx is sufficient. To allow all nodes to read and write from/to the PVC, select Read-write by many nodes for the access mode. Enter a capacity that will fit your target dataset—we will specify 10 TB for this example. Select Filesystem as the volume mode. Lastly, set the Container path to /dataset, where the PVC will be mounted inside containers.
Once the form is complete, click the CREATE DATA SOURCE button at the bottom of the page.

Note

The first time a job is launched using a new PVC, it will take longer to start as the storage gets provisioned only once the first claim to the PVC is made.

6.3.4. Creating the Training Workload

Go to the Workloads overview page and click the + New Workload button in the top left. A drop-down menu will appear. From the drop-down menu, select Training. You will be taken to the New training creation page.
Select the desired project to run your job in.
Leave the Template pane of the form set to Start from scratch.
Enter a descriptive name for your workload, such as s3-download. Click Continue. After a few seconds, the Environment pane of the creation process will appear.
To create a new environment for the workload, click the + New Environment button in the top right of the Environment pane. The Environment creation form will open.
In the Environment creation form, enter a name for the environment, such as s3-download-env, and optionally add a description.
Under Image URL, put nvcr.io/nvidia/pytorch:24.08-py3.
If desired, the image pull policy can be changed. The default value of “if not already present” is recommended for most cases, but if you are pushing new containers to the NGC org with the same tag, “always pull the image from the registry” should be used to check for updates to the image.
Under the Workload architecture & type pane, select Standard and Training if they are not selected already. These options are likely already selected and grayed out.
Do not modify any other settings, including the Runtime settings section—we will fill that in for the specific workload, keeping this environment multipurpose for future workloads.
Click Create Environment in the bottom right of the page. You should see a pop-up saying that your environment has been created. The page will refresh, and you will be taken back to the top of the Environment pane of the New training creation page. The environment you just created should now be selected.
Open the Runtime settings section for this training workload. We will execute a sequence of several commands within this workload. Set the Command field to bash -c, and the Arguments field to "wget https://github.com/peak/s5cmd/releases/download/v2.2.2/s5cmd_2.2.2_Linux-64bit.tar.gz; tar -xf s5cmd_2.2.2_Linux-64bit.tar.gz; ./s5cmd cp s3://your-s3-bucket-name-here/* /dataset"
To embed the Credentials into the workload, click the + ENVIRONMENT VARIABLE button twice.
For the first entry’s Name field, enter AWS_ACCESS_KEY_ID, set the Source to Credentials, set the Credentials name field to aws-access-keys, and set the Secret Key field to AccessKeyId.
For the second entry’s Name field, enter AWS_SECRET_ACCESS_KEY, set the Source to Credentials, set the Credentials name field to aws-access-keys, and set the Secret Key field to SecretKey.
Scroll down to the Compute resource pane and select your desired compute resource—ideally a CPU-only resource to avoid consuming GPU resources.
In the Data Sources section, select your s3-dataset PVC.
Scroll to the bottom of the page and click CREATE TRAINING. This will add your workload to the queue and it will automatically get scheduled once cluster resources are available.
After creating the training workload, you will be taken to the workload overview page, where you can view the status of your workload. Your training workload is running once the status reads “Running”. This may take a while if the container you chose has not previously been pulled to the cluster.
Once the status is “Completed”, you can review the outcome by clicking the workload and selecting the SHOW DETAILS button on the right side of the blue banner. Select the LOGS tab in the workload’s detail pane—it should indicate that data is being copied to your new PVC from your S3 bucket.

6.3.5. Cleaning up the Environment

After the workload has completed, it can be deleted from the Workloads page, or left present as a reference for a future data download workload.

6.4. Running Visual Studio Code Inside a Workload

In this example, we step through the process of creating a Workspace via the Run:ai Command-Line Interface (CLI) that runs a VS Code instance, and exposing the necessary port to our local workstation in order to access the VS Code web interface.

6.4.1. Prerequisites and Requirements

The following are required before running the VS Code workspace:

You must have the user role L1 researcher, ML Engineer, or System Administrator to be able to run through all sections of this tutorial.
Your user must be able to access a project and department.
You must have installed the Run:ai CLI tool, as detailed in the Advanced documentation.

6.4.2. Creating the Workspace via CLI on a local workstation

Open a terminal on your workstation (or other system that has the Run:ai CLI installed and configured on it).

Note

If the system you have installed the Run:ai CLI on is a remote system, you can SSH to that system and create a local port forward, connecting the port Run:ai will use with the same port on your local workstation. For example, ssh -L 8080:localhost:8080 the-ip-or-hostname-of-your-remote-system.
Make note of your target project.

Enter the following command and arguments into your terminal (inserting your target project as the argument to the --project flag) and hit enter.

runai submit vscode --project your-project-name-here --image quay.io/opendatahub-contrib/workbench-images:vscode-datascience-c9s-py311_2023c_latest --interactive

(Optional) Validate that the project is running with the following command.
runai describe job vscode --project your-project-name-here
Forward a port to your local workstation through the Run:ai CLI with the following command. The first port in the argument to port is arbitrary - it can be changed to any unused port on your system, instead of 8080.
runai port-forward vscode --project your-project-name-here --port 8080:8787
To access the VS Code web interface, navigate to http://localhost:8080.

6.4.3. Cleaning up the Environment

After you are finished with the workload, it can be stopped or deleted to free up resources for other workloads. To delete the workload from the CLI, use the following command.

runai delete --project demo-project job vscode

6.5. Distributed PyTorch Training Job

In this example, we demonstrate how to run a multi-node training job using the PyTorch training operator from Kubeflow.

6.5.1. Requirements

For running a Distributed PyTorch training job, a custom Docker container needs to be built. This is necessary to get relevant code into a container that can be run and shared for all future jobs.

To build a custom Docker container, the Docker runtime needs to be installed on a local machine with the same CPU architecture as the DGX Cloud hosts (i.e., if the DGX Cloud hosts have AMD64-based CPUs, the container needs to be built on an AMD64 machine. If the DGX Cloud hosts are arm-based, the container needs to be built on an ARM CPU). To install Docker locally, follow the Docker Engine Install guide.
Additionally, you need to authenticate with your NGC private registry using your NGC key created during the Credentials section above. To authenticate with NGC, run the following commands (note, the username must be exactly “$oauthtoken”):

  $ docker login nvcr.io
  Username: $oauthtoken
  Password: <NGC Key here>

6.5.2. Creating a Custom Docker Container

On your local machine where Docker is installed, create and navigate to a directory to save the Dockerfile, such as pytorch-distributed:
1 $ mkdir -p pytorch-distributed 2 $ cd pytorch-distributed
In the new directory, open a new file named run.sh and copy the following contents to the file:
1 #!/bin/bash 2 3 torchrun multinode.py --batch_size 32 100 25 4 cp snapshot.pt /checkpoints
This is a very simple script that uses torchrun to launch a distributed training job and copies the generated checkpoint to the /checkpoints directory inside the container so it can be used again later.
Save and close the file. Next, open another new file named Dockerfile and copy the following contents to the file:
1 FROM nvcr.io/nvidia/pytorch:24.07-py3 2 3 WORKDIR /runai-distributed 4 RUN git clone https://github.com/pytorch/examples 5 6 WORKDIR /runai-distributed/examples/distributed/ddp-tutorial-series 7 COPY run.sh .
This Dockerfile uses the 24.07 PyTorch container hosted on NGC as a base, clones the official PyTorch examples repository inside the container, and copies the run.sh file created previously into the container. Save and close the file.
Once both files have been saved locally, build a container with the following command, replacing <ORG ID> with the ID of your private registry on NGC:
docker build -t nvcr.io/<ORG ID>/pytorch-ddp-example:24.07-py3 .
This will build the custom container locally.
Once the build has finished, push the image to the NGC private registry with:
docker push nvcr.io/<ORG ID>/pytorch-ddp-example:24.07-py3
The custom container will be available in your private NGC registry and can be used immediately for jobs.

6.5.3. Creating a Data Source

To make it easier to reuse code and/or checkpoints in future jobs, a data source is created, specifically a PVC. The PVC can be mounted in jobs and will persist after the job completes so any data created can be reused.

To create a new PVC, go to the Data Sources page. Click New Data Source then PVC to open the PVC creation form.
On the new form, set the desired scope. Depending on the sensitivity of the data, the PVC can be limited in scope down to the project level or be shared more broadly with the whole org or department(s).
Give the PVC a memorable name like distributed-pytorch-checkpoint and add a description if desired.
For the data options, select a new PVC storage class that suits your needs according to the PVC recommendations here. In this example, zonal-rwx is sufficient. To allow all nodes to read and write from/to the PVC, select Read-write by many nodes for the access mode. Enter 10 TB for the size to ensure we have plenty of capacity for future jobs. Select Filesystem as the volume mode. Lastly, set the Container path to /checkpoints which is where the PVC will be mounted inside containers. This completed section should look as shown below.
Once the form is complete, create the data source using the button at the bottom of the page.

Note

The first time a job is launched using a new PVC, it will take longer to start as the storage gets provisioned only once the first claim to the PVC is made.

6.5.4. Creating the Workload via the UI

To create the training job, navigate to the Workloads page and click New Workload > Training.
On the project creation form page, select the desired project to run the job in.
Select Distributed for the workload architecture to run a multi-node job. This adds a drop-down menu which lists the available frameworks for running distributed jobs. Since we are running a PyTorch job, select the PyTorch framework.
For the distributed training configuration, there is an option for either Workers & master or Workers only.

Note

In this documentation, we will use the term ‘primary’ instead of ‘master’ to align with modern terminology. Please note that the UI and commands may still refer to it as ‘master’.
A primary node is typically required when running multi-node jobs that need to coordinate with one another, such as a job that does an all_reduce operation. The primary node can either be identical to the workers and run the training process or be a lightweight pod that only does coordination. If a job doesn’t need coordination between processes, the Workers only option can be used. For our example, we want the pods to coordinate with each other. Select the Workers & master option.
Ensure the Start from scratch option is selected for the template.
Give the job a name like “distributed-training-pytorch” and click Continue.
In the new environment form that opens, select New environment. In the new form, provide a name for the environment such as “distributed-pytorch-env” and optionally add a description.
In the image URL, provide the name of the image tag that was specified during the container build in the Creating a Custom Docker Container section above, such as nvcr.io/<ORG ID>/pytorch-ddp-example:24.07-py3. This will use the container that was pushed to your private NGC org for the job.
If desired, the image pull policy can be changed. The default value of “if not already present” is recommended for most cases, but if you are pushing new containers to the NGC org with the same tag, “always pull the image from the registry” should be used to check if there are updates to the image.
Most of the remaining settings on the form can be kept as default with the exception of the runtime settings. The runtime settings indicate the directory to open pods to and what command to launch inside the containers if something other than the default container values are desired.
In our example, we want to launch the multinode.py script with torchrun which runs a multi-process application where each process has its own unique rank. The PyTorch Training Operator coordinates with torchrun to automatically set the RANK, LOCAL_RANK, and WORLD_SIZE, among other environment variables, based on the total number of GPUs allocated for a job. In the Commands and Arguments field, enter torchrun multinode.py --batch_size 32 100 25 for the command. This will run the distributed training job on all allocated GPUs with a batch size of 32 for 100 total epochs and save a checkpoint every 25 epochs.
For the directory field, put /runai-distributed/examples/distributed/ddp-tutorial-series. This is the location of the scripts that were saved inside of the container we built and is the path we want the pod to open to inside the container when it starts.
After selecting the Create Environment button, you will be returned to the job creation page with the new environment selected. In the compute resource pane, select the desired number of workers. Note that since we are using a primary and worker configuration where we want the primary to participate in the training, we will specify one fewer worker node than the total number of nodes expected for the job. In other words, if we want to run an 8 node job where the primary will be coordinating processes as well as training, we will specify 7 workers and 1 primary (which is the next form). In our example we will train with 2 total nodes. As a result, select 1 worker in the form.
For the compute resource, we want to train on a full DGX node with all available GPU resources. Select the compute resource in your project with a GPU capacity of 8.
In the Data Sources pane, select the name of the PVC that was created in the previous section. The name of the PVC in this example is “distributed-pytorch-checkpoints”. Click Continue to navigate to the primary configuration page.
As mentioned previously, the primary pod can have a different configuration from the worker pods for distributed PyTorch jobs. For our example, we want to use the same setup for workers and the primary so ensure the Allow different setup for the master radio button is deselected.
Once finished, click Create Training to queue the training job.

6.5.5. Monitoring the Job

After the job is submitted, the status can be viewed in the Workloads page. Find the job name in the list, such as “distributed-training-pytorch” per the previous example. Clicking on the job will show a form with additional information about the job, including event history, metrics, and logs.

The event history tab shows the current status of the job, such as Creating, Initializing, Pending, Running, etc. Additionally, it shows a log with timestamps of all of the states over time including any output messages when applicable. This is useful to see when a job is running or if it failed for any reason.

The metrics tab contains several graphs showing pod resource usage, including GPU, CPU, and memory utilization. This helps determine how much a job is stressing the hardware resources or potentially seeing if the application is under-utilizing resources. At the top of the tab, there is a dropdown to select metrics for a specific pod or for an aggregate of all pods for finer granularity.

Lastly, the logs tab shows the live STDOUT for each of the pods, including the aggregate output. In our example application, there will be output showing the training progress, including the epoch, batch size, and steps on a per-GPU basis. The logs are a good place to view training progress or possibly see if there are any errors in the process.

6.5.6. Getting the Checkpoint

The end of the run.sh script we created copies the latest generated checkpoint to the PVC attached to the job. Any job that uses this same PVC will now be able to load the checkpoint at /checkpoints/snapshot.pt inside a job. Similarly, any data can be saved to the PVC in a job at the specified filesystem path. This is helpful while generating checkpoints in long-running training jobs to continue progress in future jobs or change hyperparameters between jobs.

6.5.7. Cleaning up the Environment

After the job finishes, it can be deleted to free up resources for other jobs. Note that certain jobs do not self-terminate, and jobs should be monitored to ensure resources are not sitting idle for completed jobs.

6.6. Using Zero Quota Projects for Lower Priority Workloads

In this example, we walk through the process of creating a zero quota project for lower priority workloads, ensuring that they will always be preemptible.

This example is intended for cluster administrators managing the usage of the cluster, and demonstrates that certain workloads and projects can be prioritized over others for mixed usage of the cluster. In this manner, lower priority workloads can be preempted when higher priority workloads need to be run, and resumed once higher priority workloads are completed.

Once the project and quota configuration is completed by the cluster administrator, users can then run workloads in their assigned projects that fit the lower or higher priority use cases, and their workloads will be scheduled and preempted as configured.

6.6.1. Requirements

Access to the default department or a department that has enough quota to use the entire cluster’s resources. For more information on configuring departments and projects, see Departments.
Ability to create projects and configure quotas. For more information on required user roles, see Cluster Users and Managing Users.

Note

Only users with the role of Application Administrator, Department Administrator, Editor, and Research Manager can create projects.
No workloads are currently running in the cluster (if other workloads are running, they may interfere with the scheduler behavior).

Note

The Run:ai scheduler provides more configuration capabilities, such as over-quota priority, that can influence the behavior of workloads on the cluster (such as if other workloads are also running). For more information about the Run:ai scheduler, refer to the Run:ai documentation.

6.6.2. Creating the Maximum Quota Project via the UI

In this section, we will create a project with enough quota to use the entire cluster’s compute resources.

Note

This is a setup step that only needs to be executed once. The maximum quota project will persist unless removed.

Navigate to the Projects page and click the + NEW PROJECT button.
In the Scope field, ensure the default department is selected (or an alternative department with the ability to use all of the cluster’s available resources).
Enter a name like full-quota-project in the Project name field.
Under Quota management, set the GPU devices field to use all GPUs available to the department, and set the CPUs (Cores) and CPU memory fields to Unlimited.
Change no other fields, and click the CREATE PROJECT button.

6.6.3. Creating the Zero Quota Project via the UI

In this section, we will create a project with zero quota. Workloads running in this project will still be able to use compute resources, but will always be over quota and subject to preemption.

Note

This is a setup step that only needs to be executed once. The zero quota project will persist unless removed.

Navigate to the Projects page and click the + NEW PROJECT button.
In the Scope field, ensure the same department is selected as the one used in the previous section.
Enter a name like zero-quota-project in the Project name field.
Under Quota management, set the GPU devices, CPUs (Cores), and CPU memory fields to 0.
Change no other fields, and click the CREATE PROJECT button.

6.6.4. Creating the Lower Priority Interactive Workload via the UI

In this section, we will create and start a workspace in the zero-quota project.

Note

The same scheduling behaviors exist when using a training workload in a zero quota project. Some of the exact steps and environments used will differ.

Navigate to the Workloads page and click the + NEW WORKLOAD button. Select the Workspace option.
Select the zero-quota-project option under the Projects section.
Select Start from scratch under the Template section.
Enter a name like zero-quota-workspace under the Workspace name section and click the CONTINUE button.
Select the jupyter-lab option under the Environment section.
Select the h100-1g option under the Compute resource section.
Open the General section and toggle the Allow the workspace to exceed the project's quota switch to on (it will appear blue when on). Otherwise, the workspace will not start.

Note

You can set a policy for workspaces to enable this option by default. To do so, set the spec.preemptible.rules.required: true parameter in an interactive workspace policy, following the instructions Workspaces Policy .

More details about applying policies via CLI can be found in Run:ai documentation Former Policies .
Click the CREATE WORKSPACE button.
The UI will automatically return to the Workloads page - observe the zero-quota-workspace workload enter a Running state.

6.6.5. Creating the Higher Priority Training Workload via the UI

In this section, we will create and start a training workload in the full quota project. This training workload will preempt the workload initiated in the zero quota project.

Navigate to the Workloads page and click the + NEW WORKLOAD button. Select the Training option.
Select the full-quota-project option under the Projects section.
Select the Distributed option under the Workload architecture section.
Select the PyTorch option when the Set the framework for distributed trainings dropdown menu appears in the Workload architectures section.
Select the Workers only option when the Set the distributed training configuration options appear in the Workload architectures section.
Select Start from scratch under the Template section.
Enter a name like full-quota-training under the Training name section and click the CONTINUE button.
Click the + NEW ENVIRONMENT button under the Environment section.
In the next screen, enter a name like simple-pytorch-environment under the Environment name & description section.
Set the Image URL field to nvcr.io/nvidia/pytorch:24.07-py3 under the Image section.
Click the CREATE ENVIRONMENT button.
Upon being returned to the previous screen, ensure that the simple-pytorch-environment that we just created is selected in the Environment section.
click the Runtime settings section.
Click the + COMMAND & ARGUMENTS button.
Enter sleep 5m under the Command field.
In the Set the number of workers for your training field, choose a number that uses all available GPU nodes in the cluster. For example, in a 32 GPU cluster, that number will be 4 (8 GPUs per node, 8*4=32).
Select the h100-8g option under the Compute resource section.
Click the CREATE TRAINING button.
The UI will automatically return to the Workloads page - observe the zero-quota-workspace workload entering a Pending state first, followed by the full-quota-training workload entering a Running state.
If the full-quota-training workload is completed or is stopped, the zero-quota-workspace workload will resume if there are free resources available in the cluster.

6.6.6. Cleaning up the Environment

All test workloads and projects used in this example can be safely deleted.