> This page is for Self-Hosted, version v3.0 (default).
> For other versions, use one of these documentation indexes:
> - v3.0 (default): https://docs.nvidia.com/run-ai/self-hosted/v3.0/llms.txt
> - v2.26: https://docs.nvidia.com/run-ai/self-hosted/v2.26/llms.txt
> - v2.25: https://docs.nvidia.com/run-ai/self-hosted/v2.25/llms.txt
> - v2.24: https://docs.nvidia.com/run-ai/self-hosted/v2.24/llms.txt
> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/run-ai/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/run-ai/_mcp/server.
# Run Your First Distributed Training
This quick start provides a step-by-step walkthrough for running a PyTorch distributed training workload.
Distributed training is the ability to split the training of a model among multiple processors. Each processor is called a worker. Worker nodes work in parallel to speed up model training. There is also a master which coordinates the workers.
**Note**
Multi-GPU training and distributed training are two distinct concepts. Multi-GPU training uses multiple GPUs within a single node, whereas distributed training spans multiple nodes and typically requires coordination between them.
## Prerequisites
Before you start, make sure:
* You have created a [project](/run-ai/self-hosted/documentation/platform-management/manage-ai-initiatives/organization/projects) or have one created for you.
* The project has an assigned quota of at least 1 GPU.
**Note**
Flexible workload submission is enabled by default. If unavailable, contact your administrator to enable it under **General settings** → Workloads → Flexible workload submission.
## Step 1: Logging In
Browse to the provided NVIDIA Run:ai user interface and log in with your credentials.
Run the below --help command to obtain the login options and log in according to your setup:
```sh
runai login --help
```
To use the API, you will need to obtain a token as shown in [API authentication](/run-ai/self-hosted/reference/rest-api/getting-started/how-to-authenticate-to-the-api).
## Step 2: Submitting a Distributed Training Workload
1. Go to the Workload manager → Workloads
2. Click **+NEW WORKLOAD** and select **Training**
3. Select under which **cluster** to create the workload
4. Select the **project** in which your workload will run
5. Under **Workload architecture,** select **Distributed**
6. Select **PyTorch** as the distributed framework and the distributed training configuration to **Worker & master**
7. Select **Start from scratch** to launch a new workload quickly
8. Enter a **name** for the training workload (if the name already exists in the project, you will be requested to submit a different name)
9. Click **CONTINUE**
In the next step:
10. Under **Environment**, enter the **Image URL** - `kubeflow/pytorch-dist-mnist:latest`
11. Under **Runtime settings**, click **+COMMAND & ARGUMENTS**, then set the **Arguments** to `--epochs 100`.
12. Under **Compute resources**, click the **load** icon. A side pane appears, displaying a list of available compute resources. Select the **‘small-fraction’** compute resource for your workload.
* If ‘small-fraction’ is not displayed, follow the below steps to create a one-time compute resource configuration:
* Set **GPU devices** per pod - 1
* Enable **GPU fractioning** to set the GPU memory per device:
* Select **% (of device)** - Fraction of a GPU device’s memory
* Set the memory **Request** - 10 (the workload will allocate 10% of the GPU memory)
* Optional: set the **CPU compute per pod** - 0.1 cores (default)
* Optional: set the **CPU memory per pod** - 100 MB (default)
13. Click **CONTINUE**
14. Click **CREATE TRAINING**
1. Go to the Workload manager → Workloads
2. Click **+NEW WORKLOAD** and select **Training**
3. Select under which **cluster** to create the workload
4. Select the **project** in which your workload will run
5. Under **Workload architecture,** select **Distributed**
6. Select **PyTorch** as the distributed framework and the distributed training configuration to **Worker & master**
7. Select **Start from scratch** to launch a new workload quickly
8. Enter a **name** for the training workload (if the name already exists in the project, you will be requested to submit a different name)
9. Click **CONTINUE**
In the next step:
10. Create an environment for your workload
* Click **+NEW ENVIRONMENT**
* Enter pytorch-dt as the **name** for the environment. The name must be unique.
* Enter `kubeflow/pytorch-dist-mnist:latest` as the **Image URL**
* Set the runtime settings for the environment. Click **+COMMAND & ARGUMENTS** and enter the arguments - `--epochs 100`
* Click **CREATE ENVIRONMENT**
The newly created environment will be selected automatically
11. Select the **‘small-fraction’** compute resource for your workload
* If ‘small-fraction’ is not displayed in the gallery, follow the below steps:
* Click **+NEW COMPUTE RESOURCE**
* Enter small-fraction as the **name** for the compute resource. The name must be unique.
* Set **GPU devices** per pod - 1
* Enable **GPU fractioning** to set the GPU memory per device:
* Select **% (of device)** - Fraction of a GPU device’s memory
* Set the memory **Request** - 10 (the workload will allocate 10% of the GPU memory)
* Optional: set the **CPU compute per pod** - 0.1 cores (default)
* Optional: set the **CPU memory per pod** - 100 MB (default)
* Click **CREATE COMPUTE RESOURCE**
The newly created compute resource will be selected automatically
12. Click **CONTINUE**
13. Click **CREATE TRAINING**
Copy the following command to your terminal. Make sure to update the below with the name of your project and workload. For more details, see [CLI reference](/run-ai/self-hosted/reference/cli/runai/training/runai-training-pytorch-submit):
```sh
runai project set "project-name"
runai training pytorch submit "workload-name" \
-i kubeflow/pytorch-dist-mnist:latest --workers 2 \
--gpu-request-type portion --gpu-portion-request 0.1 \
--gpu-devices-request 1 --cpu-memory-request 100M \
-- --epochs 100
```
Copy the following command to your terminal. Make sure to update the following parameters. For more details, see [Distributed](https://docs.nvidia.com/run-ai/self-hosted/v3.0/reference/rest-api/endpoints/workloads/distributed/create-distributed) API:
```bash
curl -L 'https:///api/v1/workloads/distributed' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer ' \
-d '{
"name": "workload-name",
"projectId": "",
"clusterId": "",
"spec": {
"compute": {
"cpuCoreRequest": 0.1,
"gpuRequestType": "portion",
"cpuMemoryRequest": "100M",
"gpuDevicesRequest": 1,
"gpuPortionRequest": 0.1
},
"image": "kubeflow/pytorch-dist-mnist:latest",
"args": "--epochs 100",
"numWorkers": 2,
"distributedFramework": "PyTorch"
},
"masterSpecSameAsWorker": true
}
```
* `` - The link to the NVIDIA Run:ai user interface
* `` - The API access token obtained in [Step 1](/run-ai/self-hosted/documentation/workloads-in-nvidia-run-ai/using-training/quick-starts/distributed-training-quickstart#step-1-logging-in)
* `` - The ID of the Project the workload is running on. You can get the Project ID via the [Get Projects](https://docs.nvidia.com/run-ai/self-hosted/v3.0/reference/rest-api/endpoints/organizations/projects/get-projects) API.
* `` - The unique identifier of the Cluster. You can get the Cluster UUID via the [Get Clusters](https://docs.nvidia.com/run-ai/multi-tenant/v3.0/reference/rest-api/endpoints/organizations/clusters/get-clusters) API.
**Note**
The above API snippet runs with NVIDIA Run:ai clusters of 2.18 and above only.
## Next Steps
* Manage and monitor your newly created workload using the [Workloads](/run-ai/self-hosted/documentation/workloads-in-nvidia-run-ai/workloads) table.
* After validating your training performance and results, deploy your model using [inference](/run-ai/self-hosted/documentation/workloads-in-nvidia-run-ai/using-inference/inference-overview).