Skip to navigation
Workloads in NVIDIA Run:aiTrain Models Using TrainingQuick Starts

Run Your First Distributed Training

View as Markdown

This quick start provides a step-by-step walkthrough for running a PyTorch distributed training workload.

Distributed training is the ability to split the training of a model among multiple processors. Each processor is called a worker. Worker nodes work in parallel to speed up model training. There is also a master which coordinates the workers.

Note

Multi-GPU training and distributed training are two distinct concepts. Multi-GPU training uses multiple GPUs within a single node, whereas distributed training spans multiple nodes and typically requires coordination between them.

Prerequisites

Before you start, make sure:

  • You have created a project or have one created for you.
  • The project has an assigned quota of at least 1 GPU.

Note

Flexible workload submission is enabled by default. If unavailable, contact your administrator to enable it under General settings → Workloads → Flexible workload submission.

Step 1: Logging In

Browse to the provided NVIDIA Run:ai user interface and log in with your credentials.

Step 2: Submitting a Distributed Training Workload

  1. Go to the Workload manager → Workloads

  2. Click +NEW WORKLOAD and select Training

  3. Select under which cluster to create the workload

  4. Select the project in which your workload will run

  5. Under Workload architecture, select Distributed

  6. Select PyTorch as the distributed framework and the distributed training configuration to Worker & master

  7. Select Start from scratch to launch a new workload quickly

  8. Enter a name for the training workload (if the name already exists in the project, you will be requested to submit a different name)

  9. Click CONTINUE

    In the next step:

  10. Under Environment, enter the Image URL - kubeflow/pytorch-dist-mnist:latest

  11. Under Runtime settings, click +COMMAND & ARGUMENTS, then set the Arguments to --epochs 100.

  12. Under Compute resources, click the load icon. A side pane appears, displaying a list of available compute resources. Select the ‘small-fraction’ compute resource for your workload.

    • If ‘small-fraction’ is not displayed, follow the below steps to create a one-time compute resource configuration:
      • Set GPU devices per pod - 1
      • Enable GPU fractioning to set the GPU memory per device:
        • Select % (of device) - Fraction of a GPU device’s memory
        • Set the memory Request - 10 (the workload will allocate 10% of the GPU memory)
      • Optional: set the CPU compute per pod - 0.1 cores (default)
      • Optional: set the CPU memory per pod - 100 MB (default)

  13. Click CONTINUE

  14. Click CREATE TRAINING

Next Steps

  • Manage and monitor your newly created workload using the Workloads table.
  • After validating your training performance and results, deploy your model using inference.