Run a WorkloadRun

WorkloadRun lets you run any distributed workload — training, NCCL benchmark, or custom script — without the full certification pipeline.

Basic example

1
apiVersion: cre.nvidia.com/v1alpha1
2
kind: WorkloadRun
3
metadata:
4
  name: my-workload
5
spec:
6
  image: nvcr.io/nvidia/pytorch:26.01-py3
7
  framework:
8
    mpi:
9
      mpirunPath: /usr/local/mpi/bin/mpirun
10
      binary: /usr/local/bin/all_reduce_perf_mpi
11
      args: ["-b", "8", "-e", "32G", "-f", "2", "-n", "100"]
12
  numNodes: 4

numNodes is the number of nodes per job group, not a total: the orchestrator partitions all eligible nodes into groups of that size. For example, numNodes: 4 on 16 eligible nodes produces four 4-node jobs.

$
nvcrectl workloadrun run \
>
  --workload-registry nvcr.io \
>
  --workload-registry-username '$oauthtoken' \
>
  --workload-registry-password "$NGC_API_KEY" \
>
  --wait my-workload.yaml

Targeting specific nodes

1
spec:
2
  target:
3
    nodeSelector:
4
      kubernetes.io/hostname: gpu-node-01

With bandwidth measurement

1
spec:
2
  bandwidthMeasurement:
3
    logProfileRef: nccl-bandwidth
4
    testType: all_reduce

With goodput measurement

1
spec:
2
  goodputMeasurement:
3
    logProfileRef: megatron-training

Gang scheduling

Distributed workloads can deadlock under the default scheduler when only some of their pods fit on the cluster: the placed pods hold GPUs while waiting for peers that never arrive. Set spec.gangScheduler to opt every workload pod into a gang-aware scheduler, such as KAI Scheduler, which holds all pods until the entire gang can be placed at once.

1
apiVersion: cre.nvidia.com/v1alpha1
2
kind: WorkloadRun
3
metadata:
4
  name: gang-scheduled-workload
5
spec:
6
  image: nvcr.io/nvidia/pytorch:26.01-py3
7
  framework:
8
    mpi:
9
      mpirunPath: /usr/local/mpi/bin/mpirun
10
      binary: /usr/local/bin/all_reduce_perf_mpi
11
      args: ["-b", "8", "-e", "32G", "-f", "2", "-n", "100"]
12
  numNodes: 4          # nodes per job group; all eligible nodes are partitioned into 4-node jobs
13
  gangScheduler:
14
    schedulerName: kai-scheduler   # required
15
    queue: high-priority           # optional; defaults to "default-queue"

schedulerName is required. queue is optional and defaults to default-queue; when set, it must be a valid Kubernetes label value (at most 63 characters, beginning and ending with an alphanumeric character, containing only alphanumerics, hyphens, underscores, or dots).

When gangScheduler is set, CRE modifies every pod template generated for the workload — for MPI frameworks that includes both the launcher and the worker pods:

• The configured scheduler name is injected as schedulerName in each pod spec, so the pods bypass the default scheduler.
• The queue is applied as the kai.scheduler/queue label on the pod template metadata, so a gang-aware scheduler can hold all pods in the gang until they can be placed together.

See API Reference: WorkloadRun for validation details.

Platform overrides

The controller detects the platform (from spec.providerID) and GPU architecture (from the nvidia.com/gpu.product node label) and applies platform-specific overrides automatically — the same _lib/ fragments the certification catalog uses. For MPI workloads, override mpiArgs are prepended to the launcher command ahead of your own spec.framework.mpi.mpiArgs, so your values still win under OpenMPI’s duplicate-parameter handling.

Use nvcrectl workloadrun render --platform aws my-workload.yaml to preview the exact rendered Workflow, including the platform-applied mpirun args.

View results

$
nvcrectl workloadrun report my-workload

See API Reference: WorkloadRun for the full spec.