> This page is for NVIDIA Cluster Readiness Engine, version Latest · v0.7.0 (default).
> For other versions, use one of these documentation indexes:
> - Latest · v0.7.0 (default): https://docs.nvidia.com/cluster-readiness-engine/v0.7.0/llms.txt
> - v0.6.0: https://docs.nvidia.com/cluster-readiness-engine/v0.6.0/llms.txt
> - v0.5.0: https://docs.nvidia.com/cluster-readiness-engine/v0.5.0/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/cluster-readiness-engine/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/cluster-readiness-engine/_mcp/server.

# WorkloadRun

> CRD reference for the WorkloadRun resource.

`WorkloadRun` is a user-facing simplified API for running ad-hoc distributed workloads without going through the full Certification pipeline.

## Example

```yaml
apiVersion: nvcre.nvidia.com/v1alpha1
kind: WorkloadRun
metadata:
  name: nccl-all-reduce
spec:
  image: nvcr.io/nvidia/pytorch:26.01-py3
  framework:
    mpi:
      binary: /usr/local/bin/all_reduce_perf_mpi
      args: ["-b", "8", "-e", "32G", "-f", "2", "-n", "100"]
      mpirunPath: /usr/local/mpi/bin/mpirun
  numNodes: 4   # nodes per job group; all eligible nodes are partitioned into 4-node
                # jobs. For a single 4-node job, add orchestration.placement: Unpinned
  gangScheduler:
    schedulerName: kai-scheduler
    queue: high-priority
    # On a Run:ai cluster instead:
    #   schedulerName: runai-scheduler
    #   queueLabelKey: runai/queue
    #   queue: team-a   # must name an existing Run:ai queue
  target:
    nodeSelector:
      nvidia.com/gpu.present: "true"
  bandwidthMeasurement:
    logProfileRef: nccl-bandwidth
    testType: all_reduce
  goodputMeasurement:
    logProfileRef: megatron-training
```

## Spec fields

<Warning>
The entire `spec` is **immutable** after the WorkloadRun is created (a `self == oldSelf` transition rule on the CRD rejects every update with `spec is immutable after creation`). Once `status.workflowRef` is set the controller only mirrors the existing Workflow and never rebuilds it, so mutable fields would be silently ignored. To run with different inputs, delete the WorkloadRun and create a new one.
</Warning>

_Fields documented so far:_

| Field | Type | Description |
|-------|------|-------------|
| `env` | []EnvVar | Optional. Additional environment variables for the workload containers, merged with the auto-detected NCCL/platform defaults; a user-specified value overrides a default with the same name. Set as container-level env on the workload container (for MPI, on both the launcher and worker containers). For MPI, `mpirun` launches ranks on workers through `sshd`, which gives each session a fresh environment, so the merged env (defaults and `spec.env`) is also forwarded to the ranks with `mpirun -x NAME=value`; a `valueFrom` variable is forwarded as `-x NAME`, which `mpirun` reads from the launcher container's env. At the ranks a `spec.env` value overrides both the defaults and any value a platform override forwards with `-x`, and a `-x NAME` you pass in `spec.framework.mpi.mpiArgs` overrides `spec.env` |
| `gangScheduler` | GangSchedulerSpec | Optional. Opts workload pods into a gang-aware scheduler such as KAI Scheduler. When set, the scheduler name is injected as `schedulerName` into every workload pod template (for MPI, both launcher and worker pods) and the queue is applied as a label (`gangScheduler.queueLabelKey`, `kai.scheduler/queue` when unset) on both the Job template metadata and the pod template metadata, so the scheduler holds all pods until the entire gang can be placed. On a Run:ai cluster set `schedulerName: runai-scheduler` and `queueLabelKey: runai/queue`, and make `queue` name an existing Run:ai queue |
| `gangScheduler.schedulerName` | string | Required; minimum length 1. Name of the gang-aware scheduler to use (e.g., `kai-scheduler`, or `runai-scheduler` on a Run:ai cluster). Injected as `schedulerName` in each workload pod spec |
| `gangScheduler.queue` | string | Optional. Scheduler queue to submit the workload to; defaults to `default-queue` when unset. On a Run:ai cluster it must name an existing Run:ai queue. When non-empty, must be a valid Kubernetes label value: at most 63 characters, beginning and ending with an alphanumeric character, and containing only alphanumerics, hyphens, underscores, or dots (pattern `^$\|^[a-zA-Z0-9]([a-zA-Z0-9._-]*[a-zA-Z0-9])?$`) |
| `gangScheduler.queueLabelKey` | string | Optional. Label key the queue value is written under; defaults to `kai.scheduler/queue` when unset. Set it to `runai/queue` on a Run:ai cluster, which reads that label and ignores `kai.scheduler/queue`. When non-empty, must be a valid Kubernetes label key (qualified name): an optional DNS-subdomain prefix of at most 253 characters followed by `/`, then a name of at most 63 characters; at most 317 characters in total. `app.kubernetes.io/managed-by` and any key under `nvcre.nvidia.com/` are rejected because the resolved queue label is inserted into workload metadata |
| `workloadMetadata` | WorkloadMetadata | Optional. Labels applied to the generated workload object (the `TrainJob`), copied into the generated Workflow's `spec.jobTemplate.spec.workloadMetadata`. Use it for Kueue (`kueue.x-k8s.io/queue-name`) or any other integration keyed on the submitted object's labels. See [Workload object labels](#workload-object-labels) |
| `workloadMetadata.labels` | map[string]string | Optional. At most 32 entries after merging and after any `gangScheduler` queue label is inserted — exceeding the cap fails rather than dropping labels. When `gangScheduler` is set, the queue label consumes one entry, leaving room for at most 31 user-supplied labels. Keys must be valid Kubernetes label keys and values valid Kubernetes label values; `app.kubernetes.io/managed-by` and any key under `nvcre.nvidia.com/` are rejected |
| `nicResourceName` | string | Optional. Kubernetes extended resource name of the RDMA NIC devices to request on workload containers for on-prem GB200/GB300 and x86 HGX B200/B300 targets, for example `rdma/ib` or `nvidia.com/mlnxnics`. Must be a fully qualified extended resource name (domain, slash, and a name segment of at most 63 characters); the reserved `kubernetes.io` and `k8s.io` domains are rejected. When unset, the controller detects the name automatically: if exactly one candidate resource (`rdma/*` or `nvidia.com/mlnxnics`) is allocatable at the resolved `mlnxPerNode` count on every target node, it is requested; otherwise nothing is requested and a Normal `NICResourceDetection` event on the WorkloadRun explains what was found: no candidate on any node, candidates below the requested count (naming the count), or multiple qualifying candidates. Set the field to override detection or when detection is ambiguous; requesting a resource the nodes do not advertise at the requested count leaves pods permanently Pending. Offline `nvcrectl workloadrun render` (without `--dry-run`) has no cluster to inspect and requires the field. The per-container count always comes from `mlnxPerNode` (GB200, GB300, B200, and B300 default to 8; sites running a shared-device plugin should set `mlnxPerNode: 1`, or detection will refuse a pooled resource advertised as 1) |
| `mlnxPerNode` | int32 | Optional. Overrides the auto-detected Mellanox NIC count per node used by InfiniBand/RoCE platforms and as the `nicResourceName` request count. When unset, derived from GPU architecture and platform via the catalog's `gpu-defaults.yaml`: eight for H100, H200, A100, L40S, B200, B300, GB200, and GB300, four for L40, none for RTX PRO 6000, before platform overrides such as OCI's. Setting it to `0` is a supported opt-out rather than a request for zero devices: the Azure override stops requesting `nvidia.com/mlnxnics` altogether instead of emitting a count of zero. See [Architecture-specific resources](/cluster-readiness-engine/concepts/platform-detection-overrides#architecture-specific-resources) |
| `orchestration.placement` | string | Optional, `Pinned` or `Unpinned`. Empty means `Pinned`. Under `Unpinned` the run is a single job of exactly `numNodes` nodes, with no `kubernetes.io/hostname` affinity. Rejected together with `testScale: intra-node` or `intra-rack`, which are explicit requests for a grouping strategy. See [Placement](#placement) |

`numNodes` (shown in the example above) is the number of nodes **per job group** under the default `Pinned` placement, not a total: the orchestrator partitions all eligible nodes into groups of that size. Under `placement: Unpinned` it is the total size of the one job that runs.

### `target` vs `numNodes` vs `gpusPerNode`

These three fields are often mistaken for one another. They answer different questions, and the separation is deliberate:

- **`target` says which nodes are eligible.** It is a set predicate, a label selector both the API server and the scheduler can evaluate. The default is every node labelled `nvidia.com/gpu.present: "true"`.
- **`numNodes` says how many of them to use.** It is a count. A count cannot narrow a set, because it does not say *which* members to drop, so `numNodes` never rewrites `target`. To run against a narrower set, narrow `target` itself with a tighter `nodeSelector`, `matchExpressions`, or an explicit `nodeNames` list.
- **`gpusPerNode` is a per-node resource request.** It narrows eligibility only indirectly: nodes that cannot supply the requested GPU count are dropped at discovery, and the scheduler enforces the same request again at bind time.

Which nodes the job actually lands on is the scheduler's decision under `Unpinned`, constrained to `target` by the node affinity carried onto every pod.

## Placement

`spec.orchestration.placement` decides how many jobs a WorkloadRun runs and who picks the nodes.

**`Pinned`** is the default. Every eligible node is partitioned into groups of `numNodes`, and each job carries a required `kubernetes.io/hostname` affinity naming its group's machines. A 12-node fleet with `numNodes: 2` runs six jobs.

**`Unpinned`** runs exactly one job of exactly `numNodes` nodes, whatever the target matches, with no hostname affinity. The same fleet runs one two-node job and leaves the other ten untouched.

```yaml
spec:
  numNodes: 4
  orchestration:
    testScale: full-scale
    placement: Unpinned
```

The requested size is honored exactly, or the run fails saying why. It is never clamped to fit the fleet.

`testScale: full-scale` has always been documented as "all nodes in a single group" and has never been implemented as anything but the default chunking. Pairing it with `placement: Unpinned` is what makes a single job actually happen. `intra-node` and `intra-rack` are rejected under `Unpinned`: both name a grouping strategy, which contradicts running one job.

On a contended cluster a multi-node `Unpinned` job hands placement to the scheduler, so some pods can bind while the rest stay Pending. Set `spec.gangScheduler` (KAI or Run:ai) when `numNodes > 1` so the gang schedules whole or not at all.

See [Workflow: Placement](/cluster-readiness-engine/api-reference/workflow#placement) for the full contract, including how `target` reaches the pods in both modes.

## Workload object labels

`spec.workloadMetadata.labels` sets labels on the workload object the generated Job creates — today a Kubeflow `TrainJob`. That object's own `metadata.labels` is where Kueue selects a local queue and where Kubeflow's KAI Scheduler guide documents the queue label, both read before any workload pod exists.

```yaml
spec:
  workloadMetadata:
    labels:
      kueue.x-k8s.io/queue-name: gpu-team-a
```

Kueue is never inferred from `gangScheduler`, so an explicit label is the only route to its queue. These labels are not pod labels; see [Job workload object labels](/cluster-readiness-engine/api-reference/job#workload-object-labels) for the full contract, reserved keys, limits, and immutability rules.

### Interaction with `gangScheduler`

When `gangScheduler` is set, its resolved queue is placed on the workload object too, under `gangScheduler.queueLabelKey` (`kai.scheduler/queue` when unset). This is **in addition to** the existing Job-template and pod-template placement in the generated `TrainingRuntime`, and it aligns the submitted object with Kubeflow's documented KAI placement.

<Note>
This is a behavior change for existing configurations. A WorkloadRun that sets `gangScheduler` and no `workloadMetadata` now emits the resolved queue label — the explicit `queue`, or `default-queue` when omitted — on the `TrainJob` as well as on the runtime templates.
</Note>

Setting that same key in `workloadMetadata.labels` is accepted when the value matches and rejected when it differs, naming the key, both values, and the field path. Because `kueue.x-k8s.io/queue-name` is a different key, a Kueue queue and a KAI or Run:ai queue coexist without interfering.

### Persisted scheduling contract

When `gangScheduler` is configured, the generated Workflow records the resolved intent — scheduler name, queue, and queue label key, with defaults already filled in — on `Workflow.spec.gangScheduler`. That field sits outside `jobTemplate`, `dependencies`, and `orchestration`, so no override can reach it, and it is immutable in both presence and value.

After every override resolves, and before any dependency or Job is created, NVCRE checks the effective manifests against that intent:

- The workload object's queue label is NVCRE's to write, so an override that **removed** it gets it restored, and an override that **changed** it fails.
- Runtime queue labels and pod scheduler names are assertions over the `TrainingRuntime` dependency, which an operator may legitimately own. A missing or conflicting value **fails and is never silently repaired** — rewriting someone's runtime override would hide the disagreement rather than surface it. `TrainJob` `runtimePatches` are folded in when computing the effective values, for launcher and worker templates alike.
- An override that repoints `runtimeRef` at a runtime NVCRE was not given also fails: it cannot vouch for a runtime it cannot see.

To run against a different queue, create a new WorkloadRun with a different `gangScheduler`. Offline `nvcrectl workloadrun render` without `--platform` leaves overrides unresolved, so its output keeps `spec.gangScheduler` as a template and defers the check to the point where the overrides resolve.

For example, with `gangScheduler.queue: team-a`, an override that only adds an unrelated label is accepted:

```yaml
spec:
  overrides:
    - when:
        platform:
          equals: aws
      jobTemplate:
        spec:
          workloadMetadata:
            labels:
              cost-center: hpc
```

An override that moves the workload object to another queue is rejected, and `nvcrectl workloadrun render --platform aws` reports the conflict instead of writing a Workflow:

```yaml
spec:
  overrides:
    - when:
        platform:
          equals: aws
      jobTemplate:
        spec:
          workloadMetadata:
            labels:
              kai.scheduler/queue: team-b
```

```text
spec.jobTemplate.spec.workloadMetadata.labels["kai.scheduler/queue"]: conflicting label value "team-b", expected "team-a"
```

See [Workflow: accepted and conflicting overrides](/cluster-readiness-engine/api-reference/workflow#accepted-and-conflicting-overrides) for the runtime-side cases.

## Status fields

| Field | Type | Description |
|-------|------|-------------|
| `conditions` | []Condition | Exclusive set: `InProgress`, `Succeeded`, `Failed`. Independent (additive): `ValidationFailed` — mirrored from the Workflow whenever any Job violated a performance threshold. When the threshold miss is the failure cause, `Failed` carries reason `WorkflowValidationFailed` so a threshold miss is distinguishable from an execution failure |
| `workflowRef` | WorkflowReference | Reference to the underlying `Workflow` resource |
| `detectedGPUArchitecture` | string | Auto-detected GPU type (e.g., `h100`, `gb200`) |
| `detectedPlatform` | string | Auto-detected CSP platform (e.g., `aws`, `gcp`, `azure`) |
| `resolvedGpusPerNode` | int32 | Final GPU count per node used for the workload |
| `succeededNodesRef` | TypedLocalObjectReference | ConfigMap reference for the succeeded-nodes list |
| `failedNodesRef` | TypedLocalObjectReference | ConfigMap reference for the failed-nodes list |

Bandwidth and goodput measurement results are on the `BandwidthMeasurement` and `GoodputMeasurement` child resources, which reference this WorkloadRun's underlying Job via `spec.jobRef`.