WorkloadRun

WorkloadRun is a user-facing simplified API for running ad-hoc distributed workloads without going through the full Certification pipeline.

Example

apiVersion: nvcre.nvidia.com/v1alpha1
kind: WorkloadRun
metadata:
  name: nccl-all-reduce
spec:
  image: nvcr.io/nvidia/pytorch:26.01-py3
  framework:
    mpi:
      binary: /usr/local/bin/all_reduce_perf_mpi
      args: ["-b", "8", "-e", "32G", "-f", "2", "-n", "100"]
      mpirunPath: /usr/local/mpi/bin/mpirun
  numNodes: 4   # nodes per job group; all eligible nodes are partitioned into 4-node
                # jobs. For a single 4-node job, add orchestration.placement: Unpinned
  gangScheduler:
    schedulerName: kai-scheduler
    queue: high-priority
    # On a Run:ai cluster instead:
    #   schedulerName: runai-scheduler
    #   queueLabelKey: runai/queue
    #   queue: team-a   # must name an existing Run:ai queue
  target:
    nodeSelector:
      nvidia.com/gpu.present: "true"
  bandwidthMeasurement:
    logProfileRef: nccl-bandwidth
    testType: all_reduce
  goodputMeasurement:
    logProfileRef: megatron-training

Spec fields

<Warning> The entire spec is immutable after the WorkloadRun is created (a self == oldSelf transition rule on the CRD rejects every update with spec is immutable after creation). Once status.workflowRef is set the controller only mirrors the existing Workflow and never rebuilds it, so mutable fields would be silently ignored. To run with different inputs, delete the WorkloadRun and create a new one. </Warning>

Fields documented so far:

numNodes (shown in the example above) is the number of nodes per job group under the default Pinned placement, not a total: the orchestrator partitions all eligible nodes into groups of that size. Under placement: Unpinned it is the total size of the one job that runs.

target vs numNodes vs gpusPerNode

These three fields are often mistaken for one another. They answer different questions, and the separation is deliberate:

• target says which nodes are eligible. It is a set predicate, a label selector both the API server and the scheduler can evaluate. The default is every node labelled nvidia.com/gpu.present: "true".
• numNodes says how many of them to use. It is a count. A count cannot narrow a set, because it does not say which members to drop, so numNodes never rewrites target. To run against a narrower set, narrow target itself with a tighter nodeSelector, matchExpressions, or an explicit nodeNames list.
• gpusPerNode is a per-node resource request. It narrows eligibility only indirectly: nodes that cannot supply the requested GPU count are dropped at discovery, and the scheduler enforces the same request again at bind time.

Which nodes the job actually lands on is the scheduler’s decision under Unpinned, constrained to target by the node affinity carried onto every pod.

Placement

spec.orchestration.placement decides how many jobs a WorkloadRun runs and who picks the nodes.

Pinned is the default. Every eligible node is partitioned into groups of numNodes, and each job carries a required kubernetes.io/hostname affinity naming its group’s machines. A 12-node fleet with numNodes: 2 runs six jobs.

Unpinned runs exactly one job of exactly numNodes nodes, whatever the target matches, with no hostname affinity. The same fleet runs one two-node job and leaves the other ten untouched.

spec:
  numNodes: 4
  orchestration:
    testScale: full-scale
    placement: Unpinned

The requested size is honored exactly, or the run fails saying why. It is never clamped to fit the fleet.

testScale: full-scale has always been documented as “all nodes in a single group” and has never been implemented as anything but the default chunking. Pairing it with placement: Unpinned is what makes a single job actually happen. intra-node and intra-rack are rejected under Unpinned: both name a grouping strategy, which contradicts running one job.

On a contended cluster a multi-node Unpinned job hands placement to the scheduler, so some pods can bind while the rest stay Pending. Set spec.gangScheduler (KAI or Run:ai) when numNodes > 1 so the gang schedules whole or not at all.

See Workflow: Placement for the full contract, including how target reaches the pods in both modes.

Workload object labels

spec.workloadMetadata.labels sets labels on the workload object the generated Job creates — today a Kubeflow TrainJob. That object’s own metadata.labels is where Kueue selects a local queue and where Kubeflow’s KAI Scheduler guide documents the queue label, both read before any workload pod exists.

spec:
  workloadMetadata:
    labels:
      kueue.x-k8s.io/queue-name: gpu-team-a

Kueue is never inferred from gangScheduler, so an explicit label is the only route to its queue. These labels are not pod labels; see Job workload object labels for the full contract, reserved keys, limits, and immutability rules.

Interaction with gangScheduler

When gangScheduler is set, its resolved queue is placed on the workload object too, under gangScheduler.queueLabelKey (kai.scheduler/queue when unset). This is in addition to the existing Job-template and pod-template placement in the generated TrainingRuntime, and it aligns the submitted object with Kubeflow’s documented KAI placement.

<Note> This is a behavior change for existing configurations. A WorkloadRun that sets gangScheduler and no workloadMetadata now emits the resolved queue label — the explicit queue, or default-queue when omitted — on the TrainJob as well as on the runtime templates. </Note>

Setting that same key in workloadMetadata.labels is accepted when the value matches and rejected when it differs, naming the key, both values, and the field path. Because kueue.x-k8s.io/queue-name is a different key, a Kueue queue and a KAI or Run:ai queue coexist without interfering.

Persisted scheduling contract

When gangScheduler is configured, the generated Workflow records the resolved intent — scheduler name, queue, and queue label key, with defaults already filled in — on Workflow.spec.gangScheduler. That field sits outside jobTemplate, dependencies, and orchestration, so no override can reach it, and it is immutable in both presence and value.

After every override resolves, and before any dependency or Job is created, NVCRE checks the effective manifests against that intent:

• The workload object’s queue label is NVCRE’s to write, so an override that removed it gets it restored, and an override that changed it fails.
• Runtime queue labels and pod scheduler names are assertions over the TrainingRuntime dependency, which an operator may legitimately own. A missing or conflicting value fails and is never silently repaired — rewriting someone’s runtime override would hide the disagreement rather than surface it. TrainJob runtimePatches are folded in when computing the effective values, for launcher and worker templates alike.
• An override that repoints runtimeRef at a runtime NVCRE was not given also fails: it cannot vouch for a runtime it cannot see.

To run against a different queue, create a new WorkloadRun with a different gangScheduler. Offline nvcrectl workloadrun render without --platform leaves overrides unresolved, so its output keeps spec.gangScheduler as a template and defers the check to the point where the overrides resolve.

For example, with gangScheduler.queue: team-a, an override that only adds an unrelated label is accepted:

spec:
  overrides:
    - when:
        platform:
          equals: aws
      jobTemplate:
        spec:
          workloadMetadata:
            labels:
              cost-center: hpc

An override that moves the workload object to another queue is rejected, and nvcrectl workloadrun render --platform aws reports the conflict instead of writing a Workflow:

spec:
  overrides:
    - when:
        platform:
          equals: aws
      jobTemplate:
        spec:
          workloadMetadata:
            labels:
              kai.scheduler/queue: team-b

spec.jobTemplate.spec.workloadMetadata.labels["kai.scheduler/queue"]: conflicting label value "team-b", expected "team-a"

See Workflow: accepted and conflicting overrides for the runtime-side cases.

Status fields

Bandwidth and goodput measurement results are on the BandwidthMeasurement and GoodputMeasurement child resources, which reference this WorkloadRun’s underlying Job via spec.jobRef.