WorkloadRun

WorkloadRun is a user-facing simplified API for running ad-hoc distributed workloads without going through the full Certification pipeline.

Example

apiVersion: nvcre.nvidia.com/v1alpha1
kind: WorkloadRun
metadata:
  name: nccl-all-reduce
spec:
  image: nvcr.io/nvidia/pytorch:26.01-py3
  framework:
    mpi:
      binary: /usr/local/bin/all_reduce_perf_mpi
      args: ["-b", "8", "-e", "32G", "-f", "2", "-n", "100"]
      mpirunPath: /usr/local/mpi/bin/mpirun
  numNodes: 4   # nodes per job group; all eligible nodes are partitioned into 4-node jobs
  gangScheduler:
    schedulerName: kai-scheduler
    queue: high-priority
    # On a Run:ai cluster instead:
    #   schedulerName: runai-scheduler
    #   queueLabelKey: runai/queue
    #   queue: team-a   # must name an existing Run:ai queue
  target:
    nodeSelector:
      nvidia.com/gpu.present: "true"
  bandwidthMeasurement:
    logProfileRef: nccl-bandwidth
    testType: all_reduce
  goodputMeasurement:
    logProfileRef: megatron-training

Spec fields

<Warning> The entire spec is immutable after the WorkloadRun is created (a self == oldSelf transition rule on the CRD rejects every update with spec is immutable after creation). Once status.workflowRef is set the controller only mirrors the existing Workflow and never rebuilds it, so mutable fields would be silently ignored. To run with different inputs, delete the WorkloadRun and create a new one. </Warning>

Fields documented so far:

numNodes (shown in the example above) is the number of nodes per job group, not a total: the orchestrator partitions all eligible nodes into groups of that size.

Workload object labels

spec.workloadMetadata.labels sets labels on the workload object the generated Job creates — today a Kubeflow TrainJob. That object’s own metadata.labels is where Kueue selects a local queue and where Kubeflow’s KAI Scheduler guide documents the queue label, both read before any workload pod exists.

spec:
  workloadMetadata:
    labels:
      kueue.x-k8s.io/queue-name: gpu-team-a

Kueue is never inferred from gangScheduler, so an explicit label is the only route to its queue. These labels are not pod labels; see Job workload object labels for the full contract, reserved keys, limits, and immutability rules.

Interaction with gangScheduler

When gangScheduler is set, its resolved queue is placed on the workload object too, under gangScheduler.queueLabelKey (kai.scheduler/queue when unset). This is in addition to the existing Job-template and pod-template placement in the generated TrainingRuntime, and it aligns the submitted object with Kubeflow’s documented KAI placement.

<Note> This is a behavior change for existing configurations. A WorkloadRun that sets gangScheduler and no workloadMetadata now emits the resolved queue label — the explicit queue, or default-queue when omitted — on the TrainJob as well as on the runtime templates. </Note>

Setting that same key in workloadMetadata.labels is accepted when the value matches and rejected when it differs, naming the key, both values, and the field path. Because kueue.x-k8s.io/queue-name is a different key, a Kueue queue and a KAI or Run:ai queue coexist without interfering.

Persisted scheduling contract

When gangScheduler is configured, the generated Workflow records the resolved intent — scheduler name, queue, and queue label key, with defaults already filled in — on Workflow.spec.gangScheduler. That field sits outside jobTemplate, dependencies, and orchestration, so no override can reach it, and it is immutable in both presence and value.

After every override resolves, and before any dependency or Job is created, NVCRE checks the effective manifests against that intent:

• The workload object’s queue label is NVCRE’s to write, so an override that removed it gets it restored, and an override that changed it fails.
• Runtime queue labels and pod scheduler names are assertions over the TrainingRuntime dependency, which an operator may legitimately own. A missing or conflicting value fails and is never silently repaired — rewriting someone’s runtime override would hide the disagreement rather than surface it. TrainJob runtimePatches are folded in when computing the effective values, for launcher and worker templates alike.
• An override that repoints runtimeRef at a runtime NVCRE was not given also fails: it cannot vouch for a runtime it cannot see.

To run against a different queue, create a new WorkloadRun with a different gangScheduler. Offline nvcrectl workloadrun render without --platform leaves overrides unresolved, so its output keeps spec.gangScheduler as a template and defers the check to the point where the overrides resolve.

For example, with gangScheduler.queue: team-a, an override that only adds an unrelated label is accepted:

spec:
  overrides:
    - when:
        platform:
          equals: aws
      jobTemplate:
        spec:
          workloadMetadata:
            labels:
              cost-center: hpc

An override that moves the workload object to another queue is rejected, and nvcrectl workloadrun render --platform aws reports the conflict instead of writing a Workflow:

spec:
  overrides:
    - when:
        platform:
          equals: aws
      jobTemplate:
        spec:
          workloadMetadata:
            labels:
              kai.scheduler/queue: team-b

spec.jobTemplate.spec.workloadMetadata.labels["kai.scheduler/queue"]: conflicting label value "team-b", expected "team-a"

See Workflow: accepted and conflicting overrides for the runtime-side cases.

Status fields

Bandwidth and goodput measurement results are on the BandwidthMeasurement and GoodputMeasurement child resources, which reference this WorkloadRun’s underlying Job via spec.jobRef.