API Reference

WorkloadRun

View as Markdown

WorkloadRun is a user-facing simplified API for running ad-hoc distributed workloads without going through the full Certification pipeline.

Example

1apiVersion: cre.nvidia.com/v1alpha1
2kind: WorkloadRun
3metadata:
4  name: nccl-all-reduce
5spec:
6  image: nvcr.io/nvidia/pytorch:26.01-py3
7  framework:
8    mpi:
9      binary: /usr/local/bin/all_reduce_perf_mpi
10      args: ["-b", "8", "-e", "32G", "-f", "2", "-n", "100"]
11      mpirunPath: /usr/local/mpi/bin/mpirun
12  numNodes: 4   # nodes per job group; all eligible nodes are partitioned into 4-node jobs
13  gangScheduler:
14    schedulerName: kai-scheduler
15    queue: high-priority
16  target:
17    nodeSelector:
18      nvidia.com/gpu.present: "true"
19  bandwidthMeasurement:
20    logProfileRef: nccl-bandwidth
21    testType: all_reduce
22  goodputMeasurement:
23    logProfileRef: megatron-training

Spec fields

Generated from CRD schema — coming soon. Fields documented so far:

FieldTypeDescription
gangSchedulerGangSchedulerSpecOptional. Opts workload pods into a gang-aware scheduler such as KAI Scheduler. When set, the scheduler name is injected as schedulerName into every workload pod template (for MPI, both launcher and worker pods) and the queue is applied as the kai.scheduler/queue label on the pod template metadata, so the scheduler holds all pods until the entire gang can be placed
gangScheduler.schedulerNamestringRequired; minimum length 1. Name of the gang-aware scheduler to use (e.g., kai-scheduler). Injected as schedulerName in each workload pod spec
gangScheduler.queuestringOptional. Scheduler queue to submit the workload to; defaults to default-queue when unset. When non-empty, must be a valid Kubernetes label value: at most 63 characters, beginning and ending with an alphanumeric character, and containing only alphanumerics, hyphens, underscores, or dots (pattern ^$|^[a-zA-Z0-9]([a-zA-Z0-9._-]*[a-zA-Z0-9])?$)

numNodes (shown in the example above) is the number of nodes per job group, not a total: the orchestrator partitions all eligible nodes into groups of that size.

Status fields

FieldTypeDescription
conditions[]ConditionExclusive set: InProgress, Succeeded, Failed. Independent (additive): ValidationFailed (can be True alongside Succeeded — workload finished but violated a threshold)
workflowRefWorkflowReferenceReference to the underlying Workflow resource
detectedGPUArchitecturestringAuto-detected GPU type (e.g., h100, gb200)
detectedPlatformstringAuto-detected CSP platform (e.g., aws, gcp, azure)
resolvedGpusPerNodeint32Final GPU count per node used for the workload
succeededNodesRefTypedLocalObjectReferenceConfigMap reference for the succeeded-nodes list
failedNodesRefTypedLocalObjectReferenceConfigMap reference for the failed-nodes list

Bandwidth and goodput measurement results are on the BandwidthMeasurement and GoodputMeasurement child resources, which reference this WorkloadRun’s underlying Job via spec.jobRef.