Certification

Certification is the top-level resource that defines a suite of certification categories to run against a GPU node pool.

Example

apiVersion: nvcre.nvidia.com/v1alpha1
kind: Certification
metadata:
  name: gpu-cluster-cert
  namespace: nvcre
spec:
  target:
    nodeSelector:
      nvidia.com/gpu.present: "true"
  enableMNNVL: false
  gangScheduler:
    schedulerName: kai-scheduler
    queue: high-priority
    # On a Run:ai cluster instead:
    #   schedulerName: runai-scheduler
    #   queueLabelKey: runai/queue
    #   queue: team-a   # must name an existing Run:ai queue
  categories:
    - domain: communication
      variant: nccl-all-reduce
    - domain: training
      variant: nemotron5-8b
      options:
        maxSteps: 50
        nodesPerJob: 8
        # Optional: override the DGX-class CPU/memory defaults
        # (limits: cpu "128" / memory 800Gi; requests: cpu "64" / memory 500Gi)
        # so training pods can schedule on smaller GPU nodes.
        resources:
          limits:
            cpu: "6"
            memory: 48Gi
          requests:
            cpu: "4"
            memory: 32Gi
        # Optional: when the category falls back to cloning its source
        # checkout (nothing pre-seeded in the workspace and nothing shipped
        # in the image at /opt/megatron-lm), clone from an internal mirror
        # instead of the default upstream. For the nemotron5 entries the
        # source is Megatron-LM; for air-gapped clusters, prefer baking the
        # source into the workload image so no clone runs at all.
        # sourceRepo: https://git.example.com/mirrors/Megatron-LM.git

Spec fields

Fields documented so far:

Like the rest of spec, gangScheduler is immutable after the Certification is created.

When gangScheduler is set, its resolved queue is also placed on each category’s generated workload object (the TrainJob), under gangScheduler.queueLabelKey, in addition to the runtime Job- and pod-template placement described above. See Workload object labels.

Workload object labels

workloadMetadata sets labels on the workload object each category’s Job creates — today a Kubeflow TrainJob. That object’s own metadata.labels is where Kueue selects a local queue and where Kubeflow’s KAI Scheduler guide documents the queue label, both read before any workload pod exists.

Because CertificationSpec inlines the shared category options, the one field is read at two paths: globally at spec.workloadMetadata and per category at spec.categories[].options.workloadMetadata.

spec:
  workloadMetadata:
    labels:
      environment: burn-in
  categories:
    - domain: communication
      variant: nccl-all-reduce
      options:
        workloadMetadata:
          labels:
            kueue.x-k8s.io/queue-name: nccl-queue

That nccl-all-reduce category’s TrainJob receives both environment: burn-in and kueue.x-k8s.io/queue-name: nccl-queue.

These labels are not pod labels. See Job workload object labels for the full contract, reserved keys, limits, and immutability rules.

Precedence

Unlike the other maps, slices, and pointers in category options — thresholds, resources, imagePullSecrets — workload labels merge per key rather than replacing the whole value. The order is:

1. labels already present in the resolved catalog job spec,
2. global spec.workloadMetadata.labels,
3. per-category options.workloadMetadata.labels, and
4. the resolved gangScheduler queue label.

Steps 2 and 3 replace earlier values by key. Step 4 does not overwrite: a missing key is inserted, the same value is accepted, and a different value fails.

The consequence of merging is that a category can change or add a key but cannot remove a global one — there is no deletion syntax, and an empty string is a real label value. A category needing a wholly different map should not set those keys globally.

Per-category queue conflicts

spec.gangScheduler is global, so every category receives the same resolved queue, including the default-queue default. A category that sets that same KAI or Run:ai key to a different value fails, naming the key, both values, and the field path. Per-category Kueue queues remain independent, because kueue.x-k8s.io/queue-name is a different key.

<Note> A Certification that sets gangScheduler and no workloadMetadata now emits the resolved queue label — the explicit queue, or default-queue when omitted — on every category’s TrainJob as well as on the runtime templates. This is an intended behavior change that aligns the submitted object with Kubeflow’s documented KAI placement. </Note>

Category options

spec embeds a set of workload options that apply to every category, and each spec.categories[] entry can override them via options; the per-category value wins. Options documented so far:

Coverage

A Certification always sweeps the whole target. The matching nodes are partitioned into groups of nodesPerJob, every node lands in some group, and the groups run concurrently. An 18-node target with nodesPerJob: 2 runs nine jobs.

This is deliberate and there is no opt-out. A PASSED certification means every node the target matched was tested, and the only things that reduce that coverage are the cordon, architecture, and capacity filters, each of which downgrades the run to INCOMPLETE so the operator hears about it. A mode that skipped targeted nodes on purpose would be the one way to report PASSED over untested hardware.

To run one job of a chosen size against part of a fleet, use a WorkloadRun with orchestration.placement: Unpinned. See WorkloadRun: Placement. The trade-off is that a WorkloadRun runs the image and arguments you give it, without the catalog’s per-architecture model sizing and NCCL tuning.

Spec immutability

<Warning> The entire spec is immutable after the Certification is created (a self == oldSelf transition rule on the CRD rejects every update with spec is immutable after creation). The controller never applies edits to an active run, so mutable fields would either be silently ignored or applied only to later categories. To run with different inputs, delete the Certification and create a new one. The minimum is 1 category. </Warning>

Status fields

Each categoryStatuses entry includes a failedNodesRef — a TypedLocalObjectReference pointing to a ConfigMap that stores the failed-node list (name, reason, message) for that category. To read the failed nodes:

# Get the ConfigMap name from the category status
kubectl get certification <name> -o jsonpath='{.status.categoryStatuses[0].failedNodesRef.name}'
# Read the ConfigMap contents
kubectl get configmap <ref-name> -o yaml

Lifecycle

1. Controller creates one Workflow per entry in spec.categories. The spec cannot be changed after creation — delete and recreate to modify it.
2. Workflows run sequentially — the controller processes one category at a time. maxConcurrent controls job/group parallelism within a single Workflow (how many node groups run at once), not across categories.
3. When all Workflows complete, Certification is marked Succeeded or Failed.
4. Failed nodes are recorded in ConfigMaps referenced by status.categoryStatuses[].failedNodesRef. NVCRE does not taint or cordon nodes.