> This page is for NVIDIA Cluster Readiness Engine, version Latest · v0.7.0 (default). > For other versions, use one of these documentation indexes: > - Latest · v0.7.0 (default): https://docs.nvidia.com/cluster-readiness-engine/v0.7.0/llms.txt > - v0.6.0: https://docs.nvidia.com/cluster-readiness-engine/v0.6.0/llms.txt > - v0.5.0: https://docs.nvidia.com/cluster-readiness-engine/v0.5.0/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/cluster-readiness-engine/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/cluster-readiness-engine/_mcp/server. # Certification > CRD reference for the Certification resource. `Certification` is the top-level resource that defines a suite of certification categories to run against a GPU node pool. ## Example ```yaml apiVersion: nvcre.nvidia.com/v1alpha1 kind: Certification metadata: name: gpu-cluster-cert namespace: nvcre spec: target: nodeSelector: nvidia.com/gpu.present: "true" enableMNNVL: false gangScheduler: schedulerName: kai-scheduler queue: high-priority # On a Run:ai cluster instead: # schedulerName: runai-scheduler # queueLabelKey: runai/queue # queue: team-a # must name an existing Run:ai queue categories: - domain: communication variant: nccl-all-reduce - domain: training variant: nemotron5-8b options: maxSteps: 50 nodesPerJob: 8 # Optional: override the DGX-class CPU/memory defaults # (limits: cpu "128" / memory 800Gi; requests: cpu "64" / memory 500Gi) # so training pods can schedule on smaller GPU nodes. resources: limits: cpu: "6" memory: 48Gi requests: cpu: "4" memory: 32Gi # Optional: when the category falls back to cloning its source # checkout (nothing pre-seeded in the workspace and nothing shipped # in the image at /opt/megatron-lm), clone from an internal mirror # instead of the default upstream. For the nemotron5 entries the # source is Megatron-LM; for air-gapped clusters, prefer baking the # source into the workload image so no clone runs at all. # sourceRepo: https://git.example.com/mirrors/Megatron-LM.git ``` ## Spec fields _Fields documented so far:_ | Field | Type | Description | |-------|------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `gangScheduler` | GangSchedulerSpec | Optional. Opts every category's workload pods into a gang-aware scheduler such as KAI Scheduler. When set, the scheduler name is injected as `schedulerName` into every pod template of every category's resolved `TrainingRuntime` dependency (for MPI-based categories, both the launcher and the worker pods) and the queue is applied as a label (`gangScheduler.queueLabelKey`, `kai.scheduler/queue` when unset) on both each replicated job's template metadata and its pod template metadata, so the scheduler holds all pods until the entire gang can be placed. Applied after the catalog and platform overrides resolve, so it also replaces a scheduler name a catalog entry hardcodes. On a Run:ai cluster set `schedulerName: runai-scheduler` and `queueLabelKey: runai/queue`, and make `queue` name an existing Run:ai queue | | `gangScheduler.schedulerName` | string | Required; minimum length 1. Name of the gang-aware scheduler to use (e.g., `kai-scheduler`, or `runai-scheduler` on a Run:ai cluster). Injected as `schedulerName` in each workload pod spec | | `gangScheduler.queue` | string | Optional. Scheduler queue to submit the workloads to; defaults to `default-queue` when unset. On a Run:ai cluster it must name an existing Run:ai queue. When non-empty, must be a valid Kubernetes label value: at most 63 characters, beginning and ending with an alphanumeric character, and containing only alphanumerics, hyphens, underscores, or dots (pattern `^$\|^[a-zA-Z0-9]([a-zA-Z0-9._-]*[a-zA-Z0-9])?$`) | | `gangScheduler.queueLabelKey` | string | Optional. Label key the queue value is written under; defaults to `kai.scheduler/queue` when unset. Set it to `runai/queue` on a Run:ai cluster, which reads that label and ignores `kai.scheduler/queue`. When non-empty, must be a valid Kubernetes label key (qualified name): an optional DNS-subdomain prefix of at most 253 characters followed by `/`, then a name of at most 63 characters; at most 317 characters in total. `app.kubernetes.io/managed-by` and any key under `nvcre.nvidia.com/` are rejected because the resolved queue label is inserted into workload metadata | | `sourceRepo` | string | Optional. Git repository a category clones its source checkout from when the clone actually runs. Categories with a source checkout resolve it at pod start in three steps: an existing `/mnt/workspace/megatron-lm` workspace is used unchanged, then source shipped in the workload image at `/opt/megatron-lm` is copied in, and only otherwise does the init container clone. For air-gapped or restricted-egress clusters the primary recommendation is therefore baking the source into the workload image; `sourceRepo` is for sites that run an internal Git mirror. Each catalog entry defines what its source is and its default upstream; the `training/nemotron5-8b` and `training/nemotron5-56b` entries are the current consumers, and their entry-defined default is Megatron-LM (`https://github.com/NVIDIA/Megatron-LM.git`, branch `core_v0.15.2`). The mirror must serve the entry's pinned branch; entries that clone no source ignore the field. The URL must use an authenticated remote scheme (`https://` or `ssh://`); `http://` and `git://` URLs, scp-style `git@host:path` syntax, `file://` URLs, and URLs containing whitespace or shell metacharacters are rejected (max length 2048, pattern `^(https\|ssh)://[A-Za-z0-9._~:/@%+-]+$`). `http://` and `git://` are rejected deliberately: the cloned source is executed by the workload, and those transports are unauthenticated, so an on-path attacker could substitute the code. `file://` is rejected deliberately as well: the contract is a remote git mirror. Non-TLS mirrors and local source belong in the image or on the pre-seeded checkpoint PVC instead. Settable globally on `spec` or per category under `categories[].options`; the per-category value wins. See [Training categories: Megatron-LM source](/cluster-readiness-engine/operations/deployment#training-categories-megatron-lm-source) | | `nicResourceName` | string | Optional; also settable per category via `categories[].options.nicResourceName` (per-category wins). Kubernetes extended resource name of the RDMA NIC devices to request on workload containers for on-prem GB200/GB300 and x86 HGX B200/B300 targets, for example `rdma/ib` or `nvidia.com/mlnxnics`. Must be a fully qualified extended resource name (domain, slash, and a name segment of at most 63 characters); the reserved `kubernetes.io` and `k8s.io` domains are rejected. When unset, the controller detects the name automatically: if exactly one candidate resource (`rdma/*` or `nvidia.com/mlnxnics`) is allocatable at the resolved `mlnxPerNode` count on every target node, it is requested; otherwise nothing is requested and a Normal `NICResourceDetection` event on the Certification explains what was found: no candidate on any node, candidates below the requested count (naming the count), or multiple qualifying candidates. Set the field to override detection or when detection is ambiguous; requesting a resource the nodes do not advertise at the requested count leaves pods permanently Pending. Offline `nvcrectl certification render` (without `--dry-run`) has no cluster to inspect and requires the field. The per-container count always comes from `mlnxPerNode` (GB200, GB300, B200, and B300 default to 8; sites running a shared-device plugin should set `mlnxPerNode: 1`, or detection will refuse a pooled resource advertised as 1) | | `rdmaDeviceAccess` | string | Optional; also settable per category via `categories[].options.rdmaDeviceAccess` (per-category wins). One of `hostPath` or `devicePlugin`; unset behaves as `devicePlugin`. Selects how workload containers reach the host's RDMA devices. `devicePlugin` leaves the catalog requesting NIC devices as a Kubernetes extended resource, which is what grants the container its device cgroup entries; see `mlnxPerNode` and `nicResourceName`. `hostPath` mounts the host's `/dev/infiniband` into the workload container and runs that container privileged. The two go together: kubelet builds the device cgroup allowlist from the container's requested resources, so a `hostPath` mount on its own leaves the device nodes visible but `open()` on them failing with `EPERM`, and no capability changes that. Set it on clusters that expose RDMA through host device nodes rather than through a device plugin, such as OCI GB200 shapes running no SR-IOV device plugin. It applies on every platform and architecture, since how a cluster plumbs RDMA is a property of the cluster. The value is case-sensitive and `nvcrectl` rejects anything outside the enum, so an offline render never silently omits the mount. Pods will be rejected on namespaces that enforce a `restricted` or `baseline` Pod Security Standard. A privileged container is root-equivalent on the node: it runs without the runtime's default seccomp filter and without AppArmor or SELinux confinement, so a pod-level `seccompProfile` no longer constrains it even though the pod spec still shows it, and the `/dev/infiniband` mount names the devices the workload uses rather than limiting what it can reach. Some CRI-O runtimes reject a privileged container whose GPUs come from the NVIDIA device plugin, and the pods then fail to start; see [Host RDMA device access](/cluster-readiness-engine/concepts/platform-detection-overrides#host-rdma-device-access) and [ADR-088](../designs/088-oci-gb200-roce-and-rdma-device-access.md) | | `mlnxPerNode` | int32 | Optional; also settable per category via `categories[].options.mlnxPerNode`. Overrides the auto-detected Mellanox NIC count per node used by InfiniBand/RoCE platforms and as the `nicResourceName` request count. When unset, derived from GPU architecture and platform via the catalog's `gpu-defaults.yaml`: eight for H100, H200, A100, L40S, B200, B300, GB200, and GB300, four for L40, none for RTX PRO 6000, before platform overrides such as OCI's. Setting it to `0` is a supported opt-out rather than a request for zero devices: the templates that request `nvidia.com/mlnxnics` (Azure, OCI, TogetherAI, Forge) drop the request entirely, and on OCI they also drop the `k8s.v1.cni.cncf.io/networks` annotation that names the `network-operator/sriov-net` attachments. OCI GB200 resolves to `0` by default because its shapes do not consistently expose the resource; see [Architecture-specific resources](/cluster-readiness-engine/concepts/platform-detection-overrides#architecture-specific-resources) | | `startupStallTimeoutSeconds` | int32 | Optional; also settable per category via `categories[].options.startupStallTimeoutSeconds` (per-category wins). Overrides the startup-stall window of the generated Job (`Job.spec.startupStallTimeoutSeconds`): the budget for the first parsed training step after the workload starts, after which the Job fails with reason `WorkloadStalled`. It only takes effect when the Job also sets `stallMultiplier` (the training entries set 3); the catalog entry's default applies when unset (`training/nemotron5-8b`/`nemotron5-56b`: 1200 s). | | `target` | TargetSpec | Optional. Selects which nodes are candidates for the run. A node must satisfy `nodeSelector`, `matchExpressions`, `nodeNames`, and `taintSelectors`. When unset, all nodes in the cluster are targeted. Cordoned nodes are excluded by default. See `target.taintSelectors` for how to target cordoned nodes | | `target.nodeSelector` | map[string]string | Optional. Selects nodes by label key-value pairs | | `target.matchExpressions` | []NodeSelectorRequirement | Optional. Node selector requirements (`In`, `NotIn`, `Exists`, `DoesNotExist`) | | `target.nodeNames` | []string | Optional. Explicit node names to include | | `target.taintSelectors` | []TaintSelector | Optional. Selects nodes carrying every listed taint, and tolerates those taints on the workload pods. Listing `node.kubernetes.io/unschedulable` opts cordoned nodes into the run and also clears `nodeHealthMonitor` for the whole run. As a result, no node can be reported with `HardwareFailureDetected` | | `nodesPerJob` | int32 | Optional; also settable per category via `categories[].options.nodesPerJob` (per-category wins). A per-job group size, not a total: a Certification covers every node its target matches, so N matching nodes produce ceil(N/`nodesPerJob`) concurrent jobs. When unset the controller auto-selects; when set it is clamped to `min(nodesPerJob, matchingNodes)` and snapped down to the largest count the entry's model constraints allow. To run one job of a chosen size against part of a fleet, use a WorkloadRun instead. See [Coverage](#coverage) | Like the rest of `spec`, `gangScheduler` is immutable after the Certification is created. When `gangScheduler` is set, its resolved queue is also placed on each category's generated workload object (the `TrainJob`), under `gangScheduler.queueLabelKey`, in addition to the runtime Job- and pod-template placement described above. See [Workload object labels](#workload-object-labels). ## Workload object labels `workloadMetadata` sets labels on the workload object each category's Job creates — today a Kubeflow `TrainJob`. That object's own `metadata.labels` is where Kueue selects a local queue and where Kubeflow's KAI Scheduler guide documents the queue label, both read before any workload pod exists. Because `CertificationSpec` inlines the shared category options, the one field is read at two paths: globally at `spec.workloadMetadata` and per category at `spec.categories[].options.workloadMetadata`. ```yaml spec: workloadMetadata: labels: environment: burn-in categories: - domain: communication variant: nccl-all-reduce options: workloadMetadata: labels: kueue.x-k8s.io/queue-name: nccl-queue ``` That `nccl-all-reduce` category's `TrainJob` receives both `environment: burn-in` and `kueue.x-k8s.io/queue-name: nccl-queue`. These labels are not pod labels. See [Job workload object labels](/cluster-readiness-engine/api-reference/job#workload-object-labels) for the full contract, reserved keys, limits, and immutability rules. ### Precedence Unlike the other maps, slices, and pointers in category options — `thresholds`, `resources`, `imagePullSecrets` — workload labels **merge per key** rather than replacing the whole value. The order is: 1. labels already present in the resolved catalog job spec, 2. global `spec.workloadMetadata.labels`, 3. per-category `options.workloadMetadata.labels`, and 4. the resolved `gangScheduler` queue label. Steps 2 and 3 replace earlier values by key. Step 4 does not overwrite: a missing key is inserted, the same value is accepted, and a different value fails. The consequence of merging is that a category can **change or add** a key but cannot **remove** a global one — there is no deletion syntax, and an empty string is a real label value. A category needing a wholly different map should not set those keys globally. ### Per-category queue conflicts `spec.gangScheduler` is global, so every category receives the same resolved queue, including the `default-queue` default. A category that sets that same KAI or Run:ai key to a different value fails, naming the key, both values, and the field path. Per-category Kueue queues remain independent, because `kueue.x-k8s.io/queue-name` is a different key. <Note> A Certification that sets `gangScheduler` and no `workloadMetadata` now emits the resolved queue label — the explicit `queue`, or `default-queue` when omitted — on every category's `TrainJob` as well as on the runtime templates. This is an intended behavior change that aligns the submitted object with Kubeflow's documented KAI placement. </Note> ## Category options `spec` embeds a set of workload options that apply to every category, and each `spec.categories[]` entry can override them via `options`; the per-category value wins. Options documented so far: | Field | Type | Description | |-------|------|-------------| | `workloadMetadata` | WorkloadMetadata | Optional. Labels applied to the generated workload object (the `TrainJob`). Settable globally on `spec` and per category under `categories[].options`; unlike the other options the two levels **merge per key**, with the per-category value winning. See [Workload object labels](#workload-object-labels) | | `workloadMetadata.labels` | map[string]string | Optional. At most 32 entries after merging and after any `gangScheduler` queue label is inserted — exceeding the cap fails rather than dropping labels. Keys must be valid Kubernetes label keys and values valid Kubernetes label values; `app.kubernetes.io/managed-by` and any key under `nvcre.nvidia.com/` are rejected | | `image` | string | Optional. Overrides the workload container image for catalog workloads. Replaces the trainer image in the rendered job template, the primary workload container (`containers[0]`) of every replicated job in the resolved `TrainingRuntime` dependencies, and every init container in those pods whose image exactly equals the primary's pre-override image (the workload-derived inits such as `fix-ssh-permissions` and `megatron-clone`); init containers with a distinct image (such as GCP's `tcpxo-daemon`) and any additional containers keep their catalog images. Applied after the catalog and platform overrides resolve, so it also replaces an image a platform override selects. Settable at the spec level and per category (`categories[].options.image`); a non-empty per-category value wins for that category, and the spec-level value applies to every category that does not set one. Prefer the per-category form when only some categories need a different image, because a spec-level value also replaces the image of a category that ships its own, such as the DCGM image `diagnostics/dcgm-level4` runs `dcgmi` from. **Beware**: on AWS EFA platforms (H100, GB200) the platform overrides land the workers on an `nccl-tests` image that ships the aws-ofi-nccl (EFA) plugin; setting `image` replaces that image, and you then own the EFA OFI plugin being present in the replacement. On GCP H100 the platform overrides pick the NCCL and training images from the TCPXO NCCL plugin version detected on the nodes, and a replacement's CUDA major must match that plugin: CUDA 12 for plugin v1.0.16 and earlier, CUDA 13 for v1.0.17 (see [GCP H100 clusters](/cluster-readiness-engine/concepts/platform-detection-overrides#gcp-h100-clusters)). NVCRE does not restrict which registries or images the field may reference; clusters that require trusted images should enforce that with cluster-wide admission policy (for example Kyverno or the Sigstore policy-controller), which covers this field, `WorkloadRun` `spec.image`, and every other pod alike | ## Coverage A Certification always sweeps the whole target. The matching nodes are partitioned into groups of `nodesPerJob`, every node lands in some group, and the groups run concurrently. An 18-node target with `nodesPerJob: 2` runs nine jobs. This is deliberate and there is no opt-out. A `PASSED` certification means every node the target matched was tested, and the only things that reduce that coverage are the cordon, architecture, and capacity filters, each of which downgrades the run to `INCOMPLETE` so the operator hears about it. A mode that skipped targeted nodes on purpose would be the one way to report `PASSED` over untested hardware. To run **one** job of a chosen size against part of a fleet, use a WorkloadRun with `orchestration.placement: Unpinned`. See [WorkloadRun: Placement](/cluster-readiness-engine/api-reference/workload-run#placement). The trade-off is that a WorkloadRun runs the image and arguments you give it, without the catalog's per-architecture model sizing and NCCL tuning. ## Spec immutability <Warning> The entire `spec` is **immutable** after the Certification is created (a `self == oldSelf` transition rule on the CRD rejects every update with `spec is immutable after creation`). The controller never applies edits to an active run, so mutable fields would either be silently ignored or applied only to later categories. To run with different inputs, delete the Certification and create a new one. The minimum is 1 category. </Warning> ## Status fields | Field | Type | Description | |-------|------|-------------| | `conditions` | []Condition | InProgress, Succeeded, Failed (mutually exclusive) | | `categoryStatuses` | []CertificationCategoryStatus | Per-category status including `domain`, `variant`, `status`, `workflowRef`, `succeededNodesRef`, and `failedNodesRef` | Each `categoryStatuses` entry includes a `failedNodesRef` — a `TypedLocalObjectReference` pointing to a ConfigMap that stores the failed-node list (name, reason, message) for that category. To read the failed nodes: ```bash # Get the ConfigMap name from the category status kubectl get certification -o jsonpath='{.status.categoryStatuses[0].failedNodesRef.name}' # Read the ConfigMap contents kubectl get configmap -o yaml ``` ## Lifecycle 1. Controller creates one `Workflow` per entry in `spec.categories`. The spec cannot be changed after creation — delete and recreate to modify it. 2. Workflows run **sequentially** — the controller processes one category at a time. `maxConcurrent` controls job/group parallelism *within* a single Workflow (how many node groups run at once), not across categories. 3. When all Workflows complete, Certification is marked `Succeeded` or `Failed`. 4. Failed nodes are recorded in ConfigMaps referenced by `status.categoryStatuses[].failedNodesRef`. NVCRE does not taint or cordon nodes.