Getting Started

Quick Start

View as Markdown

This guide walks through a full cluster certification: install nvcrectl, run the certification suite, and review the results.

Certification vs WorkloadRun: Use nvcrectl certification run to run a full burn-in suite — it tests multiple categories (NCCL benchmarks, training workloads) against all node groups and records pass/fail results per node. Use nvcrectl workloadrun run when you want to run a single workload ad hoc, such as a quick bandwidth check or a one-off training smoke test, without setting up a full certification. See WorkloadRun Quick Start.

Before you begin

Install nvcrectl and set up the cluster before continuing.

You will also need an NGC API key to pull the certification workload images.

Define a Certification

Create a certification.yaml targeting your GPU nodes:

1apiVersion: cre.nvidia.com/v1alpha1
2kind: Certification
3metadata:
4  name: gpu-cluster-cert
5spec:
6  target:
7    nodeSelector:
8      nvidia.com/gpu.present: "true"
9  imagePullSecrets:
10    - name: ngc-secret
11  categories:
12    - domain: communication
13      variant: nccl-all-reduce
14    - domain: training
15      variant: nemotron5-8b

Run the certification

Use nvcrectl certification run to handle the full lifecycle — apply the manifest, wait for completion, print the report, and clean up:

$nvcrectl certification run \
>  --cert-file certification.yaml \
>  --wait

Review the results

When the certification completes, nvcrectl prints a pass/fail summary per node group and category. A full report is available with:

$nvcrectl certification report gpu-cluster-cert

Failed categories indicate nodes that did not meet performance thresholds. CRE records which nodes failed and why — it does not taint or cordon them. Use kubectl cordon <node> to quarantine nodes as needed.

Next steps