Skip to navigation
CLI Reference

nvcrectl certification

View as Markdown

nvcrectl certification run

Applies a Certification manifest (or runs categories by flag), waits for completion, prints a report, and optionally cleans up.

nvcrectl certification run --cert-file <file> [flags]
nvcrectl certification run --category communication/nccl-all-reduce [flags]

Flags

FlagDefaultDescription
--cert-file—Path to a Certification YAML (mutually exclusive with --category)
--category—Category in domain/variant format; repeatable (mutually exclusive with --cert-file)
--nameautoCertification name (default: nvcrectl-<timestamp>)
--setupfalseInstall CRDs, controller, and LogProfiles before creating the certification
--image—Controller image for --setup (default: ghcr.io/nvidia/cluster-readiness-engine/manager:<version>)
--waitfalseBlock until the certification completes and print a report
--timeoutderivedTimeout for --wait. Ignored without --wait. When --wait is set, must be at least 1s. When not set, derived from the selected categories’ timeoutPerJob budgets (max across categories × iterations × 1.5), floored at 30m; the CLI prints the derived value when the watch starts. The catalog’s timeoutPerJob defaults are 1h for the communication entries and 24h for the two training entries, so a training category derives a 36h wait on its own. Pass --timeout (or set timeoutPerJob on the category) in pipelines that need a tighter bound. An explicit value always wins. On timeout, the CLI prints a partial report and leaves the Certification running unless --cleanup is set.
--cleanupfalseDelete the certification, namespace, and installed components after completion
--nodes-per-job0Nodes per job (0 = auto-select)
--gpus-per-node0GPUs per node (0 = auto-detect from GPU architecture)
--enable-checkpointfalseEnable checkpoint storage for training workloads
--enable-mnnvlfalseEnable Multi-Node NVLink (NCCL_MNNVL_ENABLE=1)
--max-steps0Max training steps for NeMo 4 workloads (0 = catalog default)
--exit-duration-mins0Training duration in minutes for NeMo 6 workloads (0 = catalog default)
--startup-stall-timeout-seconds0Startup-stall window in seconds for training workloads (0 = catalog default; negative values are invalid). Incompatible with --cert-file, including explicit 0; set a positive spec.startupStallTimeoutSeconds or spec.categories[].options.startupStallTimeoutSeconds in the Certification YAML instead.
--repeat-count0Orchestration iterations to repeat tests (0 = catalog default)
--max-restarts0Maximum checkpoint restarts for training workloads (0 = catalog default)
--storage-class—StorageClass for PVC dependencies created by catalog entries
--results-file—Write the certification report as JSON to this path (requires --wait)
--controller-pull-secret—Token for controller registry auth during --setup (e.g. GitHub PAT for ghcr.io) — separate from workload image credentials
--workload-registry—Registry server for workload image pull (e.g. nvcr.io, ghcr.io) — required when --workload-registry-password is set
--workload-registry-username—Registry username for workload image pull (e.g. $oauthtoken for NGC) — required when --workload-registry-password is set
--workload-registry-password—Registry password or API key — creates an nvcrectl-pull-<name> imagePullSecret in the namespace, deleted automatically when the Certification is deleted

When --wait reaches its timeout, the command prints the Certification’s current report, writes it to --results-file when requested, and exits with a timeout error. Without --cleanup, the Certification continues running in the cluster; the timeout stops only the CLI watch. The timeout output includes commands to watch its progress, print an updated report, or stop it. nvcrectl certification report exits nonzero while the Certification is still running. With --cleanup, the partial report is produced before the Certification and its child resources are deleted.

Examples

nvcrectl certification run \
  --cert-file certification.yaml \
  --wait

Pull workload images from NGC:

nvcrectl certification run \
  --category communication/nccl-all-reduce \
  --workload-registry nvcr.io \
  --workload-registry-username '$oauthtoken' \
  --workload-registry-password "$NGC_API_KEY" \
  --wait

nvcrectl certification render

Renders the Workflow manifests that would be created for a Certification, without applying them. Useful for inspecting override application and resource requests before running.

nvcrectl certification render [flags] <cert-file>

Flags

FlagDefaultDescription
--platformautoOverride platform detection (aws, gcp, azure, oci, onprem, togetherai, mistral, forge, nscale)
--gpu-archnoneGPU architecture for offline render, e.g. gb300 or the product name NVIDIA-GB300; an unknown architecture is rejected with the valid list. Use it for a DRA-only GPU stack whose nodes carry no nvidia.com/gpu.product label. Wins over the target nodeSelector’s label when set; cannot be combined with --dry-run, which detects the architecture from real nodes.
--dry-runfalseValidate against the live API server without creating resources. Also discovers the target nodes, so NIC resource detection (on-prem GB200/GB300 and HGX B200/B300), GKE TCPXO network detection and TCPXO plugin version detection (GCP H100) run as they do in the controller and report what they found on stderr. Plugin version detection lists kube-system pods, so it needs that permission.
--outputyamlOutput format: yaml or json

A Certification that sets spec.gangScheduler has that applied to the rendered output too: the scheduler name and the queue label (gangScheduler.queueLabelKey, kai.scheduler/queue when unset; runai/queue on a Run:ai cluster) appear in the rendered manifests on the submitted workload object, on the Job template, and on the pod template metadata, so what you inspect matches what the controller creates. There is no flag for it; the field is set in the Certification YAML.

nvcrectl certification report

Fetches a completed Certification from the cluster and generates a pass/fail report. Multiple names can be provided to combine them into a single report.

nvcrectl certification report <name> [<name>...] [flags]

Flags

FlagDefaultDescription
--results-file—Write the report as JSON to this file path

nvcrectl certification list-categories

Lists all available catalog categories that can be used in a Certification.

nvcrectl certification list-categories [flags]

Flags

FlagDefaultDescription
--outputtableOutput format: table, yaml, or json