This page is for NVIDIA Cluster Readiness Engine, version v0.8.0.
For other versions, use one of these documentation indexes:
- Latest · v0.8.2 (default): https://docs.nvidia.com/cluster-readiness-engine/v0.8.2/llms.txt
- v0.8.1: https://docs.nvidia.com/cluster-readiness-engine/v0.8.1/llms.txt
- v0.8.0: https://docs.nvidia.com/cluster-readiness-engine/v0.8.0/llms.txt
The NVIDIA Cluster Readiness Engine (NVCRE) has been validated on the following platform and GPU combinations. Each certification ran NCCL communication tests (all-reduce, all-gather, alltoall) and NeMo training. GB200/GB300 configurations include both MNNVL-enabled and MNNVL-disabled variants.
Summary
| Platform | GPU | Nodes | Interconnect | Categories | Result |
|---|---|---|---|---|---|
| AWS | H100 | 2 | EFA | 4/4 | PASSED |
| AWS | GB200 | 4 | EFA | 8/8 | PASSED |
| AWS | GB300 | 8 | RoCE | 8/8 | PASSED |
| GCP | H100 | 2 | TCPXO | 4/4 | PASSED |
| GCP | GB200 | 2 | RoCE | 8/8 | PASSED |
Catalog variant names evolve between releases. The training entry used during these validation runs (training/nemotron4-15b) has since been superseded — the current catalog offers training/nemotron5-8b and training/nemotron5-56b. List the categories available in your installed version with nvcrectl certification list-categories.
Detailed Reports
AWS H100
╔════════════════════════════════════════════════════════════════╗║ Certification Report ║╚════════════════════════════════════════════════════════════════╝Name: gpu-cluster-certPlatform: awsGPU: h100Nodes: 2┌────────────────────────────────────────────────────────────────┐│ communication/nccl-all-reduce │├────────────────────────────────────────────────────────────────┤│ Status: Succeeded ││ Duration: 11m 0s ││ Nodes/Job: 2 ││ Jobs: 1 ││ ││ Bandwidth: ││ Size AlgBW BusBW Samples ││ 16 GB 260.20 GB/s 487.87 GB/s 65 │└────────────────────────────────────────────────────────────────┘┌────────────────────────────────────────────────────────────────┐│ communication/nccl-all-gather │├────────────────────────────────────────────────────────────────┤│ Status: Succeeded ││ Duration: 7m 5s ││ Nodes/Job: 2 ││ Jobs: 1 ││ ││ Bandwidth: ││ Size AlgBW BusBW Samples ││ 16 GB 361.14 GB/s 338.57 GB/s 67 │└────────────────────────────────────────────────────────────────┘┌────────────────────────────────────────────────────────────────┐│ communication/nccl-alltoall │├────────────────────────────────────────────────────────────────┤│ Status: Succeeded ││ Duration: 22m 10s ││ Nodes/Job: 2 ││ Jobs: 1 ││ ││ Bandwidth: ││ Size AlgBW BusBW Samples ││ 16 GB 84.40 GB/s 79.11 GB/s 66 │└────────────────────────────────────────────────────────────────┘┌────────────────────────────────────────────────────────────────┐│ training/nemotron4-15b │├────────────────────────────────────────────────────────────────┤│ Status: Succeeded ││ Duration: 4m 49s ││ Nodes/Job: 2 ││ Jobs: 1 ││ ││ Avg Runtime Goodput: 0.90 (90%) ││ Avg TFLOPs/GPU: 539.8 ││ Avg Train Time: 3m 40s ││ Avg Step Time: 2.85s │└────────────────────────────────────────────────────────────────┘┌────────────────────────────────────────────────────────────────┐│ Summary │├────────────────────────────────────────────────────────────────┤│ Categories: 4/4 passed ││ Failed Nodes: none ││ Result: PASSED │└────────────────────────────────────────────────────────────────┘
AWS GB200
╔════════════════════════════════════════════════════════════════╗║ Certification Report ║╚════════════════════════════════════════════════════════════════╝Name: gpu-cluster-certPlatform: awsGPU: gb200Nodes: 4┌────────────────────────────────────────────────────────────────┐│ communication/nccl-all-reduce │├────────────────────────────────────────────────────────────────┤│ Status: Succeeded ││ Duration: 6m 47s ││ Nodes/Job: 4 ││ Jobs: 1 ││ MNNVL: Enabled ││ ││ Bandwidth: ││ Size AlgBW BusBW Samples ││ 16 GB 470.68 GB/s 882.51 GB/s 65 │└────────────────────────────────────────────────────────────────┘┌────────────────────────────────────────────────────────────────┐│ communication/nccl-all-gather │├────────────────────────────────────────────────────────────────┤│ Status: Succeeded ││ Duration: 4m 3s ││ Nodes/Job: 4 ││ Jobs: 1 ││ MNNVL: Enabled ││ ││ Bandwidth: ││ Size AlgBW BusBW Samples ││ 16 GB 649.90 GB/s 609.28 GB/s 67 │└────────────────────────────────────────────────────────────────┘┌────────────────────────────────────────────────────────────────┐│ communication/nccl-alltoall │├────────────────────────────────────────────────────────────────┤│ Status: Succeeded ││ Duration: 5m 12s ││ Nodes/Job: 4 ││ Jobs: 1 ││ MNNVL: Enabled ││ ││ Bandwidth: ││ Size AlgBW BusBW Samples ││ 16 GB 606.58 GB/s 568.66 GB/s 65 │└────────────────────────────────────────────────────────────────┘┌────────────────────────────────────────────────────────────────┐│ communication/nccl-all-reduce │├────────────────────────────────────────────────────────────────┤│ Status: Succeeded ││ Duration: 20m 20s ││ Nodes/Job: 4 ││ Jobs: 1 ││ MNNVL: Disabled ││ ││ Bandwidth: ││ Size AlgBW BusBW Samples ││ 16 GB 102.08 GB/s 191.40 GB/s 65 │└────────────────────────────────────────────────────────────────┘┌────────────────────────────────────────────────────────────────┐│ communication/nccl-all-gather │├────────────────────────────────────────────────────────────────┤│ Status: Succeeded ││ Duration: 10m 23s ││ Nodes/Job: 4 ││ Jobs: 1 ││ MNNVL: Disabled ││ ││ Bandwidth: ││ Size AlgBW BusBW Samples ││ 16 GB 204.01 GB/s 191.26 GB/s 65 │└────────────────────────────────────────────────────────────────┘┌────────────────────────────────────────────────────────────────┐│ communication/nccl-alltoall │├────────────────────────────────────────────────────────────────┤│ Status: Succeeded ││ Duration: 28m 31s ││ Nodes/Job: 4 ││ Jobs: 1 ││ MNNVL: Disabled ││ ││ Bandwidth: ││ Size AlgBW BusBW Samples ││ 16 GB 62.02 GB/s 58.15 GB/s 63 │└────────────────────────────────────────────────────────────────┘┌────────────────────────────────────────────────────────────────┐│ training/nemotron4-15b │├────────────────────────────────────────────────────────────────┤│ Status: Succeeded ││ Duration: 3m 43s ││ Nodes/Job: 4 ││ Jobs: 1 ││ MNNVL: Enabled ││ ││ Avg Runtime Goodput: 0.68 (68%) ││ Avg TFLOPs/GPU: 1186.5 ││ Avg Train Time: 2m 38s ││ Avg Step Time: 1.30s │└────────────────────────────────────────────────────────────────┘┌────────────────────────────────────────────────────────────────┐│ training/nemotron4-15b │├────────────────────────────────────────────────────────────────┤│ Status: Succeeded ││ Duration: 3m 49s ││ Nodes/Job: 4 ││ Jobs: 1 ││ MNNVL: Disabled ││ ││ Avg Runtime Goodput: 0.66 (66%) ││ Avg TFLOPs/GPU: 1118.5 ││ Avg Train Time: 3m 10s ││ Avg Step Time: 1.38s │└────────────────────────────────────────────────────────────────┘┌────────────────────────────────────────────────────────────────┐│ Summary │├────────────────────────────────────────────────────────────────┤│ Categories: 8/8 passed ││ Failed Nodes: none ││ Result: PASSED │└────────────────────────────────────────────────────────────────┘
AWS GB300
╔════════════════════════════════════════════════════════════════╗║ Certification Report ║╚════════════════════════════════════════════════════════════════╝Name: gpu-cluster-certPlatform: awsGPU: gb300Nodes: 8┌────────────────────────────────────────────────────────────────┐│ communication/nccl-all-reduce │├────────────────────────────────────────────────────────────────┤│ Status: Succeeded ││ Duration: 7m 32s ││ Nodes/Job: 8 ││ Jobs: 1 ││ MNNVL: Enabled ││ ││ Bandwidth: ││ Size AlgBW BusBW Samples ││ 16 GB 476.55 GB/s 923.32 GB/s 67 │└────────────────────────────────────────────────────────────────┘┌────────────────────────────────────────────────────────────────┐│ communication/nccl-all-gather │├────────────────────────────────────────────────────────────────┤│ Status: Succeeded ││ Duration: 5m 3s ││ Nodes/Job: 8 ││ Jobs: 1 ││ MNNVL: Enabled ││ ││ Bandwidth: ││ Size AlgBW BusBW Samples ││ 16 GB 721.28 GB/s 698.74 GB/s 69 │└────────────────────────────────────────────────────────────────┘┌────────────────────────────────────────────────────────────────┐│ communication/nccl-alltoall │├────────────────────────────────────────────────────────────────┤│ Status: Succeeded ││ Duration: 6m 40s ││ Nodes/Job: 8 ││ Jobs: 1 ││ MNNVL: Enabled ││ ││ Bandwidth: ││ Size AlgBW BusBW Samples ││ 16 GB 700.45 GB/s 678.57 GB/s 68 │└────────────────────────────────────────────────────────────────┘┌────────────────────────────────────────────────────────────────┐│ communication/nccl-all-reduce │├────────────────────────────────────────────────────────────────┤│ Status: Succeeded ││ Duration: 13m 39s ││ Nodes/Job: 8 ││ Jobs: 1 ││ MNNVL: Disabled ││ ││ Bandwidth: ││ Size AlgBW BusBW Samples ││ 16 GB 194.07 GB/s 376.01 GB/s 65 │└────────────────────────────────────────────────────────────────┘┌────────────────────────────────────────────────────────────────┐│ communication/nccl-all-gather │├────────────────────────────────────────────────────────────────┤│ Status: Succeeded ││ Duration: 6m 19s ││ Nodes/Job: 8 ││ Jobs: 1 ││ MNNVL: Disabled ││ ││ Bandwidth: ││ Size AlgBW BusBW Samples ││ 16 GB 377.48 GB/s 365.68 GB/s 63 │└────────────────────────────────────────────────────────────────┘┌────────────────────────────────────────────────────────────────┐│ communication/nccl-alltoall │├────────────────────────────────────────────────────────────────┤│ Status: Succeeded ││ Duration: 24m 26s ││ Nodes/Job: 8 ││ Jobs: 1 ││ MNNVL: Disabled ││ ││ Bandwidth: ││ Size AlgBW BusBW Samples ││ 16 GB 88.38 GB/s 85.62 GB/s 68 │└────────────────────────────────────────────────────────────────┘┌────────────────────────────────────────────────────────────────┐│ training/nemotron4-15b │├────────────────────────────────────────────────────────────────┤│ Status: Succeeded ││ Duration: 3m 49s ││ Nodes/Job: 8 ││ Jobs: 1 ││ MNNVL: Enabled ││ ││ Avg Runtime Goodput: 0.57 (57%) ││ Avg TFLOPs/GPU: 1388.5 ││ Avg Train Time: 3m 26s ││ Avg Step Time: 1.11s │└────────────────────────────────────────────────────────────────┘┌────────────────────────────────────────────────────────────────┐│ training/nemotron4-15b │├────────────────────────────────────────────────────────────────┤│ Status: Succeeded ││ Duration: 3m 54s ││ Nodes/Job: 8 ││ Jobs: 1 ││ MNNVL: Disabled ││ ││ Avg Runtime Goodput: 0.57 (57%) ││ Avg TFLOPs/GPU: 1136.9 ││ Avg Train Time: 3m 26s ││ Avg Step Time: 1.40s │└────────────────────────────────────────────────────────────────┘┌────────────────────────────────────────────────────────────────┐│ Summary │├────────────────────────────────────────────────────────────────┤│ Categories: 8/8 passed ││ Failed Nodes: none ││ Result: PASSED │└────────────────────────────────────────────────────────────────┘
GCP H100
╔════════════════════════════════════════════════════════════════╗║ Certification Report ║╚════════════════════════════════════════════════════════════════╝Name: gpu-cluster-certPlatform: gcpGPU: h100Nodes: 2┌────────────────────────────────────────────────────────────────┐│ communication/nccl-all-reduce │├────────────────────────────────────────────────────────────────┤│ Status: Succeeded ││ Duration: 13m 38s ││ Nodes/Job: 2 ││ Jobs: 1 ││ ││ Bandwidth: ││ Size AlgBW BusBW Samples ││ 16 GB 179.78 GB/s 337.09 GB/s 67 │└────────────────────────────────────────────────────────────────┘┌────────────────────────────────────────────────────────────────┐│ communication/nccl-all-gather │├────────────────────────────────────────────────────────────────┤│ Status: Succeeded ││ Duration: 11m 19s ││ Nodes/Job: 2 ││ Jobs: 1 ││ ││ Bandwidth: ││ Size AlgBW BusBW Samples ││ 16 GB 201.88 GB/s 189.27 GB/s 66 │└────────────────────────────────────────────────────────────────┘┌────────────────────────────────────────────────────────────────┐│ communication/nccl-alltoall │├────────────────────────────────────────────────────────────────┤│ Status: Succeeded ││ Duration: 40m 40s ││ Nodes/Job: 2 ││ Jobs: 1 ││ ││ Bandwidth: ││ Size AlgBW BusBW Samples ││ 16 GB 42.84 GB/s 40.16 GB/s 66 │└────────────────────────────────────────────────────────────────┘┌────────────────────────────────────────────────────────────────┐│ training/nemotron4-15b │├────────────────────────────────────────────────────────────────┤│ Status: Succeeded ││ Duration: 4m 37s ││ Nodes/Job: 2 ││ Jobs: 1 ││ ││ Avg Runtime Goodput: 0.92 (92%) ││ Avg TFLOPs/GPU: 511.7 ││ Avg Train Time: 3m 33s ││ Avg Step Time: 3.01s │└────────────────────────────────────────────────────────────────┘┌────────────────────────────────────────────────────────────────┐│ Summary │├────────────────────────────────────────────────────────────────┤│ Categories: 4/4 passed ││ Failed Nodes: none ││ Result: PASSED │└────────────────────────────────────────────────────────────────┘
GCP GB200
╔════════════════════════════════════════════════════════════════╗║ Certification Report ║╚════════════════════════════════════════════════════════════════╝Name: gpu-cluster-certPlatform: gcpGPU: gb200Nodes: 2┌────────────────────────────────────────────────────────────────┐│ communication/nccl-all-reduce │├────────────────────────────────────────────────────────────────┤│ Status: Succeeded ││ Duration: 6m 25s ││ Nodes/Job: 2 ││ Jobs: 1 ││ MNNVL: Enabled ││ ││ Bandwidth: ││ Size AlgBW BusBW Samples ││ 16 GB 478.90 GB/s 838.07 GB/s 64 │└────────────────────────────────────────────────────────────────┘┌────────────────────────────────────────────────────────────────┐│ communication/nccl-all-gather │├────────────────────────────────────────────────────────────────┤│ Status: Succeeded ││ Duration: 2m 56s ││ Nodes/Job: 2 ││ Jobs: 1 ││ MNNVL: Enabled ││ ││ Bandwidth: ││ Size AlgBW BusBW Samples ││ 16 GB 777.35 GB/s 680.18 GB/s 58 │└────────────────────────────────────────────────────────────────┘┌────────────────────────────────────────────────────────────────┐│ communication/nccl-alltoall │├────────────────────────────────────────────────────────────────┤│ Status: Succeeded ││ Duration: 3m 7s ││ Nodes/Job: 2 ││ Jobs: 1 ││ MNNVL: Enabled ││ ││ Bandwidth: ││ Size AlgBW BusBW Samples ││ 16 GB 781.51 GB/s 683.82 GB/s 67 │└────────────────────────────────────────────────────────────────┘┌────────────────────────────────────────────────────────────────┐│ communication/nccl-all-reduce │├────────────────────────────────────────────────────────────────┤│ Status: Succeeded ││ Duration: 6m 50s ││ Nodes/Job: 2 ││ Jobs: 1 ││ MNNVL: Disabled ││ ││ Bandwidth: ││ Size AlgBW BusBW Samples ││ 16 GB 478.72 GB/s 837.76 GB/s 65 │└────────────────────────────────────────────────────────────────┘┌────────────────────────────────────────────────────────────────┐│ communication/nccl-all-gather │├────────────────────────────────────────────────────────────────┤│ Status: Succeeded ││ Duration: 3m 24s ││ Nodes/Job: 2 ││ Jobs: 1 ││ MNNVL: Disabled ││ ││ Bandwidth: ││ Size AlgBW BusBW Samples ││ 16 GB 777.33 GB/s 680.16 GB/s 58 │└────────────────────────────────────────────────────────────────┘┌────────────────────────────────────────────────────────────────┐│ communication/nccl-alltoall │├────────────────────────────────────────────────────────────────┤│ Status: Succeeded ││ Duration: 3m 8s ││ Nodes/Job: 2 ││ Jobs: 1 ││ MNNVL: Disabled ││ ││ Bandwidth: ││ Size AlgBW BusBW Samples ││ 16 GB 781.54 GB/s 683.84 GB/s 65 │└────────────────────────────────────────────────────────────────┘┌────────────────────────────────────────────────────────────────┐│ training/nemotron4-15b │├────────────────────────────────────────────────────────────────┤│ Status: Succeeded ││ Duration: 5m 40s ││ Nodes/Job: 2 ││ Jobs: 1 ││ MNNVL: Enabled ││ ││ Avg Runtime Goodput: 0.68 (68%) ││ Avg TFLOPs/GPU: 1175.0 ││ Avg Train Time: 2m 16s ││ Avg Step Time: 1.31s │└────────────────────────────────────────────────────────────────┘┌────────────────────────────────────────────────────────────────┐│ training/nemotron4-15b │├────────────────────────────────────────────────────────────────┤│ Status: Succeeded ││ Duration: 2m 44s ││ Nodes/Job: 2 ││ Jobs: 1 ││ MNNVL: Disabled ││ ││ Avg Runtime Goodput: 0.69 (69%) ││ Avg TFLOPs/GPU: 1122.0 ││ Avg Train Time: 2m 21s ││ Avg Step Time: 1.37s │└────────────────────────────────────────────────────────────────┘┌────────────────────────────────────────────────────────────────┐│ Summary │├────────────────────────────────────────────────────────────────┤│ Categories: 8/8 passed ││ Failed Nodes: none ││ Result: PASSED │└────────────────────────────────────────────────────────────────┘
Machine-readable results
nvcrectl certification run --wait --results-file <path> and nvcrectl certification report <name> --results-file <path> write the same report as JSON. The file contains a single report object (or an array when multiple certifications are combined into one report):
{"name": "gpu-cluster-cert","platform": "aws","gpu": "gb200","totalNodes": 4,"categories": [{"domain": "communication","variant": "nccl-all-reduce","status": "Succeeded","runtime": "6m 47s","nodesPerJob": 4,"jobs": 1,"mnnvl": "Enabled","transport": ["Socket"],"bandwidth": [{ "size": "16 GB", "algBW": "470.68 GB/s", "busBW": "882.51 GB/s", "samples": 65 }]}],"failedNodes": [],"result": "PASSED"}
Top-level fields:
| Field | Type | Description |
|---|---|---|
name | string | Certification name |
platform | string | Detected platform (aws, gcp, azure, oci, onprem, …) |
gpu | string | Detected GPU architecture |
totalNodes | int | Number of nodes targeted by the certification |
excludedNodes | []string | Nodes that matched the target but were left untested (omitted when empty) |
exclusionReason | string | Why nodes were excluded (omitted when empty) |
categories | []object | Per-category results (see below) |
failedNodes | []string | Deduped union of failed node names across all categories |
failedNodeDetails | []object | The per-node reason and message behind those names: name, reason, and optional message (omitted when no node failed) |
result | string | PASSED, INCOMPLETE, FAILED, or RUNNING |
nodeResults | []object | Per-node pass/fail entries: name, group, rack (optional), status (Passed or Failed). Written for WorkloadRun reports only |
failedNodes stays a list of names so existing consumers keep working;
failedNodeDetails is the structured companion. reason is one of
HardwareFailureDetected, ThresholdViolation, or WorkloadFailed, and
message is the detail recorded for that node. A node appears once per distinct
(reason, message), so it can hold more than one entry when it failed different
ways across categories.
Each entry in categories:
| Field | Type | Description |
|---|---|---|
domain / variant | string | Catalog category, e.g. communication / nccl-all-reduce |
status | string | Category status (Succeeded, Failed, …) |
failureReason | string | Populated from the Workflow Failed condition message (omitted when empty) |
runtime | string | Total runtime across all iterations |
testScale | string | Test scale, when set |
nodesPerJob | int | Nodes per job group |
jobs | int | Number of jobs run |
mnnvl | string | Enabled, Disabled, or omitted when unknown |
bandwidth | []object | NCCL results per message size: size, algBW, busBW, samples, and provisional/provisionalReason when the result is not final |
transport | []string | Distinct NCCL network names recorded on the category’s BandwidthMeasurements (for example IB, Socket). Omitted when the Using network line never appeared |
groupBandwidth | []object | Per-group peak bandwidth, one row per group: group, groupName (display label), nodes, busBW, transport (optional), belowMin (true when BusBW fails a minimum busBandwidthGBps threshold such as value >= 300), failed, and provisional/provisionalReason when the result is not final |
domains | []object | Training results per topology domain: name, nodeCount, goodput, tflops, stepTime |
failedGroups | []object | Failed orchestration groups: name, nodeCount, nodes, reason, and failureLog when the Job captured one |
iterations | []object | Per-iteration results: number, status, duration |
hint | string | A suggested next step for a failure the run could not narrow on its own. Omitted when there is nothing useful to suggest |
A category that measures bandwidth, failed, and recorded no bandwidth row for a
group of two or more nodes carries a hint pointing at testScale: diagnose.
That combination means the collective never moved bytes, and bisection can
separate a node-local fault from one on the path between nodes. The hint names
a next step; it does not assert what the fault is. Categories that do not
measure bandwidth never carry it, since for them an absent bandwidth row is
normal rather than a signal.
Each failedGroups[].failureLog contains podName, nodeName, and optional
exitCode, reason, and tail. exitCode is omitted when no nonzero container
exit code was captured (including timeout, PodNotFound, and
NoTerminatedContainer captures). The JSON tail is the same bounded workload-log excerpt stored
in Job.status.failureLog. The field is omitted when no captured log is
available, such as when the failed Job no longer exists.
See also
- Certification API — the resource these reports are generated from
- Interpret Results — how to read a certification report