How-to Guides

Interpret Results

View as Markdown

Get a report

$nvcrectl certification report <name>
$nvcrectl workloadrun report <name>

Report structure

SectionContents
SummaryOverall pass/fail, node count, run duration
Category resultsPer-domain/variant measured vs. expected values with pass/fail
Node resultsPer-node breakdown — which passed, which failed, and the failure reason

Status values

StatusMeaning
PassedAll categories met thresholds on all node groups
FailedOne or more categories failed; affected nodes listed in ConfigMaps referenced by status.categoryStatuses[].failedNodesRef
InProgressStill running

Bandwidth results

  • Measured bus bandwidth (GB/s) per collective
  • Expected threshold for the detected GPU architecture
  • Pass/fail per operation

Below-threshold results indicate a network issue — degraded link, misconfigured EFA/RoCE, or faulty NIC.

Goodput results

  • Goodput ratio (0.0–1.0) — fraction of time the job was making useful forward progress
  • Expected minimum from the catalog entry
  • A ratio below ~0.95 suggests stalls, slow nodes, or framework overhead

Failed nodes

Failed nodes are stored in a ConfigMap referenced by status.categoryStatuses[].failedNodesRef for each category. The Job tier records them inline on status.failedNodes; the Workflow and Certification tiers persist them to ConfigMaps.

$# Get the ConfigMap name for the first category
$kubectl get certification <name> -o jsonpath='{.status.categoryStatuses[0].failedNodesRef.name}'
$# Read the ConfigMap contents
$kubectl get configmap <ref-name> -o yaml

Or use the CLI for a formatted report:

$nvcrectl certification report <name>

CRE records which nodes failed and why — it does not taint or cordon them. To quarantine a failed node:

$kubectl cordon <node>    # prevent new workloads
$kubectl drain <node>     # evict existing workloads

After repair, uncordon the node and re-run the relevant category to confirm it passes.