> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/cluster-readiness-engine/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/cluster-readiness-engine/_mcp/server.

# FAQ

> Frequently asked questions — common gotchas, design choices, and tips from real-world usage.

## General

### How often should I run burn-in?

Recurring cluster-wide burn-in every 3–4 weeks with real ML training workloads is recommended to catch infant mortality failures and hardware degradation over time. Synthetic benchmarks alone are not enough — faults like NVLink bandwidth degradation and NCCL collective hangs only surface under sustained distributed load.

You can declare a Certification once and re-run it on a schedule. The Cluster Readiness Engine (CRE) handles orchestration, measurement, and failure detection each time.

### Can I run CRE on non-NVIDIA GPUs?

No. CRE is built for NVIDIA GPU clusters and depends on NVIDIA-specific health signals (`nvidia.com/gpu.product` labels, DCGM diagnostics, NVLink topology). The catalog entries target NeMo training, NCCL collectives, and DCGM diagnostics — all NVIDIA tooling.

### Can I use my own training workloads instead of the catalog?

Yes. The catalog is a convenience layer that provides pre-configured WorkflowSpecs. You can bypass it entirely by creating Workflow or Job resources directly with any workload spec (`trainJob`), or run a one-off workload with [WorkloadRun](/cluster-readiness-engine/getting-started/workload-run-quick-start). See [Custom Catalog Entries](/cluster-readiness-engine/how-to-guides/custom-catalog-entries) for adding your own catalog categories.

### What is the difference between a gray failure and a hardware failure?

A **hardware failure** is detected by the node health monitor — the CEL expression evaluates to `true`, indicating a clear signal (for example, a GPU-related taint or node condition set by your cluster's health monitoring stack, or a node marked unschedulable). The controller reports this immediately in the Job's `HardwareFailed` condition and records the node with reason `HardwareFailureDetected`.

A **gray failure** is when hardware is degraded but no health monitor fires. The workload runs but underperforms — reduced NCCL bandwidth, lower goodput, or intermittent hangs. CRE detects these through performance thresholds (goodput ratio, bus bandwidth) and adaptive fault isolation. See [Adaptive Fault Isolation](/cluster-readiness-engine/how-to-guides/adaptive-fault-isolation).

### What happens to failed nodes after repair?

Failed nodes are recorded in the Certification status with a reason (`HardwareFailureDetected`, `ThresholdViolation`, or `WorkloadFailed`). CRE never modifies nodes — it does not taint, cordon, or patch them. Node quarantine and repair are handled by your platform's own tooling. After repairing or replacing the failed hardware, re-run the Certification to verify the fix.

## Development

### Why do I need to run `make manifests generate` after editing types?

CRE uses kubebuilder markers in `*_types.go` files to auto-generate CRD manifests (`helm/cluster-readiness-engine/crds/`) and DeepCopy methods (`zz_generated.deepcopy.go`). If you modify a types file and skip this step, the generated files become stale — the controller binary won't compile because the DeepCopy methods reference the old struct shape, and the CRD YAML won't match the new fields.

Always run `make manifests generate` immediately after any change to `api/v1alpha1/*_types.go`.

### Why doesn't my custom catalog entry appear?

The catalog uses Go `init()` functions for registration. `pkg/catalog` (which loads the entries in `pkg/catalog/entries/`) must be imported by every binary and test suite that resolves catalog lookups — it is blank-imported in `cmd/nvcrectl/main.go` and imported by the controller in `cmd/manager/main.go`.

If you created a new catalog entry but the `init()` registration never runs in your binary or test suite, `catalog.Lookup()` returns "not found". See [Custom Catalog Entries](/cluster-readiness-engine/how-to-guides/custom-catalog-entries).

### Why doesn't cascade deletion work in my integration tests?

Kubernetes envtest (used by the integration test suite) does not run a garbage collection controller. This means `OwnerReference`-based cascade deletion does not work — child resources are not automatically deleted when the parent is deleted.

Controllers must explicitly delete child resources in their `handleDeletion()` methods. This is by design in envtest and is documented as a critical pitfall in the project's contributor docs.

## See also

- [Troubleshooting](/cluster-readiness-engine/operations/troubleshooting) — problem-solution pairs for specific issues
- [Architecture](/cluster-readiness-engine/concepts/architecture) — understand the Certification, Workflow, Job model