> This page is for NVIDIA Cluster Readiness Engine, version Latest · v0.7.0 (default). > For other versions, use one of these documentation indexes: > - Latest · v0.7.0 (default): https://docs.nvidia.com/cluster-readiness-engine/v0.7.0/llms.txt > - v0.6.0: https://docs.nvidia.com/cluster-readiness-engine/v0.6.0/llms.txt > - v0.5.0: https://docs.nvidia.com/cluster-readiness-engine/v0.5.0/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/cluster-readiness-engine/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/cluster-readiness-engine/_mcp/server. # Health Monitoring & Failed Node Attribution > How the NVIDIA Cluster Readiness Engine detects node failures and records which nodes failed and why. ## Health monitoring The `Job` controller runs a pluggable `NodeFailureDetector` alongside each workload. Detectors evaluate node state continuously during the run and report failures before the workload exits. ### CEL detector The built-in detector evaluates [Common Expression Language (CEL)](https://cel.dev) expressions against Kubernetes `Node` objects. Conditions, labels, taints, and allocatable resources are all accessible as CEL fields. Multiple expressions can be combined with `&&` / `||`. Example expression that fires when a node's GPU ECC error count exceeds a threshold: ``` node.metadata.labels["nvidia.com/gpu.ecc.error.count"] > "0" ``` Custom detectors can be registered by implementing the `NodeFailureDetector` interface. ## Failed node attribution NVCRE does not taint, cordon, or patch nodes. Its job is to identify which nodes failed which tests and why. That signal is recorded on the Certification CR and is available for external systems (node lifecycle operators, alerting pipelines) to act on. ### How failures propagate Failed nodes bubble up through the three-tier hierarchy: ``` Job status.failedNodes[] ← inline list; set when the Job fails, one entry per node with a reason ↑ persisted to Workflow status.failedNodesRef ← ConfigMap reference; union across all failed Jobs for this category ↑ copied to Certification status.categoryStatuses[].failedNodesRef ← per-category ConfigMap reference ``` Failed nodes are stored in ConfigMaps (not inline on the status) at the Workflow and Certification tiers. To read them: ```bash # Certification tier kubectl get certification -o jsonpath='{.status.categoryStatuses[0].failedNodesRef.name}' kubectl get configmap -o yaml # Workflow tier kubectl get workflow -o jsonpath='{.status.failedNodesRef.name}' kubectl get configmap -o yaml ``` ### Failure reasons Each failed node entry carries a `reason`: | Reason | Meaning | |--------|---------| | `HardwareFailureDetected` | CEL health check detected an unhealthy node **mid-run** | | `ThresholdViolation` | A performance threshold (bandwidth, goodput, step time) was missed | | `WorkloadFailed` | The workload exited non-zero or stalled | <Note> Cordoned nodes and nodes reporting fewer allocatable GPUs than the workload requests per node are filtered **before** a Job runs by default, not attributed as `HardwareFailureDetected`. They appear in `status.orchestration.excludedNodes` with the reason in `exclusionReason` and cause the run to be marked `INCOMPLETE` rather than `Failed`. If **no** matching node can supply the requested GPU count, the run fails immediately with a message naming the requirement and the best available count instead of scheduling pods that would stay `Pending` forever. Operators can opt into targeting cordoned nodes directly by setting `target.taintSelectors` to the `node.kubernetes.io/unschedulable` taint, which also drops the hardware-failure check for the run. </Note> ### Queued (suspended) workloads A workload held by an admission controller — for example a TrainJob that Kueue suspends (`spec.suspend: true`) until quota is available — is **queued, not running**. The Job reports `InProgress` with reason `WorkloadPending` and simply waits: - No node failures are attributed while the workload is queued. - Queued time does not count against `timeoutPerJob` or stall detection. `timeoutPerJob` is measured from `status.workloadStartTime`, recorded when the workload is first observed running. The startup-stall clock is anchored on the GoodputMeasurement's timestamps but never starts before `status.workloadStartTime`. Without this distinction, a workload waiting for quota would eventually be treated as stalled or timed out and every node in the group would be recorded as `WorkloadFailed` — a false result on healthy hardware. ### Scheduling-blocked workloads A workload whose pods cannot be placed — for example a job pinned to one node whose GPUs are held by another tenant — reports `InProgress` with reason `WorkloadSchedulingBlocked`, and the condition message relays the scheduler's own diagnosis (for example `0/3 nodes are available: 1 Insufficient nvidia.com/gpu, 2 node(s) didn't match Pod's node affinity/selector`). - Detection requires the scheduler to have explicitly rejected the pod (`PodScheduled=False/Unschedulable`) and the state to persist past a grace window (default 5 minutes, overridable with `spec.schedulingStallGraceSeconds`). - Like queued workloads, blocked time does not count against `timeoutPerJob` or stall detection, from the first blocked observation (the grace window only delays the condition). `status.schedulingBlockedSince` records when the episode began; on recovery `status.workloadStartTime` is advanced by the paused interval, so the workload keeps only the budget it had left before the block. - The Workflow's `InProgress` condition carries reason `JobSchedulingBlocked` with the Job's message, and `nvcrectl certification report` shows a `Blocked:` line on the running category. - The condition clears automatically if the pod schedules. No terminal state is ever set by the detector — `timeoutPerJob` remains the bound. See [ADR-083](../designs/083-scheduling-stall-visibility.md). Different nodes in the same category can fail with different reasons. A node that fails in multiple categories appears in each category's failed-nodes ConfigMap, potentially with a different reason each time. ### Reading failed nodes Use the CLI for a structured report: ```bash nvcrectl certification report ``` Or read the raw ConfigMap directly: ```bash kubectl get certification -o jsonpath='{.status.categoryStatuses[0].failedNodesRef.name}' kubectl get configmap -o yaml ``` ### Acting on failed nodes NVCRE reports failures — quarantine is your platform's responsibility. Common patterns: - Cordon the node (`kubectl cordon `) to prevent new workloads from scheduling - Drain it (`kubectl drain `) to evict existing workloads - Trigger a node lifecycle operator or repair pipeline using the `failedNodesRef` ConfigMap data as input - After repair, uncordon the node and re-run the relevant category to confirm it passes ## Adaptive fault isolation When a run fails and multiple nodes are suspect, the controller can bisect the node pool to isolate the faulty node(s) without re-running the full suite. See [How-to: Adaptive Fault Isolation](/cluster-readiness-engine/how-to-guides/adaptive-fault-isolation).