> This page is for NVIDIA Cluster Readiness Engine, version v0.5.0. > For other versions, use one of these documentation indexes: > - Latest · v0.7.0 (default): https://docs.nvidia.com/cluster-readiness-engine/v0.7.0/llms.txt > - v0.6.0: https://docs.nvidia.com/cluster-readiness-engine/v0.6.0/llms.txt > - v0.5.0: https://docs.nvidia.com/cluster-readiness-engine/v0.5.0/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/cluster-readiness-engine/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/cluster-readiness-engine/_mcp/server. # Troubleshooting > Diagnose and resolve common issues — stuck jobs, hardware detection failures, and stalls. This page is for operators who need to diagnose problems with the NVIDIA Cluster Readiness Engine (NVCRE). Each section follows a problem-solution format: symptoms, diagnostic commands, and fixes. ## Read lifecycle Events `kubectl describe` shows lifecycle Events for Certifications, Workflows, Jobs, and WorkloadRuns. An `InProgress` or `Succeeded` transition is `Normal`; a `Failed` transition is `Warning`. The Event reason and message match the condition that was persisted on the object. ```bash kubectl describe certifications.nvcre.nvidia.com -n kubectl describe workflows.nvcre.nvidia.com -n kubectl describe jobs.nvcre.nvidia.com -n kubectl describe workloadruns.nvcre.nvidia.com -n ``` Job Events also report hardware and validation verdicts. A completed workload emits `WorkloadCompleted`. If the Job configures thresholds, it subsequently emits `ThresholdsMet` or the validation failure reason. Without thresholds, there is no validation verdict Event. A hardware verdict leaves the Job phase unchanged, but its parent Workflow treats the verdict as terminal for that group attempt. Event notes are limited to 1,024 bytes. Longer notes end with `... [truncated]`; for phase and verdict Events, inspect the object's status conditions for the full message. GoodputMeasurement and BandwidthMeasurement do not emit phase-transition Events. Inspect the owning Job for threshold verdicts; measurement-specific action failures still appear on the measurement object. Events are best-effort diagnostics, not the source of truth. Confirm the current state in `.status.conditions`. Kubernetes retains Events according to the API server's `--event-ttl` setting, which defaults to one hour, so older transitions may no longer be present. ## Job not progressing **Symptoms:** Job stays in `InProgress` for longer than expected. No `Succeeded` or `Failed` condition appears. **Diagnosis:** ```bash # Check the Job conditions kubectl describe jobs.nvcre.nvidia.com -n # Check the underlying workload (use the kind from status.workloadRef) kubectl get trainjob -n kubectl get pods -l nvcre.nvidia.com/job= -n -o wide # Check controller logs for this job kubectl logs -n nvcre deploy/nvcre-manager \ | grep ``` **Solutions:** - The `InProgress` condition has reason `WorkloadPending` — the workload is queued, not stuck. On Kueue-managed clusters the TrainJob is created with `spec.suspend: true` and held until quota is admitted. NVCRE waits without counting queued time against `timeoutPerJob` or stall detection; check the queue (`kubectl get workloads -A` for Kueue) to see why admission is not happening. - The workload resource exists but is not completing — check Kubeflow Trainer logs and pod events. - Pods are `Pending` — verify GPU resources are available on target nodes (`kubectl describe node `). - Kubeflow Trainer is not running — confirm its pods are healthy (`kubectl get pods -n kubeflow-system`). ## Workflow stuck without a Job **Symptoms:** Workflow condition is `InProgress`, but no child Job appears. **Diagnosis:** ```bash # Check Workflow status and conditions kubectl get workflows.nvcre.nvidia.com -o yaml # Look for the child Job kubectl get jobs.nvcre.nvidia.com -l nvcre.nvidia.com/workflow= # Check if dependencies were created kubectl get workflows.nvcre.nvidia.com \ -o jsonpath='{.status.dependencyRefs}' | jq . ``` **Solutions:** - Dependency creation failed — check controller logs for RBAC errors when creating ConfigMaps, PVCs, or other dependency resources. - Target nodes do not match — verify the Workflow's `nodeSelector` or `nodeNames` matches existing nodes (`kubectl get nodes -l `). - The Workflow spec is invalid — look for validation errors in the controller logs. ## Workflow cleanup or deletion pauses on terminating pods **Symptoms:** After a Job fails, times out, or the Workflow is deleted, the group stays `Running` (or the Workflow stays terminating) for a while, and dependency resources such as the ComputeDomain are not removed immediately. This is expected: the controllers wait for the workload's pods to finish terminating before deleting the dependency resources that provide their DRA allocations. Deleting a ComputeDomain while pods are still terminating would revoke NVLink/NVSwitch channel allocations under running CUDA kernels and kill every pod process with CUDA error 719. The wait is bounded to 5 minutes per Job; after that, cleanup proceeds anyway and the controller logs a warning. **Diagnosis:** ```bash # See which pods the controller is waiting on kubectl get pods -l nvcre.nvidia.com/job= -n # Confirm the wait in the controller logs kubectl logs -n nvcre deploy/nvcre-manager \ | grep -E "still terminating|drain" ``` **Solutions:** - Pods are terminating normally — no action needed; cleanup resumes as soon as they are gone. - A pod is stuck `Terminating` (for example, its node is unreachable) — cleanup proceeds automatically after the 5-minute grace period. To unblock sooner, force-delete the pod: `kubectl delete pod --grace-period=0 --force`. **Only do this when the node is confirmed unreachable or the pod's processes are known to be dead**: force deletion removes the Pod object without waiting for its processes to stop, so the controller may then delete the ComputeDomain while CUDA kernels are still running on the node — the exact CUDA error 719 failure the drain wait exists to prevent. ## Failed with a name-collision reason **Symptoms:** A Certification fails with reason `WorkflowNameCollision`, or a Workflow fails with reason `JobNameCollision` or `DependencyNameCollision`. **What it means:** The name NVCRE generated for a child resource (a Workflow, a Job, or a dependency such as a PVC, ConfigMap, or TrainingRuntime) is already taken by an object NVCRE did not create. NVCRE never adopts such an object and never deletes it during cleanup — it fails the run instead, and the pre-existing object is left untouched. **Diagnosis:** ```bash # The condition message names the colliding object kubectl get certifications.nvcre.nvidia.com -n -o yaml kubectl get workflows.nvcre.nvidia.com -n -o yaml # Inspect the colliding object's owners and labels kubectl get -n -o yaml ``` **Solutions:** - The colliding object belongs to someone else — run the certification in a dedicated namespace, or rename your Certification/Workflow so the generated child names no longer collide. - The colliding object is a leftover from an earlier run that you no longer need — delete it manually, then re-create the Certification or Workflow. If the colliding object is already being deleted (it has a `deletionTimestamp`), NVCRE does not fail: it retries with backoff until the name is released. This covers deleting and re-creating a same-named Certification or Workflow while the previous run's children are still terminating. ## Hardware failures not detected **Symptoms:** A node has a known hardware problem, but the Job does not report the `HardwareFailed` condition. **Diagnosis:** ```bash # Verify pods are running on the expected nodes kubectl get pods -l nvcre.nvidia.com/job= -o wide # Check that the pod label was injected kubectl get pods -l nvcre.nvidia.com/job= --show-labels # Inspect the node for the expected conditions/taints kubectl get node -o yaml # Check controller logs for CEL evaluation errors kubectl logs -n nvcre deploy/nvcre-manager \ | grep "CEL" ``` **Solutions:** - The CEL expression has a syntax error — look for parse errors in controller logs. Test your expression against a node object. - Pods are not scheduled on target nodes — the controller only evaluates nodes where Job pods are running. Verify pod placement. - The pod label is missing — the controller injects `nvcre.nvidia.com/job` automatically. If pods were created before the controller started, they may lack the label. - Node conditions or taints do not exist yet — NVCRE only reads node state; it relies on your cluster's health monitoring stack to set the conditions or taints your CEL expression checks. Verify with `kubectl describe node`. ## High reconciliation latency **Symptoms:** Status updates are slow. The `nvcre_reconcile_duration_seconds` P95 is above 5 seconds. **Diagnosis:** ```bash # Check controller resource usage kubectl top pod -n nvcre # Check for API server throttling (429 responses) kubectl logs -n nvcre deploy/nvcre-manager \ | grep "throttling" ``` **Solutions:** - Controller is resource-constrained — increase CPU and memory limits. See the [Deployment](/cluster-readiness-engine/v0.5.0/operations/deployment) page for sizing guidance. - Too many nodes per health check — large clusters increase CEL evaluation time. Consider splitting workloads across fewer nodes. - API server is under load — check API server metrics and reduce concurrent reconciles if needed. ## Certification not progressing **Symptoms:** Certification stays `InProgress` after Workflows have finished. **Diagnosis:** ```bash # Check category statuses kubectl get certifications.nvcre.nvidia.com \ -o jsonpath='{.status.categoryStatuses}' | jq . # List child Workflows kubectl get workflows.nvcre.nvidia.com -l nvcre.nvidia.com/certification= # Check for CategoryNotFound errors kubectl logs -n nvcre deploy/nvcre-manager \ | grep "CategoryNotFound" ``` **Solutions:** - A category is not registered in the catalog — the controller sets the Certification to `Failed` with reason `CategoryNotFound`. Verify your domain and variant values against `nvcrectl certification list-categories`. - A child Workflow is still running — the Certification waits for all Workflows to complete before transitioning. ## Stall detection not triggering **Symptoms:** A Job appears stuck but is not marked `Failed` with reason `WorkloadStalled`. **Diagnosis:** ```bash # Verify stallMultiplier is set on the Job kubectl get jobs.nvcre.nvidia.com -o jsonpath='{.spec.stallMultiplier}' # Check if a GoodputMeasurement exists and has step data kubectl get goodputmeasurement -l nvcre.nvidia.com/job= -o yaml # Look for avgStepTimeSec and lastStepTimestamp kubectl get goodputmeasurement -l nvcre.nvidia.com/job= \ -o jsonpath='{.items[0].status.avgStepTimeSec}' ``` **Solutions:** - `stallMultiplier` is not set — stall detection is opt-in. Add `stallMultiplier` to the Job spec (e.g., `10` means stalled if no step for 10x average step time). - No GoodputMeasurement exists — stall detection requires a GoodputMeasurement to provide `avgStepTimeSec` and `lastStepTimestamp`. Configure `goodputMeasurement` on the Job. - Not enough training steps — the controller needs at least two non-warmup steps to compute `avgStepTimeSec`. Wait for the workload to produce more log output. ## BandwidthMeasurement not reporting results **Symptoms:** A BandwidthMeasurement exists but `status.results` is empty. **Diagnosis:** ```bash # Check the BandwidthMeasurement status kubectl get bandwidthmeasurement -o yaml # Verify the LogProfile has a bandwidthResult pattern kubectl get logprofile -o jsonpath='{.spec.patterns.bandwidthResult}' # Check pod logs for NCCL output kubectl logs | head -50 ``` **Solutions:** - The LogProfile is missing the `bandwidthResult` pattern — BandwidthMeasurement requires a LogProfile with a `bandwidthResult` regex. Add it to the LogProfile spec. - The regex does not match the NCCL output format — verify the regex against actual log output. NCCL test output format varies between versions. - The workload has not produced output yet — bandwidth results appear only after the NCCL test completes its message-size sweep. - The `replicatedJobName` is wrong — for MPI workloads, set `workerStrategy.replicatedJobName: launcher` in the LogProfile since NCCL output goes to the launcher pod. ## Enable debug logging For deeper investigation, enable debug-level logging by adding the flag to the manager container args: ```yaml # In the Deployment spec args: - --zap-log-level=1 ``` This surfaces detailed reconciliation traces and CEL evaluation results. ## Next steps - [Monitoring](/cluster-readiness-engine/v0.5.0/operations/monitoring) — Set up alerts so you catch issues before they need manual diagnosis. - [Deployment](/cluster-readiness-engine/v0.5.0/operations/deployment) — Review resource sizing and RBAC. - [Health Monitoring & Failed Node Attribution](/cluster-readiness-engine/v0.5.0/concepts/health-monitoring-failed-node-attribution) — Understand how CEL-based hardware detection works. - [Certify a Cluster](/cluster-readiness-engine/v0.5.0/how-to-guides/certify-a-cluster) — End-to-end certification walkthrough.