Skip to navigation
Operations

Troubleshooting

View as Markdown

This page is for operators who need to diagnose problems with the NVIDIA Cluster Readiness Engine (NVCRE). Each section follows a problem-solution format: symptoms, diagnostic commands, and fixes.

Read lifecycle Events

kubectl describe shows lifecycle Events for Certifications, Workflows, Jobs, and WorkloadRuns. An InProgress or Succeeded transition is Normal; a Failed transition is Warning. The Event reason and message match the condition that was persisted on the object.

kubectl describe certifications.nvcre.nvidia.com <name> -n <namespace>
kubectl describe workflows.nvcre.nvidia.com <name> -n <namespace>
kubectl describe jobs.nvcre.nvidia.com <name> -n <namespace>
kubectl describe workloadruns.nvcre.nvidia.com <name> -n <namespace>

Job Events also report hardware and validation verdicts. A completed workload emits WorkloadCompleted. If the Job configures thresholds, it subsequently emits ThresholdsMet or the validation failure reason. Without thresholds, there is no validation verdict Event. A hardware verdict leaves the Job phase unchanged, but its parent Workflow treats the verdict as terminal for that group attempt.

Event notes are limited to 1,024 bytes. Longer notes end with ... [truncated]; for phase and verdict Events, inspect the object’s status conditions for the full message.

GoodputMeasurement and BandwidthMeasurement do not emit phase-transition Events. Inspect the owning Job for threshold verdicts; measurement-specific action failures still appear on the measurement object.

Events are best-effort diagnostics, not the source of truth. Confirm the current state in .status.conditions. Kubernetes retains Events according to the API server’s --event-ttl setting, which defaults to one hour, so older transitions may no longer be present.

Job not progressing

Symptoms: Job stays in InProgress for longer than expected. No Succeeded or Failed condition appears.

Diagnosis:

# Check the Job conditions
kubectl describe jobs.nvcre.nvidia.com <name> -n <namespace>
# Check the underlying workload (use the kind from status.workloadRef)
kubectl get trainjob -n <namespace>
kubectl get pods -l nvcre.nvidia.com/job=<name> -n <namespace> -o wide
# Check controller logs for this job
kubectl logs -n nvcre deploy/nvcre-manager \
  | grep <name>

Solutions:

  • The InProgress condition has reason WorkloadPending — the workload is queued, not stuck. On Kueue-managed clusters the TrainJob is created with spec.suspend: true and held until quota is admitted. NVCRE waits without counting queued time against timeoutPerJob or stall detection; check the queue (kubectl get workloads -A for Kueue) to see why admission is not happening.
  • The workload resource exists but is not completing — check Kubeflow Trainer logs and pod events.
  • Pods are Pending — verify GPU resources are available on target nodes (kubectl describe node <node>).
  • Kubeflow Trainer is not running — confirm its pods are healthy (kubectl get pods -n kubeflow-system).

Workflow stuck without a Job

Symptoms: Workflow condition is InProgress, but no child Job appears.

Diagnosis:

# Check Workflow status and conditions
kubectl get workflows.nvcre.nvidia.com <name> -o yaml
# Look for the child Job
kubectl get jobs.nvcre.nvidia.com -l nvcre.nvidia.com/workflow=<name>
# Check if dependencies were created
kubectl get workflows.nvcre.nvidia.com <name> \
  -o jsonpath='{.status.dependencyRefs}' | jq .

Solutions:

  • Dependency creation failed — check controller logs for RBAC errors when creating ConfigMaps, PVCs, or other dependency resources.
  • Target nodes do not match — verify the Workflow’s nodeSelector or nodeNames matches existing nodes (kubectl get nodes -l <selector>).
  • The Workflow spec is invalid — look for validation errors in the controller logs.

Workflow cleanup or deletion pauses on terminating pods

Symptoms: After a Job fails, times out, or the Workflow is deleted, the group stays Running (or the Workflow stays terminating) for a while, and dependency resources such as the ComputeDomain are not removed immediately.

This is expected: the controllers wait for the workload’s pods to finish terminating before deleting the dependency resources that provide their DRA allocations. Deleting a ComputeDomain while pods are still terminating would revoke NVLink/NVSwitch channel allocations under running CUDA kernels and kill every pod process with CUDA error 719. The wait is bounded to 5 minutes per Job; after that, cleanup proceeds anyway and the controller logs a warning.

Diagnosis:

# See which pods the controller is waiting on
kubectl get pods -l nvcre.nvidia.com/job=<name> -n <namespace>
# Confirm the wait in the controller logs
kubectl logs -n nvcre deploy/nvcre-manager \
  | grep -E "still terminating|drain"

Solutions:

  • Pods are terminating normally — no action needed; cleanup resumes as soon as they are gone.
  • A pod is stuck Terminating (for example, its node is unreachable) — cleanup proceeds automatically after the 5-minute grace period. To unblock sooner, force-delete the pod: kubectl delete pod <pod> --grace-period=0 --force. Only do this when the node is confirmed unreachable or the pod’s processes are known to be dead: force deletion removes the Pod object without waiting for its processes to stop, so the controller may then delete the ComputeDomain while CUDA kernels are still running on the node — the exact CUDA error 719 failure the drain wait exists to prevent.

Failed with a name-collision reason

Symptoms: A Certification fails with reason WorkflowNameCollision, or a Workflow fails with reason JobNameCollision or DependencyNameCollision.

What it means: The name NVCRE generated for a child resource (a Workflow, a Job, or a dependency such as a PVC, ConfigMap, or TrainingRuntime) is already taken by an object NVCRE did not create. NVCRE never adopts such an object and never deletes it during cleanup — it fails the run instead, and the pre-existing object is left untouched.

Diagnosis:

# The condition message names the colliding object
kubectl get certifications.nvcre.nvidia.com <name> -n <namespace> -o yaml
kubectl get workflows.nvcre.nvidia.com <name> -n <namespace> -o yaml
# Inspect the colliding object's owners and labels
kubectl get <kind> <colliding-name> -n <namespace> -o yaml

Solutions:

  • The colliding object belongs to someone else — run the certification in a dedicated namespace, or rename your Certification/Workflow so the generated child names no longer collide.
  • The colliding object is a leftover from an earlier run that you no longer need — delete it manually, then re-create the Certification or Workflow.

If the colliding object is already being deleted (it has a deletionTimestamp), NVCRE does not fail: it retries with backoff until the name is released. This covers deleting and re-creating a same-named Certification or Workflow while the previous run’s children are still terminating.

Hardware failures not detected

Symptoms: A node has a known hardware problem, but the Job does not report the HardwareFailed condition.

Diagnosis:

# Verify pods are running on the expected nodes
kubectl get pods -l nvcre.nvidia.com/job=<name> -o wide
# Check that the pod label was injected
kubectl get pods -l nvcre.nvidia.com/job=<name> --show-labels
# Inspect the node for the expected conditions/taints
kubectl get node <node> -o yaml
# Check controller logs for CEL evaluation errors
kubectl logs -n nvcre deploy/nvcre-manager \
  | grep "CEL"

Solutions:

  • The CEL expression has a syntax error — look for parse errors in controller logs. Test your expression against a node object.
  • Pods are not scheduled on target nodes — the controller only evaluates nodes where Job pods are running. Verify pod placement.
  • The pod label is missing — the controller injects nvcre.nvidia.com/job automatically. If pods were created before the controller started, they may lack the label.
  • Node conditions or taints do not exist yet — NVCRE only reads node state; it relies on your cluster’s health monitoring stack to set the conditions or taints your CEL expression checks. Verify with kubectl describe node.

High reconciliation latency

Symptoms: Status updates are slow. The nvcre_reconcile_duration_seconds P95 is above 5 seconds.

Diagnosis:

# Check controller resource usage
kubectl top pod -n nvcre
# Check for API server throttling (429 responses)
kubectl logs -n nvcre deploy/nvcre-manager \
  | grep "throttling"

Solutions:

  • Controller is resource-constrained — increase CPU and memory limits. See the Deployment page for sizing guidance.
  • Too many nodes per health check — large clusters increase CEL evaluation time. Consider splitting workloads across fewer nodes.
  • API server is under load — check API server metrics and reduce concurrent reconciles if needed.

Certification not progressing

Symptoms: Certification stays InProgress after Workflows have finished.

Diagnosis:

# Check category statuses
kubectl get certifications.nvcre.nvidia.com <name> \
  -o jsonpath='{.status.categoryStatuses}' | jq .
# List child Workflows
kubectl get workflows.nvcre.nvidia.com -l nvcre.nvidia.com/certification=<name>
# Check for CategoryNotFound errors
kubectl logs -n nvcre deploy/nvcre-manager \
  | grep "CategoryNotFound"

Solutions:

  • A category is not registered in the catalog — the controller sets the Certification to Failed with reason CategoryNotFound. Verify your domain and variant values against nvcrectl certification list-categories.
  • A child Workflow is still running — the Certification waits for all Workflows to complete before transitioning.

Stall detection not triggering

Symptoms: A Job appears stuck but is not marked Failed with reason WorkloadStalled.

Diagnosis:

# Verify stallMultiplier is set on the Job
kubectl get jobs.nvcre.nvidia.com <name> -o jsonpath='{.spec.stallMultiplier}'
# Check if a GoodputMeasurement exists and has step data
kubectl get goodputmeasurement -l nvcre.nvidia.com/job=<name> -o yaml
# Look for avgStepTimeSec and lastStepTimestamp
kubectl get goodputmeasurement -l nvcre.nvidia.com/job=<name> \
  -o jsonpath='{.items[0].status.avgStepTimeSec}'

Solutions:

  • stallMultiplier is not set — stall detection is opt-in. Add stallMultiplier to the Job spec (e.g., 10 means stalled if no step for 10x average step time).
  • No GoodputMeasurement exists — stall detection requires a GoodputMeasurement to provide avgStepTimeSec and lastStepTimestamp. Configure goodputMeasurement on the Job.
  • Not enough training steps — the controller needs at least two non-warmup steps to compute avgStepTimeSec. Wait for the workload to produce more log output.

BandwidthMeasurement not reporting results

Symptoms: A BandwidthMeasurement exists but status.results is empty.

Diagnosis:

# Check the BandwidthMeasurement status
kubectl get bandwidthmeasurement <name> -o yaml
# Verify the LogProfile has a bandwidthResult pattern
kubectl get logprofile <profile-name> -o jsonpath='{.spec.patterns.bandwidthResult}'
# Check pod logs for NCCL output
kubectl logs <pod-name> | head -50

Solutions:

  • The LogProfile is missing the bandwidthResult pattern — BandwidthMeasurement requires a LogProfile with a bandwidthResult regex. Add it to the LogProfile spec.
  • The regex does not match the NCCL output format — verify the regex against actual log output. NCCL test output format varies between versions.
  • The workload has not produced output yet — bandwidth results appear only after the NCCL test completes its message-size sweep.
  • The replicatedJobName is wrong — for MPI workloads, set workerStrategy.replicatedJobName: launcher in the LogProfile since NCCL output goes to the launcher pod.

Enable debug logging

For deeper investigation, enable debug-level logging by adding the flag to the manager container args:

# In the Deployment spec
args:
  - --zap-log-level=1

This surfaces detailed reconciliation traces and CEL evaluation results.

Next steps