Skip to navigation
Operations

Troubleshooting

View as Markdown

This page is for operators who need to diagnose problems with the NVIDIA Cluster Readiness Engine (NVCRE). Each section follows a problem-solution format: symptoms, diagnostic commands, and fixes.

Read lifecycle Events

kubectl describe shows lifecycle Events for Certifications, Workflows, Jobs, and WorkloadRuns. An InProgress or Succeeded transition is Normal; a Failed transition is Warning. The Event reason and message match the condition that was persisted on the object.

kubectl describe certifications.nvcre.nvidia.com <name> -n <namespace>
kubectl describe workflows.nvcre.nvidia.com <name> -n <namespace>
kubectl describe jobs.nvcre.nvidia.com <name> -n <namespace>
kubectl describe workloadruns.nvcre.nvidia.com <name> -n <namespace>

Job Events also report hardware and validation verdicts. A completed workload emits WorkloadCompleted. If the Job configures thresholds, it subsequently emits ThresholdsMet or the validation failure reason. Without thresholds, there is no validation verdict Event. A hardware verdict leaves the Job phase unchanged, but its parent Workflow treats the verdict as terminal for that group attempt.

Event notes are limited to 1,024 bytes. Longer notes end with ... [truncated]; for phase and verdict Events, inspect the object’s status conditions for the full message.

GoodputMeasurement and BandwidthMeasurement do not emit phase-transition Events. Inspect the owning Job for threshold verdicts; measurement-specific action failures still appear on the measurement object.

Events are best-effort diagnostics, not the source of truth. Confirm the current state in .status.conditions. Kubernetes retains Events according to the API server’s --event-ttl setting, which defaults to one hour, so older transitions may no longer be present.

Job not progressing

Symptoms: Job stays in InProgress for longer than expected. No Succeeded or Failed condition appears.

Diagnosis:

# Check the Job conditions
kubectl describe jobs.nvcre.nvidia.com <name> -n <namespace>
# Check the underlying workload (use the kind from status.workloadRef)
kubectl get trainjob -n <namespace>
kubectl get pods -l nvcre.nvidia.com/job=<name> -n <namespace> -o wide
# Check controller logs for this job
kubectl logs -n nvcre deploy/nvcre-manager \
  | grep <name>

Solutions:

  • The InProgress condition has reason WorkloadPending — the workload is queued, not stuck. On Kueue-managed clusters the TrainJob is created with spec.suspend: true and held until quota is admitted. NVCRE waits without counting queued time against timeoutPerJob or stall detection; check the queue (kubectl get workloads -A for Kueue) to see why admission is not happening.
  • The workload resource exists but is not completing — check Kubeflow Trainer logs and pod events.
  • Pods are Pending — verify GPU resources are available on target nodes (kubectl describe node <node>).
  • The InProgress condition has reason WorkloadSchedulingBlocked — the workload is admitted but its pods cannot be placed (e.g. GPUs held by another tenant). The condition message relays the scheduler’s own diagnosis (also shown on the Workflow as reason JobSchedulingBlocked and as a Blocked: line in nvcrectl certification report); blocked time does not count against timeoutPerJob or stall detection. Tune with spec.schedulingStallGraceSeconds (default 5m).
  • Kubeflow Trainer is not running — confirm its pods are healthy (kubectl get pods -n kubeflow-system).

MPI workers refuse SSH sessions

Symptoms: On a release that predates the fix for #461, the node pods of a multi-node NCCL category or a WorkloadRun with framework.mpi reach 1/1 Running, the launcher starts, and mpirun fails at once with the generic ORTE banner (“ORTE was unable to reliably start one or more daemons”). On a current release the same cluster shows the workers stuck at 0/1 with no launcher pod, and the Job stays InProgress until timeoutPerJob. In both cases the worker log shows Server listening on 0.0.0.0 port 22 followed, for every inbound connection, by fatal: chroot("/run/sshd"): Operation not permitted [preauth].

This is the container runtime’s default capability set. sshd runs every session through privilege separation, which calls chroot() before authenticating; that needs CAP_SYS_CHROOT. Docker and containerd grant it by default. CRI-O does not, so on OpenShift the listener comes up and every session dies. The MPI worker container now requests SYS_CHROOT explicitly, and its readiness probe opens a real loopback SSH session instead of a TCP connect, so a worker in this state stays NotReady and the launcher’s dependsOn gate never releases.

Diagnosis:

# Workers never become Ready: the probe output names the failure
kubectl describe pod <node-pod> -n <namespace> | grep -A3 "Readiness probe failed"
# Confirm the chroot failure in the worker log
kubectl logs <node-pod> -n <namespace> | grep -E "chroot|preauth"
# Confirm the capability was admitted into the pod
kubectl get pod <node-pod> -n <namespace> \
  -o jsonpath='{.spec.containers[0].securityContext.capabilities.add}'

Solutions:

  • The capability list does not include SYS_CHROOT: the release predates the fix. Upgrade; the fix is in the catalog NCCL entries, the WorkloadRun MPI builder, and the AWS platform overrides.
  • The pod was rejected at admission with a message naming SYS_CHROOT: the SecurityContextConstraints granted to the workload’s service account (OpenShift), or the namespace’s Pod Security level, does not allow it. Grant an SCC whose allowedCapabilities lists both IPC_LOCK and SYS_CHROOT. Under Pod Security admission only the privileged level admits these workers: baseline allows SYS_CHROOT but not the IPC_LOCK they already needed, and restricted (like the stock restricted-v2 SCC) allows neither.
  • The workers are Ready but mpirun still prints the ORTE banner on a current release: the worker image ships sshd without an ssh client, so the probe fell back to the listening check and could not see the refused sessions. Prebake openssh-client alongside openssh-server; the recipe in the air-gapped FAQ does both, and on Debian and Ubuntu the server package pulls the client in on its own.

Workflow stuck without a Job

Symptoms: Workflow condition is InProgress, but no child Job appears.

Diagnosis:

# Check Workflow status and conditions
kubectl get workflows.nvcre.nvidia.com <name> -o yaml
# Look for the child Job
kubectl get jobs.nvcre.nvidia.com -l nvcre.nvidia.com/workflow=<name>
# Check if dependencies were created
kubectl get workflows.nvcre.nvidia.com <name> \
  -o jsonpath='{.status.dependencyRefs}' | jq .

Solutions:

  • Dependency creation failed — check controller logs for RBAC errors when creating ConfigMaps, PVCs, or other dependency resources.
  • Target nodes do not match — verify the Workflow’s nodeSelector or nodeNames matches existing nodes (kubectl get nodes -l <selector>).
  • The Workflow spec is invalid — look for validation errors in the controller logs.

Workflow cleanup or deletion pauses on terminating pods

Symptoms: After a Job fails, times out, or the Workflow is deleted, the group stays Running (or the Workflow stays terminating) for a while, and dependency resources such as the ComputeDomain are not removed immediately.

This is expected: the controllers wait for the workload’s pods to finish terminating before deleting the dependency resources that provide their DRA allocations. Deleting a ComputeDomain while pods are still terminating would revoke NVLink/NVSwitch channel allocations under running CUDA kernels and kill every pod process with CUDA error 719. The wait is bounded to 5 minutes per Job; after that, cleanup proceeds anyway and the controller logs a warning.

Diagnosis:

# See which pods the controller is waiting on
kubectl get pods -l nvcre.nvidia.com/job=<name> -n <namespace>
# Confirm the wait in the controller logs
kubectl logs -n nvcre deploy/nvcre-manager \
  | grep -E "still terminating|drain"

Solutions:

  • Pods are terminating normally — no action needed; cleanup resumes as soon as they are gone.
  • A pod is stuck Terminating (for example, its node is unreachable) — cleanup proceeds automatically after the 5-minute grace period. To unblock sooner, force-delete the pod: kubectl delete pod <pod> --grace-period=0 --force. Only do this when the node is confirmed unreachable or the pod’s processes are known to be dead: force deletion removes the Pod object without waiting for its processes to stop, so the controller may then delete the ComputeDomain while CUDA kernels are still running on the node — the exact CUDA error 719 failure the drain wait exists to prevent.

Failed with a name-collision reason

Symptoms: A Certification fails with reason WorkflowNameCollision, or a Workflow fails with reason JobNameCollision or DependencyNameCollision.

What it means: The name NVCRE generated for a child resource (a Workflow, a Job, or a dependency such as a PVC, ConfigMap, or TrainingRuntime) is already taken by an object NVCRE did not create. NVCRE never adopts such an object and never deletes it during cleanup — it fails the run instead, and the pre-existing object is left untouched.

Diagnosis:

# The condition message names the colliding object
kubectl get certifications.nvcre.nvidia.com <name> -n <namespace> -o yaml
kubectl get workflows.nvcre.nvidia.com <name> -n <namespace> -o yaml
# Inspect the colliding object's owners and labels
kubectl get <kind> <colliding-name> -n <namespace> -o yaml

Solutions:

  • The colliding object belongs to someone else — run the certification in a dedicated namespace, or rename your Certification/Workflow so the generated child names no longer collide.
  • The colliding object is a leftover from an earlier run that you no longer need — delete it manually, then re-create the Certification or Workflow.

If the colliding object is already being deleted (it has a deletionTimestamp), NVCRE does not fail: it retries with backoff until the name is released. This covers deleting and re-creating a same-named Certification or Workflow while the previous run’s children are still terminating.

Hardware failures not detected

Symptoms: A node has a known hardware problem, but the Job does not report the HardwareFailed condition.

Diagnosis:

# Verify pods are running on the expected nodes
kubectl get pods -l nvcre.nvidia.com/job=<name> -o wide
# Check that the pod label was injected
kubectl get pods -l nvcre.nvidia.com/job=<name> --show-labels
# Inspect the node for the expected conditions/taints
kubectl get node <node> -o yaml
# Check controller logs for CEL evaluation errors
kubectl logs -n nvcre deploy/nvcre-manager \
  | grep "CEL"

Solutions:

  • The CEL expression has a syntax error — look for parse errors in controller logs. Test your expression against a node object.
  • Pods are not scheduled on target nodes — the controller only evaluates nodes where Job pods are running. Verify pod placement.
  • The pod label is missing — the controller injects nvcre.nvidia.com/job automatically. If pods were created before the controller started, they may lack the label.
  • Node conditions or taints do not exist yet — NVCRE only reads node state; it relies on your cluster’s health monitoring stack to set the conditions or taints your CEL expression checks. Verify with kubectl describe node.

High reconciliation latency

Symptoms: Status updates are slow. The nvcre_reconcile_duration_seconds P95 is above 5 seconds.

Diagnosis:

# Check controller resource usage
kubectl top pod -n nvcre
# Check for API server throttling (429 responses)
kubectl logs -n nvcre deploy/nvcre-manager \
  | grep "throttling"

Solutions:

  • Controller is resource-constrained — increase CPU and memory limits. See the Deployment page for sizing guidance.
  • Too many nodes per health check — large clusters increase CEL evaluation time. Consider splitting workloads across fewer nodes.
  • API server is under load — check API server metrics and reduce concurrent reconciles if needed.

Certification not progressing

Symptoms: Certification stays InProgress after Workflows have finished.

Diagnosis:

# Check category statuses
kubectl get certifications.nvcre.nvidia.com <name> \
  -o jsonpath='{.status.categoryStatuses}' | jq .
# List child Workflows
kubectl get workflows.nvcre.nvidia.com -l nvcre.nvidia.com/certification=<name>
# Check for CategoryNotFound errors
kubectl logs -n nvcre deploy/nvcre-manager \
  | grep "CategoryNotFound"

Solutions:

  • A category is not registered in the catalog — the controller sets the Certification to Failed with reason CategoryNotFound. Verify your domain and variant values against nvcrectl certification list-categories.
  • A child Workflow is still running — the Certification waits for all Workflows to complete before transitioning.

Stall detection not triggering

Symptoms: A Job appears stuck but is not marked Failed with reason WorkloadStalled.

Diagnosis:

# Verify stallMultiplier is set on the Job
kubectl get jobs.nvcre.nvidia.com <name> -o jsonpath='{.spec.stallMultiplier}'
# Check if a GoodputMeasurement exists and has step data
kubectl get goodputmeasurement -l nvcre.nvidia.com/job=<name> -o yaml
# Look for avgStepTimeSec and lastStepTimestamp
kubectl get goodputmeasurement -l nvcre.nvidia.com/job=<name> \
  -o jsonpath='{.items[0].status.avgStepTimeSec}'

Solutions:

  • stallMultiplier is not set — stall detection is opt-in. Add stallMultiplier to the Job spec (e.g., 10 means stalled if no step for 10x average step time).
  • No GoodputMeasurement exists — stall detection requires a GoodputMeasurement to provide avgStepTimeSec and lastStepTimestamp. Configure goodputMeasurement on the Job.
  • Not enough training steps — the controller needs at least two non-warmup steps to compute avgStepTimeSec. Wait for the workload to produce more log output.

BandwidthMeasurement not reporting results

Symptoms: A BandwidthMeasurement exists but status.results is empty.

Diagnosis:

# Check the BandwidthMeasurement status
kubectl get bandwidthmeasurement <name> -o yaml
# Verify the LogProfile has a bandwidthResult pattern
kubectl get logprofile <profile-name> -o jsonpath='{.spec.patterns.bandwidthResult}'
# Check pod logs for NCCL output
kubectl logs <pod-name> | head -50

Solutions:

  • The LogProfile is missing the bandwidthResult pattern — BandwidthMeasurement requires a LogProfile with a bandwidthResult regex. Add it to the LogProfile spec.
  • The regex does not match the NCCL output format — verify the regex against actual log output. NCCL test output format varies between versions.
  • The workload has not produced output yet — bandwidth results appear only after the NCCL test completes its message-size sweep.
  • The replicatedJobName is wrong — for MPI workloads, set workerStrategy.replicatedJobName: launcher in the LogProfile since NCCL output goes to the launcher pod.
  • The logProfileRef does not name an existing LogProfile — check kubectl get logprofile <name>; LogProfiles are cluster-scoped.
  • The LogProfile’s containerName names no container in the launcher pod — the log read fails with an error from the API server. Leave it empty to read the pod’s only container.

Bandwidth threshold fails with MeasurementTimeout

Symptoms: A Job succeeded, but ValidationFailed is True with reason MeasurementTimeout for busBandwidthGBps or algBandwidthGBps, and its BandwidthMeasurement is Complete with reason LogsUnavailable or NoDataCollected.

Bandwidth thresholds are evaluated only against final results, read from the launcher’s full log after the Job succeeds. The measurement could not produce them, so the value stays unmeasured and validation fails once measurementTimeout expires.

Diagnosis:

# The Complete condition's message names the cause
kubectl get bandwidthmeasurement <name> -o jsonpath='{.status.conditions[?(@.type=="Complete")]}'
# Retries of the final read are recorded as Warning events
kubectl get events --field-selector involvedObject.name=<name>

Solutions:

  • LogsUnavailable with an error reading the pod log — the launcher pod or its log was gone or unreachable for the whole retry period, or the LogProfile’s containerName names no container in the pod. Check that nothing deletes the TrainJob or its pods before thresholds are evaluated, and that the controller can reach the kubelet for pods/log. If the log is only slow to become readable, raise measurementTimeout; the read is retried for as long as the Job waits (see BandwidthMeasurement for where to set it).
  • LogsUnavailable with an error naming the LogProfile (getting LogProfile ..., a pattern that does not compile) or a missing pod (no pods found for ... replicatedJob ...) — the measurement never found the log to read. Check logProfileRef and workerStrategy.replicatedJobName as described in BandwidthMeasurement not reporting results.
  • LogsUnavailable because the log cannot be read in full — for example a line longer than 1 MiB. Retrying cannot help; reduce the workload’s log output.
  • LogsUnavailable because the referenced Job no longer exists — it was deleted, or replaced when its group was retried. The replacement Job gets its own BandwidthMeasurement.
  • NoDataCollected — the whole log was read but no line matched the bandwidthResult pattern. See BandwidthMeasurement not reporting results.

Enable debug logging

For deeper investigation, enable debug-level logging by adding the flag to the manager container args:

# In the Deployment spec
args:
  - --zap-log-level=1

This surfaces detailed reconciliation traces and CEL evaluation results.

Next steps