Skip to navigation
Operations

Troubleshooting

View as Markdown

This page is for operators who need to diagnose problems with the NVIDIA Cluster Readiness Engine (NVCRE). Each section follows a problem-solution format: symptoms, diagnostic commands, and fixes.

Read lifecycle Events

kubectl describe shows lifecycle Events for Certifications, Workflows, Jobs, and WorkloadRuns. An InProgress or Succeeded transition is Normal; a Failed transition is Warning. The Event reason and message match the condition that was persisted on the object.

kubectl describe certifications.nvcre.nvidia.com <name> -n <namespace>
kubectl describe workflows.nvcre.nvidia.com <name> -n <namespace>
kubectl describe jobs.nvcre.nvidia.com <name> -n <namespace>
kubectl describe workloadruns.nvcre.nvidia.com <name> -n <namespace>

Job Events also report hardware and validation verdicts. A completed workload emits WorkloadCompleted. If the Job configures thresholds, it subsequently emits ThresholdsMet or the validation failure reason. Without thresholds, there is no validation verdict Event. A hardware verdict leaves the Job phase unchanged, but its parent Workflow treats the verdict as terminal for that group attempt.

Event notes are limited to 1,024 bytes. Longer notes end with ... [truncated]; for phase and verdict Events, inspect the object’s status conditions for the full message.

GoodputMeasurement and BandwidthMeasurement do not emit phase-transition Events. Inspect the owning Job for threshold verdicts; measurement-specific action failures still appear on the measurement object.

Events are best-effort diagnostics, not the source of truth. Confirm the current state in .status.conditions. Kubernetes retains Events according to the API server’s --event-ttl setting, which defaults to one hour, so older transitions may no longer be present.

Job not progressing

Symptoms: Job stays in InProgress for longer than expected. No Succeeded or Failed condition appears.

Diagnosis:

# Check the Job conditions
kubectl describe jobs.nvcre.nvidia.com <name> -n <namespace>
# Check the underlying workload (use the kind from status.workloadRef)
kubectl get trainjob -n <namespace>
kubectl get pods -l nvcre.nvidia.com/job=<name> -n <namespace> -o wide
# Check controller logs for this job
kubectl logs -n nvcre deploy/nvcre-manager \
  | grep <name>

Solutions:

  • The InProgress condition has reason WorkloadPending — the workload is queued, not stuck. On Kueue-managed clusters the TrainJob is created with spec.suspend: true and held until quota is admitted. NVCRE waits without counting queued time against timeoutPerJob or stall detection; check the queue (kubectl get workloads -A for Kueue) to see why admission is not happening.
  • The workload resource exists but is not completing — check Kubeflow Trainer logs and pod events.
  • Pods are Pending — verify GPU resources are available on target nodes (kubectl describe node <node>).
  • The InProgress condition has reason WorkloadSchedulingBlocked — the workload is admitted but its pods cannot be placed (e.g. GPUs held by another tenant). The condition message relays the scheduler’s own diagnosis (also shown on the Workflow as reason JobSchedulingBlocked and as a Blocked: line in nvcrectl certification report); blocked time does not count against timeoutPerJob or stall detection. Tune with spec.schedulingStallGraceSeconds (default 5m).
  • The InProgress condition has reason WorkloadCreationRejected: the API server is refusing the workload create, usually an admission webhook, a ValidatingAdmissionPolicy or Kyverno/Gatekeeper rule, or an exhausted ResourceQuota. The condition message carries the API server’s own error, and the Workflow relays it under JobSchedulingBlocked as the Blocked: line in nvcrectl certification report. Check kubectl get events -n <namespace> --field-selector reason=WorkloadCreationRejected. NVCRE keeps retrying, so a rejection that clears (a TrainingRuntime created late, quota freed) recovers on its own. One that lasts past the controller’s --workload-creation-deadline (Helm manager.workloadCreationDeadline, default 10m, independent of timeoutPerJob) fails the Job with reason WorkloadCreationError and leaves its status.failedNodes empty, since nothing ran on them. The deadline is per Job attempt: with spec.orchestration.execution.retryFailedGroups set, each retry gets a fresh Job and a fresh clock, so a permanent denial costs (retries + 1) times the deadline before the group gives up. A missing RoleBinding is not a rejection: the controller confirms a Forbidden create with a SelfSubjectAccessReview, and an RBAC denial keeps retrying without this deadline.
  • Kubeflow Trainer is not running — confirm its pods are healthy (kubectl get pods -n kubeflow-system).

NCCL aborts on an NVLS multicast bind failure

Symptoms: A communication category fails early, before any bandwidth number is produced. The launcher log ends with this, repeated once per rank:

transport/nvls.cc:284 NCCL WARN Failed to bind NVLink SHARP (NVLS) Multicast memory of size 2097152 :
CUDA error 401 'the operation cannot be performed in the present state'.
This is usually caused by a system or configuration error in the Fabric Manager or NVSwitches.
Disable NVLS (NCCL_NVLS_ENABLE=0) if you wish to avoid this error in the future.

Every rank reports it at the same instant, and the Job fails with the generic reason WorkloadFailed. The text above is also captured to Job.status.failureLog.

What it means: NVLS is NVLink SHARP, the NVSwitch’s in-network reduction. NCCL binds GPU memory to a multicast object the switch can operate on; this is that bind being refused. Four things about it are easy to misread:

  • On a chassis-local NVSwitch the failing call is intra-node. Where the log reports MNNVL 0 — an HGX baseboard, or an NVL72 tray with MNNVL off — the multicast team is built over the local ranks only and the handle never leaves the host, so a failure here is not a fabric, network, or multi-node problem even when the job spans nodes. The absence of via NET/IB channel lines is then a consequence of the abort happening during init, not independent evidence of a network fault. On GB200 or GB300 NVL72 with MNNVL 1, the multicast group spans the NVLink domain through IMEX and this does not hold: the team crosses trays, so the per-node checks below are the wrong place to start. Check the IMEX domain and the NVLink Subnet Manager first, and read the rack as one fabric rather than one node at a time.
  • The NVLS Head table is not a cross-chassis group. On a multi-node job NCCL prints NVLS Head 0: 0 4, NVLS Head 1: 1 5, and so on, which reads as if ranks on different nodes were joined into one multicast team. The row lists one rank per node: these are the per-node heads of an NVLS tree, not members of a multicast group. The block is printed only by the first node’s ranks, so its absence elsewhere is not an asymmetry.
  • The capability probe passing does not mean multicast works. NVLS multicast support is available on dev N is the driver answering CU_DEVICE_ATTRIBUTE_MULTICAST_SUPPORTED. Seeing that line followed by a failed bind is the signature of a host whose switch-side multicast state is wrong, not of hardware that lacks the feature.
  • NCCL 2.29 made this fatal. Through NCCL 2.28 a failed bind at the default setting logged Proceeding without NVLS support. and the job continued. From 2.29 the fallback is gone and the failure aborts the job. The same cluster state that was survivable on an older image fails on a newer one with no change on the NVCRE side. The last line distinguishes the two: Disable NVLS (NCCL_NVLS_ENABLE=0) is 2.29 or later, Do not force-enable NVLS (NCCL_NVLS_ENABLE=1) is 2.28 or earlier.

Diagnosis:

# Confirm the signature and which NCCL version produced it
kubectl logs <launcher-pod> -n <namespace> | grep -E "nvls\.cc|NCCL version|multicast support"
# The captured tail is on the Job as well
kubectl get jobs.nvcre.nvidia.com <name> -n <namespace> -o jsonpath='{.status.failureLog}'

On each affected node, which is where the answer usually is:

systemctl status nvidia-fabricmanager
nvidia-smi -q | grep -i -A4 fabric
grep -i error /var/log/fabricmanager.log

Solutions:

  • Fabric Manager was restarted without a subsequent GPU reset — multicast stays non-functional while capability detection still reports success. Rebooting the node is the simplest fix. To reset in place instead, follow the order the Fabric Manager User Guide gives: drain the GPU workloads, stop Fabric Manager, run nvidia-smi -r, start Fabric Manager, then resume the workloads. Note that nvidia-smi -r with no -i resets every GPU on the host, and on an HGX system that can reset the NVSwitches with them.
  • Fabric Manager is not running, or its version does not match the driver — start it and align the versions. On a fourth-generation NVSwitch system (HGX B100, B200 and B300 8-GPU, and DGX B200/B300) Fabric Manager also needs NVLSM, the NVLink Subnet Manager, so confirm that service is installed and running too. Third-generation systems such as H100 and H200 have no such dependency.
  • The run must proceed before the host can be fixed — a WorkloadRun can disable NVLS with NCCL_NVLS_ENABLE=0 in spec.env, or -x NCCL_NVLS_ENABLE=0 in spec.framework.mpi.mpiArgs. A Certification has no equivalent field today (#501). Treat this as a way to keep working, not as a fix: a certification that passes with NVLS disabled has not exercised NVLink SHARP, and the underlying fault is still there.

Check what your own render carries before assuming either way. The WorkloadRun base environment no longer sets NCCL_NVLS_ENABLE, so where nothing else sets it NCCL’s default decides from the device attribute. Several platform fragments still set 1 explicitly, as field-tuned values for their hardware: onprem GB200/GB300, nscale B200/GB300, togetherai, azure H100, and the AWS GB300 transport profile; GCP H100 and the AWS GB200 training profile set 0. nvcrectl certification render and nvcrectl workloadrun render show the resolved value for your platform and GPU architecture.

MPI workers refuse SSH sessions

Symptoms: On a release that predates the fix for #461, the node pods of a multi-node NCCL category or a WorkloadRun with framework.mpi reach 1/1 Running, the launcher starts, and mpirun fails at once with the generic ORTE banner (“ORTE was unable to reliably start one or more daemons”). On a current release the same cluster shows the workers stuck at 0/1 with no launcher pod, and the Job stays InProgress until timeoutPerJob. In both cases the worker log shows Server listening on 0.0.0.0 port 22 followed, for every inbound connection, by fatal: chroot("/run/sshd"): Operation not permitted [preauth].

This is the container runtime’s default capability set. sshd runs every session through privilege separation, which calls chroot() before authenticating; that needs CAP_SYS_CHROOT. Docker and containerd grant it by default. CRI-O does not, so on OpenShift the listener comes up and every session dies. The MPI worker container now requests SYS_CHROOT explicitly, and its readiness probe opens a real loopback SSH session instead of a TCP connect, so a worker in this state stays NotReady and the launcher’s dependsOn gate never releases.

Diagnosis:

# Workers never become Ready: the probe output names the failure
kubectl describe pod <node-pod> -n <namespace> | grep -A3 "Readiness probe failed"
# Confirm the chroot failure in the worker log
kubectl logs <node-pod> -n <namespace> | grep -E "chroot|preauth"
# Confirm the capability was admitted into the pod
kubectl get pod <node-pod> -n <namespace> \
  -o jsonpath='{.spec.containers[0].securityContext.capabilities.add}'

Solutions:

  • The capability list does not include SYS_CHROOT: the release predates the fix. Upgrade; the fix is in the catalog NCCL entries, the WorkloadRun MPI builder, and the AWS platform overrides.
  • The pod was rejected at admission with a message naming SYS_CHROOT: the SecurityContextConstraints granted to the workload’s service account (OpenShift), or the namespace’s Pod Security level, does not allow it. Grant an SCC whose allowedCapabilities lists both IPC_LOCK and SYS_CHROOT. Under Pod Security admission only the privileged level admits these workers: baseline allows SYS_CHROOT but not the IPC_LOCK they already needed, and restricted (like the stock restricted-v2 SCC) allows neither.
  • The workers are Ready but mpirun still prints the ORTE banner on a current release: the worker image ships sshd without an ssh client, so the probe fell back to the listening check and could not see the refused sessions. Prebake openssh-client alongside openssh-server; the recipe in the air-gapped FAQ does both, and on Debian and Ubuntu the server package pulls the client in on its own.

MPI workers stuck installing sshd

Symptoms: A multi-node NCCL category or a WorkloadRun with framework.mpi has one or more node pods at 0/1 and no launcher pod. The worker log shows apt-get update or apt-get install ... openssh-server, and may show sshd install attempt N of 3 failed or timed out. Or the worker exits at once with sshd is not installed and the image has no apt-get or timeout to install it.

When the workload image has no /usr/sbin/sshd, each worker installs openssh-server from the Ubuntu archive at start-up. Each apt-get step is bounded at 120 seconds, and the worker makes three attempts. Before each retry it attempts to recover an install a timeout interrupted, with dpkg --configure -a, itself bounded at 120 seconds. A step that reaches its limit is killed together with its process group. Dpkg::Use-Pty=0 keeps apt’s dpkg child in that group instead of starting a separate terminal session. A hung apt-get therefore costs one attempt instead of the whole timeoutPerJob (#527): at most 4 minutes for the first attempt and 6 for a retry. If every step of every attempt ran to its limit, the three attempts would take 16 minutes. If all three attempts fail, the worker container exits with status 1 and its restart policy decides what happens next.

Diagnosis:

# Find the worker that is not Ready
kubectl get pods -n <namespace> -l nvcre.nvidia.com/job=<job-name> -o wide
# See how far the install got and how many attempts failed
kubectl logs <node-pod> -n <namespace> | grep -E "apt-get|Ign:|Err:|install attempt"
# Check that the pod can reach the Ubuntu archive its apt uses. Run it while
# the log shows install attempts: an image that gets that far has bash and
# timeout, while one that exits at once has nothing to check here.
# arm64 nodes (GB200, GB300) use ports.ubuntu.com
kubectl exec <node-pod> -n <namespace> -- \
  bash -c 'timeout 15 bash -c "exec 3<>/dev/tcp/ports.ubuntu.com/80" && echo connect-ok'
# amd64 nodes use archive.ubuntu.com
kubectl exec <node-pod> -n <namespace> -- \
  bash -c 'timeout 15 bash -c "exec 3<>/dev/tcp/archive.ubuntu.com/80" && echo connect-ok'

Solutions:

  • A failed attempt followed by a successful one needs no action. If it happens on many workers or in every run, look at the mirror or the egress path, or prebake sshd as below.
  • Every attempt fails: the nodes cannot reach the Ubuntu archive, or a proxy is required. Fix the egress path, or prebake sshd into the workload image and set image on the Certification or WorkloadRun, as described in the air-gapped FAQ. A prebaked image skips the install entirely, which also removes the start-up delay.
  • The worker exits at once saying the image has no apt-get or timeout: the image is not Debian-based, or lacks coreutils, so the bootstrap cannot install sshd safely. Prebake sshd into the image.

Workflow stuck without a Job

Symptoms: Workflow condition is InProgress, but no child Job appears.

Diagnosis:

# Check Workflow status and conditions
kubectl get workflows.nvcre.nvidia.com <name> -o yaml
# Look for the child Job
kubectl get jobs.nvcre.nvidia.com -l nvcre.nvidia.com/workflow=<name>
# Check if dependencies were created
kubectl get workflows.nvcre.nvidia.com <name> \
  -o jsonpath='{.status.dependencyRefs}' | jq .

Solutions:

  • The InProgress reason is NamespacePermissionsMissing or NamespaceAccessCheckFailed — the controller has no write access in the namespace yet. See Workflow waits with NamespacePermissionsMissing.
  • The Workflow failed with DependencyCreationError and a forbidden message — an admission policy, webhook or ResourceQuota rejected the dependency. The controller checked that its own RBAC allows the write before failing, so the rejection’s message names the policy that blocked it.
  • Target nodes do not match — verify the Workflow’s nodeSelector or nodeNames matches existing nodes (kubectl get nodes -l <selector>).
  • The Workflow spec is invalid — look for validation errors in the controller logs.

Workflow waits with NamespacePermissionsMissing

Symptoms: a Workflow, or the Certification category above it, stays InProgress with reason NamespacePermissionsMissing or NamespaceAccessCheckFailed. Nothing is created in the namespace. kubectl describe shows a NamespaceAccessDenied or NamespaceAccessReviewError Warning, and certification run --wait shows the same message on the category’s Blocked: line. After manager.namespaceAccessTimeout (default 10 minutes) the Workflow fails with the same reason.

Cause: the chart is installed with rbac.scope: namespaced, and the namespace has no RoleBinding granting nvcre-run-role to the controller (ADR-092). Before it writes anything, the controller checks its own access with SelfSubjectAccessReviews. A denied review means the binding is missing (NamespacePermissionsMissing). A review that could not be created at all means the check could not run (NamespaceAccessCheckFailed); an unanswered check is never treated as permission.

Diagnosis:

# The message names the ServiceAccount and every permission it lacks
kubectl get workflows.nvcre.nvidia.com <name> -n <namespace> \
  -o jsonpath='{.status.conditions[?(@.type=="NamespaceAccess")]}' | jq .
# Is there a binding, and does it name the right ServiceAccount?
kubectl get rolebinding nvcre-manager-run -n <namespace> -o yaml
# Ask the API server what the controller may do there
kubectl auth can-i create configmaps -n <namespace> \
  --as=system:serviceaccount:nvcre:nvcre-manager
# Which scope is the chart installed with?
helm get values nvcre -n nvcre -a | grep -A2 '^rbac:'

Solutions:

  • Apply the binding. The Workflow resumes on its own within a minute, and its NamespaceAccess condition turns True:

    nvcrectl rbac print --namespace &lt;namespace> | kubectl apply -f -

  • The binding exists but names another ServiceAccount: the release was installed under another name or namespace. Compare the subject with kubectl get clusterrolebinding nvcre-manager-rolebinding -o yaml and print the binding with --service-account and --controller-namespace.

  • NamespaceAccessCheckFailed for any other reason: the reviews themselves fail, usually because the API server or an admission webhook is struggling. The Workflow keeps rechecking, backing off to once every two minutes, and resumes on its own once reviews succeed again.

  • NamespaceAccessCheckFailed with selfsubjectaccessreviews ... is forbidden: Kubernetes normally grants these reviews to authenticated identities, including ServiceAccounts, through system:basic-user. This error means that default grant was removed or narrowed and the controller has no other grant. Check the system:basic-user role and binding, and nvcre-manager-role. Upgrading the NVCRE release adds an explicit create grant on selfsubjectaccessreviews to nvcre-manager-role; an older chart alone does not cause this error.

  • For a long-lived namespace, list it in rbac.runNamespaces so the chart keeps the binding.

  • The Workflow already failed after the timeout: apply the binding and run again. A failed Workflow does not retry.

Workflow cleanup or deletion pauses on terminating pods

Symptoms: After a Job fails, times out, or the Workflow is deleted, the group stays Running (or the Workflow stays terminating) for a while, and dependency resources such as the ComputeDomain are not removed immediately.

This is expected: the controllers wait for the workload’s pods to finish terminating before deleting the dependency resources that provide their DRA allocations. Deleting a ComputeDomain while pods are still terminating would revoke NVLink/NVSwitch channel allocations under running CUDA kernels and kill every pod process with CUDA error 719. The wait is bounded to 5 minutes per Job; after that, cleanup proceeds anyway and the controller logs a warning.

Diagnosis:

# See which pods the controller is waiting on
kubectl get pods -l nvcre.nvidia.com/job=<name> -n <namespace>
# Confirm the wait in the controller logs
kubectl logs -n nvcre deploy/nvcre-manager \
  | grep -E "still terminating|drain"

Solutions:

  • Pods are terminating normally — no action needed; cleanup resumes as soon as they are gone.
  • A pod is stuck Terminating (for example, its node is unreachable) — cleanup proceeds automatically after the 5-minute grace period. To unblock sooner, force-delete the pod: kubectl delete pod <pod> --grace-period=0 --force. Only do this when the node is confirmed unreachable or the pod’s processes are known to be dead: force deletion removes the Pod object without waiting for its processes to stop, so the controller may then delete the ComputeDomain while CUDA kernels are still running on the node — the exact CUDA error 719 failure the drain wait exists to prevent.

Failed with a name-collision reason

Symptoms: A Certification fails with reason WorkflowNameCollision, or a Workflow fails with reason JobNameCollision or DependencyNameCollision.

What it means: The name NVCRE generated for a child resource (a Workflow, a Job, or a dependency such as a PVC, ConfigMap, or TrainingRuntime) is already taken by an object NVCRE did not create. NVCRE never adopts such an object and never deletes it during cleanup — it fails the run instead, and the pre-existing object is left untouched.

Diagnosis:

# The condition message names the colliding object
kubectl get certifications.nvcre.nvidia.com <name> -n <namespace> -o yaml
kubectl get workflows.nvcre.nvidia.com <name> -n <namespace> -o yaml
# Inspect the colliding object's owners and labels
kubectl get <kind> <colliding-name> -n <namespace> -o yaml

Solutions:

  • The colliding object belongs to someone else — run the certification in a dedicated namespace, or rename your Certification/Workflow so the generated child names no longer collide.
  • The colliding object is a leftover from an earlier run that you no longer need — delete it manually, then re-create the Certification or Workflow.

If the colliding object is already being deleted (it has a deletionTimestamp), NVCRE does not fail: it retries with backoff until the name is released. This covers deleting and re-creating a same-named Certification or Workflow while the previous run’s children are still terminating.

Hardware failures not detected

Symptoms: A node has a known hardware problem, but the Job does not report the HardwareFailed condition.

Diagnosis:

# Verify pods are running on the expected nodes
kubectl get pods -l nvcre.nvidia.com/job=<name> -o wide
# Check that the pod label was injected
kubectl get pods -l nvcre.nvidia.com/job=<name> --show-labels
# Inspect the node for the expected conditions/taints
kubectl get node <node> -o yaml
# Check controller logs for CEL evaluation errors
kubectl logs -n nvcre deploy/nvcre-manager \
  | grep "CEL"

Solutions:

  • The CEL expression has a syntax error — look for parse errors in controller logs. Test your expression against a node object.
  • Pods are not scheduled on target nodes — the controller only evaluates nodes where Job pods are running. Verify pod placement.
  • The pod label is missing — the controller injects nvcre.nvidia.com/job automatically. If pods were created before the controller started, they may lack the label.
  • Node conditions or taints do not exist yet — NVCRE only reads node state; it relies on your cluster’s health monitoring stack to set the conditions or taints your CEL expression checks. Verify with kubectl describe node.

High reconciliation latency

Symptoms: Status updates are slow. The nvcre_reconcile_duration_seconds P95 is above 5 seconds.

Diagnosis:

# Check controller resource usage
kubectl top pod -n nvcre
# Check for API server throttling (429 responses)
kubectl logs -n nvcre deploy/nvcre-manager \
  | grep "throttling"

Solutions:

  • Controller is resource-constrained — increase CPU and memory limits. See the Deployment page for sizing guidance.
  • Too many nodes per health check — large clusters increase CEL evaluation time. Consider splitting workloads across fewer nodes.
  • API server is under load — check API server metrics and reduce concurrent reconciles if needed.

Certification not progressing

Symptoms: Certification stays InProgress after Workflows have finished.

Diagnosis:

# Check category statuses
kubectl get certifications.nvcre.nvidia.com <name> \
  -o jsonpath='{.status.categoryStatuses}' | jq .
# List child Workflows
kubectl get workflows.nvcre.nvidia.com -l nvcre.nvidia.com/certification=<name>
# Check for CategoryNotFound errors
kubectl logs -n nvcre deploy/nvcre-manager \
  | grep "CategoryNotFound"

Solutions:

  • A category is not registered in the catalog — the controller sets the Certification to Failed with reason CategoryNotFound. Verify your domain and variant values against nvcrectl certification list-categories.
  • A child Workflow is still running — the Certification waits for all Workflows to complete before transitioning.

Stall detection not triggering

Symptoms: A Job appears stuck but is not marked Failed with reason WorkloadStalled.

Diagnosis:

# Verify stallMultiplier is set on the Job
kubectl get jobs.nvcre.nvidia.com <name> -o jsonpath='{.spec.stallMultiplier}'
# Check if a GoodputMeasurement exists and has step data
kubectl get goodputmeasurement -l nvcre.nvidia.com/job=<name> -o yaml
# Look for avgStepTimeSec and lastStepTimestamp
kubectl get goodputmeasurement -l nvcre.nvidia.com/job=<name> \
  -o jsonpath='{.items[0].status.avgStepTimeSec}'

Solutions:

  • stallMultiplier is not set — stall detection is opt-in. Add stallMultiplier to the Job spec (e.g., 10 means stalled if no step for 10x average step time).
  • No GoodputMeasurement exists — stall detection requires a GoodputMeasurement to provide avgStepTimeSec and lastStepTimestamp. Configure goodputMeasurement on the Job.
  • Not enough training steps — the controller needs at least two non-warmup steps to compute avgStepTimeSec. Wait for the workload to produce more log output.

BandwidthMeasurement not reporting results

Symptoms: A BandwidthMeasurement exists but status.results is empty.

Diagnosis:

# Check the BandwidthMeasurement status
kubectl get bandwidthmeasurement <name> -o yaml
# Verify the LogProfile has a bandwidthResult pattern
kubectl get logprofile <profile-name> -o jsonpath='{.spec.patterns.bandwidthResult}'
# Check pod logs for NCCL output
kubectl logs <pod-name> | head -50

Solutions:

  • The LogProfile is missing the bandwidthResult pattern — BandwidthMeasurement requires a LogProfile with a bandwidthResult regex. Add it to the LogProfile spec.
  • The regex does not match the NCCL output format — verify the regex against actual log output. NCCL test output format varies between versions.
  • The workload has not produced output yet — bandwidth results appear only after the NCCL test completes its message-size sweep.
  • The replicatedJobName is wrong — for MPI workloads, set workerStrategy.replicatedJobName: launcher in the LogProfile since NCCL output goes to the launcher pod.
  • The logProfileRef does not name an existing LogProfile — check kubectl get logprofile <name>; LogProfiles are cluster-scoped.
  • The LogProfile’s containerName names no container in the launcher pod — the log read fails with an error from the API server. Leave it empty to read the pod’s only container.

Bandwidth threshold fails with MeasurementTimeout

Symptoms: A Job succeeded, but ValidationFailed is True with reason MeasurementTimeout for busBandwidthGBps or algBandwidthGBps, and its BandwidthMeasurement is Complete with reason LogsUnavailable or NoDataCollected.

Bandwidth thresholds are evaluated only against final results, read from the launcher’s full log after the Job succeeds. The measurement could not produce them, so the value stays unmeasured and validation fails once measurementTimeout expires.

Diagnosis:

# The Complete condition's message names the cause
kubectl get bandwidthmeasurement <name> -o jsonpath='{.status.conditions[?(@.type=="Complete")]}'
# Retries of the final read are recorded as Warning events
kubectl get events --field-selector involvedObject.name=<name>

Solutions:

  • LogsUnavailable with an error reading the pod log — the launcher pod or its log was gone or unreachable for the whole retry period, or the LogProfile’s containerName names no container in the pod. Check that nothing deletes the TrainJob or its pods before thresholds are evaluated, and that the controller can reach the kubelet for pods/log. If the log is only slow to become readable, raise measurementTimeout; the read is retried for as long as the Job waits (see BandwidthMeasurement for where to set it).
  • LogsUnavailable with an error naming the LogProfile (getting LogProfile ..., a pattern that does not compile) or a missing pod (no pods found for ... replicatedJob ...) — the measurement never found the log to read. Check logProfileRef and workerStrategy.replicatedJobName as described in BandwidthMeasurement not reporting results.
  • LogsUnavailable because the log cannot be read in full — for example a line longer than 1 MiB. Retrying cannot help; reduce the workload’s log output.
  • LogsUnavailable because the referenced Job no longer exists — it was deleted, or retired when its group was retried. The retry’s Job has its own name (…-retry-<n>) and its own BandwidthMeasurement.
  • NoDataCollected — the whole log was read but no line matched the bandwidthResult pattern. See BandwidthMeasurement not reporting results.

Enable debug logging

For deeper investigation, enable debug-level logging by adding the flag to the manager container args:

# In the Deployment spec
args:
  - --zap-log-level=1

This surfaces detailed reconciliation traces and CEL evaluation results.

Next steps