Troubleshooting#

The components deploy as Argo CD Applications in health-gated waves: a wave does not start until the previous one is Healthy. When the rollout stalls, one Application is failing and the rest are waiting on it. Find the failing Application, read why it failed, then go to that component’s page for the fix.

Find the failing Application#

List every Application with its sync and health status:

kubectl -n argocd get applications

Each row shows a sync status (Synced or OutOfSync) and a health status (Healthy, Progressing, Degraded, or Missing). The Application that is Degraded or stuck in Progressing is the one to investigate. Applications still Missing are waiting on it, not broken. Component Application names carry the cluster-name prefix, for example k8s-admin-openbao.

Read why it failed#

Show the failing Application’s sync result, health, and conditions:

kubectl -n argocd describe application <name>

The Conditions and the last sync Message usually name the cause: a Helm render error, a failed hook Job, or a resource that never becomes healthy. When the Application is Degraded, inspect its workload in the component’s namespace:

kubectl -n <namespace> get pods
kubectl -n <namespace> describe pod <pod>
kubectl -n <namespace> logs <pod>

The pod events and logs show the runtime cause, such as an image-pull failure, a CrashLoopBackOff, or a missing Secret. The Argo CD web UI shows the same Application status, resource tree, and events once the gateway serves it.

Once you know which component is failing, refer to that component’s page for the fix. The following table maps the common cases.

Common failures#

Symptom	Likely cause and where to look
An Application stays `Missing`, or Argo CD reports `not allowed to download code`.	The repository token lacks read access. Refer to Argo CD Installation for the required role and scope.
An Application stays `OutOfSync` after you publish.	Argo CD has not polled yet. Trigger a refresh with `kubectl -n argocd annotate application/root argocd.argoproj.io/refresh=hard --overwrite`.
OpenBao is sealed or in `CrashLoopBackOff`.	The `openbao-seal` Secret, created by the seed step, is missing. Refer to OpenBao.
A workload cannot mount its volume or find its persistent data.	Check the PersistentVolumeClaim, pod events, and shared mounts. A missing or inconsistent NFS mount can leave data inaccessible when a pod moves to another node. Refer to Check Persistent Storage.
A UI does not load even though every Application is `Healthy`.	The host name does not resolve to the address for the access path you chose, the head-node gateway proxy is not configured, or the gateway certificate does not cover the host name. Refer to Reach the UIs.
Sign-in fails or bounces back to the sign-in page.	The realm import is off, `keycloakFQDN` does not match the Keycloak host, or the Launchpad pod cannot reach Keycloak because the head-node gateway proxy is not configured. Refer to Reach the UIs, Keycloak, and NMC Launchpad.
The seed step fails.	The required NGC image-pull token is missing from `secrets.yaml`. Refer to NMC Launchpad.
The OpenBao configure Job fails.	A `values.yaml` setting tells OpenBao to expect a credential that `secrets.yaml` does not supply. Refer to Installing GitOps-Managed Components for which settings need a matching secret.
BCM LDAP users do not appear in the realm.	`ldapHost` does not match the LDAP server certificate SAN, or `ldapHostIp` does not point at your LDAP server. Read the KeycloakX log for LDAP, TLS, and bind errors. Refer to BCM LDAP authentication (optional).
Argo CD fails to pull its Redis image with `429 Too Many Requests` or `data limit exceeded`.	Argo CD pulls Redis from AWS ECR Public, which rate-limits anonymous pulls. Set `useDockerHubRedisImage` and `k8s_argocd_use_docker_hub_redis_image` to `true` to pull Redis from Docker Hub. Refer to Argo CD and Argo CD Installation.

Check Persistent Storage#

On the active BCM head node, inspect the failing workload’s PersistentVolumeClaim (PVC) and pod events:

module load kubernetes/k8s-admin
kubectl -n <namespace> get pvc
kubectl -n <namespace> describe pvc <pvc-name>
kubectl -n <namespace> describe pod <pod-name>

A Pending PVC can indicate a provisioning failure or that it is waiting for a consumer pod. Read the events before choosing a recovery action. A Bound PVC confirms that Kubernetes assigned a volume; it does not prove that the same data is accessible on every node.

If a pod cannot find previously initialized data, repeat Verify Shared Storage on every node in the cluster.

If the mounts are missing or inconsistent, stop the installation. Do not mount over directories used by running workloads. Establish how to stop the affected workloads and preserve any required data before correcting storage in BCM.