Runbooks

Runbook: Health Monitor UDS Communication Failures

View as Markdown

Overview

Health monitors (GPU, NVSwitch, syslog, CSP) publish events via gRPC over Unix Domain Socket (UDS) to platform-connector. Communication failures block all health event reporting.

Key points:

  • Platform-connector runs as a DaemonSet (one pod per node)
  • Each node has its own UDS socket (/var/run/nvsentinel.sock)
  • Both platform-connector and health monitors must mount /var/run/nvsentinel from hostPath
  • Platform-connector requires MongoDB connection during startup

Symptoms

  • Metric health_events_insertion_to_uds_error or trigger_uds_send_errors_total increasing
  • Health monitor logs show gRPC errors (code 14: Unavailable)
  • No health events in MongoDB despite monitors running

Procedure

1. Identify Affected Node

# Find which node has the failing health monitor
kubectl get pods -n nvsentinel -l app.kubernetes.io/name=gpu-health-monitor -o wide
# Check health monitor logs for UDS errors
kubectl logs -n nvsentinel {HEALTH_MONITOR_POD}

Look for:

  • "code = Unavailable" → Socket closed or platform-connector not running
  • "connection refused" → Socket doesn’t exist
  • "broken pipe" → Socket was closed mid-communication

2. Check Platform Connector on That Node

# Find platform-connector pod on the same node
kubectl get pods -n nvsentinel -l app.kubernetes.io/name=nvsentinel -o wide | grep {NODE_NAME}
# Check platform-connector logs
kubectl logs -n nvsentinel {PLATFORM_CONNECTOR_POD}

Look for errors:

  • "failed to initialize database store connector" → MongoDB connection failed
  • "failed to create database client" → MongoDB authentication or network issue
  • "failed to listen on unix socket" → Volume mount issue

3. Verify MongoDB Connectivity

Platform-connector requires MongoDB connection during startup. If MongoDB is unavailable, platform-connector will fail to start.

# Check MongoDB pods are running
kubectl get pods -n nvsentinel -l app.kubernetes.io/name=mongodb
# All pods should be Running and Ready
# Check certificates (mongo-root-ca, mongo-app-client-cert, mongo-server-cert-*)
kubectl get certificates -n nvsentinel
# All should show READY = True
# If certificates not ready, check cert-manager
kubectl get pods -n cert-manager
# Check MongoDB database creation job
kubectl get job -n nvsentinel -l app.kubernetes.io/name=create-mongodb-database
# Should show COMPLETIONS: 1/1

If the MongoDB job needs to be rerun:

# Delete by label (name is create-mongodb-database-<seconds>-<scriptHash>), then re-sync / helm upgrade
kubectl delete job -n nvsentinel -l app.kubernetes.io/name=create-mongodb-database

Platform-connector connects to MongoDB on port 27017 with TLS. Check network policies:

kubectl get networkpolicies -n nvsentinel -o yaml

4. Verify Volume Mounts

# Check platform-connector mount
kubectl get daemonset platform-connectors -n nvsentinel -o yaml | grep -A 3 "/var/run/nvsentinel"
# Check health monitor mount
kubectl get daemonset gpu-health-monitor -n nvsentinel -o yaml | grep -A 3 "/var/run/nvsentinel"

Both should be mounted from hostPath at /var/run/nvsentinel.

6. Verify Resolution

# Watch health monitor logs for successful sends
kubectl logs -n nvsentinel {HEALTH_MONITOR_POD} -f