Site Controller Health

Use this playbook when the problem affects site-controller nodes, core NICo services, cloud sync, or shared infrastructure rather than a single managed host.

Site-controller nodes run the NICo control plane. They are not managed hosts.

Quick Health Checklist

kubectl get nodes -o wide
kubectl get pods -A
kubectl get svc -A | grep LoadBalancer

Check critical namespaces for the site:

kubectl get pods -n <nico-namespace>
kubectl get pods -n postgres
kubectl get pods -n vault

Node and Kubernetes Layer

Core NICo Services

Postgres health needs more than kubectl get pods:

kubectl -n postgres exec pod/<postgres-pod> -c postgres -- patronictl list

Control-Plane Networking

Check:

• MetalLB BGP peers
• IP pools
• LoadBalancer services
• FRR speaker status
• DNS and service routing

kubectl get svc -n <nico-namespace> | grep LoadBalancer

Site Agent and Cloud Sync

Cloud-to-site sync failures can make the cloud UI and site state disagree.

Check site-agent logs:

kubectl logs -n <site-agent-namespace> -l app.kubernetes.io/name=<site-agent-label> | grep NicoClient

Common causes:

• site agent cannot reach nico-api
• mTLS cert projection problem
• DNS cold-cache or startup race
• cloud API connectivity issue
• site agent crash loop

Check a Site Agent pod

Each Site Agent pod reports its own state on its http port, 8080 unless the chart’s service.port changes it. The Site Agent Service picks a pod for each request and leaves out pods that aren’t Ready. So port-forward to the http port of the pod you want to check, such as nico-rest-site-agent-0:

kubectl port-forward -n <site-agent-namespace> pod/nico-rest-site-agent-0 8080:http
curl -s localhost:8080/readyz
curl -s localhost:8080/status | jq

/readyz returns ok, or a 503 with one line for each of Temporal, Core gRPC, and Flow gRPC that isn’t healthy, such as Temporal: Unhealthy. It checks Flow gRPC only when FLOW_GRPC_ENABLED is true, as it is by default in the chart. Any client that can reach the pod can read these endpoints, so /readyz, /healthz, and /status leave out the errors themselves. Find them in the pod’s logs with kubectl logs -n <site-agent-namespace> pod/nico-rest-site-agent-0. The Site Agent installation guide describes when /readyz and /healthz fail.

/status returns JSON. Only the pod whose name ends in -0 runs the bootstrap, so every other pod reports the bootstrap as disabled. This example comes from that pod with Flow gRPC enabled:

{
  "pod": {"name": "nico-rest-site-agent-0", "role": "Master"},
  "health": "Healthy",
  "bootstrap": {
    "enabled": true,
    "message": null,
    "credentialDownloadsAttempted": 1,
    "credentialDownloadsSucceeded": 1
  },
  "temporal": {
    "health": "Healthy",
    "connectionsAttempted": 1,
    "connectionsSucceeded": 1,
    "lastConnectionAttempt": "2026-10-06T20:50:00Z"
  },
  "coreGrpc": {"health": "Healthy", "requestsSucceeded": 120, "requestsFailed": 0},
  "flowGrpc": {"health": "Healthy", "requestsSucceeded": 40, "requestsFailed": 0}
}

Upgrades and Configuration

For config or upgrade issues:

• lint changed TOML where possible
• confirm generated ConfigMaps contain expected values
• confirm ArgoCD or deployment sync completed
• confirm required secrets were projected

Certificate and Secret Rotation

Credential and certificate issues often surface as unrelated BMC, API, or probe failures.

Check:

• Vault pod health
• nico-api to Vault connectivity
• certificate renewal on every control-plane node
• projected secrets in affected pods
• carbide_api_vault_requests_failed_total

The metric prefix may remain carbide_* even when the service is now named NICo.