Skip to navigation
Observability

Monitor Policy and Service Health

View as Markdown

Monitoring APF means answering three questions: are the services up, is every host enforcing the policy you published, and is anything being refused that should not be. APF answers them through health endpoints, audit records, and logs.

Service Health

CheckEndpointHealthy response
Policy service livenessGET /healthz on the HTTP surface200 with {"status":"ok","version":"<version>"}
Policy service readinessGET /readyz on the HTTP surface200 with {"status":"ok"}. 503 with {"status":"not_ready","reason":"no bundle loaded"} until a bundle is loaded.
Policy service over the socketPolicyService.Health gRPC on the Unix socketHEALTH_STATE_OK
Policy gatewayThe EvaluationService socket and the OpenShell listener portsThe socket file exists and the listeners accept connections.

The HTTP surface is optional. Without it, use the gRPC Health RPC from a process running as the service account, or check that the socket exists. The policy gateway has no HTTP endpoint. The local container stack checks it with test -S on its socket.

Health routes need no token. Keep the HTTP port bound to loopback, and do not expose it off the host for monitoring. Use a local agent or a narrow proxy that forwards only the health routes.

Policy Freshness

A host is enforcing the policy you published when the same revision appears at every stage. Compare policy_revision:

StageWhere to read the revision
PublishedYour artifact store.
Verified and active in the policy serviceThe latest policy_activation record with outcome success in apf-policy-service-audit.jsonl.
Active in the gatewayThe latest policy_activation record in apf-policy-gateway-audit.jsonl.
Applied to a sandboxThe sandbox’s apf.nvidia.com/policy-version annotation.

A gateway revision that stays lower than the policy service revision for longer than the gateway’s refresh interval has several possible causes. The update changed no rules, annotators, effects, or findings, so the gateway saw no change, which is expected. The gateway could not fetch, decode, or build the new policy set; check its log for get_document failed on hot-swap, PolicySet decode failed on hot-swap, or evaluator rebuild failed. The change landed before the gateway’s first poll after startup. Or bundle refresh is disabled with --bundle-refresh-interval-secs 0, or the gateway cannot reach the policy service; check its log for digest watcher: get_document failed. Refer to Policy Updates Do Not Take Effect. Sandboxes with a lower revision were admitted before the update and have not been refreshed.

Also track expires_at for the active bundle. APF does not warn before a bundle expires. Refer to Expiry and Rollback Protection.

Signals to Alert On

From Audit Records

SignalQueryMeaning
Weakened startupevent_type == "policy_service_startup" and security_state == "degraded"The policy service started with a weakened control, such as an ephemeral document key, an unauthenticated bundle source, a token in process arguments, or a non-loopback HTTP bind.
Rejected bundleevent_type == "policy_verification" and outcome == "refused"A delivered bundle failed verification.
Activation refusedevent_type == "policy_activation" and outcome == "refused"A verified bundle’s revision could not be persisted, or an HTTP source poll fetched a bundle that is not newer than the active one.
Unsigned document servedevent_type == "provenance_signing"A document was served without a provenance token.
Authentication refusalsevent_type == "authentication"Callers failing the trust boundary, such as extension_auth.*, auth.uds_peer_rejected, or auth.http_bearer_invalid. A spike can mean misconfiguration or probing.
Evaluation failuresreason_code == "middleware.evaluation_failed"Requests denied because evaluation could not complete.
Admission refusalsevent_type == "policy_injection" or "admission_validation" with outcome == "refused"Sandboxes that could not be admitted.

From Logs

Some conditions appear only in the services’ logs, which go to standard output and are filtered with RUST_LOG, info by default:

  • The gateway logs evaluator rebuild failed, PolicySet decode failed on hot-swap, or get_document failed on hot-swap when it keeps its previous evaluator after a bad update.
  • The gateway logs digest watcher: get_document failed when it cannot reach the policy service.
  • The policy service logs poll failed for variant when the HTTP source cannot fetch.
  • The gateway exits if it cannot record a policy hot swap. Alert on unexpected restarts.

Metrics and Capacity

In development. APF does not expose a metrics endpoint in this release, and capacity and performance guidance is not yet in the repository. Use the health checks, audit records, and logs above.

Next Steps