Skip to navigation
Reference

Observability Metrics for Harmonizer NIM

View as Markdown

Harmonizer NIM exposes standard NIM health and readiness endpoints over HTTP. For endpoint details, refer to the API Reference.

General Observability Metrics

The GET /v1/metrics endpoint returns Prometheus text on the HTTP port. The route is available with either HTTP or gRPC inference transport. The serving stack records HTTP and gRPC request metrics; inspect the response from the running release for its available metric names.

To inspect the metrics exposed by your deployment, run:

curl http://localhost:8000/v1/metrics

Readiness Signals

The signals cover different layers of the serving stack:

  • GET /v1/health/live reports that the HTTP process, or the gRPC server queried by the companion HTTP process, is live.
  • GET /v1/health/ready is the deployment readiness signal. In HTTP mode it checks the Harmonizer backend’s internal ready state. In gRPC mode it reflects the standard gRPC health service.
  • grpc.health.v1.Health/Check reports SERVING after the gRPC server has started and NOT_SERVING during shutdown or after an unrecoverable CUDA error is detected.

Model loading and warmup occur before either inference server begins accepting traffic. A normal cold start therefore appears as connection failure or an incomplete startup probe until initialization succeeds. After the HTTP server is reachable, readiness returns 200. In HTTP mode it can return 503 if the backend later transitions to an unready state. A startup error such as a missing model artifact normally terminates startup before a health endpoint is reachable.

Fatal CUDA Error Recovery

Harmonizer treats CUDA error 226 (cudaErrorContained), including an invalid peer GPU memory access over NVLink or a hardware error, as process-fatal. After this error, the CUDA context cannot safely continue serving inference requests.

When Harmonizer detects this condition while serving, it performs a bounded shutdown:

  • The backend is marked unready. GET /v1/health/ready returns 503, and the gRPC health service reports NOT_SERVING.
  • New GPU work is rejected and queued work is settled with an error. Where the request transport is still able to respond, HTTP callers receive 503 and gRPC callers receive UNAVAILABLE.
  • After a 250 ms grace period for health propagation and in-flight responses, the inference process exits with status 70 and the container terminates.

If the failure occurs during model loading or warmup, the server and health endpoints have not started. In that case, the inference process exits with status 70 immediately, without an observable readiness transition.

The unready or NOT_SERVING state is transitional, not a degraded steady state. The readiness endpoint changes immediately, but the transition is shorter than a typical Kubernetes readiness-probe cadence. The subsequent process exit makes the pod unready and removes it from Service endpoints. The exit, together with the pod’s restart policy, causes the container to restart; the readiness probe itself does not restart the container.

NIM management endpoints expose service version, model, manifest, license, and metrics information independently of inference responses. These are /v1/version, /v1/models, /v1/manifest, /v1/license, /v1/metadata, and /v1/metrics.

Continue with these related guides: