> This page is for version 1.0.0 (Latest) (default).
> For other versions, use one of these documentation indexes:
> - 1.0.0 (Latest) (default): https://docs.nvidia.com/nim/harmonizer/1.0.0/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nim/harmonizer/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nim/harmonizer/_mcp/server.

# Observability Metrics for Harmonizer NIM

> Find Harmonizer NIM metrics and health endpoints, readiness behavior, and fatal CUDA error recovery.

Harmonizer NIM exposes standard NIM health and readiness endpoints over HTTP.
For endpoint details, refer to the [API Reference](/nim/harmonizer/api-reference).

## General Observability Metrics

The `GET /v1/metrics` endpoint returns Prometheus text on the HTTP port.
The route is available with either HTTP or gRPC inference transport.
The serving stack records HTTP and gRPC request metrics; inspect the
response from the running release for its available metric names.

To inspect the metrics exposed by your deployment, run:

```bash
curl http://localhost:8000/v1/metrics
```

## Readiness Signals

The signals cover different layers of the serving stack:

* `GET /v1/health/live` reports that the HTTP process, or the gRPC server
  queried by the companion HTTP process, is live.
* `GET /v1/health/ready` is the deployment readiness signal. In HTTP mode it
  checks the Harmonizer backend's internal `ready` state. In gRPC mode it
  reflects the standard gRPC health service.
* `grpc.health.v1.Health/Check` reports `SERVING` after the gRPC server has
  started and `NOT_SERVING` during shutdown or after an unrecoverable CUDA
  error is detected.

Model loading and warmup occur before either inference server begins accepting
traffic. A normal cold start therefore appears as connection failure or an
incomplete startup probe until initialization succeeds. After the HTTP server is
reachable, readiness returns `200`. In HTTP mode it can return `503` if the
backend later transitions to an unready state. A startup error such as a missing
model artifact normally terminates startup before a health endpoint is reachable.

## Fatal CUDA Error Recovery

Harmonizer treats CUDA error 226 (`cudaErrorContained`), including an invalid
peer GPU memory access over NVLink or a hardware error, as process-fatal. After
this error, the CUDA context cannot safely continue serving inference requests.

When Harmonizer detects this condition while serving, it performs a bounded
shutdown:

* The backend is marked unready. `GET /v1/health/ready` returns `503`, and
  the gRPC health service reports `NOT_SERVING`.
* New GPU work is rejected and queued work is settled with an error. Where the
  request transport is still able to respond, HTTP callers receive `503` and
  gRPC callers receive `UNAVAILABLE`.
* After a 250 ms grace period for health propagation and in-flight responses,
  the inference process exits with status `70` and the container terminates.

If the failure occurs during model loading or warmup, the server and health
endpoints have not started. In that case, the inference process exits with
status `70` immediately, without an observable readiness transition.

The unready or `NOT_SERVING` state is transitional, not a degraded steady
state. The readiness endpoint changes immediately, but the transition is shorter
than a typical Kubernetes readiness-probe cadence. The subsequent process exit
makes the pod unready and removes it from Service endpoints. The exit, together
with the pod's restart policy, causes the container to restart; the readiness
probe itself does not restart the container.

NIM management endpoints expose service version, model, manifest, license, and
metrics information independently of inference responses. These are
`/v1/version`, `/v1/models`, `/v1/manifest`, `/v1/license`,
`/v1/metadata`, and `/v1/metrics`.

## Related Topics

Continue with these related guides:

* [Deploy with Helm](/nim/harmonizer/helm)
* [Troubleshooting Harmonizer NIM](/nim/harmonizer/troubleshooting)