> This page is for version 1.14.0 (default).
> For other versions, use one of these documentation indexes:
> - 1.14.0 (default): https://docs.nvidia.com/nim/llama-3-1-nemotron-safety-guard-8b/1.14.0/llms.txt
> - 1.10.1: https://docs.nvidia.com/nim/llama-3-1-nemotron-safety-guard-8b/1.10.1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nim/llama-3-1-nemotron-safety-guard-8b/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nim/llama-3-1-nemotron-safety-guard-8b/_mcp/server.

# Observability for Nemotron Safety Guard NIM

> Collect and visualize Prometheus metrics from Nemotron Safety Guard NIM.

NIM provides [Prometheus](https://prometheus.io/) metrics for request statistics. You can use these metrics to create dashboards with [Grafana](https://grafana.com/docs/grafana/latest/dashboards/use-dashboards/). By default, metrics are available at <code>[http://localhost:8000/v1/metrics](http://localhost:8000/v1/metrics)</code>.

| Category | Metric                 | Metric name                     | Description                                                                                | Granularity | Frequency     |
| -------- | ---------------------- | ------------------------------- | ------------------------------------------------------------------------------------------ | ----------- | ------------- |
| KV Cache | GPU Cache Usage        | `gpu_cache_usage_perc`          | GPU KV-cache usage. A value of 1 means 100% usage.                                         | Per model   | Per iteration |
| Count    | Running Count          | `num_requests_running`          | Number of requests currently running on GPU.                                               | Per model   | Per iteration |
| Count    | Waiting Count          | `num_requests_waiting`          | Number of requests waiting to be processed.                                                | Per model   | Per iteration |
| Count    | Max Request Count      | `num_request_max`               | Maximum number of requests that can run concurrently by the model.                         | Per model   | Per iteration |
| Count    | Prompt Token Count     | `prompt_tokens_total`           | Number of prefill tokens processed.                                                        | Per model   | Per iteration |
| Count    | Generation Token Count | `generation_tokens_total`       | Number of generation tokens processed.                                                     | Per model   | Per iteration |
| Latency  | Time to First Token    | `time_to_first_token_seconds`   | Histogram of time to first token in seconds.                                               | Per model   | Per request   |
| Latency  | Time per Output Token  | `time_per_output_token_seconds` | Histogram of time per output token in seconds.                                             | Per model   | Per request   |
| Latency  | End to End             | `e2e_request_latency_seconds`   | Histogram of end-to-end request latency in seconds.                                        | Per model   | Per request   |
| Count    | Prompt Token Count     | `request_prompt_tokens`         | Histogram of prefill tokens processed.                                                     | Per model   | Per request   |
| Count    | Generation Token Count | `request_generation_tokens`     | Histogram of generation tokens processed.                                                  | Per model   | Per request   |
| Count    | Finished Request Count | `request_finish_total`          | Number of finished requests, with a label indicating the finish reason.                    | Per model   | Per request   |
| Count    | Success Request Count  | `request_success_total`         | Number of successful requests. Requests with finish reason `stop` or `length` are counted. | Per model   | Per request   |
| Count    | Failure Request Count  | `request_failure_total`         | Number of failed requests. Requests with other finish reasons are counted.                 | Per model   | Per request   |

## Prometheus

Install [Prometheus](https://prometheus.io/download/) and configure its `prometheus.yml` file to scrape NIM at `localhost:8000`:

```yaml
scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:8000"]
```

Start Prometheus:

```bash
./prometheus --config.file=./prometheus.yml
```

Open [http://localhost:9090/targets](http://localhost:9090/targets) to verify that Prometheus detects the NIM target.

## Grafana

Install [Grafana](https://grafana.com/grafana/download/), then start its server:

```bash
./bin/grafana-server
```

Open [http://localhost:3000](http://localhost:3000) and sign in with the default `admin` / `admin` credentials. Add Prometheus as a data source with the URL `localhost:9090`.