> This page is for version v0.25.0.
> For other versions, use one of these documentation indexes:
> - Latest (default): https://docs.nvidia.com/nemo/guardrails/latest/llms.txt
> - v0.25.0: https://docs.nvidia.com/nemo/guardrails/v0.25.0/llms.txt
> - v0.24.1: https://docs.nvidia.com/nemo/guardrails/v0.24.1/llms.txt
> - v0.24.0: https://docs.nvidia.com/nemo/guardrails/v0.24.0/llms.txt
> - v0.23.0: https://docs.nvidia.com/nemo/guardrails/v0.23.0/llms.txt
> - v0.22.0: https://docs.nvidia.com/nemo/guardrails/v0.22.0/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/guardrails/llms.txt.

# Prometheus and Grafana for the Guardrails Server

> Expose the non-streaming admission-queue metrics from nemoguardrails server on a Prometheus scrape endpoint and chart them in Grafana.

Use `nemoguardrails server` to expose non-streaming admission-queue metrics on a Prometheus endpoint and visualize them in Grafana.
The server configures the OpenTelemetry SDK and serves the endpoint on a dedicated port, so you do not need a Python bootstrap script.
This exporter includes only the three non-streaming admission-queue metrics listed below.
The server drops every other metric the IORails engine records, so the scrape output stays small and stable while the GenAI semantic conventions are still evolving.
For application-side setups that export every metric, refer to [OpenTelemetry Metrics Integration](/nemo/guardrails/v0.25.0/observability/metrics/opentelemetry-integration).
The Guardrails hooks server, `nemoguardrails hooks-server`, exports the same metrics with its own defaults, as described in [The Hooks Server](#the-hooks-server).

> **Experimental Feature**
>
> Metrics currently require the opt-in IORails engine.
> Set `NEMO_GUARDRAILS_IORAILS_ENGINE=1` before starting the server; configurations that fall back to LLMRails emit no metrics.

## Exported Metrics

| OpenTelemetry name                | Prometheus name                         | Type    | Description                                                                                |
| --------------------------------- | --------------------------------------- | ------- | ------------------------------------------------------------------------------------------ |
| `guardrails.nonstream.queued`     | `guardrails_nonstream_queued`           | Gauge   | Requests buffered in the admission queue, not yet picked up by a worker.                   |
| `guardrails.nonstream.active`     | `guardrails_nonstream_active`           | Gauge   | Requests currently executing on a worker.                                                  |
| `guardrails.nonstream.rejections` | `guardrails_nonstream_rejections_total` | Counter | Submissions rejected with HTTP `503` because the admission queue exceeded its depth limit. |

Prometheus replaces dots with underscores and appends `_total` to counters.
Every series also carries the `otel_scope_*` labels, and a `target_info` series carries `service_name`.
For the full metric semantics, refer to [Metric Reference](/nemo/guardrails/v0.25.0/observability/metrics/reference).

## Prerequisites

1. Install the server extra. It includes the OpenTelemetry SDK and the Prometheus exporter.

   ```bash
   pip install "nemoguardrails[server]"
   ```

2. Enable metrics in every guardrails configuration that should report.
   The exporter is server-wide, but each `IORails` instance only records when its own configuration opts in.

   ```yaml
   metrics:
     enabled: true
   ```

   In multi-config mode, add this block to each configuration sub-folder you want to monitor.
   In single-config mode, add it to the root `config.yml`.

   > **Note**
   >
   > The two gauges observe the admission queue of the first `IORails` instance that starts in the process.
   > When a server hosts several configurations, each configuration gets its own instance and queue, but the OpenTelemetry SDK registers one observable gauge per name, so `guardrails_nonstream_queued` and `guardrails_nonstream_active` report only the first instance.
   > The binding lasts for the life of the process: OpenTelemetry has no API to unregister an observable gauge, so replacing a configuration does not move the gauges to the new instance.
   > With `--auto-reload`, or any other change that rebuilds a configuration, the two gauges keep observing the retired instance's idle queue and read a flat `0` while the live instance does the work.
   > `guardrails_nonstream_rejections_total` is unaffected and counts rejections from every instance, so rejections can climb while the queue depth still reads `0`.
   > Run one configuration per server process when you need per-configuration queue depth, and restart the server after a configuration change to rebind the gauges.

3. Select the IORails engine.

   ```bash
   export NEMO_GUARDRAILS_IORAILS_ENGINE=1
   ```

## Start the Server with the Exporter

Pass `--metrics-exporter prometheus` or set the equivalent environment variable.
The API keeps serving on `--port` (default `8000`); the scrape endpoint listens separately.

#### CLI flags

```bash
nemoguardrails server --config ./config --metrics-exporter prometheus --metrics-port 9464
```

#### Environment variables

```bash
export NEMO_GUARDRAILS_SERVER_METRICS_EXPORTER=prometheus
export NEMO_GUARDRAILS_SERVER_METRICS_PORT=9464
nemoguardrails server --config ./config
```

The environment variables also work when you run the app directly with `uvicorn nemoguardrails.server.api:app`, because the server lifespan reads them.

#### Docker

Install Docker and [build the local `nemoguardrails` image](/nemo/guardrails/v0.25.0/more-deployment-options/using-docker#build-the-docker-images) before running this example.
If your model provider requires credentials, pass its environment variables or mount its credential files in the container as described in the [Docker deployment guide](/nemo/guardrails/v0.25.0/more-deployment-options/using-docker).

```bash
docker run -p 8000:8000 -p 127.0.0.1:9464:9464 \
  -e NEMO_GUARDRAILS_IORAILS_ENGINE=1 \
  -e NEMO_GUARDRAILS_SERVER_METRICS_EXPORTER=prometheus \
  -v ./config:/config \
  nemoguardrails server --config /config
```

The metrics port is published on the loopback interface only, which is enough for a Prometheus instance on the same host.
When Prometheus also runs in Docker, attach both containers to one Docker network and scrape the container name instead of publishing the port at all.

| Setting                           | CLI flag             | Environment variable                                                    | Default                           |
| --------------------------------- | -------------------- | ----------------------------------------------------------------------- | --------------------------------- |
| Exporter                          | `--metrics-exporter` | `NEMO_GUARDRAILS_SERVER_METRICS_EXPORTER`                               | `none` (metrics are not exported) |
| Listen interface                  | `--metrics-host`     | `NEMO_GUARDRAILS_SERVER_METRICS_HOST`                                   | `0.0.0.0`                         |
| Listen port                       | `--metrics-port`     | `NEMO_GUARDRAILS_SERVER_METRICS_PORT`                                   | `9464`                            |
| `service.name` resource attribute | none                 | `NEMO_GUARDRAILS_SERVER_METRICS_SERVICE_NAME`, then `OTEL_SERVICE_NAME` | `nemoguardrails-server`           |

The CLI flags take precedence over the environment variables.
Supported exporter values are `prometheus` and `none`.
Use `--metrics-exporter none` to disable the exporter, including when `NEMO_GUARDRAILS_SERVER_METRICS_EXPORTER=prometheus` is set.

> **One process per metrics port**
>
> The exporter binds the metrics port once per process, so it does not work with `uvicorn --workers N` for `N > 1`: every worker tries to bind the same port, only the first succeeds, and the others fail to start.
> `nemoguardrails server` runs a single worker and is unaffected.
> To scale out, run several server processes with distinct `NEMO_GUARDRAILS_SERVER_METRICS_PORT` values (or one process per container) and scrape each one.

> **The endpoint is unauthenticated**
>
> The scrape endpoint has no authentication, and the default listen interface is `0.0.0.0`.
> Restrict who can reach it: set `--metrics-host 127.0.0.1` when Prometheus runs on the same host, or keep the default and allow inbound traffic on the metrics port only from your Prometheus servers with a firewall or network policy.
> Do not expose the metrics port to the public internet.

The server starts the exporter before it binds the API port and shuts it down with the server, so a misconfiguration fails at startup with the reason printed to the console.

## Verify the Endpoint

In another terminal, [send a non-streaming chat request](/nemo/guardrails/v0.25.0/run-guardrailed-inference/using-fastapi-server/chat-with-guardrailed-model#basic-request) using your configuration ID and configured model.
This starts the metrics-enabled `IORails` instance and registers the queue gauges.
Then scrape the endpoint:

```bash
curl -s http://localhost:9464/metrics | grep guardrails_
```

```text
# TYPE guardrails_nonstream_queued gauge
guardrails_nonstream_queued{otel_scope_name="nemo-guardrails",...} 0.0
# TYPE guardrails_nonstream_active gauge
guardrails_nonstream_active{otel_scope_name="nemo-guardrails",...} 0.0
```

`guardrails_nonstream_rejections_total` appears after the first rejected submission.

## Configure Prometheus

Add a scrape job that targets the metrics port.

```yaml
scrape_configs:
  - job_name: nemoguardrails
    scrape_interval: 5s
    static_configs:
      - targets: ["localhost:9464"]
```

When Prometheus runs in a container and the server runs on the host, target `host.docker.internal:9464` instead.

## Connect Grafana

The `examples/observability/prometheus-grafana/` bundle starts Prometheus and Grafana, provisions the Prometheus data source, and imports a dashboard for the three metrics.
To use it, install Docker with Docker Compose and clone the [Guardrails repository](https://github.com/NVIDIA-NeMo/Guardrails), or use an existing checkout.
Run the following commands from the repository root:

```bash
cd examples/observability/prometheus-grafana
GRAFANA_ADMIN_PASSWORD='<choose a password>' docker compose up -d
```

The Compose file refuses to start without `GRAFANA_ADMIN_PASSWORD`, anonymous access is disabled, and both containers publish their ports on `127.0.0.1` only.
Open `http://localhost:3000`, sign in as `admin` with the password you chose, and select the **NeMo Guardrails - Non-Streaming Admission Queue** dashboard in the **NeMo Guardrails** folder.

To connect an existing Grafana instance instead:

1. In Grafana, go to **Connections > Data sources > Add new data source** and choose **Prometheus**.

2. Set **Prometheus server URL** to the address of your Prometheus server, for example `http://localhost:9090`, and select **Save & test**.

3. Create a dashboard and add panels with the following queries.

   | Panel                            | PromQL                                                                |
   | -------------------------------- | --------------------------------------------------------------------- |
   | Queued requests                  | `sum by (instance) (guardrails_nonstream_queued)`                     |
   | Active requests                  | `sum by (instance) (guardrails_nonstream_active)`                     |
   | Rejections in the last 5 minutes | `sum(increase(guardrails_nonstream_rejections_total[5m]))`            |
   | Rejection rate                   | `sum by (instance) (rate(guardrails_nonstream_rejections_total[1m]))` |

4. Alternatively, import `examples/observability/prometheus-grafana/grafana/provisioning/dashboards/guardrails-admission-queue.json` through **Dashboards** > **New** > **Import**.
   Before importing, replace both `datasource.uid` values of `nemoguardrails-prometheus` in the JSON with your Prometheus data source's UID.
   These references configure the panel and the `instance` variable.
   The imported dashboard contains one time-series panel for queued requests, active requests, and rejection rate; it does not include the five-minute rejection total.

A useful alert rule is any sustained non-zero rejection rate: it means the server is shedding load and clients are receiving HTTP `503` responses with a `Retry-After` header.
`guardrails_nonstream_queued` approaching the queue depth (default `256`) is the leading indicator.

## The Hooks Server

The [Guardrails hooks server](/nemo/guardrails/v0.25.0/run-guardrailed-inference/using-hooks-server/overview) (`nemoguardrails hooks-server`) exports the same three metrics with the same flags and environment variables.
The following defaults let both servers export metrics on the same host:

| Setting                           | Guardrails API Server Default | Hooks Server Default          |
| --------------------------------- | ----------------------------- | ----------------------------- |
| Listen interface                  | `0.0.0.0`                     | `127.0.0.1`                   |
| Listen port                       | `9464`                        | `9465`                        |
| `service.name` resource attribute | `nemoguardrails-server`       | `nemoguardrails-hooks-server` |

The hooks server also differs from the Guardrails API server in the following ways:

* It always runs on `IORails`, so it needs no `NEMO_GUARDRAILS_IORAILS_ENGINE`.
* It serves one configuration, which must set `metrics.enabled: true`.
  When the exporter runs and the configuration does not set it, the server logs a warning at startup, and the scrape endpoint shows no `guardrails_` series.
* Its engine starts with the server, so the queue gauges appear before the first request.
* Only calls that run per-tool rails pass through the admission queue.
  A call that the tool policy decides, or a call to a tool with no per-tool rails, does not change the metrics.

Start it with the exporter:

```bash
nemoguardrails hooks-server --config ./hooks-config --metrics-exporter prometheus
```

Check the scrape endpoint with the following command:

```bash
curl -s http://127.0.0.1:9465/metrics | grep guardrails_
```

The environment variables override the hooks server's defaults as they override the Guardrails API server's, and both servers read the same variables.
A variable exported for one server in a shell, a container, or a service unit therefore also applies to a hooks server started from it:

* `NEMO_GUARDRAILS_SERVER_METRICS_EXPORTER=prometheus` turns on the hooks server's export too.
  Pass `--metrics-exporter none` to turn it off for one server.
* `NEMO_GUARDRAILS_SERVER_METRICS_HOST=0.0.0.0` moves the hooks server's scrape endpoint off `127.0.0.1`, onto every interface.
* `NEMO_GUARDRAILS_SERVER_METRICS_PORT=9464` gives both servers the same port, so the one that starts second exits with `Cannot bind the Prometheus metrics endpoint`.
* `NEMO_GUARDRAILS_SERVER_METRICS_SERVICE_NAME` or `OTEL_SERVICE_NAME` replaces the hooks server's `service.name`, which has no flag.

Configure each server's exporter with its own flags rather than exported variables, since a flag applies to one server only.

To scrape both servers on one host, add a target for each:

```yaml
scrape_configs:
  - job_name: nemoguardrails
    static_configs:
      - targets: ["localhost:9464"]
  - job_name: nemoguardrails-hooks
    static_configs:
      - targets: ["localhost:9465"]
```

The hooks server has no authentication, and neither does its scrape endpoint.
Keep the endpoint on `127.0.0.1` unless Prometheus runs on another host, and then restrict who can reach the port.

## Troubleshooting

* **The gauges read `0` while the server is clearly busy.**
  They bind to the first `IORails` instance that starts in the process and cannot be rebound, so `--auto-reload` and any other configuration swap leave them observing the retired instance.
  Restart the server to rebind them; `guardrails_nonstream_rejections_total` keeps counting correctly throughout.
* **Workers fail to start with `Cannot bind the Prometheus metrics endpoint`.**
  You are running `uvicorn --workers N` with `N > 1`; only one process can own the metrics port. Run one worker per process and give each process its own `NEMO_GUARDRAILS_SERVER_METRICS_PORT`.
* **`curl` returns no `guardrails_` series.**
  The gauges register when the first `IORails` instance starts, which happens on the first request for that configuration.
  Confirm that `metrics.enabled: true` is set in that configuration and that `NEMO_GUARDRAILS_IORAILS_ENGINE=1` was exported before the server started.
* **`A global OpenTelemetry MeterProvider is already configured`.**
  Another component, such as a root `config.py` or an auto-instrumentation agent, installed a `MeterProvider` before the server did.
  Unset `NEMO_GUARDRAILS_SERVER_METRICS_EXPORTER` and export metrics through that provider, or remove the other setup.
* **`Cannot bind the Prometheus metrics endpoint`.**
  The port is in use. Pick another one with `--metrics-port` or `NEMO_GUARDRAILS_SERVER_METRICS_PORT`.
* **`requires the OpenTelemetry SDK and Prometheus exporter`.**
  The server extra is not installed in this environment. Run `pip install "nemoguardrails[server]"`.
* **`A Prometheus metrics endpoint is already running on <URL>; serving it instead of <host>:<port> (service.name=<name>)`.**
  Two applications run in one process with different exporter settings, for example when an application or a test serves both the Guardrails API server's app and the hooks server's app.
  A process has one scrape endpoint, so the first application's exporter serves the metrics of both, with its own port and `service.name`.
  Run each server as its own process, or give both applications the same exporter settings.

## Related Topics

* [Run the Guardrails Server](/nemo/guardrails/v0.25.0/run-guardrailed-inference/using-fastapi-server/run-guardrails-server)
* [Run the Hooks Server](/nemo/guardrails/v0.25.0/run-guardrailed-inference/using-hooks-server/run-hooks-server)
* [Metric Reference](/nemo/guardrails/v0.25.0/observability/metrics/reference)
* [OpenTelemetry Metrics Integration](/nemo/guardrails/v0.25.0/observability/metrics/opentelemetry-integration)