Skip to navigation

Prometheus and Grafana for the Guardrails Server

View as MarkdownOpen in Claude

Use nemoguardrails server to expose non-streaming admission-queue metrics on a Prometheus endpoint and visualize them in Grafana. The server configures the OpenTelemetry SDK and serves the endpoint on a dedicated port, so you do not need a Python bootstrap script. This exporter includes only the three non-streaming admission-queue metrics listed below. The server drops every other metric the IORails engine records, so the scrape output stays small and stable while the GenAI semantic conventions are still evolving. For application-side setups that export every metric, refer to OpenTelemetry Metrics Integration. The Guardrails hooks server, nemoguardrails hooks-server, exports the same metrics with its own defaults, as described in The Hooks Server.

Experimental Feature

Metrics currently require the opt-in IORails engine. Set NEMO_GUARDRAILS_IORAILS_ENGINE=1 before starting the server; configurations that fall back to LLMRails emit no metrics.

Exported Metrics

OpenTelemetry namePrometheus nameTypeDescription
guardrails.nonstream.queuedguardrails_nonstream_queuedGaugeRequests buffered in the admission queue, not yet picked up by a worker.
guardrails.nonstream.activeguardrails_nonstream_activeGaugeRequests currently executing on a worker.
guardrails.nonstream.rejectionsguardrails_nonstream_rejections_totalCounterSubmissions rejected with HTTP 503 because the admission queue exceeded its depth limit.

Prometheus replaces dots with underscores and appends _total to counters. Every series also carries the otel_scope_* labels, and a target_info series carries service_name. For the full metric semantics, refer to Metric Reference.

Prerequisites

  1. Install the server extra. It includes the OpenTelemetry SDK and the Prometheus exporter.

    pip install "nemoguardrails[server]"
  2. Enable metrics in every guardrails configuration that should report. The exporter is server-wide, but each IORails instance only records when its own configuration opts in.

    metrics:
    enabled: true

    In multi-config mode, add this block to each configuration sub-folder you want to monitor. In single-config mode, add it to the root config.yml.

    The two gauges observe the admission queue of the first IORails instance that starts in the process. When a server hosts several configurations, each configuration gets its own instance and queue, but the OpenTelemetry SDK registers one observable gauge per name, so guardrails_nonstream_queued and guardrails_nonstream_active report only the first instance. The binding lasts for the life of the process: OpenTelemetry has no API to unregister an observable gauge, so replacing a configuration does not move the gauges to the new instance. With --auto-reload, or any other change that rebuilds a configuration, the two gauges keep observing the retired instance’s idle queue and read a flat 0 while the live instance does the work. guardrails_nonstream_rejections_total is unaffected and counts rejections from every instance, so rejections can climb while the queue depth still reads 0. Run one configuration per server process when you need per-configuration queue depth, and restart the server after a configuration change to rebind the gauges.

  3. Select the IORails engine.

    export NEMO_GUARDRAILS_IORAILS_ENGINE=1

Start the Server with the Exporter

Pass --metrics-exporter prometheus or set the equivalent environment variable. The API keeps serving on --port (default 8000); the scrape endpoint listens separately.

nemoguardrails server --config ./config --metrics-exporter prometheus --metrics-port 9464
SettingCLI flagEnvironment variableDefault
Exporter--metrics-exporterNEMO_GUARDRAILS_SERVER_METRICS_EXPORTERnone (metrics are not exported)
Listen interface--metrics-hostNEMO_GUARDRAILS_SERVER_METRICS_HOST0.0.0.0
Listen port--metrics-portNEMO_GUARDRAILS_SERVER_METRICS_PORT9464
service.name resource attributenoneNEMO_GUARDRAILS_SERVER_METRICS_SERVICE_NAME, then OTEL_SERVICE_NAMEnemoguardrails-server

The CLI flags take precedence over the environment variables. Supported exporter values are prometheus and none. Use --metrics-exporter none to disable the exporter, including when NEMO_GUARDRAILS_SERVER_METRICS_EXPORTER=prometheus is set.

One process per metrics port

The exporter binds the metrics port once per process, so it does not work with uvicorn --workers N for N > 1: every worker tries to bind the same port, only the first succeeds, and the others fail to start. nemoguardrails server runs a single worker and is unaffected. To scale out, run several server processes with distinct NEMO_GUARDRAILS_SERVER_METRICS_PORT values (or one process per container) and scrape each one.

The endpoint is unauthenticated

The scrape endpoint has no authentication, and the default listen interface is 0.0.0.0. Restrict who can reach it: set --metrics-host 127.0.0.1 when Prometheus runs on the same host, or keep the default and allow inbound traffic on the metrics port only from your Prometheus servers with a firewall or network policy. Do not expose the metrics port to the public internet.

The server starts the exporter before it binds the API port and shuts it down with the server, so a misconfiguration fails at startup with the reason printed to the console.

Verify the Endpoint

In another terminal, send a non-streaming chat request using your configuration ID and configured model. This starts the metrics-enabled IORails instance and registers the queue gauges. Then scrape the endpoint:

curl -s http://localhost:9464/metrics | grep guardrails_
# TYPE guardrails_nonstream_queued gauge
guardrails_nonstream_queued{otel_scope_name="nemo-guardrails",...} 0.0
# TYPE guardrails_nonstream_active gauge
guardrails_nonstream_active{otel_scope_name="nemo-guardrails",...} 0.0

guardrails_nonstream_rejections_total appears after the first rejected submission.

Configure Prometheus

Add a scrape job that targets the metrics port.

scrape_configs:
- job_name: nemoguardrails
scrape_interval: 5s
static_configs:
- targets: ["localhost:9464"]

When Prometheus runs in a container and the server runs on the host, target host.docker.internal:9464 instead.

Connect Grafana

The examples/observability/prometheus-grafana/ bundle starts Prometheus and Grafana, provisions the Prometheus data source, and imports a dashboard for the three metrics. To use it, install Docker with Docker Compose and clone the Guardrails repository, or use an existing checkout. Run the following commands from the repository root:

cd examples/observability/prometheus-grafana
GRAFANA_ADMIN_PASSWORD='<choose a password>' docker compose up -d

The Compose file refuses to start without GRAFANA_ADMIN_PASSWORD, anonymous access is disabled, and both containers publish their ports on 127.0.0.1 only. Open http://localhost:3000, sign in as admin with the password you chose, and select the NeMo Guardrails - Non-Streaming Admission Queue dashboard in the NeMo Guardrails folder.

To connect an existing Grafana instance instead:

  1. In Grafana, go to Connections > Data sources > Add new data source and choose Prometheus.

  2. Set Prometheus server URL to the address of your Prometheus server, for example http://localhost:9090, and select Save & test.

  3. Create a dashboard and add panels with the following queries.

    PanelPromQL
    Queued requestssum by (instance) (guardrails_nonstream_queued)
    Active requestssum by (instance) (guardrails_nonstream_active)
    Rejections in the last 5 minutessum(increase(guardrails_nonstream_rejections_total[5m]))
    Rejection ratesum by (instance) (rate(guardrails_nonstream_rejections_total[1m]))
  4. Alternatively, import examples/observability/prometheus-grafana/grafana/provisioning/dashboards/guardrails-admission-queue.json through Dashboards > New > Import. Before importing, replace both datasource.uid values of nemoguardrails-prometheus in the JSON with your Prometheus data source’s UID. These references configure the panel and the instance variable. The imported dashboard contains one time-series panel for queued requests, active requests, and rejection rate; it does not include the five-minute rejection total.

A useful alert rule is any sustained non-zero rejection rate: it means the server is shedding load and clients are receiving HTTP 503 responses with a Retry-After header. guardrails_nonstream_queued approaching the queue depth (default 256) is the leading indicator.

The Hooks Server

The Guardrails hooks server (nemoguardrails hooks-server) exports the same three metrics with the same flags and environment variables. The following defaults let both servers export metrics on the same host:

SettingGuardrails API Server DefaultHooks Server Default
Listen interface0.0.0.0127.0.0.1
Listen port94649465
service.name resource attributenemoguardrails-servernemoguardrails-hooks-server

The hooks server also differs from the Guardrails API server in the following ways:

  • It always runs on IORails, so it needs no NEMO_GUARDRAILS_IORAILS_ENGINE.
  • It serves one configuration, which must set metrics.enabled: true. When the exporter runs and the configuration does not set it, the server logs a warning at startup, and the scrape endpoint shows no guardrails_ series.
  • Its engine starts with the server, so the queue gauges appear before the first request.
  • Only calls that run per-tool rails pass through the admission queue. A call that the tool policy decides, or a call to a tool with no per-tool rails, does not change the metrics.

Start it with the exporter:

nemoguardrails hooks-server --config ./hooks-config --metrics-exporter prometheus

Check the scrape endpoint with the following command:

curl -s http://127.0.0.1:9465/metrics | grep guardrails_

The environment variables override the hooks server’s defaults as they override the Guardrails API server’s, and both servers read the same variables. A variable exported for one server in a shell, a container, or a service unit therefore also applies to a hooks server started from it:

  • NEMO_GUARDRAILS_SERVER_METRICS_EXPORTER=prometheus turns on the hooks server’s export too. Pass --metrics-exporter none to turn it off for one server.
  • NEMO_GUARDRAILS_SERVER_METRICS_HOST=0.0.0.0 moves the hooks server’s scrape endpoint off 127.0.0.1, onto every interface.
  • NEMO_GUARDRAILS_SERVER_METRICS_PORT=9464 gives both servers the same port, so the one that starts second exits with Cannot bind the Prometheus metrics endpoint.
  • NEMO_GUARDRAILS_SERVER_METRICS_SERVICE_NAME or OTEL_SERVICE_NAME replaces the hooks server’s service.name, which has no flag.

Configure each server’s exporter with its own flags rather than exported variables, since a flag applies to one server only.

To scrape both servers on one host, add a target for each:

scrape_configs:
- job_name: nemoguardrails
static_configs:
- targets: ["localhost:9464"]
- job_name: nemoguardrails-hooks
static_configs:
- targets: ["localhost:9465"]

The hooks server has no authentication, and neither does its scrape endpoint. Keep the endpoint on 127.0.0.1 unless Prometheus runs on another host, and then restrict who can reach the port.

Troubleshooting

  • The gauges read 0 while the server is clearly busy. They bind to the first IORails instance that starts in the process and cannot be rebound, so --auto-reload and any other configuration swap leave them observing the retired instance. Restart the server to rebind them; guardrails_nonstream_rejections_total keeps counting correctly throughout.
  • Workers fail to start with Cannot bind the Prometheus metrics endpoint. You are running uvicorn --workers N with N > 1; only one process can own the metrics port. Run one worker per process and give each process its own NEMO_GUARDRAILS_SERVER_METRICS_PORT.
  • curl returns no guardrails_ series. The gauges register when the first IORails instance starts, which happens on the first request for that configuration. Confirm that metrics.enabled: true is set in that configuration and that NEMO_GUARDRAILS_IORAILS_ENGINE=1 was exported before the server started.
  • A global OpenTelemetry MeterProvider is already configured. Another component, such as a root config.py or an auto-instrumentation agent, installed a MeterProvider before the server did. Unset NEMO_GUARDRAILS_SERVER_METRICS_EXPORTER and export metrics through that provider, or remove the other setup.
  • Cannot bind the Prometheus metrics endpoint. The port is in use. Pick another one with --metrics-port or NEMO_GUARDRAILS_SERVER_METRICS_PORT.
  • requires the OpenTelemetry SDK and Prometheus exporter. The server extra is not installed in this environment. Run pip install "nemoguardrails[server]".
  • A Prometheus metrics endpoint is already running on <URL>; serving it instead of <host>:<port> (service.name=<name>). Two applications run in one process with different exporter settings, for example when an application or a test serves both the Guardrails API server’s app and the hooks server’s app. A process has one scrape endpoint, so the first application’s exporter serves the metrics of both, with its own port and service.name. Run each server as its own process, or give both applications the same exporter settings.