Prometheus and Grafana for the Guardrails Server
Use nemoguardrails server to expose non-streaming admission-queue metrics on a Prometheus endpoint and visualize them in Grafana.
The server configures the OpenTelemetry SDK and serves the endpoint on a dedicated port, so you do not need a Python bootstrap script.
This exporter includes only the three non-streaming admission-queue metrics listed below.
The server drops every other metric the IORails engine records, so the scrape output stays small and stable while the GenAI semantic conventions are still evolving.
For application-side setups that export every metric, refer to OpenTelemetry Metrics Integration.
The Guardrails hooks server, nemoguardrails hooks-server, exports the same metrics with its own defaults, as described in The Hooks Server.
Metrics currently require the opt-in IORails engine.
Set NEMO_GUARDRAILS_IORAILS_ENGINE=1 before starting the server; configurations that fall back to LLMRails emit no metrics.
Exported Metrics
Prometheus replaces dots with underscores and appends _total to counters.
Every series also carries the otel_scope_* labels, and a target_info series carries service_name.
For the full metric semantics, refer to Metric Reference.
Prerequisites
-
Install the server extra. It includes the OpenTelemetry SDK and the Prometheus exporter.
-
Enable metrics in every guardrails configuration that should report. The exporter is server-wide, but each
IORailsinstance only records when its own configuration opts in.In multi-config mode, add this block to each configuration sub-folder you want to monitor. In single-config mode, add it to the root
config.yml.The two gauges observe the admission queue of the first
IORailsinstance that starts in the process. When a server hosts several configurations, each configuration gets its own instance and queue, but the OpenTelemetry SDK registers one observable gauge per name, soguardrails_nonstream_queuedandguardrails_nonstream_activereport only the first instance. The binding lasts for the life of the process: OpenTelemetry has no API to unregister an observable gauge, so replacing a configuration does not move the gauges to the new instance. With--auto-reload, or any other change that rebuilds a configuration, the two gauges keep observing the retired instance’s idle queue and read a flat0while the live instance does the work.guardrails_nonstream_rejections_totalis unaffected and counts rejections from every instance, so rejections can climb while the queue depth still reads0. Run one configuration per server process when you need per-configuration queue depth, and restart the server after a configuration change to rebind the gauges. -
Select the IORails engine.
Start the Server with the Exporter
Pass --metrics-exporter prometheus or set the equivalent environment variable.
The API keeps serving on --port (default 8000); the scrape endpoint listens separately.
CLI flags
Environment variables
Docker
The CLI flags take precedence over the environment variables.
Supported exporter values are prometheus and none.
Use --metrics-exporter none to disable the exporter, including when NEMO_GUARDRAILS_SERVER_METRICS_EXPORTER=prometheus is set.
The exporter binds the metrics port once per process, so it does not work with uvicorn --workers N for N > 1: every worker tries to bind the same port, only the first succeeds, and the others fail to start.
nemoguardrails server runs a single worker and is unaffected.
To scale out, run several server processes with distinct NEMO_GUARDRAILS_SERVER_METRICS_PORT values (or one process per container) and scrape each one.
The scrape endpoint has no authentication, and the default listen interface is 0.0.0.0.
Restrict who can reach it: set --metrics-host 127.0.0.1 when Prometheus runs on the same host, or keep the default and allow inbound traffic on the metrics port only from your Prometheus servers with a firewall or network policy.
Do not expose the metrics port to the public internet.
The server starts the exporter before it binds the API port and shuts it down with the server, so a misconfiguration fails at startup with the reason printed to the console.
Verify the Endpoint
In another terminal, send a non-streaming chat request using your configuration ID and configured model.
This starts the metrics-enabled IORails instance and registers the queue gauges.
Then scrape the endpoint:
guardrails_nonstream_rejections_total appears after the first rejected submission.
Configure Prometheus
Add a scrape job that targets the metrics port.
When Prometheus runs in a container and the server runs on the host, target host.docker.internal:9464 instead.
Connect Grafana
The examples/observability/prometheus-grafana/ bundle starts Prometheus and Grafana, provisions the Prometheus data source, and imports a dashboard for the three metrics.
To use it, install Docker with Docker Compose and clone the Guardrails repository, or use an existing checkout.
Run the following commands from the repository root:
The Compose file refuses to start without GRAFANA_ADMIN_PASSWORD, anonymous access is disabled, and both containers publish their ports on 127.0.0.1 only.
Open http://localhost:3000, sign in as admin with the password you chose, and select the NeMo Guardrails - Non-Streaming Admission Queue dashboard in the NeMo Guardrails folder.
To connect an existing Grafana instance instead:
-
In Grafana, go to Connections > Data sources > Add new data source and choose Prometheus.
-
Set Prometheus server URL to the address of your Prometheus server, for example
http://localhost:9090, and select Save & test. -
Create a dashboard and add panels with the following queries.
-
Alternatively, import
examples/observability/prometheus-grafana/grafana/provisioning/dashboards/guardrails-admission-queue.jsonthrough Dashboards > New > Import. Before importing, replace bothdatasource.uidvalues ofnemoguardrails-prometheusin the JSON with your Prometheus data source’s UID. These references configure the panel and theinstancevariable. The imported dashboard contains one time-series panel for queued requests, active requests, and rejection rate; it does not include the five-minute rejection total.
A useful alert rule is any sustained non-zero rejection rate: it means the server is shedding load and clients are receiving HTTP 503 responses with a Retry-After header.
guardrails_nonstream_queued approaching the queue depth (default 256) is the leading indicator.
The Hooks Server
The Guardrails hooks server (nemoguardrails hooks-server) exports the same three metrics with the same flags and environment variables.
The following defaults let both servers export metrics on the same host:
The hooks server also differs from the Guardrails API server in the following ways:
- It always runs on
IORails, so it needs noNEMO_GUARDRAILS_IORAILS_ENGINE. - It serves one configuration, which must set
metrics.enabled: true. When the exporter runs and the configuration does not set it, the server logs a warning at startup, and the scrape endpoint shows noguardrails_series. - Its engine starts with the server, so the queue gauges appear before the first request.
- Only calls that run per-tool rails pass through the admission queue. A call that the tool policy decides, or a call to a tool with no per-tool rails, does not change the metrics.
Start it with the exporter:
Check the scrape endpoint with the following command:
The environment variables override the hooks server’s defaults as they override the Guardrails API server’s, and both servers read the same variables. A variable exported for one server in a shell, a container, or a service unit therefore also applies to a hooks server started from it:
NEMO_GUARDRAILS_SERVER_METRICS_EXPORTER=prometheusturns on the hooks server’s export too. Pass--metrics-exporter noneto turn it off for one server.NEMO_GUARDRAILS_SERVER_METRICS_HOST=0.0.0.0moves the hooks server’s scrape endpoint off127.0.0.1, onto every interface.NEMO_GUARDRAILS_SERVER_METRICS_PORT=9464gives both servers the same port, so the one that starts second exits withCannot bind the Prometheus metrics endpoint.NEMO_GUARDRAILS_SERVER_METRICS_SERVICE_NAMEorOTEL_SERVICE_NAMEreplaces the hooks server’sservice.name, which has no flag.
Configure each server’s exporter with its own flags rather than exported variables, since a flag applies to one server only.
To scrape both servers on one host, add a target for each:
The hooks server has no authentication, and neither does its scrape endpoint.
Keep the endpoint on 127.0.0.1 unless Prometheus runs on another host, and then restrict who can reach the port.
Troubleshooting
- The gauges read
0while the server is clearly busy. They bind to the firstIORailsinstance that starts in the process and cannot be rebound, so--auto-reloadand any other configuration swap leave them observing the retired instance. Restart the server to rebind them;guardrails_nonstream_rejections_totalkeeps counting correctly throughout. - Workers fail to start with
Cannot bind the Prometheus metrics endpoint. You are runninguvicorn --workers NwithN > 1; only one process can own the metrics port. Run one worker per process and give each process its ownNEMO_GUARDRAILS_SERVER_METRICS_PORT. curlreturns noguardrails_series. The gauges register when the firstIORailsinstance starts, which happens on the first request for that configuration. Confirm thatmetrics.enabled: trueis set in that configuration and thatNEMO_GUARDRAILS_IORAILS_ENGINE=1was exported before the server started.A global OpenTelemetry MeterProvider is already configured. Another component, such as a rootconfig.pyor an auto-instrumentation agent, installed aMeterProviderbefore the server did. UnsetNEMO_GUARDRAILS_SERVER_METRICS_EXPORTERand export metrics through that provider, or remove the other setup.Cannot bind the Prometheus metrics endpoint. The port is in use. Pick another one with--metrics-portorNEMO_GUARDRAILS_SERVER_METRICS_PORT.requires the OpenTelemetry SDK and Prometheus exporter. The server extra is not installed in this environment. Runpip install "nemoguardrails[server]".A Prometheus metrics endpoint is already running on <URL>; serving it instead of <host>:<port> (service.name=<name>). Two applications run in one process with different exporter settings, for example when an application or a test serves both the Guardrails API server’s app and the hooks server’s app. A process has one scrape endpoint, so the first application’s exporter serves the metrics of both, with its own port andservice.name. Run each server as its own process, or give both applications the same exporter settings.