Skip to navigation

About Running Guardrailed Inference

View as MarkdownOpen in Claude

After you configure your guardrails, you can run guardrailed inference using the tools provided by the NeMo Guardrails library: the Python API and the Guardrails API server.

These tools enable you to interact with your application’s main LLM as usual by sending prompts and receiving responses while the guardrails system monitors and controls all communication in the background. The guardrails intercept inputs and outputs between uses and the LLM and the execution of actions done by the LLM. The tools apply your configured guardrails and ensure that any inputs, generated responses, or actions remain within the boundaries you defined.

To guard a coding agent such as Claude Code or Codex, which runs its own model, use the Guardrails hooks server instead: it checks each tool call the agent is about to run.

To apply checks around supported existing model HTTP exchanges, use NeMo Guardrails Gateway.


Install the NeMo Guardrails Library

To use the Python API or the Guardrails API server, you need to install the NeMo Guardrails library. See Installation for instructions.


Choosing the Right Tool

Both the Python API and the Guardrails API server are production-ready approaches for integrating guardrails into your application. The Guardrails hooks server is for coding agents, which you connect through their hook settings rather than your own code.

Choose an interface based on the following tasks:

TaskInterface
Call guardrails from Python code in your applicationThe Python API
Send chat requests or checks over HTTP, from any languageThe Guardrails API server (nemoguardrails server)
Check supported existing provider HTTP exchanges while preserving accepted JSON and server-sent event bytesNeMo Guardrails Gateway (python -m nemoguardrails.server.experimental)
Check a coding agent’s tool calls before they run, from the agent’s own hooksThe Guardrails hooks server (nemoguardrails hooks-server)

Python API for Edge and Embedded Applications

Build guardrails directly into your Python application. You call guardrails functions in your code, and everything runs in the same process with no network overhead.

from nemoguardrails import Guardrails, RailsConfig
config = RailsConfig.from_path("path/to/config")
rails = Guardrails(config)
response = rails.generate(messages=[
{"role": "user", "content": "Hello!"}
])

LLMRails, which earlier versions of this example used, is deprecated in 0.25.0; removal is planned no earlier than 0.27.0. Guardrails runs the IORails engine when that engine supports the configuration. For the steps, refer to Migrating to 0.25.

This approach is best for:

  • Edge deployments where no networked backend is available.
  • Applications that require low latency and no external dependencies.
  • Rapid prototyping and development in notebooks or scripts.
  • Fine-grained control over generation and custom streaming handlers.

Guardrails API Server for Networked and Multi-Client Applications

The Guardrails API server provides a RESTful API interface to the guardrails library. Instead of calling Python functions directly, you make HTTP requests to the server endpoints, making integration straightforward for any language or platform.

curl -X POST http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model": "meta/llama-3.1-8b-instruct", "messages": [{"role": "user", "content": "Hello!"}], "guardrails": {"config_id": "content_safety"}}'

This approach is best for:

  • Networked systems where multiple clients need access to guardrails.
  • Non-Python clients such as JavaScript, Go, and Java that communicate through HTTP.

NeMo Guardrails Gateway for Existing Provider Exchanges

Use NeMo Guardrails Gateway when your application already calls a model service over HTTP. The gateway checks inputs and outputs for supported Chat Completions or Responses traffic using the selected Rails configuration, and forwards generation to the configured upstream.

The current bindings support one user text input and one assistant text target; they do not cover arbitrary conversation histories, tools, or multimodal content. Use its own manifest, library server extra, and module launcher, as described in the quickstart.

Guardrails Hooks Server for Coding Agents

The Guardrails hooks server is a separate server that checks the tool calls of Claude Code, Codex, and NeMo Relay before they run. The agent’s own pre-tool hook sends each call to the server, which applies a tool policy and per-tool rails and answers with allow or deny.

nemoguardrails hooks-server --config examples/configs/agent_hooks/claude_code

This approach is best for:

  • Coding agents whose code you do not control, and that can call out only from a hook.
  • Blocking risky tools, arguments, or commands, such as a force push or a read of a secrets file, before the agent runs them.

Next Steps

After choosing the right tool for your implementation model, proceed to the corresponding guide from the following.