Skip to navigation

This guide demonstrates how to integrate the NeMo Guardrails library into LangChain applications using the RunnableRails class. The class implements the full Runnable Protocol with comprehensive support for synchronous and asynchronous operations, streaming, and batch processing.


Overview

RunnableRails provides a complete LangChain-native interface that wraps guardrail configurations around LLMs or entire chains. It supports all Runnable methods including invoke(), ainvoke(), stream(), astream(), batch(), and abatch() with full metadata preservation.


Prerequisites

The examples on this page use OpenAI chat models through LangChain. Set up your environment before you run them.

  1. Install the NVIDIA NeMo Guardrails library and the LangChain OpenAI integration:

    pip install nemoguardrails langchain-openai

    RunnableRails requires langchain-core, which the NVIDIA NeMo Guardrails library does not install by default. The langchain-openai package installs it.

  2. Set your OpenAI API key:

    export OPENAI_API_KEY="your_openai_api_key"

Getting Started

To get started, load a guardrail configuration and create a RunnableRails instance.

from nemoguardrails import RailsConfig
from nemoguardrails.integrations.langchain.runnable_rails import RunnableRails
config = RailsConfig.from_path("path/to/config")
guardrails = RunnableRails(config)

To add guardrails around an LLM model inside a chain, wrap the LLM model with a RunnableRails instance. For example, (guardrails | ...).

The following is an example of using a prompt, model, and output parser:

from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
prompt = ChatPromptTemplate.from_template("tell me a short joke about {topic}")
model = ChatOpenAI()
output_parser = StrOutputParser()
chain = prompt | model | output_parser

Add guardrails around the LLM model in the example above with the following code:

chain_with_guardrails = prompt | (guardrails | model) | output_parser

Using the extra parenthesis is essential to enforce the order in which the | (pipe) operator is applied.

To add guardrails to an existing chain or any Runnable, wrap it similarly.

rag_chain = (
{"context": retriever | format_docs, "question": RunnablePassthrough()}
| prompt
| llm
| StrOutputParser()
)
rag_chain_with_guardrails = guardrails | rag_chain

You can also use the same approach to add guardrails only around certain parts of your chain. The following example from the RunnableBranch Documentation adds guardrails around the "anthropic" and "general" branches inside a RunnableBranch.

from langchain_core.runnables import RunnableBranch
branch = RunnableBranch(
(lambda x: "anthropic" in x["topic"].lower(), guardrails | anthropic_chain),
(lambda x: "langchain" in x["topic"].lower(), langchain_chain),
guardrails | general_chain,
)

In general, you can wrap any part of a runnable chain with guardrails.

chain = runnable_1 | runnable_2 | runnable_3 | runnable_4 | ...
chain_with_guardrails = runnable_1 | (guardrails | (runnable_2 | runnable_3)) | runnable_4 | ...

Streaming Support

RunnableRails provides full streaming support with both synchronous and asynchronous methods. This enables responsive applications that stream LLM outputs as they are generated.

# Synchronous streaming
for chunk in guardrails.stream("What is machine learning?"):
print(chunk, end="", flush=True)
# Asynchronous streaming
async def stream_example():
async for chunk in guardrails.astream("What is machine learning?"):
print(chunk, end="", flush=True)

Metadata in Streaming: RunnableRails preserves all metadata during streaming, including response_metadata, usage_metadata, and additional_kwargs in AIMessageChunk objects.

Latency and Cost: Guardrails add latency and token usage to streamed responses. Input rails finish before the first token streams, so every input rail that calls a model adds to the time to first token. If you enable output rail streaming, output rails check the response in chunks while it streams. See Output Rail Streaming for the settings that control the chunk size.


Batch Processing

RunnableRails supports efficient batch processing for multiple inputs. The following example shows how to use the batch and abatch methods.

inputs = [
"What is Python?",
"Explain machine learning",
"How does AI work?"
]
# Synchronous batch processing
results = guardrails.batch(inputs)
# Asynchronous batch processing
results = await guardrails.abatch(inputs)
# Control concurrency
from langchain_core.runnables import RunnableConfig
config = RunnableConfig(max_concurrency=3)
results = await guardrails.abatch(inputs, config=config)

Each input in a batch runs through the full set of rails, so the rail model calls, latency, and token usage scale with the number of inputs, the same as calling invoke() once per input.


Input/Output Formats

RunnableRails intelligently handles various input and output formats with automatic transformation.

LLM Wrapping Formats

Input FormatOutput FormatDescription
strAIMessageString prompts → AI messages with full metadata
StringPromptValueAIMessagePrompt values → AI messages
ChatPromptValueAIMessageChat prompts → AI messages
List[BaseMessage]AIMessageMessage lists → AI messages
HumanMessageAIMessageHuman messages → AI messages

Chain Wrapping Formats

Input FormatOutput FormatBehavior
dict with input keydict with output keyDictionary passthrough
dict with custom keydict with custom keyConfigurable via input_key/output_key
strstrString passthrough
Mixed formatsIntelligently detectedAutomatic format detection

Metadata Preservation

RunnableRails maintains complete metadata compatibility with LangChain components. All AIMessage responses include the following:

  • response_metadata: Token usage, model info, finish reasons.
  • usage_metadata: Input/output token counts, total tokens.
  • additional_kwargs: Custom fields from the LLM provider.
  • id: Unique message identifiers.
  • tool_calls: Tool call information when applicable.
result = guardrails.invoke("Hello world")
print(result.response_metadata) # {'token_usage': {...}, 'model_name': '...', ...}
print(result.usage_metadata) # {'input_tokens': 10, 'output_tokens': 5, ...}
print(result.additional_kwargs) # Provider-specific fields
print(result.id) # 'msg_abc123...'

This ensures seamless integration with LangChain components that depend on message metadata.


Configuration Options

Passthrough Mode

The role of a guardrail configuration is to validate user input, check LLM output, and guide the LLM model on how to respond. See the Configuration Guide for more details on the different types of rails.

To achieve this, the guardrail configuration might make additional calls to the LLM or other models/APIs (for example, for fact-checking and content moderation).

By default, when the guardrail configuration decides that it is safe to prompt the LLM, it uses the exact prompt that was provided as the input, such as a string, StringPromptValue or ChatPromptValue. However, to enforce specific rails, for example, dialog rails, general instructions, the guardrails configuration needs to alter the prompt used to generate the response.

The passthrough parameter controls this behavior.

  • passthrough=True (default): Uses the exact input prompt with minimal guardrail intervention.
  • passthrough=False: Allows guardrails to modify prompts for enhanced protection.
# Minimal intervention
guardrails = RunnableRails(config, passthrough=True)
# Enhanced guardrails (modifies prompts as needed)
guardrails = RunnableRails(config, passthrough=False)

Tool Calling: Tool calls work with either passthrough setting when the guardrails configuration has no user-intent definitions (user_messages). Enabling rails.dialog.single_call.enabled alone does not block tool calls.

What passthrough=False changes about the prompt: instead of sending your original messages to the LLM, the library renders its own prompt from the general task template, which prepends general instructions and reformats the conversation as a completion-style transcript. For example, a ChatPromptValue built from HumanMessage(content="Hello!") is not sent to the LLM as-is. The library instead sends a prompt similar to the following:

Below is a conversation between a helpful AI assistant and a user. The bot is
designed to generate human-like text based on the input that it receives. The
bot is talkative and provides lots of specific details. If the bot does not
know the answer to a question, it truthfully says it does not know.
User: Hello!
Assistant:

The exact instructions and format depend on your configuration and enabled rails. To inspect the transformed prompt for debugging, set verbose=True when creating RunnableRails. See Verbose Logging, which logs every Prompt :: and Completion :: pair sent to the LLM.

Rejection Behavior

When an input or output rail blocks a request, RunnableRails does not raise an exception. It returns the rail’s configured refusal message in the same output shape you would otherwise get back.

This behavior applies when enable_rails_exceptions is off, which is the default. If you set enable_rails_exceptions: true, the result depends on what you wrap and which rail blocks:

  • A blocked LLM-wrapping call raises a pydantic ValidationError, a ValueError subclass, because the exception dictionary is not valid AIMessage content.
  • A chain-wrapping call blocked by an input rail returns the rail exception dictionary under the output key instead of refusal text.
  • A chain-wrapping call blocked by an output rail returns the chain’s original output unchanged, with no exception dictionary and no refusal text.

Do not set enable_rails_exceptions: true when you use RunnableRails to wrap a chain with output rails. A response that an output rail blocks reaches your caller as if it had passed, and nothing in the result marks it as blocked.

See Exceptions for the exception format.

For chain-wrapping (dict) inputs, only a dictionary with the output key is returned:

{"answer": "I can't assist with that request."}

For LLM-wrapping inputs, such as a list[BaseMessage] or ChatPromptValue, an AIMessage is returned with the refusal text as content, no tool_calls, and empty response_metadata/additional_kwargs, since no underlying model call for a final answer was made:

from langchain_core.messages import HumanMessage
result = guarded_model.invoke([HumanMessage(content="You are stupid")])
print(result)
# AIMessage(content="I'm sorry, I can't respond to that.")
print(result.tool_calls) # []
print(result.response_metadata) # {}

RunnableRails does not report which rail blocked a request. If your application must tell a rejection apart from a normal answer, for example to return a structured error to a client, call LLMRails.generate() with the log.activated_rails generation option instead. See Detailed Logging Information.

Error Handling

RunnableRails.invoke()/ainvoke() wrap the underlying rails call in a try/except, but whether a failure reaches that except block depends on where the failure happens:

  • Failures that RunnableRails itself surfaces raise a ValueError, with the original exception chained through from e. This covers LLM provider failures (the main model or a safety-check model call fails, for example a timeout or an API error), unsupported input types, and any other error that escapes the underlying rails call. Some recognized errors are re-raised with a more specific message.
  • A tool or guardrail action registered with the guardrail configuration that raises (a bug in your tool function, a non-LLM network call failing, and so on) does not propagate. The action dispatcher catches it, and the runtime returns an ordinary-looking response with the content "I'm sorry, an internal error has occurred." This response has the same shape as a normal, non-blocked answer. LLM call failures are the exception: the dispatcher forwards those, so they surface as in the first case.

Because this second response is indistinguishable from a normal answer, do not use its content as an error sentinel; a legitimate model or rail response can contain the same text. If you need a distinguishable outcome, handle the failure inside the tool or action itself. Your calling code still needs to catch the exceptions that do propagate; RunnableRails does not return a structured error payload for those.

In a LangGraph node, an uncaught exception from runnable_with_guardrails.invoke(state) propagates as a normal Python exception through graph.invoke(). LangGraph does not catch node exceptions for you. Wrap the node body in a try/except ValueError if you want the graph to continue with a fallback message instead of raising. See Failure Handling in Tool-Calling Graphs for an example, including how tools executed by a LangGraph ToolNode differ from tools registered with the guardrail configuration.

Custom Input/Output Keys

When you use a guardrail configuration to wrap a chain or a Runnable, the input and output are either dictionaries or strings. However, a guardrail configuration always operates on a text input from the user and a text output from the LLM. To achieve this, when dictionaries are used, one of the keys from the input dictionary must be designated as the "input text" and one of the keys from the output as the "output text".

By default, these keys are input and output. To customize these keys, provide the input_key and output_key parameters when creating the RunnableRails instance.

The following examples show how to customize the input and output keys with “question” and “answer” keys.

# Custom keys for specialized chains
guardrails = RunnableRails(
config,
input_key="question", # Default: "input"
output_key="answer" # Default: "output"
)
# Usage with RAG chain
rag_chain_with_guardrails = guardrails | rag_chain

When a guardrail is triggered and predefined messages must be returned instead of the output from the LLM, only a dictionary with the output key is returned.

{"answer": "I can't assist with that request."}

Tool Calling

RunnableRails supports LangChain tool calling with full metadata preservation and streaming. Tool calling works with or without passthrough when the guardrails configuration has no user-intent definitions (user_messages). Configurations with user-intent definitions drop tool calls and return only the generated text.

bind_tools() is a LangChain method that attaches tool/function definitions to a chat model so the model can request tool calls in its response instead of only returning text. ToolMessage is LangChain’s message type for returning a tool’s result back to the model, keyed to the originating call through tool_call_id. See the LangChain tool calling documentation for the full concept. The NVIDIA NeMo Guardrails library does not change either of these; RunnableRails only adds safety checks around the model calls that use them.

To use tool calling with RunnableRails:

  • Use bind_tools() to attach tools to your model.
  • Handle tool execution in your application logic.
  • Avoid guardrail configurations that define dialog flows if you need tool calls to surface in the response.

Basic Tool Setup

Avoid Python’s built-in eval() for evaluating LLM-provided expressions, even with a __builtins__ whitelist. Sandbox-escape techniques (for example, attribute traversal through __class__) can bypass a manually constructed whitelist. Use a restricted expression evaluator such as simpleeval instead. simpleeval is already an NVIDIA NeMo Guardrails library dependency, so no extra install is required.

Do not map a custom pow function into functions. simpleeval’s built-in ** operator is guarded by simpleeval.MAX_POWER against oversized exponents, but that guard does not apply to a function you supply yourself, so a mapped pow lets an expression like pow(2, 999999999) consume seconds of CPU time and hundreds of megabytes of memory. Let callers use ** for exponentiation instead.

from langchain_core.tools import tool
from langchain_openai import ChatOpenAI
from simpleeval import SimpleEval
import math
from nemoguardrails import RailsConfig
from nemoguardrails.integrations.langchain.runnable_rails import RunnableRails
@tool
def calculator(expression: str) -> str:
"""Evaluates mathematical expressions like '2 + 2' or 'sqrt(16)'."""
evaluator = SimpleEval(functions={"sqrt": math.sqrt})
try:
return str(evaluator.eval(expression))
except Exception as e:
return f"Error: {e}"
tools = [calculator]
model = ChatOpenAI(model="gpt-5").bind_tools(tools)
config = RailsConfig.from_path("path/to/config")
guardrails = RunnableRails(config=config, passthrough=True)
guarded_model = guardrails | model

Two-Call Tool Pattern

The standard flow for two-call tool calling is to get tool calls, execute them, and synthesize results.

from langchain_core.messages import AIMessage, HumanMessage, ToolMessage
# First call: Get tool calls
messages = [HumanMessage(content="What is 2 + 2?")]
result = guarded_model.invoke(messages)
# Execute tools
tools_by_name = {tool.name: tool for tool in tools}
messages_with_tools = [
messages[0],
AIMessage(content=result.content or "", tool_calls=result.tool_calls),
]
for tool_call in result.tool_calls:
tool_result = tools_by_name[tool_call["name"]].invoke(tool_call["args"])
messages_with_tools.append(
ToolMessage(
content=str(tool_result),
name=tool_call["name"],
tool_call_id=tool_call["id"],
)
)
# Second call: Synthesize results
final_result = guarded_model.invoke(messages_with_tools)
print(final_result.content)

Single-Call with Pre-processed Messages

Use single-call tool calling when you already have a complete message history with tool results.

messages = [
HumanMessage(content="What is 2 + 2?"),
AIMessage(
content="",
tool_calls=[
{
"name": "calculator",
"args": {"expression": "2 + 2"},
"id": "call_001",
"type": "tool_call",
}
],
),
ToolMessage(
content="4",
name="calculator",
tool_call_id="call_001",
),
]
result = guarded_model.invoke(messages)
print(result.content) # "2 + 2 equals 4."

Composition and Chaining

RunnableRails integrates with complex LangChain compositions. The following example shows how to use RunnableRails with a conditional branching chain.

from langchain_core.runnables import RunnablePassthrough, RunnableBranch
from langchain_core.output_parsers import StrOutputParser
# Complex chain with guardrails
chain = (
{"context": retriever | format_docs, "question": RunnablePassthrough()}
| prompt
| (guardrails | llm)
| StrOutputParser()
)
# Conditional branching with guardrails
branch = RunnableBranch(
(lambda x: "technical" in x["topic"], guardrails | technical_chain),
(lambda x: "creative" in x["topic"], creative_chain),
guardrails | general_chain,
)

Key Benefits of RunnableRails:

  • Maintains full Runnable protocol compatibility.
  • Preserves metadata throughout the chain.
  • Supports all async/sync operations.
  • Works with streaming and batch processing.