LangGraph Integration
This guide demonstrates how to integrate the NeMo Guardrails library with LangGraph to build safe and controlled multi-agent workflows. LangGraph enables you to create sophisticated agent architectures with state management, conditional routing, and tool calling, while NeMo Guardrails provides the safety layer to ensure responsible AI behavior.
Overview
LangGraph is a library for building stateful, multi-actor applications with LLMs. When combined with the NeMo Guardrails library, you can create complex agent workflows that maintain safety and compliance throughout the entire conversation flow.
Key Benefits
- Stateful Safety: Guardrails persist across conversation turns and agent interactions.
- Tool Call Protection: Optional tool rails validate both tool calls and tool results.
- Multi-Agent Coordination: Each agent can have its own guardrail configuration.
- Graph-Based Control: Conditional routing with safety considerations.
- Conversation Memory: Maintained context with continuous safety monitoring.
The following diagram shows the basic data flow through a guarded LangGraph agent:
Prerequisites
Install the required dependencies and set up your environment.
-
Install the required dependencies:
-
Make sure that you have OpenAI API keys set up in your environment:
Basic Integration Pattern
The simplest integration involves wrapping your LangGraph nodes with the NeMo Guardrails library using the RunnableRails interface.
Configuration Setup
First, create a simple guardrails configuration for your LangGraph integration. You will use this configuration throughout this section. Create two files:
config.yml:
prompts.yml:
Set the path to the configuration file as a Python variable called config_path.
Then load the configuration:
Basic Agent with Guardrails
The following is a complete example of a basic LangGraph agent with guardrails:
Tool Calling Integration
To enhance the functionality of your LangGraph agents, you can combine tool calling with guardrails. Guardrails run on every model call in the graph, and tool rails can also validate the tool calls the model makes and the tool results your tools return.
bind_tools() is a LangChain method that attaches tool/function definitions to a chat model so the model can request tool calls in its response instead of only returning text. ToolMessage is LangChain’s message type for returning a tool’s result back to the model, keyed to the originating call through tool_call_id. See the LangChain tool calling documentation for the full concept. The NVIDIA NeMo Guardrails library does not change either of these; RunnableRails only adds safety checks around the model calls that use them.
The following diagram shows where guardrails sit relative to the ToolNode/tools_condition conditional edge:
tools_condition inspects the AIMessage that chatbot returns.
Because guardrails wrap the model call inside chatbot, the configured rails run on every trip through the chatbot node, not only on the final answer.
Input rails check the user message, not the ToolMessage results that ToolNode adds to the conversation, so tool output reaches the model unchecked by default.
To validate tool calls and tool results, add flows to the rails.tool_call and rails.tool_result sections of your configuration.
If you don’t configure tool result rails, validate untrusted tool output inside the tool before you return it.
For the available flows and engine support, see Tool Calling and Security Considerations.
Tool Definition
Define the following simplified example tools that demonstrate the integration pattern with the NeMo Guardrails library.
The first tool, search_knowledge, searches a predefined knowledge base for information matching the user’s query. It performs matching against keywords like "capital", "weather", and "python", returning relevant information or a generic response if no match is found.
The second tool, multiply, performs multiplication of two integers.
Tool Calling Graph with Guardrails
Create a tool calling graph with guardrails using the tools defined in the previous section.
Failure Handling in Tool-Calling Graphs
The chatbot node above has no exception handling around runnable_with_guardrails.invoke(state), and the graph has three distinct failure modes that behave differently:
- Failures that
RunnableRailssurfaces in thechatbotnode raise aValueError, which crashesgraph.invoke()if uncaught. See Error Handling; this covers LLM provider failures and other errors alike, soexcept ValueErrorin the node catches all of them, not just provider failures. - Failures in a tool executed by the
toolsnode are LangGraph’s concern, not the guardrail configuration’s. In this example the tools run insideToolNode, outsideRunnableRails, so the guardrail runtime never sees them. LangGraph’s defaulthandle_tool_errorshandler returns a message only for tool-invocation errors, such as malformed arguments, and re-raises everything else out ofgraph.invoke(). PassToolNode(tools=tools, handle_tool_errors=...)to change that. Atry/exceptinchatbotdoes not catch these, because they are raised from a different node. - Failures in a tool or action registered with the guardrail configuration (for example, tools passed as
RunnableRails(config=config, tools=[...])) are caught by the action dispatcher, which converts them into a normal-looking response with the content"I'm sorry, an internal error has occurred."The dispatcher forwards LLM call failures rather than catching them, so those still surface as in the first case.
The following variant catches the first case. Do not extend it by matching on the response content to detect the third case: that response is indistinguishable from a normal answer, and a legitimate model or rail response can contain the same text. Handle those failures inside the tool or action itself if you need a fallback for them. Everything else about create_tool_calling_agent stays the same; only chatbot changes.
With the default configuration, a rail that blocks the request, as opposed to an outright error, does not raise. It returns the normal AIMessage shape described in Rejection Behavior, with the rail’s configured refusal text.
If you set enable_rails_exceptions: true, a blocked request raises a ValueError instead, and the except ValueError branch above returns the fallback message.
Stateful Conversations
The checkpointing feature in LangGraph allows you to maintain conversation state across multiple interactions while keeping guardrails active throughout.
Multi-Agent Workflows
You can create sophisticated multi-agent systems where each agent has specialized roles and all are protected by guardrails.
Best Practices
1. Passthrough Mode Configuration
Use passthrough=True to maintain the original prompt structure for complex flows.
2. Verbose Logging
Enable verbose mode during development to understand the guardrails flow:
3. Performance Considerations
- Guardrails add latency due to additional LLM calls for safety checks.
- Consider caching strategies for repeated safety validations.
- Monitor token usage as guardrails consume additional tokens.
Debugging and Troubleshooting
Common Issues
- Empty Content with Tool Calls: Check that the guardrails configuration does not define dialog flows.
- Authorization Errors: Verify API keys for both main model and safety models.
- Configuration Not Found: Ensure guardrail config paths are correct.
Debugging Tips
- Enable verbose logging to see guardrail execution flow.
- Test without guardrails first to isolate integration issues.
- Check token limits for safety model calls.
- Validate configuration syntax.
Streaming Support
What Works
- Direct RunnableRails async streaming provides true token-by-token streaming.
What Does Not Work
- LangGraph integration with RunnableRails produces single large chunks after processing delays.
- Token-level streaming is not preserved when RunnableRails is integrated into LangGraph nodes.
Why
This is a RunnableRails-specific limitation, not a general LangGraph one. LangGraph itself supports token-level streaming from inside a node through stream_mode="messages", which hooks the LLM’s own streaming callbacks independently of what the node returns.
The chatbot node in the examples above calls runnable_with_guardrails.invoke(state). RunnableRails.invoke() calls LLMRails.generate(), which makes a single blocking call and returns one fully formatted result; no incremental LLM callback events fire during it for LangGraph’s messages mode to observe. RunnableRails.astream() does stream token-by-token, but it yields chunks directly to whatever calls it; it does not forward those chunks into LangGraph’s callback-based streaming machinery. Calling astream() from inside a node, instead of invoke(), would still not surface as messages-mode events without additional wiring.
Workarounds
- Stream directly from
RunnableRailsor the LLM outside the LangGraph node (guardrails.astream(...), as shown earlier in this guide), and use LangGraph only for the non-streaming orchestration and tool-calling parts of your flow. - Consider a hybrid architecture: stream the final synthesis call directly to the client, and keep guardrail-gated intermediate steps inside the graph non-streaming.
RunnableRails supports streaming when you use it directly, but not when a LangGraph node calls it through invoke().