Chat with Guardrailed Model
Use the /v1/chat/completions endpoint to send messages and receive guarded responses from the server.
The endpoint is compatible with the OpenAI Chat Completions API,
with additional guardrails-specific fields nested under a guardrails object.
Basic Request
Send a POST request to the chat completions endpoint.
The model field is required and specifies which LLM to use.
Guardrails-specific fields such as config_id are nested under the guardrails object.
Response
The response follows the standard OpenAI ChatCompletion format, with an additional guardrails object containing guardrails-specific output data.
The guardrails response object may include additional fields depending on your request options:
llm_output— Additional LLM output data (whenguardrails.options.llm_outputistrue).output_data— Values for requested context variables (whenguardrails.options.output_varsis set).log— Logging information (whenguardrails.options.logis configured).
The HTTP endpoint does not accept caller-supplied runtime state. To continue a conversation, send the full message history or use a thread_id. Trusted in-process Python callers using Colang 1.x can pass state directly to LLMRails.generate_async().
Using the OpenAI Python SDK
Since the server is OpenAI-compatible, you can use the OpenAI Python SDK to interact with it.
Pass guardrails-specific fields using the extra_body parameter.
Using Python Requests
Combine Multiple Configurations
You can combine multiple guardrails configurations in a single request using config_ids inside the guardrails object.
Use either config_id or config_ids, but not both — they are mutually exclusive.
The configurations combine in the order specified. If there are conflicts, the last configuration takes precedence.
All configurations must use the same model type and engine.
Example: Atomic Configurations
Create reusable atomic configurations that you can combine as needed:
input_checking: Uses the self-check input railoutput_checking: Uses the self-check output railmain: Uses the base LLM with no guardrails
Without input checking:
With input checking:
The input rail blocks the inappropriate message before it reaches the LLM.
Use the Default Configuration
If the server was started with --default-config-id, you can omit the guardrails object:
Streaming Responses
Enable streaming to receive partial responses as server-sent events (SSE). Each chunk follows the OpenAI streaming format.
Using curl
The server sends chunks in SSE format:
Using the OpenAI Python SDK
Using Python Requests
Tool Calling
Send tools, tool_choice, and parallel_tool_calls at the top level of your
request. Tool requests require a Colang 1.0 configuration without user-intent
definitions (user_messages). Both LLMRails and IORails support this path;
passthrough: true is not required. Unsupported configurations return HTTP 422.
You can set stream: true for these requests. When the request includes a
non-empty tools list, tool_choice, or parallel_tool_calls, the server waits
for the complete answer and all enabled rail checks before sending any SSE.
This includes text returned instead of, or alongside, tool calls. After approval,
the server splits text and tool arguments into successive SSE deltas. These are
fragments of the completed answer, not live model tokens. The server adds no
artificial delay between fragments; clients or intermediaries may deliver them
in batches.
The response uses the OpenAI streaming format:
- The first delta identifies the assistant role. Later deltas contain text or
indexed
tool_calls. Each call identifies its ID, type, and function name once; subsequent deltas carry fragments of its JSON argument string. Accumulate argument fragments by tool index before parsing them. - A separate terminal choice has
finish_reason: "tool_calls"when calls are returned, or"stop"for a textual answer or refusal. [DONE]ends the response.
The selected engine applies the same configured rails as a non-streaming request. Text output rails do not substitute for tool-call rails. A blocked call is omitted from the response; a generation or rail failure returns the usual HTTP error before SSE begins. Invalid provider responses, including malformed tool arguments, return HTTP 502. This status does not disable client retries. This path does not require output-rail streaming to be enabled.
After receiving the complete response, execute approved calls in your application
and send their results as role: tool messages with matching tool_call_id
values. Include the assistant tool-call message in the next request’s history,
or use guardrails.thread_id to retain it on the server. Continue supplying the
tool parameters on subsequent streaming requests to use delayed, checked delivery.
Requests without these tool parameters retain the existing incremental streaming behavior. For rail configuration and engine differences, see Tools Integration.
Conversation Threads
Use thread_id inside the guardrails object to maintain conversation history on the server.
This is useful when you can only send the latest message rather than the full history.
The thread_id must be between 16 and 255 characters long.
The thread_id is currently not implemented in the NeMo Guardrails microservices.
Configure Thread Storage
To use threads, register a datastore in the server’s config.py:
To use RedisStore, install aioredis >= 2.0.1.
Thread Limitations
- Incremental text streams do not save the generated response to thread history. Delayed tool streams save it through the same path as non-streaming requests.
- Threads are stored indefinitely with no automatic cleanup.
Add Context
Include additional context data in your request using the context field inside the guardrails object:
Control Generation Options
Use the options field inside the guardrails object to control which rails are applied and what information is returned:
Standard OpenAI Parameters
You can also pass standard OpenAI parameters such as temperature, max_tokens, max_completion_tokens, top_p, stop, presence_penalty, and frequency_penalty at the top level:
Use max_completion_tokens to set a completion-token budget, including reasoning tokens, when your provider supports it.
Both LLMRails and IORails forward this parameter for streaming and non-streaming requests.
A non-null top-level value overrides the same parameter in guardrails.options.llm_params.
For complete details on generation options, see Create Chat Completion.