> This page is for version v0.24.0. > For other versions, use one of these documentation indexes: > - Latest (default): https://docs.nvidia.com/nemo/guardrails/latest/llms.txt > - v0.24.1: https://docs.nvidia.com/nemo/guardrails/v0.24.1/llms.txt > - v0.24.0: https://docs.nvidia.com/nemo/guardrails/v0.24.0/llms.txt > - v0.23.0: https://docs.nvidia.com/nemo/guardrails/v0.23.0/llms.txt > - v0.22.0: https://docs.nvidia.com/nemo/guardrails/v0.22.0/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/guardrails/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/guardrails/_mcp/server. # Generation Options Reference > Configure logging, LLM parameters, and rail selection for generation. Generation options let you control which rails run, which parameters the main large language model (LLM) receives, and which context and logging data the NVIDIA NeMo Guardrails library returns. To use generation options, provide the `options` keyword argument to the `generate()` or `generate_async()` methods: ```python messages = [{ "role": "user", "content": "..." }] rails.generate(messages=messages, options={...}) ``` > **Note** > > Generation options are also available through [Chat Completions: Control Generation Options](/run-guardrailed-inference/using-fastapi-server/chat-with-guardrailed-model#control-generation-options). ## Engine Support Both the `LLMRails` and `IORails` engines accept generation options, and both return a structured `GenerationResponse` object when you pass `options`. Because `IORails` does not run the Colang runtime, the options that read or write Colang state are unavailable on it, and a few `GenerationResponse` fields are built differently. Each section below documents the shared behavior first and then the `IORails` difference. For the complete field-by-field comparison, refer to [IORails and LLMRails Differences](#iorails-and-llmrails-differences). > **Warning** > > Passing `options` to `generate()` or `generate_async()` on `IORails` returns a `GenerationResponse` object instead of an OpenAI-style message dictionary. > This behavior also applies to a `Guardrails` facade that selected `IORails`. > Earlier versions honored the `llm_params` and rail toggles inside `options` but always returned a message dictionary. > Update code that reads `result["content"]` to read `result.response[0]["content"]`. ```python # No options: an OpenAI-style message dictionary. message = rails.generate(messages=messages) print(message["content"]) # With options: a GenerationResponse. result = rails.generate(messages=messages, options={"log": {"llm_calls": True}}) print(result.response[0]["content"]) print(result.reasoning_content) ``` > **Note** > > An empty `options={}` behaves differently on the two engines. > `IORails` treats it as an all-defaults `GenerationOptions` and returns a `GenerationResponse`. > `LLMRails` raises `TypeError`, because an empty dictionary is falsy and does not match its dictionary branch. > Pass `options=GenerationOptions()` when you want the structured return with default settings on either engine. ## Disabling Rails Use the `rails` generation option to choose which rail categories to apply. The supported categories are `input`, `output`, `dialog`, `retrieval`, `tool_input`, and `tool_output`. By default, all categories are enabled. `IORails` runs input, output, and tool rails only. It ignores the `dialog` and `retrieval` categories rather than raising, because a configuration that declares dialog or retrieval rails routes to `LLMRails` in the first place. ```python res = rails.generate(messages=messages) ``` runs the same rails as: ```python res = rails.generate(messages=messages, options={ "rails": ["input", "output", "dialog", "retrieval", "tool_input", "tool_output"] }) ``` The two calls return different types because passing `options` selects the structured `GenerationResponse` return. The list form is exhaustive: every category you leave out of the list is disabled. To disable one category and keep the rest, use the dictionary form instead. ```python res = rails.generate(messages=messages, options={ "rails": {"dialog": False} }) ``` > **Warning** > > The rails-only patterns in the following three sections depend on disabling the dialog rails. > This setting suppresses the main LLM call and echoes the supplied message back as the response. > That behavior is specific to `LLMRails`. > `IORails` has no dialog rails to disable, so it always calls the main model and returns a generated answer. > To validate messages without generating on either engine, use [`check()` and `check_async()`](/run-guardrailed-inference/using-python-apis/check-messages) instead. ### Input Rails Only To check a user's input with only the input rails from a guardrails configuration, disable all other rail categories: ```python res = rails.generate(messages=[{ "role": "user", "content": "Some user input." }], options={ "rails": ["input"] }) ``` The response will be the same string if the input was allowed "as is": ```json { "role": "assistant", "content": "Some user input." } ``` If some of the rails alter the input, for example, to mask sensitive information, then the returned value is the altered input. ```json { "role": "assistant", "content": "Some altered user input." } ``` If the input was blocked, you will get the predefined response `bot refuse to respond` (by default "I'm sorry, I can't respond to that"). ```json { "role": "assistant", "content": "I'm sorry, I can't respond to that." } ``` For more details on what rails was triggered, use the `log.activated_rails` generation option. ### Input and Output Rails Only If you want to check both the user input and an output that was generated outside of the guardrails configuration, you must disable the dialog rails and the retrieval rails, and provide a bot message as well when making the call: ```python res = rails.generate(messages=[{ "role": "user", "content": "Some user input." }, { "role": "assistant", "content": "Some bot output." }], options={ "rails": ["input", "output"] }) ``` The response will be the exact bot message provided, if allowed, an altered version if an output rail decides to change it, for example, to remove sensitive information, or the predefined message for `bot refuse to respond`, if the message was blocked. For receive details on what rails are triggered, use the `log.activated_rails` generation option. #### Worked Example: Compare All Rails to Input and Output Rails The [topical rails tutorial](/configure-guardrails/colang/colang-1/tutorials/6-topical-rails) uses an ABC bot configuration with input, dialog, generation, and output rails. When all rails are enabled, a simple greeting can activate several rails and trigger multiple LLM calls: ```text {'type': 'input', 'name': 'self check input'} {'type': 'dialog', 'name': 'generate user intent'} {'type': 'dialog', 'name': 'generate next step'} {'type': 'generation', 'name': 'generate bot message'} {'type': 'output', 'name': 'self check output'} {'type': 'output', 'name': 'check blocked terms'} ``` The `explain()` method can show the corresponding LLM call count: ```python info = rails.explain() info.print_llm_calls_summary() ``` ```text Summary: 5 LLM call(s) took 3.54 seconds and used 1621 tokens. ``` If you only need to validate an already-generated assistant message, provide both the user and assistant messages and set `options={"rails": ["input", "output"]}`. This skips dialog, retrieval, and generation rails while still applying the configured input and output checks. For validation-only use cases, prefer the [`check()` and `check_async()` APIs](/run-guardrailed-inference/using-python-apis/check-messages), which run input and output rails without invoking full generation. ### Output Rails Only To apply output rails exclusively to an LLM response, disable the input rails and provide an empty input. ```python res = rails.generate(messages=[{ "role": "user", "content": "" }, { "role": "assistant", "content": "Some bot output." }], options={ "rails": ["output"] }) ``` ## Detailed Logging Information You can obtain detailed information about what happened under the hood during the generation process by setting the `log` generation option. This option has four different inner-options: * `activated_rails`: Include detailed information about the rails that were activated during generation. * `llm_calls`: Include the prompt, completion, timing, token usage, request ID, model, and provider information available for each LLM call. * `internal_events`: Include the array of internal generated events. * `colang_history`: Include the history of the conversation in Colang format. ```python res = rails.generate(messages=messages, options={ "log": { "activated_rails": True, "llm_calls": True, "internal_events": True, "colang_history": True } }) ``` ```text { "response": [...], "log": { "activated_rails": { ... }, "stats": {...}, "llm_calls": [...], "internal_events": [...], "colang_history": "..." } } ``` When using the Python API, the `log` is an object that also has a `print_summary` method. When called, it will print a simplified version of the log information. Below is a sample output. ```python res.log.print_summary() ``` ```markdown # General stats - Total time: 2.85s - [0.56s][19.64%]: INPUT Rails - [1.40s][49.02%]: DIALOG Rails - [0.58s][20.22%]: GENERATION Rails - [0.31s][10.98%]: OUTPUT Rails - 5 LLM calls, 2.74s total duration, 1641 total prompt tokens, 103 total completion tokens, 1744 total tokens. # Detailed stats - [0.56s] INPUT (self check input): 1 actions (self_check_input), 1 llm calls [0.56s] - [0.43s] DIALOG (generate user intent): 1 actions (generate_user_intent), 1 llm calls [0.43s] - [0.96s] DIALOG (generate next step): 1 actions (generate_next_step), 1 llm calls [0.95s] - [0.58s] GENERATION (generate bot message): 2 actions (retrieve_relevant_chunks, generate_bot_message), 1 llm calls [0.49s] - [0.31s] OUTPUT (self check output): 1 actions (self_check_output), 1 llm calls [0.31s] ``` ### Log Options on IORails `IORails` supports `activated_rails` and `llm_calls`, and it includes `stats` whenever you request either one, exactly as `LLMRails` does. `internal_events` and `colang_history` describe Colang runtime state, so `IORails` raises `NotImplementedError` when you set either to `True` rather than returning an empty value. The following example requests the supported log fields: ```python res = rails.generate(messages=messages, options={ "log": { "activated_rails": True, "llm_calls": True } }) ``` `IORails` synthesizes the log from the record each rail leaves behind, so the structure matches `LLMRails` but the contents reflect the way `IORails` executes rails: * Each activated rail carries exactly one `ExecutedAction`, holding the rail's structured verdict in `return_value` and up to one LLM or API call. `IORails` runs one model-backed check per rail instead of a Colang action chain. * `activated_rails[].decisions` is always an empty list, because decisions are a Colang construct. * The main model call appears as a rail with `type` `generation`, `name` `generation`, and action name `generate_bot_message`. On `LLMRails` the equivalent entries come from the Colang dialog rails, such as `generate user intent`. * `llm_calls[].prompt` is the conversation serialized as `": "` lines. The content matches what `LLMRails` logs, but the formatting differs. * Each call record includes timing and the request ID, token usage, model name, and provider name available from the provider. The `stats` object aggregates the available values the same way as `LLMRails`. * A request stopped by a rail still returns the log for the rails that ran before processing stopped. ## Output Variables > **Warning** > > Output variables are an `LLMRails` capability. > `IORails` raises `ValueError` when you pass `output_vars`, and its `GenerationResponse.output_data` is always `None`. > There is no Colang context for `IORails` to read. Some rails can store additional information in [Colang 1.0 Language Syntax: Variables](/configure-guardrails/colang/colang-1/colang-language-syntax-guide#variables). You can return the content of these variables by setting the `output_vars` generation option to the list of names for all the variables that you are interested in. If you want to return the complete context (this will also include some predefined variables), you can set `output_vars` to `True`. ```python rails.generate(messages=messages, options={ "output_vars": ["some_input_rail_score", "some_output_rail_score"] }) ``` You can find the returned data in the `output_data` key of the response: ```text { "response": [...], "output_data": { "some_input_rail_score": 0.7, "some_output_rail_score": 0.8 } } ``` ## Additional LLM Parameters To supply additional parameters to the LLM call during final message generation, utilize the `llm_params` option. The following example demonstrates how to apply a lower value for `temperature`: ```python rails.generate(messages=messages, options={ "llm_params": { "temperature": 0.2 } }) ``` The available parameters are determined by the specific LLM engine in use. The NeMo Guardrails library transmits values defined in the options parameter without modification. Both engines apply `llm_params` to the main model call only, not to the models that back the rails. `IORails` also accepts `llm_params` in `stream_async()`, and it is the supported way to pass tool definitions to the main model. ## Additional LLM Output You can receive additional output from the LLM generation by setting `llm_output` to `True` through the `options` parameter. ```python rails.generate(messages=messages, options={ "llm_output": True }) ``` > **Note** > > `GenerationResponse.llm_output` is currently `None` on both engines. > It is populated from `LLMCallInfo.raw_response`, which no code path assigns. > `IORails` accepts `llm_output: True` and ignores it. > To inspect provider-specific response data, use `llm_metadata` or the `log.llm_calls` entries instead. ## IORails and LLMRails Differences This section collects the differences between the two engines in one place. For capability differences outside generation options, refer to [Engine Feature Support](/reference/engine-feature-support). ### Method Signature `IORails.generate()` and `IORails.generate_async()` take the same first three parameters as `LLMRails`. ```python def generate( self, prompt: Optional[str] = None, messages: Optional[LLMMessages] = None, options: Optional[Union[dict, GenerationOptions]] = None, **kwargs, ) -> Union[LLMMessage, GenerationResponse]: ``` Three argument-handling details differ from `LLMRails`: * When you pass both `prompt` and `messages`, `IORails` uses `messages` and ignores `prompt`. `LLMRails` raises `ValueError`. * When you pass a `prompt` and `options`, `LLMRails` sets `GenerationResponse.response` to a plain string. `IORails` always sets it to a one-element list holding the assistant message. * An empty `options={}` returns a `GenerationResponse` on `IORails` and raises `TypeError` on `LLMRails`. Pass `options=GenerationOptions()` for the structured return with default settings on either engine. Passing a message list as the first positional argument raises `TypeError` on `IORails`, because that slot is `prompt`. Pass the list with the `messages=` keyword. ```python # Raises TypeError on IORails. rails.generate([{"role": "user", "content": "Hello"}]) # Correct. rails.generate(messages=[{"role": "user", "content": "Hello"}]) ``` `IORails` does not support the `state` and `streaming_handler` keyword arguments that `LLMRails` accepts. `state` raises `ValueError`, because `IORails` is a stateless input and output rails engine with nothing to persist or resume. `streaming_handler` is accepted and ignored. Use [`stream_async()`](/run-guardrailed-inference/using-python-apis/streaming) to stream from `IORails`. ### GenerationOptions Support `stream_async()` accepts `options` too, but reads only a subset of it and never yields a `GenerationResponse`. Refer to [Generation Options and Streaming](/run-guardrailed-inference/using-python-apis/streaming#generation-options-and-streaming). Legend: ✓ supported · ✗ not supported (raises) · ◐ accepted but not populated · N/A not applicable to the engine. The following table compares generation option support for `generate()` and `generate_async()`: | Field | Type | LLMRails | IORails | Notes | | ------------------- | --------------------------- | :------: | :-----: | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `rails.input` | `bool \| list[str]` | ✓ | ✓ | `IORails` runs only the named rails when given a list. `LLMRails` tests truthiness only, so any non-empty list means "all enabled". | | `rails.output` | `bool \| list[str]` | ✓ | ✓ | Same list-selection difference as `rails.input`. | | `rails.tool_input` | `bool \| list[str]` | ✓ | ✓ | Gates the tool-result rails. Same list-selection difference. | | `rails.tool_output` | `bool \| list[str]` | ✓ | ✓ | Gates the tool-call rails. Same list-selection difference. | | `rails.dialog` | `bool` | ✓ | N/A | `IORails` runs no dialog rails and ignores the toggle. On `LLMRails`, `dialog: False` suppresses the main LLM call and echoes the supplied message back. | | `rails.retrieval` | `bool \| list[str]` | ✓ | N/A | `IORails` runs no retrieval rails and ignores the toggle. | | `llm_params` | `dict \| None` | ✓ | ✓ | Applied to the main model call only, never to the rail models. On `IORails`, this is also how you pass tool definitions, and it is accepted by `stream_async()`. | | `llm_output` | `bool` | ◐ | ◐ | Accepted by both. The resulting `llm_output` field is `None` on both. | | `output_vars` | `bool \| list[str] \| None` | ✓ | ✗ | `IORails` raises `ValueError`: it has no Colang context to read. | | `log` | `GenerationLogOptions` | ✓ | ◐ | Refer to [GenerationLogOptions Support](#generationlogoptions-support). | `IORails` raises for Colang-dependent options that it cannot honor, including `output_vars`, `log.internal_events`, and `log.colang_history`. It accepts and ignores `llm_output`, `rails.dialog`, and `rails.retrieval`. `state` is a `generate()` keyword argument rather than a `GenerationOptions` field. `enforce` is commented out in the source and is not a live field on either engine. ### GenerationLogOptions Support The following table compares generation log field support: | Field | Type | LLMRails | IORails | Notes | | ----------------- | ------ | :------: | :-----: | ------------------------------------------------------------------------------------------------------------ | | `activated_rails` | `bool` | ✓ | ✓ | `IORails` synthesizes one entry per rail, each with a single `ExecutedAction`. `decisions` is always empty. | | `llm_calls` | `bool` | ✓ | ✓ | Both engines include the timing, usage, request ID, model, and provider information available for each call. | | `internal_events` | `bool` | ✓ | ✗ | `IORails` raises `NotImplementedError`: no Colang runtime. | | `colang_history` | `bool` | ✓ | ✗ | `IORails` raises `NotImplementedError`: no Colang runtime. | `GenerationLog.stats` is not a `GenerationLogOptions` field. Both engines populate it whenever you request either `activated_rails` or `llm_calls`. For the `IORails` log structure and how it differs from the Colang-derived log, refer to [Log Options on IORails](#log-options-on-iorails). ### GenerationResponse Field Support `stream_async()` never yields or returns a `GenerationResponse` on either engine. The following table compares the fields returned when you pass `options` to `generate()` or `generate_async()`: | Field | Type | LLMRails | IORails | Notes | | ------------------- | ----------------------- | :------: | :-----: | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `response` | `str \| list[dict]` | ✓ | ✓ | `IORails` always returns a one-element `[assistant_msg]` list. `LLMRails` returns a plain string when called with `prompt=` instead of `messages=`. | | `tool_calls` | `list \| None` | ✓ | ✓ | `IORails` uses the canonical `ToolCall.to_dict()` shape with dictionary arguments. `LLMRails` can preserve a supplied assistant tool call in the OpenAI wire shape, including JSON-string arguments, when dialog rails are disabled. The bare-message and streaming paths on `IORails` also use the OpenAI wire shape. | | `reasoning_content` | `str \| None` | ✓ | ✓ | Provider `reasoning` field, or content extracted from `` tags. On this path `IORails` keeps the message content clean. The bare dictionary returned without `options` inlines reasoning as a `` prefix. No streamed equivalent exists on either engine. | | `log` | `GenerationLog \| None` | ✓ | ◐ | `IORails` populates `activated_rails`, `llm_calls`, and `stats` only. Refer to [Log Options on IORails](#log-options-on-iorails). | | `llm_metadata` | `dict \| None` | ✓ | ◐ | Provider metadata blob verbatim, such as `response_headers`. `IORails` reports the main model call. `LLMRails` reports the most recent call, usually the last output rail. The library does not add normalized usage here. Read token counts from `log`. | | `llm_output` | `dict \| None` | ◐ | ◐ | Always `None` on both engines, because `LLMCallInfo.raw_response` is never assigned. | | `output_data` | `dict \| None` | ✓ | ✗ | Always `None` on `IORails`. Passing `output_vars` raises `ValueError`. | | `state` | `dict \| None` | ◐ | ✗ | `LLMRails` supports public state through Colang 1.0. `IORails` leaves it `None`, and passing `state` raises `ValueError`. | Reasoning bypasses the output rails on both engines, so output rails check the final answer rather than the reasoning trace. ### Requests Stopped by Rails When a rail blocks content, `LLMRails` returns the configured refusal message in `response`. `IORails` returns its fixed refusal message for a policy block and an internal-error message for a rail execution failure. In either `IORails` case, `tool_calls`, `reasoning_content`, and `llm_metadata` are `None`. To inspect records through the rail that stopped processing, request `log.activated_rails`. That rail has `stop` set to `True`, and its action's `return_value["failed"]` value distinguishes a failure from a policy block. ## Limitations * Only supported for the `generate`/`generate_async` methods (not for `generate_events`/`generate_events_async`). * On `LLMRails`, specifying which individual rails of a particular type to activate is not yet supported. `IORails` supports it: pass a list of rail names to `rails.input`, `rails.output`, `rails.tool_input`, or `rails.tool_output`. * On `LLMRails` with a Colang 2.x configuration, `output_vars`, `log`, and `llm_output` raise `ValueError`.