> This page is for version v0.25.0.
> For other versions, use one of these documentation indexes:
> - Latest (default): https://docs.nvidia.com/nemo/guardrails/latest/llms.txt
> - v0.25.0: https://docs.nvidia.com/nemo/guardrails/v0.25.0/llms.txt
> - v0.24.1: https://docs.nvidia.com/nemo/guardrails/v0.24.1/llms.txt
> - v0.24.0: https://docs.nvidia.com/nemo/guardrails/v0.24.0/llms.txt
> - v0.23.0: https://docs.nvidia.com/nemo/guardrails/v0.23.0/llms.txt
> - v0.22.0: https://docs.nvidia.com/nemo/guardrails/v0.22.0/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/guardrails/llms.txt.

# nemoguardrails.actions.llm.generation

A set of actions for generating various types of completions using an LLMs.

## Module Contents

### Classes

| Name                                                                                            | Description                                                                        |
| ----------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- |
| [`LLMGenerationActions`](#nemoguardrails-actions-llm-generation-LLMGenerationActions)           | A container objects for multiple related actions.                                  |
| [`SingleCallPayload`](#nemoguardrails-actions-llm-generation-SingleCallPayload)                 | The bot intent/message events computed by the single-call generation.              |
| [`_StreamingHandoffRegistry`](#nemoguardrails-actions-llm-generation-_StreamingHandoffRegistry) | Encodes the `&lt;&lt;STREAMING[uid]&gt;&gt;` handoff used by the single-call path. |

### Functions

| Name                                                                                              | Description                                                                           |
| ------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- |
| [`_bot_turn_output_events`](#nemoguardrails-actions-llm-generation-_bot_turn_output_events)       | Build the output events for a bot turn: an optional BotThinking then the final event. |
| [`_spec_op_as_dict`](#nemoguardrails-actions-llm-generation-_spec_op_as_dict)                     | Return a `SpecOp`'s spec as a dict.                                                   |
| [`_streaming_pattern_for`](#nemoguardrails-actions-llm-generation-_streaming_pattern_for)         | Return the `(prefix, suffix)` streaming pattern for the bot message.                  |
| [`_unpack_passthrough_output`](#nemoguardrails-actions-llm-generation-_unpack_passthrough_output) | Split a passthrough fn result into `(text, passthrough_output)`.                      |
| [`build_single_call_payload`](#nemoguardrails-actions-llm-generation-build_single_call_payload)   | Build the single-call cache carried on the `UserIntent` event.                        |
| [`clean_utterance_content`](#nemoguardrails-actions-llm-generation-clean_utterance_content)       | Clean an utterance by performing the following operations:                            |
| [`read_single_call_payload`](#nemoguardrails-actions-llm-generation-read_single_call_payload)     | Read the single-call cache back from a `UserIntent` event.                            |

### Data

[`_STREAMING_MARKER_PREFIX`](#nemoguardrails-actions-llm-generation-_STREAMING_MARKER_PREFIX)

[`_STREAMING_MARKER_SUFFIX`](#nemoguardrails-actions-llm-generation-_STREAMING_MARKER_SUFFIX)

[`_streaming_handoff`](#nemoguardrails-actions-llm-generation-_streaming_handoff)

[`log`](#nemoguardrails-actions-llm-generation-log)

### API

```python
class nemoguardrails.actions.llm.generation.LLMGenerationActions(
    config: nemoguardrails.rails.llm.config.RailsConfig,
    llm: typing.Optional[nemoguardrails.types.LLMModel],
    llm_task_manager: nemoguardrails.llm.taskmanager.LLMTaskManager,
    get_embedding_search_provider_instance: typing.Callable[[Optional[EmbeddingSearchProvider]], nemoguardrails.embeddings.index.EmbeddingsIndex],
    verbose: bool = False
)
```

A container objects for multiple related actions.

**`_init_lock`** `= asyncio.Lock()`

---

**`_passthrough_fn`** `Optional[Callable[..., Awaitable[str]]] = None`

---

**`bot_messages`** `= config.bot_messages.copy()`

---

**`env`** `= SandboxedEnvironment()`

---

**`user_messages`** `= config.user_messages.copy()`

---

```python
nemoguardrails.actions.llm.generation.LLMGenerationActions._bot_message_from_single_call_cache(
    user_intent_event: dict,
    events: typing.List[dict],
    streaming_handler: typing.Optional[nemoguardrails.streaming.StreamingHandler]
) -> typing.Optional[nemoguardrails.actions.actions.ActionResult]
```

async

Return the bot message cached by the single LLM call, or None to fall back.

Returns None when the cache cannot be used -- the last user-intent event
is not a `UserIntent`, or its cached bot intent does not match the bot
intent now being generated -- in which case the caller regenerates.

```python
nemoguardrails.actions.llm.generation.LLMGenerationActions._build_intent_steps_examples(
    text: str
) -> typing.Tuple[typing.List[str], typing.List[str]]
```

async

Build the few-shot examples and candidate intents for the single call.

Searches the user-message index for utterances similar to `text`, pairs
each candidate intent with a flow (and its bot message, if any) from the
flows / bot-message indexes, and returns up to five formatted examples
plus the list of candidate user intents.

```python
nemoguardrails.actions.llm.generation.LLMGenerationActions._detect_user_intent(
    events: typing.List[dict],
    event: dict,
    config: nemoguardrails.rails.llm.config.RailsConfig,
    generation_llm: typing.Optional[nemoguardrails.types.LLMModel]
) -> nemoguardrails.actions.actions.ActionResult
```

async

Detect the canonical form (user intent) for the given user message.

Covers both the embeddings-only lookup and the LLM-based canonical-form
generation. Always returns a single `UserIntent` event.

```python
nemoguardrails.actions.llm.generation.LLMGenerationActions._discard_single_call_handoff(
    events: typing.List[dict]
) -> None
```

Evict a pending single-call streaming handoff for a bypassed bot message.

When `generate_bot_message` short-circuits to a predefined message or a
`$context_var` it never consumes the single-call cache, so a handoff
registered while streaming the (now unused) cached bot message would leak.
This releases it. Safe to call when there is no pending handoff.

```python
nemoguardrails.actions.llm.generation.LLMGenerationActions._emit_general_bot_turn(
    events: typing.List[dict],
    context: dict,
    event: dict,
    generation_llm: typing.Optional[nemoguardrails.types.LLMModel],
    streaming_handler: typing.Optional[nemoguardrails.streaming.StreamingHandler],
    kb: typing.Optional[nemoguardrails.kb.kb.KnowledgeBase]
) -> nemoguardrails.actions.actions.ActionResult
```

async

Generate and package a general bot turn when there are no user messages.

Handles the passthrough (with or without a passthrough fn) and the
non-passthrough general paths, then packages the result into the
BotMessage/BotToolCalls/BotThinking events and context updates that
`generate_user_intent` returns in this mode.

```python
nemoguardrails.actions.llm.generation.LLMGenerationActions._ensure_bot_message_index()
```

async

```python
nemoguardrails.actions.llm.generation.LLMGenerationActions._ensure_flows_index()
```

async

```python
nemoguardrails.actions.llm.generation.LLMGenerationActions._ensure_user_message_index()
```

async

```python
nemoguardrails.actions.llm.generation.LLMGenerationActions._extract_bot_message_example(
    flow: nemoguardrails.colang.v2_x.lang.colang_ast.Flow
)
```

```python
nemoguardrails.actions.llm.generation.LLMGenerationActions._extract_user_message_example(
    flow: nemoguardrails.colang.v2_x.lang.colang_ast.Flow
) -> None
```

Heuristic to extract user message examples from a flow.

```python
nemoguardrails.actions.llm.generation.LLMGenerationActions._generate_general_response(
    generation_llm: typing.Optional[nemoguardrails.types.LLMModel],
    prompt,
    streaming_handler: typing.Optional[nemoguardrails.streaming.StreamingHandler] = None,
    stream_during_call: bool = False,
    stop: typing.Optional[typing.List[str]] = None,
    llm_call_task: nemoguardrails.llm.types.Task = Task.GENERAL,
    parse_task: nemoguardrails.llm.types.Task = Task.GENERAL,
    llm_params: typing.Optional[dict] = None
) -> str
```

async

Make a single general-response LLM call and parse its output.

This is the duplicated core of the four `Task.GENERAL` call sites: the
general user-intent fallback, the passthrough completion, the single-call
general branch and the passthrough bot-message branch. Prompt rendering,
chunk retrieval and any output stripping stay at the call sites; only the
`LLMCallInfo` set, the `llm_call` and the parse are shared here. The
`stop`, `stream_during_call`, `llm_call_task` and `parse_task`
parameters preserve the per-site differences (e.g. the passthrough
bot-message branch reports `GENERATE_BOT_MESSAGE` but parses as
`GENERAL`).

```python
nemoguardrails.actions.llm.generation.LLMGenerationActions._get_general_instructions()
```

Helper to extract the general instruction.

```python
nemoguardrails.actions.llm.generation.LLMGenerationActions._get_sample_conversation_two_turns()
```

Helper to extract only the two turns from the sample conversation.

This is needed to be included to "seed" the conversation so that the model
can follow the format more easily.

```python
nemoguardrails.actions.llm.generation.LLMGenerationActions._init_bot_message_index()
```

async

Initializes the index of bot messages.

```python
nemoguardrails.actions.llm.generation.LLMGenerationActions._init_flows_index()
```

async

Initializes the index of flows.

```python
nemoguardrails.actions.llm.generation.LLMGenerationActions._init_user_message_index()
```

async

Initializes the index of user messages.

```python
nemoguardrails.actions.llm.generation.LLMGenerationActions._process_flows()
```

Process the provided flows to extract the user utterance examples.

```python
nemoguardrails.actions.llm.generation.LLMGenerationActions._render_string(
    template_str: str,
    context: typing.Optional[dict] = None
) -> str
```

Render a string using the provided context information.

**Parameters:**

**`template_str`** `str`

The string template to render.

---

**`context`** `Optional[dict]` — default: None

The context for rendering.

---

**Returns:** `str`

The rendered string.

```python
nemoguardrails.actions.llm.generation.LLMGenerationActions._search_flows_index(
    text,
    max_results
)
```

async

Search the index of flows.

```python
nemoguardrails.actions.llm.generation.LLMGenerationActions.generate_bot_message(
    events: typing.List[dict],
    context: dict,
    llm: typing.Optional[nemoguardrails.types.LLMModel] = None
)
```

async

Generate a bot message based on the desired bot intent.

```python
nemoguardrails.actions.llm.generation.LLMGenerationActions.generate_intent_steps_message(
    events: typing.List[dict],
    context: dict,
    llm: typing.Optional[nemoguardrails.types.LLMModel] = None,
    kb: typing.Optional[nemoguardrails.kb.kb.KnowledgeBase] = None
)
```

async

Generate all three main Guardrails phases with a single LLM call.
The three phases are: user canonical from (user intent), next flow steps (i.e. bot canonical form)
and bot message.

```python
nemoguardrails.actions.llm.generation.LLMGenerationActions.generate_next_steps(
    events: typing.List[dict],
    llm: typing.Optional[nemoguardrails.types.LLMModel] = None
)
```

async

Generate the next step in the current conversation flow.

Currently, only generates a next step after a user intent.

```python
nemoguardrails.actions.llm.generation.LLMGenerationActions.generate_user_intent(
    events: typing.List[dict],
    context: dict,
    config: nemoguardrails.rails.llm.config.RailsConfig,
    llm: typing.Optional[nemoguardrails.types.LLMModel] = None,
    kb: typing.Optional[nemoguardrails.kb.kb.KnowledgeBase] = None
)
```

async

Generate the canonical form for what the user said i.e. user intent.

```python
nemoguardrails.actions.llm.generation.LLMGenerationActions.generate_value(
    instructions: str,
    events: typing.List[dict],
    var_name: typing.Optional[str] = None,
    llm: typing.Optional[nemoguardrails.types.LLMModel] = None
)
```

async

Generate a value in the context of the conversation.

**Parameters:**

**`instructions`** `str`

The instructions to generate the value.

---

**`events`** `List[dict]`

The full stream of events so far.

---

**`var_name`** `Optional[str]` — default: None

The name of the variable to generate. If not specified, it will use the `action_result_key` as the name of the variable.

---

**`llm`** `Optional[LLMModel]` — default: None

`Optional[` [`LLMModel`](/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/types#nemoguardrails-types-LLMModel) `]` — Custom llm model to generate_value

---

```python
class nemoguardrails.actions.llm.generation.SingleCallPayload(
    bot_intent_event: dict,
    bot_message_event: dict
)
```

Dataclass

The bot intent/message events computed by the single-call generation.

Carried on the `additional_info` of the `UserIntent` event so that the
later `generate_next_steps` and `generate_bot_message` phases can reuse
the results of the single LLM call instead of calling the LLM again.

**`bot_intent_event`** `dict`

---

**`bot_message_event`** `dict`

---

```python
class nemoguardrails.actions.llm.generation._StreamingHandoffRegistry()
```

Encodes the `<<STREAMING[uid]>>` handoff used by the single-call path.

The single-call streaming path generates the bot message on an inner
`StreamingHandler`, registers it here and leaves a marker on the cached bot
message. The later bot-message phase parses the marker, takes the handler and
pipes it to the main handler. Each handler is removed when taken, so handlers
do not accumulate for the module lifetime.

**`_handlers`** `= {}`

---

```python
nemoguardrails.actions.llm.generation._StreamingHandoffRegistry.parse_marker(
    text: str
) -> typing.Optional[str]
```

Return the handler uid encoded in `text`, or None if not a marker.

```python
nemoguardrails.actions.llm.generation._StreamingHandoffRegistry.register(
    handler: nemoguardrails.streaming.StreamingHandler
) -> str
```

Register an inner handler and return the marker text for it.

```python
nemoguardrails.actions.llm.generation._StreamingHandoffRegistry.take(
    uid: str
) -> typing.Optional[nemoguardrails.streaming.StreamingHandler]
```

Remove and return the handler for `uid`, or None if already consumed.

```python
nemoguardrails.actions.llm.generation._bot_turn_output_events(
    final_event: dict,
    context_updates: typing.Optional[dict] = None
) -> typing.List[dict]
```

Build the output events for a bot turn: an optional BotThinking then the final event.

Pops any reasoning trace into a `BotThinking` event placed before
`final_event`. When `context_updates` is provided, the reasoning trace is
also recorded as `bot_thinking` (the generated bot-message path does this;
the single-call cached path does not).

```python
nemoguardrails.actions.llm.generation._spec_op_as_dict(
    spec_op: nemoguardrails.colang.v2_x.lang.colang_ast.SpecOp
) -> typing.Dict[str, typing.Any]
```

Return a `SpecOp`'s spec as a dict.

`SpecOp.spec` is `Union[Spec, dict]`; callers that need dict-only fields
(such as `elements`) use this to normalize it.

```python
nemoguardrails.actions.llm.generation._streaming_pattern_for(
    output_parser,
    include_bot_message_parser: bool
)
```

Return the `(prefix, suffix)` streaming pattern for the bot message.

The two call sites differ: `generate_bot_message` treats both `verbose_v1`
and `bot_message` as the verbose pattern, while
`generate_intent_steps_message` treats only `verbose_v1` that way;
`include_bot_message_parser` selects which rule applies.

```python
nemoguardrails.actions.llm.generation._unpack_passthrough_output(
    raw_output
)
```

Split a passthrough fn result into `(text, passthrough_output)`.

A tuple/list result carries an explicit passthrough output as its second
element; any other return value is treated as the text output alone.

```python
nemoguardrails.actions.llm.generation.build_single_call_payload(
    bot_intent_event: dict,
    bot_message_event: dict
) -> dict
```

Build the single-call cache carried on the `UserIntent` event.

Returns the plain nested `additional_info` dict (not a dataclass) so the
event serializes byte-identically as it passes between actions.

```python
nemoguardrails.actions.llm.generation.clean_utterance_content(
    utterance: str
) -> str
```

Clean an utterance by performing the following operations:

* replacing "\n" with "
  ".

Args:
utterance (str): The utterance to clean.

Returns:
str: The cleaned utterance.

```python
nemoguardrails.actions.llm.generation.read_single_call_payload(
    event: dict
) -> nemoguardrails.actions.llm.generation.SingleCallPayload
```

Read the single-call cache back from a `UserIntent` event.

Mirrors the existing direct dict access (raising `KeyError` when the
payload is absent); guarding that is a separate follow-up.

```python
nemoguardrails.actions.llm.generation._STREAMING_MARKER_PREFIX = 'Bot message: "<<STREAMING['
```

```python
nemoguardrails.actions.llm.generation._STREAMING_MARKER_SUFFIX = ']>>"'
```

```python
nemoguardrails.actions.llm.generation._streaming_handoff = _StreamingHandoffRegistry()
```

```python
nemoguardrails.actions.llm.generation.log = logging.getLogger(__name__)
```