Skip to navigation

nemoguardrails.actions.llm.generation

View as MarkdownOpen in Claude

A set of actions for generating various types of completions using an LLMs.

Module Contents

Classes

NameDescription
LLMGenerationActionsA container objects for multiple related actions.
SingleCallPayloadThe bot intent/message events computed by the single-call generation.
_StreamingHandoffRegistryEncodes the <<STREAMING[uid]>> handoff used by the single-call path.

Functions

NameDescription
_bot_turn_output_eventsBuild the output events for a bot turn: an optional BotThinking then the final event.
_spec_op_as_dictReturn a SpecOp’s spec as a dict.
_streaming_pattern_forReturn the (prefix, suffix) streaming pattern for the bot message.
_unpack_passthrough_outputSplit a passthrough fn result into (text, passthrough_output).
build_single_call_payloadBuild the single-call cache carried on the UserIntent event.
clean_utterance_contentClean an utterance by performing the following operations:
read_single_call_payloadRead the single-call cache back from a UserIntent event.

Data

_STREAMING_MARKER_PREFIX

_STREAMING_MARKER_SUFFIX

_streaming_handoff

log

API

class nemoguardrails.actions.llm.generation.LLMGenerationActions(
llm: typing.Optional[nemoguardrails.types.LLMModel],
get_embedding_search_provider_instance: typing.Callable[[Optional[EmbeddingSearchProvider]], nemoguardrails.embeddings.index.EmbeddingsIndex],
verbose: bool = False
)

A container objects for multiple related actions.

_init_lock
= asyncio.Lock()
_passthrough_fn
Optional[Callable[..., Awaitable[str]]] = None
bot_messages
= config.bot_messages.copy()
env
= SandboxedEnvironment()
user_messages
= config.user_messages.copy()
nemoguardrails.actions.llm.generation.LLMGenerationActions._bot_message_from_single_call_cache(
user_intent_event: dict,
events: typing.List[dict],
streaming_handler: typing.Optional[nemoguardrails.streaming.StreamingHandler]
async

Return the bot message cached by the single LLM call, or None to fall back.

Returns None when the cache cannot be used — the last user-intent event is not a UserIntent, or its cached bot intent does not match the bot intent now being generated — in which case the caller regenerates.

nemoguardrails.actions.llm.generation.LLMGenerationActions._build_intent_steps_examples(
text: str
) -> typing.Tuple[typing.List[str], typing.List[str]]
async

Build the few-shot examples and candidate intents for the single call.

Searches the user-message index for utterances similar to text, pairs each candidate intent with a flow (and its bot message, if any) from the flows / bot-message indexes, and returns up to five formatted examples plus the list of candidate user intents.

nemoguardrails.actions.llm.generation.LLMGenerationActions._detect_user_intent(
events: typing.List[dict],
event: dict,
generation_llm: typing.Optional[nemoguardrails.types.LLMModel]
async

Detect the canonical form (user intent) for the given user message.

Covers both the embeddings-only lookup and the LLM-based canonical-form generation. Always returns a single UserIntent event.

nemoguardrails.actions.llm.generation.LLMGenerationActions._discard_single_call_handoff(
events: typing.List[dict]
) -> None

Evict a pending single-call streaming handoff for a bypassed bot message.

When generate_bot_message short-circuits to a predefined message or a $context_var it never consumes the single-call cache, so a handoff registered while streaming the (now unused) cached bot message would leak. This releases it. Safe to call when there is no pending handoff.

nemoguardrails.actions.llm.generation.LLMGenerationActions._emit_general_bot_turn(
events: typing.List[dict],
context: dict,
event: dict,
generation_llm: typing.Optional[nemoguardrails.types.LLMModel],
streaming_handler: typing.Optional[nemoguardrails.streaming.StreamingHandler],
async

Generate and package a general bot turn when there are no user messages.

Handles the passthrough (with or without a passthrough fn) and the non-passthrough general paths, then packages the result into the BotMessage/BotToolCalls/BotThinking events and context updates that generate_user_intent returns in this mode.

nemoguardrails.actions.llm.generation.LLMGenerationActions._ensure_bot_message_index()
async
nemoguardrails.actions.llm.generation.LLMGenerationActions._ensure_flows_index()
async
nemoguardrails.actions.llm.generation.LLMGenerationActions._ensure_user_message_index()
async
nemoguardrails.actions.llm.generation.LLMGenerationActions._extract_bot_message_example(
)
nemoguardrails.actions.llm.generation.LLMGenerationActions._extract_user_message_example(
) -> None

Heuristic to extract user message examples from a flow.

nemoguardrails.actions.llm.generation.LLMGenerationActions._generate_general_response(
generation_llm: typing.Optional[nemoguardrails.types.LLMModel],
prompt,
streaming_handler: typing.Optional[nemoguardrails.streaming.StreamingHandler] = None,
stream_during_call: bool = False,
stop: typing.Optional[typing.List[str]] = None,
llm_call_task: nemoguardrails.llm.types.Task = Task.GENERAL,
parse_task: nemoguardrails.llm.types.Task = Task.GENERAL,
llm_params: typing.Optional[dict] = None
) -> str
async

Make a single general-response LLM call and parse its output.

This is the duplicated core of the four Task.GENERAL call sites: the general user-intent fallback, the passthrough completion, the single-call general branch and the passthrough bot-message branch. Prompt rendering, chunk retrieval and any output stripping stay at the call sites; only the LLMCallInfo set, the llm_call and the parse are shared here. The stop, stream_during_call, llm_call_task and parse_task parameters preserve the per-site differences (e.g. the passthrough bot-message branch reports GENERATE_BOT_MESSAGE but parses as GENERAL).

nemoguardrails.actions.llm.generation.LLMGenerationActions._get_general_instructions()

Helper to extract the general instruction.

nemoguardrails.actions.llm.generation.LLMGenerationActions._get_sample_conversation_two_turns()

Helper to extract only the two turns from the sample conversation.

This is needed to be included to “seed” the conversation so that the model can follow the format more easily.

nemoguardrails.actions.llm.generation.LLMGenerationActions._init_bot_message_index()
async

Initializes the index of bot messages.

nemoguardrails.actions.llm.generation.LLMGenerationActions._init_flows_index()
async

Initializes the index of flows.

nemoguardrails.actions.llm.generation.LLMGenerationActions._init_user_message_index()
async

Initializes the index of user messages.

nemoguardrails.actions.llm.generation.LLMGenerationActions._process_flows()

Process the provided flows to extract the user utterance examples.

nemoguardrails.actions.llm.generation.LLMGenerationActions._render_string(
template_str: str,
context: typing.Optional[dict] = None
) -> str

Render a string using the provided context information.

Parameters:

template_str
str

The string template to render.

context
Optional[dict]Defaults to None

The context for rendering.

Returns: str

The rendered string.

nemoguardrails.actions.llm.generation.LLMGenerationActions._search_flows_index(
text,
max_results
)
async

Search the index of flows.

nemoguardrails.actions.llm.generation.LLMGenerationActions.generate_bot_message(
events: typing.List[dict],
context: dict,
llm: typing.Optional[nemoguardrails.types.LLMModel] = None
)
async

Generate a bot message based on the desired bot intent.

nemoguardrails.actions.llm.generation.LLMGenerationActions.generate_intent_steps_message(
events: typing.List[dict],
context: dict,
llm: typing.Optional[nemoguardrails.types.LLMModel] = None,
kb: typing.Optional[nemoguardrails.kb.kb.KnowledgeBase] = None
)
async

Generate all three main Guardrails phases with a single LLM call. The three phases are: user canonical from (user intent), next flow steps (i.e. bot canonical form) and bot message.

nemoguardrails.actions.llm.generation.LLMGenerationActions.generate_next_steps(
events: typing.List[dict],
llm: typing.Optional[nemoguardrails.types.LLMModel] = None
)
async

Generate the next step in the current conversation flow.

Currently, only generates a next step after a user intent.

nemoguardrails.actions.llm.generation.LLMGenerationActions.generate_user_intent(
events: typing.List[dict],
context: dict,
llm: typing.Optional[nemoguardrails.types.LLMModel] = None,
kb: typing.Optional[nemoguardrails.kb.kb.KnowledgeBase] = None
)
async

Generate the canonical form for what the user said i.e. user intent.

nemoguardrails.actions.llm.generation.LLMGenerationActions.generate_value(
instructions: str,
events: typing.List[dict],
var_name: typing.Optional[str] = None,
llm: typing.Optional[nemoguardrails.types.LLMModel] = None
)
async

Generate a value in the context of the conversation.

Parameters:

instructions
str

The instructions to generate the value.

events
List[dict]

The full stream of events so far.

var_name
Optional[str]Defaults to None

The name of the variable to generate. If not specified, it will use the action_result_key as the name of the variable.

llm
Optional[LLMModel]Defaults to None

Optional[ LLMModel ] — Custom llm model to generate_value

class nemoguardrails.actions.llm.generation.SingleCallPayload(
bot_intent_event: dict,
bot_message_event: dict
)
Dataclass

The bot intent/message events computed by the single-call generation.

Carried on the additional_info of the UserIntent event so that the later generate_next_steps and generate_bot_message phases can reuse the results of the single LLM call instead of calling the LLM again.

bot_intent_event
dict
bot_message_event
dict
class nemoguardrails.actions.llm.generation._StreamingHandoffRegistry()

Encodes the <<STREAMING[uid]>> handoff used by the single-call path.

The single-call streaming path generates the bot message on an inner StreamingHandler, registers it here and leaves a marker on the cached bot message. The later bot-message phase parses the marker, takes the handler and pipes it to the main handler. Each handler is removed when taken, so handlers do not accumulate for the module lifetime.

_handlers
= {}
nemoguardrails.actions.llm.generation._StreamingHandoffRegistry.parse_marker(
text: str
) -> typing.Optional[str]

Return the handler uid encoded in text, or None if not a marker.

nemoguardrails.actions.llm.generation._StreamingHandoffRegistry.register(
) -> str

Register an inner handler and return the marker text for it.

nemoguardrails.actions.llm.generation._StreamingHandoffRegistry.take(
uid: str

Remove and return the handler for uid, or None if already consumed.

nemoguardrails.actions.llm.generation._bot_turn_output_events(
final_event: dict,
context_updates: typing.Optional[dict] = None
) -> typing.List[dict]

Build the output events for a bot turn: an optional BotThinking then the final event.

Pops any reasoning trace into a BotThinking event placed before final_event. When context_updates is provided, the reasoning trace is also recorded as bot_thinking (the generated bot-message path does this; the single-call cached path does not).

nemoguardrails.actions.llm.generation._spec_op_as_dict(
) -> typing.Dict[str, typing.Any]

Return a SpecOp’s spec as a dict.

SpecOp.spec is Union[Spec, dict]; callers that need dict-only fields (such as elements) use this to normalize it.

nemoguardrails.actions.llm.generation._streaming_pattern_for(
output_parser,
include_bot_message_parser: bool
)

Return the (prefix, suffix) streaming pattern for the bot message.

The two call sites differ: generate_bot_message treats both verbose_v1 and bot_message as the verbose pattern, while generate_intent_steps_message treats only verbose_v1 that way; include_bot_message_parser selects which rule applies.

nemoguardrails.actions.llm.generation._unpack_passthrough_output(
raw_output
)

Split a passthrough fn result into (text, passthrough_output).

A tuple/list result carries an explicit passthrough output as its second element; any other return value is treated as the text output alone.

nemoguardrails.actions.llm.generation.build_single_call_payload(
bot_intent_event: dict,
bot_message_event: dict
) -> dict

Build the single-call cache carried on the UserIntent event.

Returns the plain nested additional_info dict (not a dataclass) so the event serializes byte-identically as it passes between actions.

nemoguardrails.actions.llm.generation.clean_utterance_content(
utterance: str
) -> str

Clean an utterance by performing the following operations:

  • replacing “\n” with ” ”.

Args: utterance (str): The utterance to clean.

Returns: str: The cleaned utterance.

nemoguardrails.actions.llm.generation.read_single_call_payload(
event: dict

Read the single-call cache back from a UserIntent event.

Mirrors the existing direct dict access (raising KeyError when the payload is absent); guarding that is a separate follow-up.

nemoguardrails.actions.llm.generation._STREAMING_MARKER_PREFIX = 'Bot message: "<<STREAMING['
nemoguardrails.actions.llm.generation._STREAMING_MARKER_SUFFIX = ']>>"'
nemoguardrails.actions.llm.generation._streaming_handoff = _StreamingHandoffRegistry()
nemoguardrails.actions.llm.generation.log = logging.getLogger(__name__)