> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/llm/llms.txt. # nemoguardrails.llm ## Subpackages * **[`nemoguardrails.llm.cache`](/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/llm/cache)** * **[`nemoguardrails.llm.clients`](/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/llm/clients)** * **[`nemoguardrails.llm.frameworks`](/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/llm/frameworks)** ## Submodules * **[`nemoguardrails.llm.call`](/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/llm/call)** * **[`nemoguardrails.llm.completion_parsing`](/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/llm/completion_parsing)** * **[`nemoguardrails.llm.constants`](/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/llm/constants)** * **[`nemoguardrails.llm.filters`](/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/llm/filters)** * **[`nemoguardrails.llm.openai_reasoning`](/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/llm/openai_reasoning)** * **[`nemoguardrails.llm.output_parsers`](/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/llm/output_parsers)** * **[`nemoguardrails.llm.prompts`](/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/llm/prompts)** * **[`nemoguardrails.llm.providers`](/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/llm/providers)** * **[`nemoguardrails.llm.taskmanager`](/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/llm/taskmanager)** * **[`nemoguardrails.llm.telemetry`](/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/llm/telemetry)** * **[`nemoguardrails.llm.types`](/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/llm/types)** ## Package Contents ### Classes | Name | Description | | ------------------------------------------------------------ | ---------------------------------------------- | | [`ChatMessage`](#nemoguardrails-types-ChatMessage) | - | | [`LLMFramework`](#nemoguardrails-types-LLMFramework) | Protocol for pluggable LLM framework backends. | | [`LLMModel`](#nemoguardrails-types-LLMModel) | Protocol that all LLM backends must implement. | | [`LLMResponse`](#nemoguardrails-types-LLMResponse) | - | | [`LLMResponseChunk`](#nemoguardrails-types-LLMResponseChunk) | - | | [`Role`](#nemoguardrails-types-Role) | - | | [`ToolCall`](#nemoguardrails-types-ToolCall) | - | | [`ToolCallFunction`](#nemoguardrails-types-ToolCallFunction) | - | | [`UsageInfo`](#nemoguardrails-types-UsageInfo) | - | ### Functions | Name | Description | | ---------------------------------------------------------------------------------------- | ------------------------------------------------- | | [`get_default_framework`](#nemoguardrails-llm-frameworks-registry-get_default_framework) | - | | [`register_framework`](#nemoguardrails-llm-frameworks-registry-register_framework) | Register an `LLMFramework` instance under `name`. | | [`register_provider`](#nemoguardrails-llm-providers-register_provider) | - | | [`set_default_framework`](#nemoguardrails-llm-frameworks-registry-set_default_framework) | - | ### Data [`FinishReason`](#nemoguardrails-types-FinishReason) ### API ```python class nemoguardrails.ChatMessage( role: nemoguardrails.types.Role, content: typing.Optional[typing.Union[str, typing.List[typing.Dict[str, typing.Any]]]] = None, tool_calls: typing.Optional[typing.List[nemoguardrails.types.ToolCall]] = None, tool_call_id: typing.Optional[str] = None, name: typing.Optional[str] = None, provider_metadata: typing.Dict[str, typing.Any] = dict() ) ``` Dataclass **`content`** `Optional[Union[str, List[Dict[str, Any]]]] = None` --- **`name`** `Optional[str] = None` --- **`provider_metadata`** `Dict[str, Any] = field(default_factory=dict)` --- **`role`** `Role` --- **`tool_call_id`** `Optional[str] = None` --- **`tool_calls`** `Optional[List[ToolCall]] = None` --- ```python nemoguardrails.ChatMessage.from_assistant( content: str, kwargs = {} ) -> nemoguardrails.types.ChatMessage ``` classmethod ```python nemoguardrails.ChatMessage.from_dict( d: typing.Dict[str, typing.Any] ) -> nemoguardrails.types.ChatMessage ``` classmethod Create a ChatMessage from a dict. Accepts both the canonical nested tool call format (`{"function": {"name": ..., "arguments": ...}}`) and the legacy flat format (`{"name": ..., "args": ...}`). JSON string arguments are parsed automatically. Role aliases like "bot", "human", and "developer" are mapped to canonical Role values. Unknown keys are captured into `provider_metadata`. ```python nemoguardrails.ChatMessage.from_system( content: str, kwargs = {} ) -> nemoguardrails.types.ChatMessage ``` classmethod ```python nemoguardrails.ChatMessage.from_tool( content: str, tool_call_id: str, kwargs = {} ) -> nemoguardrails.types.ChatMessage ``` classmethod ```python nemoguardrails.ChatMessage.from_user( content: str, kwargs = {} ) -> nemoguardrails.types.ChatMessage ``` classmethod ```python nemoguardrails.ChatMessage.to_dict() -> typing.Dict[str, typing.Any] ``` ```python class nemoguardrails.LLMFramework() ``` Protocol Protocol for pluggable LLM framework backends. Each framework (LangChain, LiteLLM, etc.) implements this protocol to provide a factory for creating `LLMModel` instances and managing its own set of providers. `model_kwargs` carries all provider-specific configuration. Framework implementations extract what they need (e.g. LangChain pops `mode` to choose between chat and text completion models). ```python nemoguardrails.LLMFramework.create_model( model_name: str, provider_name: str, model_kwargs: typing.Optional[typing.Dict[str, typing.Any]] = None ) -> nemoguardrails.types.LLMModel ``` ```python nemoguardrails.LLMFramework.get_provider_names() -> typing.List[str] ``` ```python nemoguardrails.LLMFramework.register_provider( name: str, provider_cls: typing.Any ) -> None ``` ```python nemoguardrails.LLMFramework.reset() -> None ``` async Release all framework-owned resources and clear all registered state. Implementations should close any pooled connections, clear registered providers, and return the framework to its initial state. Callers can continue using the instance after reset; new resources will be created on demand. ```python class nemoguardrails.LLMModel() ``` Protocol Protocol that all LLM backends must implement. Adapters wrap provider-specific SDKs (LangChain, LiteLLM, OpenAI, etc.) behind this interface so the core pipeline remains framework-agnostic. `prompt` accepts either a plain string or a list of `ChatMessage` objects. Adapters convert `ChatMessage` to whatever their SDK expects. `**kwargs` are forwarded to the underlying SDK (e.g. temperature, max_tokens). **`model_name`** `str` --- **`provider_name`** `Optional[str]` --- **`provider_url`** `Optional[str]` --- ```python nemoguardrails.LLMModel.generate_async( prompt: typing.Union[str, typing.List[nemoguardrails.types.ChatMessage]], stop: typing.Optional[typing.List[str]] = None, kwargs = {} ) -> nemoguardrails.types.LLMResponse ``` async ```python nemoguardrails.LLMModel.stream_async( prompt: typing.Union[str, typing.List[nemoguardrails.types.ChatMessage]], stop: typing.Optional[typing.List[str]] = None, kwargs = {} ) -> typing.AsyncIterator[nemoguardrails.types.LLMResponseChunk] ``` Implementations must be async generator functions (use `yield`). ```python class nemoguardrails.LLMResponse( content: str, reasoning: typing.Optional[str] = None, tool_calls: typing.Optional[typing.List[nemoguardrails.types.ToolCall]] = None, model: typing.Optional[str] = None, finish_reason: typing.Optional[nemoguardrails.types.FinishReason] = None, stop_sequence: typing.Optional[str] = None, request_id: typing.Optional[str] = None, usage: typing.Optional[nemoguardrails.types.UsageInfo] = None, provider_metadata: typing.Optional[typing.Dict[str, typing.Any]] = None ) ``` Dataclass **`content`** `str` --- **`finish_reason`** `Optional[FinishReason] = None` --- **`model`** `Optional[str] = None` --- **`provider_metadata`** `Optional[Dict[str, Any]] = None` --- **`reasoning`** `Optional[str] = None` --- **`request_id`** `Optional[str] = None` --- **`stop_sequence`** `Optional[str] = None` --- **`tool_calls`** `Optional[List[ToolCall]] = None` --- **`usage`** `Optional[UsageInfo] = None` --- ```python class nemoguardrails.LLMResponseChunk( delta_content: typing.Optional[str] = None, delta_reasoning: typing.Optional[str] = None, delta_tool_calls: typing.Optional[typing.List[nemoguardrails.types.ToolCall]] = None, model: typing.Optional[str] = None, finish_reason: typing.Optional[nemoguardrails.types.FinishReason] = None, request_id: typing.Optional[str] = None, usage: typing.Optional[nemoguardrails.types.UsageInfo] = None, provider_metadata: typing.Optional[typing.Dict[str, typing.Any]] = None, tool_call_deltas: typing.Optional[typing.List[nemoguardrails.types.ToolCallDelta]] = None ) ``` Dataclass **`delta_content`** `Optional[str] = None` --- **`delta_reasoning`** `Optional[str] = None` --- **`delta_tool_calls`** `Optional[List[ToolCall]] = None` --- **`finish_reason`** `Optional[FinishReason] = None` --- **`model`** `Optional[str] = None` --- **`provider_metadata`** `Optional[Dict[str, Any]] = None` --- **`request_id`** `Optional[str] = None` --- **`tool_call_deltas`** `Optional[List[ToolCallDelta]] = None` --- **`usage`** `Optional[UsageInfo] = None` --- ```python class nemoguardrails.Role ``` **Bases:** `enum.Enum` **`ASSISTANT`** `= 'assistant'` --- **`SYSTEM`** `= 'system'` --- **`TOOL`** `= 'tool'` --- **`USER`** `= 'user'` --- ```python class nemoguardrails.ToolCall( id: str, type: str = 'function', function: nemoguardrails.types.ToolCallFunction = (lambda: ToolCallFunction(n... ) ``` Dataclass **`function`** `ToolCallFunction` --- **`id`** `str` --- **`type`** `str = 'function'` --- ```python nemoguardrails.ToolCall.to_dict() -> typing.Dict[str, typing.Any] ``` ```python class nemoguardrails.ToolCallFunction( name: str, arguments: typing.Dict[str, typing.Any] ) ``` Dataclass **`arguments`** `Dict[str, Any]` --- **`name`** `str` --- ```python class nemoguardrails.UsageInfo( input_tokens: int = 0, output_tokens: int = 0, total_tokens: int = 0, reasoning_tokens: typing.Optional[int] = None, cached_tokens: typing.Optional[int] = None ) ``` Dataclass **`cached_tokens`** `Optional[int] = None` --- **`input_tokens`** `int = 0` --- **`output_tokens`** `int = 0` --- **`reasoning_tokens`** `Optional[int] = None` --- **`total_tokens`** `int = 0` --- ```python nemoguardrails.get_default_framework() -> str ``` ```python nemoguardrails.register_framework( name: str, framework: nemoguardrails.types.LLMFramework ) -> None ``` Register an `LLMFramework` instance under `name`. Validates two invariants before storing the instance: 1. The object structurally matches the `LLMFramework` Protocol (`create_model`, `register_provider`, `get_provider_names`, `reset`). 2. Its `reset` attribute is an `async` coroutine function. The registry awaits it directly during shutdown / test teardown. **Raises:** * `ValueError`: a framework is already registered under `name`. * `TypeError`: `framework` does not implement the `LLMFramework` Protocol, or its `reset` is not an async coroutine function. ```python nemoguardrails.register_provider( name: str, provider_cls: typing.Any ) -> None ``` ```python nemoguardrails.set_default_framework( name: str ) -> None ``` ```python nemoguardrails.types.FinishReason = Literal['stop', 'length', 'tool_calls', 'content_filter', 'error', 'other'] ``` ## Docs - [nemoguardrails.llm.cache](https://docs.nvidia.com/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/llm/cache.md) - [nemoguardrails.llm.cache.interface](https://docs.nvidia.com/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/llm/cache/interface.md) - [nemoguardrails.llm.cache.lfu](https://docs.nvidia.com/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/llm/cache/lfu.md) - [nemoguardrails.llm.cache.utils](https://docs.nvidia.com/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/llm/cache/utils.md) - [nemoguardrails.llm.call](https://docs.nvidia.com/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/llm/call.md) - [nemoguardrails.llm.clients](https://docs.nvidia.com/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/llm/clients.md) - [nemoguardrails.llm.clients.base](https://docs.nvidia.com/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/llm/clients/base.md) - [nemoguardrails.llm.clients.constants](https://docs.nvidia.com/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/llm/clients/constants.md) - [nemoguardrails.llm.clients.openai_compatible](https://docs.nvidia.com/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/llm/clients/openai_compatible.md) - [nemoguardrails.llm.completion_parsing](https://docs.nvidia.com/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/llm/completion_parsing.md) - [nemoguardrails.llm.constants](https://docs.nvidia.com/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/llm/constants.md) - [nemoguardrails.llm.filters](https://docs.nvidia.com/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/llm/filters.md) - [nemoguardrails.llm.frameworks](https://docs.nvidia.com/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/llm/frameworks.md) - [nemoguardrails.llm.frameworks.default](https://docs.nvidia.com/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/llm/frameworks/default.md) - [nemoguardrails.llm.frameworks.registry](https://docs.nvidia.com/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/llm/frameworks/registry.md) - [nemoguardrails.llm.openai_reasoning](https://docs.nvidia.com/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/llm/openai_reasoning.md) - [nemoguardrails.llm.output_parsers](https://docs.nvidia.com/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/llm/output_parsers.md) - [nemoguardrails.llm.prompts](https://docs.nvidia.com/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/llm/prompts.md) - [nemoguardrails.llm.providers](https://docs.nvidia.com/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/llm/providers.md) - [nemoguardrails.llm.taskmanager](https://docs.nvidia.com/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/llm/taskmanager.md) - [nemoguardrails.llm.telemetry](https://docs.nvidia.com/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/llm/telemetry.md) - [nemoguardrails.llm.types](https://docs.nvidia.com/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/llm/types.md)