nemoguardrails.guardrails.model_engine | NVIDIA NeMo Guardrails Library Developer Guide

Model engine for IORails.

Wraps a single Model config and makes raw HTTP calls to its OpenAI-compatible /v1/chat/completions endpoint via aiohttp. Retries are handled by aiohttp-retry (ExponentialRetry).

Module Contents

Classes

Name	Description
`ModelEngine`	Wraps a single Model config and makes HTTP calls to its endpoint.
`ModelEngineError`	Raised when a model engine call fails.
`_RequestParams`	Pre-built parameters for an HTTP request to the completions endpoint.

Functions

Name	Description
`_accumulate_tool_call_delta`	Update the tool-call accumulator with any tool_call deltas from a raw SSE chunk.
`_extract_tool_exchanges_nim`	Extract NIM tool exchanges. NIM uses the OpenAI Chat Completions shape.
`_extract_tool_exchanges_openai`	Group an OpenAI Chat Completions conversation into per-turn `ToolExchange`es.
`_extract_tool_results_nim`	Extract NIM tool results. NIM uses the OpenAI Chat Completions shape.
`_extract_tool_results_openai`	Extract OpenAI Chat Completions tool results into `ToolResult` objects.
`_finalize_tool_calls`	Assemble accumulated tool-call fragments into ToolCall objects.
`_parse_chat_completion`	Convert a /v1/chat/completions response dict into an LLMResponse.
`_parse_chat_completion_chunk`	Build an LLMResponseChunk from an SSE chunk dict.
`_parse_tools_nim`	Parse NIM tool definitions. NIM uses the OpenAI Chat Completions tool shape.
`_parse_tools_openai`	Parse OpenAI Chat Completions tool definitions into `Tool` objects.
`_parse_usage`	Build UsageInfo from an OpenAI-format usage dict.
`_to_wire_messages`	Normalize an `LLMModel` prompt into OpenAI-format message dicts.
`_tool_calls_from_message`	Extract the tool calls from one assistant message into `ToolCall` objects.
`_tool_result_from_message`	Normalize one OpenAI Chat Completions `role:"tool"` message into a `ToolResult`.

Data

_CHAT_COMPLETIONS_ENDPOINT

_ENGINE_BASE_URLS

_ERROR_BODY_MAX_CHARS

_OPERATION_NAME

_RESERVED_LLM_PARAMETERS

_RESULT_EXTRACTORS

_STANDARD_CHUNK_KEYS

_TOOL_EXCHANGE_EXTRACTORS

_TOOL_PARSERS

log

API

class nemoguardrails.guardrails.model_engine.ModelEngine(
    model_config: nemoguardrails.rails.llm.config.Model,
    tracer: typing.Optional[opentelemetry.trace.Tracer] = None,
    metrics_enabled: bool = False,
    content_capture_enabled: bool = False
)

Bases: BaseEngine

Wraps a single Model config and makes HTTP calls to its endpoint.

Each ModelEngine owns its own RetryClient with per-model timeout, retry, and connection pool settings.

Implements the :class:~nemoguardrails.types.LLMModel protocol through generate_async / stream_async, so library rail actions can reach an IORails-configured model through nemoguardrails.llm.call.llm_call the same way they reach any other backend.

_error_context

ErrorContext

Identify this engine’s model to the shared error classifier.

api_key

Optional[str] = self._resolve_api_key(model_config.engine)

base_url

str = self._resolve_base_url()

body_param_defaults

Mapping[str, Any]

default_headers

Mapping[str, str]

default_query

tuple[tuple[str, str], ...]

model_name

str = model_config.model or ''

provider_name

str

The provider this model is served by (‘nim’, ‘openai’, …).

Falls back to "unknown" so telemetry always carries a gen_ai.provider.name label rather than dropping the attribute.

provider_url

str

The OpenAI-compatible API root for this model.

base_url is stored without the /v1 suffix (see _resolve_base_url), so it is re-appended here to match what OpenAIChatModel.provider_url reports for the same endpoint.

nemoguardrails.guardrails.model_engine.ModelEngine._classify_error_response(
    status: int,
    body: str,
    headers: typing.Any,
    req_id: str
) -> nemoguardrails.exceptions.LLMClientError | None

Return the typed provider error for a failed response, or None if it cannot be classified.

nemoguardrails.guardrails.model_engine.ModelEngine._classify_transport_failure(
    exc: Exception
) -> nemoguardrails.exceptions.LLMClientError | None

Return the typed client error for a timeout or connection failure, or None for anything else.

Mirrors the httpx mapping in nemoguardrails.llm.clients.base, so the same transport failure reads the same to a caller whichever engine served the request. No HTTP response arrived, hence status 0. Exception details stay on the cause chain instead of entering the client-facing message.

nemoguardrails.guardrails.model_engine.ModelEngine._duration_metric()

Return the operation-duration metric context, or a no-op when metrics are off.

nemoguardrails.guardrails.model_engine.ModelEngine._ensure_running() -> None

Raise if the engine has not been started.

nemoguardrails.guardrails.model_engine.ModelEngine._generic_client_error(
    status_code: int,
    message: str
) -> nemoguardrails.exceptions.LLMClientError

Build a caller-safe client error for a failure the shared classifier cannot type.

Every ModelEngineError needs one of these, because client_facing_message falls back to str(exception) without it, and this engine’s messages name the model. The model context still travels for diagnostics: it reaches logs through LLMClientError.__str__, never the client envelope, which renders error_message alone.

nemoguardrails.guardrails.model_engine.ModelEngine._get_environment_variable(
    variable_name: str
) -> str | None

Return the value stored in environment variable variable_name.

nemoguardrails.guardrails.model_engine.ModelEngine._merge_params(
    stop: typing.Optional[list[str]],
    kwargs: dict[str, typing.Any]
) -> dict[str, typing.Any]

Merge the model’s configured body defaults under the per-call kwargs.

stop=None means “not specified by this call”, so a model-level stop default survives rather than being overwritten with None.

nemoguardrails.guardrails.model_engine.ModelEngine._prepare_request(
    messages: nemoguardrails.guardrails.guardrails_types.LLMMessages,
    kwargs: typing.Any = {}
) -> nemoguardrails.guardrails.model_engine._RequestParams

Build the client, URL, headers, query params, and body common to every request.

nemoguardrails.guardrails.model_engine.ModelEngine._raise_for_sse_error(
    raw_chunk: dict,
    headers: typing.Any
) -> None

Raise ModelEngineError for a provider error frame carried inside a stream.

Mirrors BaseClient._check_sse_error. Anything raise_for_sse_error raises other than an LLMClientError still terminates the stream through stream_call’s catch-all, so the frame is never silently dropped.

nemoguardrails.guardrails.model_engine.ModelEngine._raise_for_status(
    response: aiohttp.ClientResponse,
    req_id: str,
    t0: float
) -> None

async

Raise ModelEngineError if the HTTP status indicates an error.

nemoguardrails.guardrails.model_engine.ModelEngine._resolve_api_key(
    engine: str | None
) -> typing.Optional[str]

Resolve the API key from model config or environment.

nemoguardrails.guardrails.model_engine.ModelEngine._resolve_base_url() -> str

Resolve the base URL from model parameters or engine type.

Strips an optional trailing “/v1” so users can follow the OpenAI / LLMRails convention of including “/v1” in base_url without producing a doubled “/v1/v1/chat/completions” path when _CHAT_COMPLETIONS_ENDPOINT is appended.

nemoguardrails.guardrails.model_engine.ModelEngine._wrap_client_error(
    client_error: nemoguardrails.exceptions.LLMClientError,
    summary: str
) -> nemoguardrails.guardrails.model_engine.ModelEngineError

Carry a classified provider error inside the ModelEngineError this engine raises.

The message keeps the model name for the server log; it is the inner error, not this text, that the server renders for the caller, so the raw upstream body is left out rather than concatenated in.

nemoguardrails.guardrails.model_engine.ModelEngine._wrap_exception(
    exc: Exception,
    req_id: str,
    t0: float,
    label: str = 'Request'
) -> nemoguardrails.guardrails.model_engine.ModelEngineError

Wrap an unexpected exception in a ModelEngineError.

nemoguardrails.guardrails.model_engine.ModelEngine.call(
    messages: nemoguardrails.guardrails.guardrails_types.LLMMessages,
    kwargs: typing.Any = {}
) -> dict

async

Make a POST request to the /v1/chat/completions endpoint.

Retries on transient failures (429, 5xx, connection errors) are handled automatically by the RetryClient with exponential backoff.

Parameters:

messages

LLMMessages

List of message dicts in OpenAI format.

**kwargs

AnyDefaults to {}

Additional parameters for the request body (temperature, max_tokens, etc.)

Returns: dict

The parsed JSON response dict from the API.

Raises:

ModelEngineError: If the request fails after all retries.

nemoguardrails.guardrails.model_engine.ModelEngine.chat_completion(
    messages: nemoguardrails.guardrails.guardrails_types.LLMMessages,
    kwargs: typing.Any = {}
) -> nemoguardrails.types.LLMResponse

async

Generate a chat completion and return a structured LLMResponse.

Calls the /v1/chat/completions endpoint and parses the OpenAI-format response into an LLMResponse carrying content, reasoning (when the provider exposes reasoning_content), usage, finish reason, and request id.

Raises:

ModelEngineError: If the request fails or the response format is unexpected.

nemoguardrails.guardrails.model_engine.ModelEngine.extract_tool_exchanges(
    messages: nemoguardrails.guardrails.guardrails_types.LLMMessages
) -> list[nemoguardrails.guardrails.tool_schema.ToolExchange]

Group messages into per-turn (tool_calls, tool_results) exchanges.

Each exchange pairs one assistant turn’s tool calls with the tool results that answer it, so RailsManager.are_tool_results_safe can validate call_id linkage turn-locally rather than across the whole flattened history (the latter falsely flags ids reused across turns, which the OpenAI spec permits). Keyed on the model’s engine (_TOOL_EXCHANGE_EXTRACTORS); OpenAI and NIM share the Chat Completions shape and an engine with no registered extractor falls back to it.

nemoguardrails.guardrails.model_engine.ModelEngine.extract_tool_results(
    messages: nemoguardrails.guardrails.guardrails_types.LLMMessages
) -> list[nemoguardrails.guardrails.tool_schema.ToolResult]

Extract incoming tool results from messages into ToolResult objects.

Pulls the provider’s tool-result messages out of the conversation and normalizes them into the internal ToolResult shape the ToolResultRail consumes, keyed on the model’s engine (_RESULT_EXTRACTORS). OpenAI and NIM share the Chat Completions shape (role:"tool" messages); an engine with no registered extractor falls back to it. Returns an empty list when there are no tool results.

nemoguardrails.guardrails.model_engine.ModelEngine.generate_async(
    prompt: str | list,
    stop: typing.Optional[list[str]] = None,
    kwargs: typing.Any = {}
) -> nemoguardrails.types.LLMResponse

async

Generate a completion, implementing the LLMModel protocol.

Adapter only: it normalizes prompt — a plain string or a list of ChatMessage, which is what llm_call passes on behalf of library rail actions — into wire messages and delegates. All the work is in generate_from_messages.

Raises:

ModelEngineError: If the request fails or the response format is unexpected.

nemoguardrails.guardrails.model_engine.ModelEngine.generate_from_messages(
    messages: nemoguardrails.guardrails.guardrails_types.LLMMessages,
    stop: typing.Optional[list[str]] = None,
    kwargs: typing.Any = {}
) -> nemoguardrails.types.LLMResponse

async

Generate a completion from OpenAI-format messages.

The instrumented entry point every caller ends up at: rails through generate_async, main generation directly from EngineRegistry.model_call, which already holds wire messages and so does not need the protocol adapter. The model’s configured parameters supply request-body defaults which kwargs override.

Raises:

ModelEngineError: If the request fails or the response format is unexpected.

nemoguardrails.guardrails.model_engine.ModelEngine.parse_tools(
    llm_params: typing.Optional[dict]
) -> nemoguardrails.guardrails.tool_schema.Toolset

Parse the provider tool block in llm_params into a Toolset.

Reads the opaque tools block forwarded via GenerationOptions.llm_params and normalizes it into the internal Toolset the tool rails validate against, keyed on the model’s engine (_TOOL_PARSERS). OpenAI and NIM share the Chat Completions shape; an engine with no registered parser falls back to it. Returns an empty Toolset when no tools are declared.

nemoguardrails.guardrails.model_engine.ModelEngine.stream_async(
    prompt: str | list,
    stop: typing.Optional[list[str]] = None,
    kwargs: typing.Any = {}
) -> collections.abc.AsyncGenerator[nemoguardrails.types.LLMResponseChunk, None]

async

Stream a completion, implementing the LLMModel protocol.

The streaming counterpart of generate_async: normalize prompt into wire messages, then delegate to stream_from_messages.

Raises:

ModelEngineError: If the request fails after all retries.

nemoguardrails.guardrails.model_engine.ModelEngine.stream_call(
    messages: nemoguardrails.guardrails.guardrails_types.LLMMessages,
    kwargs: typing.Any = {}
) -> collections.abc.AsyncGenerator[nemoguardrails.types.LLMResponseChunk, None]

async

Make a streaming POST request to the /v1/chat/completions endpoint.

Sends stream=True and yields one LLMResponseChunk per SSE event that carries a content delta, reasoning delta, OR a usage payload. Role-only, finish-only, and empty-choices events without usage are skipped. Retries are handled by the RetryClient (same as call()).

Note: when the upstream payload includes stream_options.include_usage=true (default for the OpenAI-compatible client), the provider sends a final usage-only chunk with empty choices after the last content chunk. That terminal chunk is yielded as LLMResponseChunk(usage=...) with both delta_content and delta_reasoning unset — callers that only care about content should gate on chunk.delta_content rather than assuming every yielded chunk carries one.

Tool calls (when the request declared tools) are accumulated from streamed delta.tool_calls fragments and surfaced as a single LLMResponseChunk whose delta_tool_calls carries the COMPLETE finalized list exactly once — on the first chunk with a finish_reason ("tool_calls" for a free choice, "stop" for a forced tool_choice), or via a post-loop safety net if the provider omits a parseable finish frame. No other chunk carries delta_tool_calls, so consumers may treat it as last-write-wins.

Parameters:

messages

LLMMessages

List of message dicts in OpenAI format.

**kwargs

AnyDefaults to {}

Additional parameters for the request body (temperature, max_tokens, etc.)

Raises:

ModelEngineError: If the request fails after all retries.

nemoguardrails.guardrails.model_engine.ModelEngine.stream_chat_completion(
    messages: nemoguardrails.guardrails.guardrails_types.LLMMessages,
    kwargs: typing.Any = {}
) -> collections.abc.AsyncGenerator[nemoguardrails.types.LLMResponseChunk, None]

async

Stream a chat completion and yield LLMResponseChunk objects.

Thin pass-through over stream_call — see that method’s docstring for the contract, including the terminal usage-only chunk emitted when stream_options.include_usage is on.

Raises:

ModelEngineError: If the request fails after all retries.

nemoguardrails.guardrails.model_engine.ModelEngine.stream_from_messages(
    messages: nemoguardrails.guardrails.guardrails_types.LLMMessages,
    stop: typing.Optional[list[str]] = None,
    kwargs: typing.Any = {}
) -> collections.abc.AsyncGenerator[nemoguardrails.types.LLMResponseChunk, None]

async

Stream a completion from OpenAI-format messages.

The streaming counterpart of generate_from_messages. Parameter merging matches it; the chunk contract is stream_call’s, including the terminal usage-only chunk.

Raises:

ModelEngineError: If the request fails after all retries.

class nemoguardrails.guardrails.model_engine.ModelEngineError(
    message: str,
    model_name: str,
    status: int | None = None,
    inner_exception: BaseException | None = None
)

Exception

Bases: Exception

Raised when a model engine call fails.

status_code

int | None

The upstream HTTP status, under the name status extractors look for.

A rail reaching the model through llm_call has this error wrapped into LLMCallException, and nemoguardrails.llm.call._extract_http_status duck-types on status_code — the OpenAI SDK and httpx spelling. Without this alias the upstream status is dropped and the server reports 500 for what was really a 429 or 503, which SDKs retry differently. Callers that hold the concrete type can keep reading status.

class nemoguardrails.guardrails.model_engine._RequestParams()

Bases: NamedTuple

Pre-built parameters for an HTTP request to the completions endpoint.

body

dict[str, Any]

client

RetryClient

headers

dict[str, str]

params

tuple[tuple[str, str], ...]

url

str

nemoguardrails.guardrails.model_engine._accumulate_tool_call_delta(
    tool_calls: dict[int, dict],
    raw_chunk: dict
) -> None

Update the tool-call accumulator with any tool_call deltas from a raw SSE chunk.

OpenAI streams argument JSON as fragments across many chunks; NIM delivers complete arguments in one delta. Both are handled uniformly: tool_calls is keyed by the OpenAI index field and mutated in place on every call. Finalize with _finalize_tool_calls once finish_reason=="tool_calls".

nemoguardrails.guardrails.model_engine._extract_tool_exchanges_nim(
    messages: nemoguardrails.guardrails.guardrails_types.LLMMessages
) -> list[nemoguardrails.guardrails.tool_schema.ToolExchange]

Extract NIM tool exchanges. NIM uses the OpenAI Chat Completions shape.

nemoguardrails.guardrails.model_engine._extract_tool_exchanges_openai(
    messages: nemoguardrails.guardrails.guardrails_types.LLMMessages
) -> list[nemoguardrails.guardrails.tool_schema.ToolExchange]

Group an OpenAI Chat Completions conversation into per-turn ToolExchangees.

nemoguardrails.guardrails.model_engine._extract_tool_results_nim(
    messages: nemoguardrails.guardrails.guardrails_types.LLMMessages
) -> list[nemoguardrails.guardrails.tool_schema.ToolResult]

Extract NIM tool results. NIM uses the OpenAI Chat Completions shape.

nemoguardrails.guardrails.model_engine._extract_tool_results_openai(
    messages: nemoguardrails.guardrails.guardrails_types.LLMMessages
) -> list[nemoguardrails.guardrails.tool_schema.ToolResult]

Extract OpenAI Chat Completions tool results into ToolResult objects.

Chat Completions carries each tool result as a top-level {"role": "tool", "tool_call_id", "content"} message (optionally name).

nemoguardrails.guardrails.model_engine._finalize_tool_calls(
    tool_calls: dict[int, dict]
) -> list[nemoguardrails.types.ToolCall]

Assemble accumulated tool-call fragments into ToolCall objects.

Called once when the stream emits finish_reason=‘tool_calls’. An empty buffer (no argument fragments streamed) is a no-argument call and becomes {}; a non-empty buffer that is not a valid JSON object (e.g. arguments truncated mid-stream) raises ValueError so the malformed call fails closed rather than silently degrading to empty arguments that could pass the tool-call rail. This mirrors the non-streaming parser (ChatMessage.from_dict), which raises on the same bytes; stream_call wraps the error into ModelEngineError exactly as the non-streaming path does.

nemoguardrails.guardrails.model_engine._parse_chat_completion(
    response: dict
) -> nemoguardrails.types.LLMResponse

Convert a /v1/chat/completions response dict into an LLMResponse.

Reasoning is read from message.reasoning_content when the provider exposes it (NIM, DeepSeek-style). Tool calls are parsed from message.tool_calls (OpenAI shape) into LLMResponse.tool_calls via ChatMessage.from_dict, which normalizes JSON-string arguments into a dict. content is None on both a tool-call-only response and a reasoning-only frame, and normalizes to an empty string in each case; a None content with neither is treated as a malformed response.

nemoguardrails.guardrails.model_engine._parse_chat_completion_chunk(
    chunk: dict
) -> typing.Optional[nemoguardrails.types.LLMResponseChunk]

Build an LLMResponseChunk from an SSE chunk dict.

Returns None for chunks without one of: content delta, reasoning delta, a usage payload, or a finish_reason. Role-only first events map to None.

Finish-only frames are preserved: a delta with no content/reasoning (OpenAI sends delta: {}, NIM sends delta: {"content": ""}) and no usage, carrying only a finish_reason. Dropping them would strip gen_ai.response.finish_reasons from the LLM span. (Some providers instead attach finish_reason to the final content chunk — that case is already captured, since content keeps the chunk alive.) When stream_options.include_usage=true the usage payload arrives in a separate later frame with empty choices — so finish_reason and usage do not share a frame.

Last chunk from OpenAI-compatible providers has a usage field when stream_options.include_usage=true. This is passed through to capture the token usage metadata.

nemoguardrails.guardrails.model_engine._parse_tools_nim(
    tools: list
) -> list[nemoguardrails.guardrails.tool_schema.Tool]

Parse NIM tool definitions. NIM uses the OpenAI Chat Completions tool shape.

nemoguardrails.guardrails.model_engine._parse_tools_openai(
    tools: list
) -> list[nemoguardrails.guardrails.tool_schema.Tool]

Parse OpenAI Chat Completions tool definitions into Tool objects.

Each entry has the nested shape {"type": "function", "function": {"name", "description", "parameters", "strict"}}; function.parameters (the JSON Schema) maps to Tool.arguments_schema. Entries that are not a dict, lack a function block, or whose function has no non-empty name are skipped.

nemoguardrails.guardrails.model_engine._parse_usage(
    usage_dict: dict
) -> nemoguardrails.types.UsageInfo

Build UsageInfo from an OpenAI-format usage dict.

Picks up reasoning_tokens from completion_tokens_details (OpenAI reasoning models) and cached_tokens from prompt_tokens_details when present.

nemoguardrails.guardrails.model_engine._to_wire_messages(
    prompt: str | list
) -> nemoguardrails.guardrails.guardrails_types.LLMMessages

Normalize an LLMModel prompt into OpenAI-format message dicts.

This is the LLMModel protocol boundary. Library rail actions render a task prompt with LLMTaskManager.render_task_prompt, which returns either a string or a list of {"type": ..., "content": ...} dicts depending on whether the task prompt was configured with content: or messages:; llm_call then hands over a string or a list of ChatMessage. Both shapes are converted here. A list of dicts that is already in wire form passes through untouched.

ChatMessage conversion drops provider_metadata — it is an internal field the completions endpoint would reject — and re-encodes tool-call arguments as a JSON string, which is the wire shape the API expects. It also maps the task manager’s type key onto role, which the API requires.

nemoguardrails.guardrails.model_engine._tool_calls_from_message(
    message: dict
) -> list[nemoguardrails.types.ToolCall]

Extract the tool calls from one assistant message into ToolCall objects. Malformed tool calls fall back to just id, type, function

nemoguardrails.guardrails.model_engine._tool_result_from_message(
    message: dict
) -> nemoguardrails.guardrails.tool_schema.ToolResult

Normalize one OpenAI Chat Completions role:"tool" message into a ToolResult.

This shape has no error flag, so is_error is always False.

nemoguardrails.guardrails.model_engine._CHAT_COMPLETIONS_ENDPOINT = '/v1/chat/completions'

nemoguardrails.guardrails.model_engine._ENGINE_BASE_URLS = {'nim': 'https://integrate.api.nvidia.com', 'openai': 'https://api.openai.com'}

nemoguardrails.guardrails.model_engine._ERROR_BODY_MAX_CHARS = 8192

nemoguardrails.guardrails.model_engine._OPERATION_NAME = 'chat'

nemoguardrails.guardrails.model_engine._RESERVED_LLM_PARAMETERS = frozenset({'base_url', 'timeout', 'timeout_connect', 'max_attempts', 'api_key', ...

nemoguardrails.guardrails.model_engine._RESULT_EXTRACTORS = {'openai': _extract_tool_results_openai, 'nim': _extract_tool_results_nim}

nemoguardrails.guardrails.model_engine._STANDARD_CHUNK_KEYS = frozenset({'model', 'choices', 'usage', 'id', 'object', 'created'})

nemoguardrails.guardrails.model_engine._TOOL_EXCHANGE_EXTRACTORS = {'openai': _extract_tool_exchanges_openai, 'nim': _extract_tool_exchanges_nim}

nemoguardrails.guardrails.model_engine._TOOL_PARSERS = {'openai': _parse_tools_openai, 'nim': _parse_tools_nim}

nemoguardrails.guardrails.model_engine.log = logging.getLogger(__name__)