Skip to navigation

Public testing surface for NeMo Guardrails.

This subpackage ships utilities that help users write fast, deterministic tests for their guardrails configurations. The two main building blocks are:

  • FakeLLMModel: a scriptable implementation of the LLMModel protocol that returns canned responses.
  • RecordingHTTPClient: a scriptable HTTP client that records requests and returns queued responses.
  • TestChat: an ergonomic helper for asserting bot replies against a scripted conversation.

Pytest fixtures are exposed via the nemoguardrails.testing.fixtures plugin. Add it to your conftest.py to opt in:

pytest_plugins = ["nemoguardrails.testing.fixtures"]

Submodules

Package Contents

Classes

NameDescription
FakeLLMModelFramework-agnostic fake LLM for testing. Implements the LLMModel protocol.
RecordingHTTPClientRecord requests and return queued responses or exceptions in order.
TestChatHelper class for easily writing tests.

API

class nemoguardrails.testing.FakeLLMModel(
responses: typing.Optional[typing.List[str]] = None,
llm_responses: typing.Optional[typing.List[nemoguardrails.types.LLMResponse]] = None,
llm_exception: typing.Optional[Exception] = None,
token_usage: typing.Optional[typing.List[typing.Dict[str, int]]] = None,
should_return_token_usage: bool = False
)

Framework-agnostic fake LLM for testing. Implements the LLMModel protocol.

Parameters:

responses
Optional[List[str]]Defaults to None

A list of plain string responses. Each call to generate_async (or stream_async) consumes the next entry. Mutually exclusive with llm_responses.

llm_responses
Optional[List[LLMResponse]]Defaults to None

Optional[List[ LLMResponse ]] — A list of LLMResponse objects. Useful when tool calls or structured fields need to be asserted. Takes precedence over responses when provided.

llm_exception
Optional[Exception]Defaults to None

An exception instance to raise on every generation, useful for exercising error-handling paths.

token_usage
Optional[List[Dict[str, int]]]Defaults to None

Optional list of token usage dictionaries (one per response). Each entry may include prompt_tokens, completion_tokens and total_tokens keys.

should_return_token_usage
boolDefaults to False

When True, populate LLMResponse.usage from token_usage.

inference_count
= 0
model_name
str
provider_name
Optional[str]
provider_url
Optional[str]
responses
nemoguardrails.testing.FakeLLMModel._get_usage() -> typing.Optional[nemoguardrails.types.UsageInfo]
nemoguardrails.testing.FakeLLMModel._next_response() -> nemoguardrails.types.LLMResponse
nemoguardrails.testing.FakeLLMModel.generate_async(
prompt,
stop = None,
kwargs = {}
async
nemoguardrails.testing.FakeLLMModel.stream_async(
prompt,
stop = None,
kwargs = {}
)
async
class nemoguardrails.testing.RecordingHTTPClient(
responses: collections.abc.Iterable[nemoguardrails.http.types.HTTPResponse | BaseException] = ()
)

Record requests and return queued responses or exceptions in order.

_responses
= deque(responses)
close_calls
= 0
requests
list[HTTPRequest] = []
nemoguardrails.testing.RecordingHTTPClient.add_response(
response: nemoguardrails.http.types.HTTPResponse | BaseException
) -> None
nemoguardrails.testing.RecordingHTTPClient.close() -> None
async

Record a close call without discarding captured requests.

nemoguardrails.testing.RecordingHTTPClient.request(
method: str,
url: str,
headers: typing.Mapping[str, str] | None = None,
params: typing.Mapping[str, typing.Any] | None = None,
json: typing.Any = None,
content: bytes | str | None = None,
timeout: float | None = None
async

Record a request and consume the next queued result.

Raises:

  • RuntimeError: If no queued result is available.
  • BaseException: The queued exception, when the next result is an exception instance.
class nemoguardrails.testing.TestChat(
config: nemoguardrails.RailsConfig,
llm_completions: typing.Optional[typing.List[str]] = None,
streaming: bool = False,
llm_exception: typing.Optional[Exception] = None,
token_usage: typing.Optional[typing.List[typing.Dict[str, int]]] = None,
)

Helper class for easily writing tests.

Usage: config = RailsConfig.from_path(…) chat = TestChat( config, llm_completions=[ “Hello! How can I help you today?”, ], )

chat.user(“Hello! How are you?”) chat.bot(“Hello! How can I help you today?“)

app
= LLMRails(config, llm=(self.llm))
history
= []
input_events
= []
nemoguardrails.testing.TestChat.__lshift__(
msg: str
)
nemoguardrails.testing.TestChat.__rshift__(
msg: typing.Union[str, dict]
)
nemoguardrails.testing.TestChat.bot(
expected: typing.Union[str, dict, list]
)
nemoguardrails.testing.TestChat.bot_async(
msg: str
)
async
nemoguardrails.testing.TestChat.user(
msg: typing.Union[str, dict]
)