> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/testing/llms.txt. # nemoguardrails.testing Public testing surface for NeMo Guardrails. This subpackage ships utilities that help users write fast, deterministic tests for their guardrails configurations. The two main building blocks are: * `FakeLLMModel`: a scriptable implementation of the `LLMModel` protocol that returns canned responses. * `RecordingHTTPClient`: a scriptable HTTP client that records requests and returns queued responses. * `TestChat`: an ergonomic helper for asserting bot replies against a scripted conversation. Pytest fixtures are exposed via the `nemoguardrails.testing.fixtures` plugin. Add it to your `conftest.py` to opt in: ```python pytest_plugins = ["nemoguardrails.testing.fixtures"] ``` ## Submodules * **[`nemoguardrails.testing.chat_harness`](/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/testing/chat_harness)** * **[`nemoguardrails.testing.fake_model`](/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/testing/fake_model)** * **[`nemoguardrails.testing.fixtures`](/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/testing/fixtures)** * **[`nemoguardrails.testing.http`](/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/testing/http)** ## Package Contents ### Classes | Name | Description | | ------------------------------------------------------------------------- | ---------------------------------------------------------------------------- | | [`FakeLLMModel`](#nemoguardrails-testing-fake_model-FakeLLMModel) | Framework-agnostic fake LLM for testing. Implements the `LLMModel` protocol. | | [`RecordingHTTPClient`](#nemoguardrails-testing-http-RecordingHTTPClient) | Record requests and return queued responses or exceptions in order. | | [`TestChat`](#nemoguardrails-testing-chat_harness-TestChat) | Helper class for easily writing tests. | ### API ```python class nemoguardrails.testing.FakeLLMModel( responses: typing.Optional[typing.List[str]] = None, llm_responses: typing.Optional[typing.List[nemoguardrails.types.LLMResponse]] = None, llm_exception: typing.Optional[Exception] = None, token_usage: typing.Optional[typing.List[typing.Dict[str, int]]] = None, should_return_token_usage: bool = False ) ``` Framework-agnostic fake LLM for testing. Implements the `LLMModel` protocol. **Parameters:** **`responses`** `Optional[List[str]]` — default: None A list of plain string responses. Each call to `generate_async` (or `stream_async`) consumes the next entry. Mutually exclusive with `llm_responses`. --- **`llm_responses`** `Optional[List[LLMResponse]]` — default: None `Optional[List[` [`LLMResponse`](/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/types#nemoguardrails-types-LLMResponse) `]]` — A list of `LLMResponse` objects. Useful when tool calls or structured fields need to be asserted. Takes precedence over `responses` when provided. --- **`llm_exception`** `Optional[Exception]` — default: None An exception instance to raise on every generation, useful for exercising error-handling paths. --- **`token_usage`** `Optional[List[Dict[str, int]]]` — default: None Optional list of token usage dictionaries (one per response). Each entry may include `prompt_tokens`, `completion_tokens` and `total_tokens` keys. --- **`should_return_token_usage`** `bool` — default: False When `True`, populate `LLMResponse.usage` from `token_usage`. --- **`inference_count`** `= 0` --- **`model_name`** `str` --- **`provider_name`** `Optional[str]` --- **`provider_url`** `Optional[str]` --- **`responses`** --- ```python nemoguardrails.testing.FakeLLMModel._get_usage() -> typing.Optional[nemoguardrails.types.UsageInfo] ``` ```python nemoguardrails.testing.FakeLLMModel._next_response() -> nemoguardrails.types.LLMResponse ``` ```python nemoguardrails.testing.FakeLLMModel.generate_async( prompt, stop = None, kwargs = {} ) -> nemoguardrails.types.LLMResponse ``` async ```python nemoguardrails.testing.FakeLLMModel.stream_async( prompt, stop = None, kwargs = {} ) ``` async ```python class nemoguardrails.testing.RecordingHTTPClient( responses: collections.abc.Iterable[nemoguardrails.http.types.HTTPResponse | BaseException] = () ) ``` Record requests and return queued responses or exceptions in order. **`_responses`** `= deque(responses)` --- **`close_calls`** `= 0` --- **`requests`** `list[HTTPRequest] = []` --- ```python nemoguardrails.testing.RecordingHTTPClient.add_response( response: nemoguardrails.http.types.HTTPResponse | BaseException ) -> None ``` ```python nemoguardrails.testing.RecordingHTTPClient.close() -> None ``` async Record a close call without discarding captured requests. ```python nemoguardrails.testing.RecordingHTTPClient.request( method: str, url: str, headers: typing.Mapping[str, str] | None = None, params: typing.Mapping[str, typing.Any] | None = None, json: typing.Any = None, content: bytes | str | None = None, timeout: float | None = None ) -> nemoguardrails.http.types.HTTPResponse ``` async Record a request and consume the next queued result. **Raises:** * `RuntimeError`: If no queued result is available. * `BaseException`: The queued exception, when the next result is an exception instance. ```python class nemoguardrails.testing.TestChat( config: nemoguardrails.RailsConfig, llm_completions: typing.Optional[typing.List[str]] = None, streaming: bool = False, llm_exception: typing.Optional[Exception] = None, token_usage: typing.Optional[typing.List[typing.Dict[str, int]]] = None, llm: typing.Optional[nemoguardrails.testing.fake_model.FakeLLMModel] = None ) ``` Helper class for easily writing tests. Usage: config = RailsConfig.from_path(...) chat = TestChat( config, llm_completions=\[ "Hello! How can I help you today?", ], ) chat.user("Hello! How are you?") chat.bot("Hello! How can I help you today?") **`app`** `= LLMRails(config, llm=(self.llm))` --- **`history`** `= []` --- **`input_events`** `= []` --- ```python nemoguardrails.testing.TestChat.__lshift__( msg: str ) ``` ```python nemoguardrails.testing.TestChat.__rshift__( msg: typing.Union[str, dict] ) ``` ```python nemoguardrails.testing.TestChat.bot( expected: typing.Union[str, dict, list] ) ``` ```python nemoguardrails.testing.TestChat.bot_async( msg: str ) ``` async ```python nemoguardrails.testing.TestChat.user( msg: typing.Union[str, dict] ) ``` ## Docs - [nemoguardrails.testing.chat_harness](https://docs.nvidia.com/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/testing/chat_harness.md) - [nemoguardrails.testing.fake_model](https://docs.nvidia.com/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/testing/fake_model.md) - [nemoguardrails.testing.fixtures](https://docs.nvidia.com/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/testing/fixtures.md) - [nemoguardrails.testing.http](https://docs.nvidia.com/nemo/guardrails/v0.25.0/guardrails-python-sdk/nemoguardrails/testing/http.md)