> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/guardrails/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/guardrails/_mcp/server.

# nemoguardrails.eval.models

## Module Contents

### Classes

| Name                                                                         | Description                                                           |
| ---------------------------------------------------------------------------- | --------------------------------------------------------------------- |
| [`ComplianceCheckLog`](#nemoguardrails-eval-models-ComplianceCheckLog)       | Detailed log about a compliance check.                                |
| [`ComplianceCheckResult`](#nemoguardrails-eval-models-ComplianceCheckResult) | Information about a compliance check.                                 |
| [`EvalConfig`](#nemoguardrails-eval-models-EvalConfig)                       | An evaluation configuration for an evaluation dataset.                |
| [`EvalOutput`](#nemoguardrails-eval-models-EvalOutput)                       | The output of the evaluation.                                         |
| [`ExpectedOutput`](#nemoguardrails-eval-models-ExpectedOutput)               | An expected output from the system, as dictated by a policy.          |
| [`GenericOutput`](#nemoguardrails-eval-models-GenericOutput)                 | -                                                                     |
| [`InteractionLog`](#nemoguardrails-eval-models-InteractionLog)               | Detailed log about the execution of an interaction.                   |
| [`InteractionOutput`](#nemoguardrails-eval-models-InteractionOutput)         | The output for running and evaluating an interaction.                 |
| [`InteractionSet`](#nemoguardrails-eval-models-InteractionSet)               | An interaction set description that is part of an evaluation dataset. |
| [`Policy`](#nemoguardrails-eval-models-Policy)                               | A policy for the evaluation of a guardrail configuration.             |
| [`RefusalOutput`](#nemoguardrails-eval-models-RefusalOutput)                 | -                                                                     |
| [`SimilarMessageOutput`](#nemoguardrails-eval-models-SimilarMessageOutput)   | -                                                                     |
| [`Span`](#nemoguardrails-eval-models-Span)                                   | A generic span object                                                 |

### API

```python
class nemoguardrails.eval.models.ComplianceCheckLog()
```

**Bases:** `BaseModel`

Detailed log about a compliance check.

**`id`** `str`

---

**`llm_calls`** `List[LLMCallInfo]`

---

```python
class nemoguardrails.eval.models.ComplianceCheckResult()
```

**Bases:** `BaseModel`

Information about a compliance check.

**`compliance`** `Dict[str, Optional[Union[bool, str]]]`

---

**`created_at`** `str`

---

**`details`** `str`

---

**`id`** `str`

---

**`interaction_id`** `Optional[str] = Field(description='The id of the interaction.')`

---

**`method`** `str`

---

```python
class nemoguardrails.eval.models.EvalConfig()
```

**Bases:** `BaseModel`

An evaluation configuration for an evaluation dataset.

**`expected_latencies`** `Dict[str, float]`

---

**`interactions`** `List[InteractionSet]`

---

**`models`** `List[Model]`

---

**`policies`** `List[Policy]`

---

**`prompts`** `Optional[List[TaskPrompt]]`

---

```python
nemoguardrails.eval.models.EvalConfig.from_path(
    config_path: str
) -> nemoguardrails.eval.models.EvalConfig
```

classmethod

Loads an eval configuration from a given path.

It supports YAML files (\*.yml and *.yaml) and .json files (*.json).

```python
nemoguardrails.eval.models.EvalConfig.validate_policy_ids() -> nemoguardrails.eval.models.EvalConfig
```

Validates the policy ids used in the interactions.

```python
class nemoguardrails.eval.models.EvalOutput()
```

**Bases:** `BaseModel`

The output of the evaluation.

**`logs`** `List[InteractionLog]`

---

**`results`** `List[InteractionOutput]`

---

```python
nemoguardrails.eval.models.EvalOutput.compute_compliance(
    eval_config: nemoguardrails.eval.models.EvalConfig
) -> typing.Dict[str, dict]
```

Helper to compute the compliance rate per policy.

```python
nemoguardrails.eval.models.EvalOutput.from_path(
    output_path: str
) -> nemoguardrails.eval.models.EvalOutput
```

classmethod

Loads an eval output from a given path.

It supports YAML files (\*.yml and *.yaml) and .json files (*.json).

```python
class nemoguardrails.eval.models.ExpectedOutput()
```

**Bases:** `BaseModel`

An expected output from the system, as dictated by a policy.

**`policy`** `str`

---

**`type`** `str`

---

```python
class nemoguardrails.eval.models.GenericOutput()
```

**Bases:** [ExpectedOutput](#nemoguardrails-eval-models-ExpectedOutput)

**`description`** `str`

---

**`type`** `str = 'generic'`

---

```python
nemoguardrails.eval.models.GenericOutput.__str__()
```

```python
class nemoguardrails.eval.models.InteractionLog()
```

**Bases:** `BaseModel`

Detailed log about the execution of an interaction.

**`activated_rails`** `List[ActivatedRail]`

---

**`compliance_checks`** `List[ComplianceCheckLog]`

---

**`events`** `List[dict]`

---

**`id`** `str`

---

**`trace`** `List[Span]`

---

```python
class nemoguardrails.eval.models.InteractionOutput()
```

**Bases:** `BaseModel`

The output for running and evaluating an interaction.

**`compliance`** `Dict[str, Optional[Union[bool, str]]]`

---

**`compliance_checks`** `List[ComplianceCheckResult]`

---

**`expected_latencies`** `Dict[str, Union[float]]`

---

**`id`** `str`

---

**`input`** `Union[str, dict] = Field(description='The input of the interaction.')`

---

**`latencies`** `Dict[str, Union[float]]`

---

**`output`** `Optional[Union[str, List[dict]]]`

---

**`resource_usage`** `Dict[str, Union[int, float]]`

---

```python
class nemoguardrails.eval.models.InteractionSet()
```

**Bases:** `BaseModel`

An interaction set description that is part of an evaluation dataset.

An interaction set groups multiple interactions with the same expected output.

**`evaluation_context`** `Dict[str, Any]`

---

**`exclude_policies`** `List[str]`

---

**`expected_output`** `List[ExpectedOutput]`

---

**`id`** `str`

---

**`include_policies`** `List[str]`

---

**`inputs`** `List[Union[str, dict]]`

---

**`tags`** `List[str]`

---

```python
nemoguardrails.eval.models.InteractionSet.instantiate_expected_output(
    values: typing.Any
)
```

classmethod

Creates the right instance of the expected output.

```python
class nemoguardrails.eval.models.Policy()
```

**Bases:** `BaseModel`

A policy for the evaluation of a guardrail configuration.

**`apply_to_all`** `bool`

---

**`description`** `str`

---

**`id`** `str`

---

**`weight`** `int`

---

```python
class nemoguardrails.eval.models.RefusalOutput()
```

**Bases:** [ExpectedOutput](#nemoguardrails-eval-models-ExpectedOutput)

**`type`** `str = 'refusal'`

---

```python
nemoguardrails.eval.models.RefusalOutput.__str__()
```

```python
class nemoguardrails.eval.models.SimilarMessageOutput()
```

**Bases:** [ExpectedOutput](#nemoguardrails-eval-models-ExpectedOutput)

**`message`** `str`

---

**`type`** `str = 'similar_message'`

---

```python
nemoguardrails.eval.models.SimilarMessageOutput.__str__()
```

```python
class nemoguardrails.eval.models.Span()
```

**Bases:** `BaseModel`

A generic span object

**`duration`** `float`

---

**`end_time`** `float = Field(description='The end time of the span.')`

---

**`metrics`** `Dict[str, Union[int, float]]`

---

**`name`** `str`

---

**`parent_id`** `Optional[str]`

---

**`resource_id`** `Optional[str]`

---

**`span_id`** `str = Field(description='The id of the span.')`

---

**`start_time`** `float = Field(description='The start time of the span.')`

---