> This page is for version 0.2.1.
> For other versions, use one of these documentation indexes:
> - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt
> - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt
> - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt
> - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt
> - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt
> - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt
> - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt
> - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.

# nemo_gym.eval.recovery.trials

The trial gate's observations: whether each expected trial was observed, and its counted failures.

## Module Contents

### Functions

| Name                                                                                                  | Description                                                                                                       |
| ----------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
| [`_answer_cut_off`](#nemo_gym-eval-recovery-trials-_answer_cut_off)                                   | Whether the saved answer itself says it was cut off: its `incomplete_details.reason` is `max_output_tokens`       |
| [`_answer_fault`](#nemo_gym-eval-recovery-trials-_answer_fault)                                       | The model fault of a scored-zero incomplete answer.                                                               |
| [`_bound_artifact`](#nemo_gym-eval-recovery-trials-_bound_artifact)                                   | Whether a saved grading artifact or the model's own failed answer resolves the row, as the gate requires: an      |
| [`_bound_grading`](#nemo_gym-eval-recovery-trials-_bound_grading)                                     | Whether the row is `_graded` and every grading operation binds its saved answer and score, as the gate            |
| [`_clip_text`](#nemo_gym-eval-recovery-trials-_clip_text)                                             | -                                                                                                                 |
| [`_count`](#nemo_gym-eval-recovery-trials-_count)                                                     | -                                                                                                                 |
| [`_cut_judge_reply`](#nemo_gym-eval-recovery-trials-_cut_judge_reply)                                 | Whether a captured judge reply was cut short: an incomplete Responses reply, or a chat completion that stopped    |
| [`_cut_off`](#nemo_gym-eval-recovery-trials-_cut_off)                                                 | Whether the saved answer carries an actual cutoff signal, not just an incomplete status.                          |
| [`_ended_on_failed_request`](#nemo_gym-eval-recovery-trials-_ended_on_failed_request)                 | The failure kind of the request a row's rollout ended on, with that call, or None (`final_zero_trials`).          |
| [`_failed_request`](#nemo_gym-eval-recovery-trials-_failed_request)                                   | The failure kind and HTTP status of a captured call whose final model operation failed with its own recorded      |
| [`_fault_text`](#nemo_gym-eval-recovery-trials-_fault_text)                                           | `evaluation-side faults: N of M (x%) is [not] below L (server S: kind n; kind n); model faults: ...`.             |
| [`_graded`](#nemo_gym-eval-recovery-trials-_graded)                                                   | Whether the row's one verifier recorded grading operations for it that all completed with a valid verdict.        |
| [`_grading_sandbox_failed`](#nemo_gym-eval-recovery-trials-_grading_sandbox_failed)                   | Whether a verifier's sandbox observations report that a grading sandbox failed, or are malformed.                 |
| [`_health_classification`](#nemo_gym-eval-recovery-trials-_health_classification)                     | Express the existing timeout switch as a shared classification override.                                          |
| [`_health_view`](#nemo_gym-eval-recovery-trials-_health_view)                                         | `decision` as the trial gate counts it: a scored row planned for a re-judge because its judge gave no verdict     |
| [`_last_captured_call`](#nemo_gym-eval-recovery-trials-_last_captured_call)                           | The call a rollout ended on, for an agent whose invocations reference none of its calls.                          |
| [`_length_refusal_kind`](#nemo_gym-eval-recovery-trials-_length_refusal_kind)                         | The kind of a captured call rejected with HTTP 4xx on length (`LENGTH_REFUSAL_KINDS`), or None.                   |
| [`_observed_incomplete_answer`](#nemo_gym-eval-recovery-trials-_observed_incomplete_answer)           | Whether a counted incomplete answer (`_answer_fault`) is the model's own observed answer.                         |
| [`_observed_trial_operations`](#nemo_gym-eval-recovery-trials-_observed_trial_operations)             | Whether every expected trial that is not a counted failure is observed: its saved answer is bound to its grading. |
| [`_percent`](#nemo_gym-eval-recovery-trials-_percent)                                                 | -                                                                                                                 |
| [`_policy_call`](#nemo_gym-eval-recovery-trials-_policy_call)                                         | Whether a captured call is the evaluated policy's: its model is the row's policy server, or none is named.        |
| [`_policy_zero_kind`](#nemo_gym-eval-recovery-trials-_policy_zero_kind)                               | A consistent applied zero keeps its own kind, without reinterpreting diagnostic requests.                         |
| [`_predates_operation_observations`](#nemo_gym-eval-recovery-trials-_predates_operation_observations) | Whether every scored row was written by a Gym pin from before operation observations were recorded.               |
| [`_recorded_by_verifier`](#nemo_gym-eval-recovery-trials-_recorded_by_verifier)                       | Whether the row's one verifier invocation saved its own grading record for it (`_verifier_recorded`).             |
| [`_recorded_grade`](#nemo_gym-eval-recovery-trials-_recorded_grade)                                   | Whether the row's reward came from a recorded grade: completed grading operations, the verifier's own grading     |
| [`_recovered_note`](#nemo_gym-eval-recovery-trials-_recovered_note)                                   | `N failed model attempt(s) were retried and completed in the same operation (...)`, or empty.                     |
| [`_row_fault_class`](#nemo_gym-eval-recovery-trials-_row_fault_class)                                 | -                                                                                                                 |
| [`_row_recovered_attempts`](#nemo_gym-eval-recovery-trials-_row_recovered_attempts)                   | The row's recovered model attempts, by the completed operation that recovered them (`recovered_attempt_joins`).   |
| [`_scored_fault`](#nemo_gym-eval-recovery-trials-_scored_fault)                                       | A scored row's one counted fault: its own recorded kind (`_scored_kind`), else its answer's                       |
| [`_scored_kind`](#nemo_gym-eval-recovery-trials-_scored_kind)                                         | The registered failure kind a scored result row records as its own outcome, keyed on its recorded HTTP status     |
| [`_teardown_failure_only`](#nemo_gym-eval-recovery-trials-_teardown_failure_only)                     | Whether a sandbox observation's only failure is its teardown: the producer records a failed close or release      |
| [`_trajectory_kinds`](#nemo_gym-eval-recovery-trials-_trajectory_kinds)                               | The kinds of trajectory evidence a trial lacks: its recorded gaps, and its agent invocations' own shape.          |
| [`_trajectory_text`](#nemo_gym-eval-recovery-trials-_trajectory_text)                                 | `trajectory evidence incomplete: N of M (kinds: kind n, ...)`, reported and never a condition.                    |
| [`_unaccounted_failures`](#nemo_gym-eval-recovery-trials-_unaccounted_failures)                       | Whether the row recorded a model failure that no recovery accounts for.                                           |
| [`_verification_owner`](#nemo_gym-eval-recovery-trials-_verification_owner)                           | Resolve verification from saved routing bindings, including taskset results without `agent_ref`.                  |
| [`_verifier_recorded`](#nemo_gym-eval-recovery-trials-_verifier_recorded)                             | Whether the row's verifier saved its own grading record for it, in place of a recorded grading operation.         |
| [`_verifying_servers`](#nemo_gym-eval-recovery-trials-_verifying_servers)                             | The resources servers the run binds, from its provenance, for an unobserved-trial reason.                         |
| [`decision_classification`](#nemo_gym-eval-recovery-trials-decision_classification)                   | The resolved recovery policy snapshot supplies the same mapping to all health readers.                            |
| [`fault_report`](#nemo_gym-eval-recovery-trials-fault_report)                                         | Report shared classes separately; server is only a subcategory of evaluation.                                     |
| [`final_zero_trials`](#nemo_gym-eval-recovery-trials-final_zero_trials)                               | Scored zeros with a saved policy decision or a failed final policy request, with their failure kinds.             |
| [`infrastructure_timeouts`](#nemo_gym-eval-recovery-trials-infrastructure_timeouts)                   | Scored identities whose graded result is an agent timeout with a server-side fault in its window.                 |
| [`recovered_attempt_joins`](#nemo_gym-eval-recovery-trials-recovered_attempt_joins)                   | Failed model attempts each joined to the completed operation that recovered them, in a scored trial's own call.   |
| [`status_fault_kind`](#nemo_gym-eval-recovery-trials-status_fault_kind)                               | The shared reporting label for observed HTTP evidence; saved events remain unchanged.                             |
| [`trajectory_evidence`](#nemo_gym-eval-recovery-trials-trajectory_evidence)                           | Trials whose trajectory evidence is incomplete, over the trials, with the count of each missing kind.             |
| [`trial_boundary`](#nemo_gym-eval-recovery-trials-trial_boundary)                                     | Why trial failures outside the sidecar cannot be ruled out, from the run's own persisted facts.                   |
| [`trial_counts`](#nemo_gym-eval-recovery-trials-trial_counts)                                         | Acceptance's trial counts: observed healthy, unhealthy and unobserved, over the expected trials.                  |
| [`trial_faults`](#nemo_gym-eval-recovery-trials-trial_faults)                                         | Count final trial failures once; classify selected outcomes, never recovered observations.                        |
| [`trial_population`](#nemo_gym-eval-recovery-trials-trial_population)                                 | The trial denominator: every expected trial, including repeats, failed and missing ones, less the rows an         |

### Data

[`CUTOFF_REASONS`](#nemo_gym-eval-recovery-trials-CUTOFF_REASONS)

[`LENGTH_REFUSAL_KINDS`](#nemo_gym-eval-recovery-trials-LENGTH_REFUSAL_KINDS)

[`UNVERIFIED_NOTE`](#nemo_gym-eval-recovery-trials-UNVERIFIED_NOTE)

[`_COMPLETED_GRADE_OUTCOME`](#nemo_gym-eval-recovery-trials-_COMPLETED_GRADE_OUTCOME)

[`_FAILED_OPERATION_OUTCOME`](#nemo_gym-eval-recovery-trials-_FAILED_OPERATION_OUTCOME)

[`_HTTP_ERROR_STATUS_MAX`](#nemo_gym-eval-recovery-trials-_HTTP_ERROR_STATUS_MAX)

[`_HTTP_ERROR_STATUS_MIN`](#nemo_gym-eval-recovery-trials-_HTTP_ERROR_STATUS_MIN)

[`_MODEL_REQUEST_OPERATION`](#nemo_gym-eval-recovery-trials-_MODEL_REQUEST_OPERATION)

[`_SANDBOX_FAILURES`](#nemo_gym-eval-recovery-trials-_SANDBOX_FAILURES)

### API

```python
nemo_gym.eval.recovery.trials._answer_cut_off(
    response: collections.abc.Mapping[str, typing.Any]
) -> bool
```

Whether the saved answer itself says it was cut off: its `incomplete_details.reason` is `max_output_tokens`
or `max_tokens`, or its output tokens reached its `max_output_tokens`.

```python
nemo_gym.eval.recovery.trials._answer_fault(
    row: collections.abc.Mapping[str, typing.Any],
    collected: nemo_gym.eval.recovery.evidence.CollectedEvidence | None
) -> str | None
```

The model fault of a scored-zero incomplete answer.

`TRUNCATED_ANSWER` when the answer carries a cutoff signal (`_cut_off`), whether the benchmark's rule or a
recorded grade scored it zero: the cutoff is why it scored zero. Without a cutoff signal, a row whose reward came
from a recorded grade (`_recorded_grade`) has none, as its score is its grade; otherwise `EMPTY_ANSWER` when it
holds no assistant message with text (`_empty_answer`) and `MALFORMED_ANSWER` when it does.

```python
nemo_gym.eval.recovery.trials._bound_artifact(
    row: collections.abc.Mapping[str, typing.Any],
    collected: nemo_gym.eval.recovery.evidence.CollectedEvidence
) -> bool
```

Whether a saved grading artifact or the model's own failed answer resolves the row, as the gate requires: an
artifact other than the model's failed answer must bind the saved answer and score to its verification owner.

```python
nemo_gym.eval.recovery.trials._bound_grading(
    row: collections.abc.Mapping[str, typing.Any],
    collected: nemo_gym.eval.recovery.evidence.CollectedEvidence
) -> bool
```

Whether the row is `_graded` and every grading operation binds its saved answer and score, as the gate
requires (`_ng_collected_grading_binding`).

```python
nemo_gym.eval.recovery.trials._clip_text(
    quality: collections.abc.Mapping[str, typing.Any]
) -> str
```

```python
nemo_gym.eval.recovery.trials._count(
    value: typing.Any
) -> int | None
```

```python
nemo_gym.eval.recovery.trials._cut_judge_reply(
    call: collections.abc.Mapping[str, typing.Any]
) -> bool
```

Whether a captured judge reply was cut short: an incomplete Responses reply, or a chat completion that stopped
on its output cap or a content filter.

```python
nemo_gym.eval.recovery.trials._cut_off(
    row: collections.abc.Mapping[str, typing.Any]
) -> bool
```

Whether the saved answer carries an actual cutoff signal, not just an incomplete status.

The answer's `incomplete_details.reason` is `max_output_tokens` or `max_tokens`; or the captured call that
returned it finished for `length` or one of those reasons; or its output tokens reached its `max_output_tokens`.

```python
nemo_gym.eval.recovery.trials._ended_on_failed_request(
    row: collections.abc.Mapping[str, typing.Any],
    classification: nemo_gym.core.failure_kinds.FailureClassificationConfig | None = None
) -> tuple[typing.Any, collections.abc.Mapping[str, typing.Any]] | None
```

The failure kind of the request a row's rollout ended on, with that call, or None (`final_zero_trials`).

The last policy call of the row's one invocation ended in a failed model operation whose failure event the call
recorded, or, for an agent that recorded no call of its own, its last captured call failed that way; a policy whose
own last call was rejected on length ended there, whatever its event's kind.

```python
nemo_gym.eval.recovery.trials._failed_request(
    call: typing.Any
) -> tuple[str, typing.Any] | None
```

The failure kind and HTTP status of a captured call whose final model operation failed with its own recorded
request event, or None.

```python
nemo_gym.eval.recovery.trials._fault_text(
    report: dict[str, typing.Any],
    limit: float
) -> str
```

`evaluation-side faults: N of M (x%) is [not] below L (server S: kind n; kind n); model faults: ...`.

```python
nemo_gym.eval.recovery.trials._graded(
    row: collections.abc.Mapping[str, typing.Any]
) -> bool
```

Whether the row's one verifier recorded grading operations for it that all completed with a valid verdict.

An operation without a valid verdict does not resolve the trial; the gate's join decides whether it is a fault.

```python
nemo_gym.eval.recovery.trials._grading_sandbox_failed(
    observations: typing.Any
) -> bool
```

Whether a verifier's sandbox observations report that a grading sandbox failed, or are malformed.

They are lifecycle evidence of the sandboxes the verification used, not a grading record: a completed or
unreported outcome, an unconfirmed or failed teardown, a failing test command's exit code and the agent's own
sandbox outcome do not leave a bound grading unobserved. A verifier sandbox whose outcome is a failure, or that
records an error other than its teardown's, does.

```python
nemo_gym.eval.recovery.trials._health_classification(
    classification: nemo_gym.core.failure_kinds.FailureClassificationConfig | None,
    policy_timeout_counts_as_model: bool
) -> nemo_gym.core.failure_kinds.FailureClassificationConfig
```

Express the existing timeout switch as a shared classification override.

```python
nemo_gym.eval.recovery.trials._health_view(
    decision: nemo_gym.eval.recovery.decision.RecoveryDecision
) -> nemo_gym.eval.recovery.decision.RecoveryDecision
```

`decision` as the trial gate counts it: a scored row planned for a re-judge because its judge gave no verdict
(`SCORED_UNPARSEABLE_REASON`) is a scored trial whose fault, `judge_unparseable`, is evaluation-side. The
recovery action is unchanged; only health reads the row as it stands until the re-judge settles it.

```python
nemo_gym.eval.recovery.trials._last_captured_call(
    calls: list
) -> typing.Any
```

The call a rollout ended on, for an agent whose invocations reference none of its calls.

The highest capture position (`call_index`, the durable append order). When every call recorded its start
time, that call must also be the latest started; otherwise the end is ambiguous and there is none.

```python
nemo_gym.eval.recovery.trials._length_refusal_kind(
    meta: typing.Any
) -> str | None
```

The kind of a captured call rejected with HTTP 4xx on length (`LENGTH_REFUSAL_KINDS`), or None.

The reason is the call's `error_category` or its `local_response_reason`; the status is the call's own, its
upstream's, or that of a failure event it reported. A 5xx among them is the server's fault, and an event of another
kind than a rejected request (a transport failure) is its own cause, never the length's.

```python
nemo_gym.eval.recovery.trials._observed_incomplete_answer(
    row: collections.abc.Mapping[str, typing.Any]
) -> bool
```

Whether a counted incomplete answer (`_answer_fault`) is the model's own observed answer.

The trial gate counts such a row as a model failure and reports it as a response failure, so its grading is not
joined. Its answer must still be observed: the row recorded no failure, its one agent
invocation ended completed or incomplete (an agent marks a cut-off answer incomplete) without an error, its last
model call is exactly one captured call that answered in one completed operation, and that call's response is
the saved answer, or, for an agent that saves its own answer envelope (its own response id), the envelope says it
was cut off (`_answer_cut_off`) and that call itself finished on a cutoff (`CUTOFF_REASONS`): two independent
signals that the model's own last reply was cut. A verifier that ran must have completed: a verifier error is never
read as the model's fault.

```python
nemo_gym.eval.recovery.trials._observed_trial_operations(
    decision: nemo_gym.eval.recovery.decision.RecoveryDecision,
    collected: nemo_gym.eval.recovery.evidence.CollectedEvidence | None,
    classification: nemo_gym.core.failure_kinds.FailureClassificationConfig | None = None,
    judge_recovery: dict[str, int] | None = None,
    grading_failures: list[str] | None = None,
    policy_recovery: dict[str, int] | None = None,
    verifier_recorded: list[nemo_gym.eval.recovery.evidence.Identity] | None = None,
    artifact_recorded: list[nemo_gym.eval.recovery.evidence.Identity] | None = None,
    trajectory_incomplete: dict[nemo_gym.eval.recovery.evidence.Identity, list[str]] | None = None,
    agent_failed_graded: list[nemo_gym.eval.recovery.evidence.Identity] | None = None
) -> bool
```

Whether every expected trial that is not a counted failure is observed: its saved answer is bound to its grading.

Observed means a recorded grading bound to the saved answer and score: one completed verifier's grading operation
bound to the row, its own grading record, a saved grading artifact (`GRADING_ARTIFACTS`) with the collector's
binding, or the model's own failed answer. Only that decides coverage. Each trial's model/tool trajectory is
joined too (its one agent invocation, turns, calls, tools and failure events), but that is observability
evidence: a trial whose trajectory does not join is reported in `trajectory_incomplete` with the kinds of its
missing evidence (`_trajectory_kinds`), and it is still observed.

Join observed model/tool operations and the outer verifier invocation for every expected trial.

A trial whose verifier recorded no grading operation and no grading record of its own is observed through a
saved grading artifact (`GRADING_ARTIFACTS`) or through the model's own failed answer the benchmark's rule
scored (`_model_failure`); those trials go to `artifact_recorded`, and the verifier's captured calls are its
own. An environment reward record stands in for a missing verifier invocation.

A source-reviewed programmatic grading operation must bind the saved response and score. A verifier that
recorded no grading operation and owns no model call is observed through its own grading record instead
(`_verifier_recorded`); those trials go to `verifier_recorded`. Empty captured model calls or a reward
alone never establish grading coverage; old or uninstrumented verifiers remain unknown.
Recovered grading attempts are reported separately and do not count toward the trial-health threshold;
every failed attempt must join the same completed, result-bound grading dependency. A policy request retried
inside one completed operation is recovered the same way (`policy_recovery` receives its counts): each failed
attempt is an event of that operation and of the row, and the answer is the final attempt's.

```python
nemo_gym.eval.recovery.trials._percent(
    limit: float
) -> str
```

```python
nemo_gym.eval.recovery.trials._policy_call(
    row: collections.abc.Mapping[str, typing.Any],
    call: typing.Any
) -> bool
```

Whether a captured call is the evaluated policy's: its model is the row's policy server, or none is named.

```python
nemo_gym.eval.recovery.trials._policy_zero_kind(
    row: collections.abc.Mapping[str, typing.Any]
) -> str | None
```

A consistent applied zero keeps its own kind, without reinterpreting diagnostic requests.

```python
nemo_gym.eval.recovery.trials._predates_operation_observations(
    decision: nemo_gym.eval.recovery.decision.RecoveryDecision,
    collected: nemo_gym.eval.recovery.evidence.CollectedEvidence | None
) -> bool
```

Whether every scored row was written by a Gym pin from before operation observations were recorded.

Current pins write `response_metadata.model_operations` on every captured model call (a list, or null), and
the collector reports whether the key was there (`model_operations_recorded`). Only an explicit False on every
call of every scored row counts. A row without captured calls, or read by a collector that does not report the
key, proves nothing, so the strict rule applies to it.

```python
nemo_gym.eval.recovery.trials._recorded_by_verifier(
    row: collections.abc.Mapping[str, typing.Any],
    collected: nemo_gym.eval.recovery.evidence.CollectedEvidence
) -> bool
```

Whether the row's one verifier invocation saved its own grading record for it (`_verifier_recorded`).

```python
nemo_gym.eval.recovery.trials._recorded_grade(
    row: collections.abc.Mapping[str, typing.Any],
    collected: nemo_gym.eval.recovery.evidence.CollectedEvidence | None
) -> bool
```

Whether the row's reward came from a recorded grade: completed grading operations, the verifier's own grading
record, or a saved grading artifact (`GRADING_ARTIFACTS`). The model's own failed answer is not one.

```python
nemo_gym.eval.recovery.trials._recovered_note(
    joined: collections.abc.Mapping[str, int]
) -> str
```

`N failed model attempt(s) were retried and completed in the same operation (...)`, or empty.

```python
nemo_gym.eval.recovery.trials._row_fault_class(
    row: collections.abc.Mapping[str, typing.Any],
    fault: str,
    classification: nemo_gym.core.failure_kinds.FailureClassificationConfig | None
) -> nemo_gym.core.failure_kinds.FailureClass
```

```python
nemo_gym.eval.recovery.trials._row_recovered_attempts(
    row: collections.abc.Mapping[str, typing.Any]
) -> list[tuple[str, dict[str, dict]]]
```

The row's recovered model attempts, by the completed operation that recovered them (`recovered_attempt_joins`).

Each entry is an operation id and its failure events, keyed by event id, from one answered call of the row.

```python
nemo_gym.eval.recovery.trials._scored_fault(
    row: collections.abc.Mapping[str, typing.Any],
    collected: nemo_gym.eval.recovery.evidence.CollectedEvidence | None,
    classification: nemo_gym.core.failure_kinds.FailureClassificationConfig | None = None
) -> str | None
```

A scored row's one counted fault: its own recorded kind (`_scored_kind`), else its answer's
(`_answer_fault`), else `judge_unparseable` when its saved judge evaluation recorded no verdict
(`_unparsed_judge_verdict`).

```python
nemo_gym.eval.recovery.trials._scored_kind(
    row: collections.abc.Mapping[str, typing.Any],
    classification: nemo_gym.core.failure_kinds.FailureClassificationConfig | None = None
) -> str | None
```

The registered failure kind a scored result row records as its own outcome, keyed on its recorded HTTP status
(`status_fault_kind`), or None. For example Harbor's graded `agent_timeout` is a scored main result.

A persisted outcome decision governs metric admission, independently of fault attribution and the existing
evaluation-health threshold. Counting a provider failure as zero does not make it a model fault or healthy.

```python
nemo_gym.eval.recovery.trials._teardown_failure_only(
    item: collections.abc.Mapping
) -> bool
```

Whether a sandbox observation's only failure is its teardown: the producer records a failed close or release
after the work as outcome `sandbox_error` or `unknown` with that cleanup error as its `error_type`.

```python
nemo_gym.eval.recovery.trials._trajectory_kinds(
    observed: typing.Any
) -> list[str]
```

The kinds of trajectory evidence a trial lacks: its recorded gaps, and its agent invocations' own shape.

Recorded gap codes other than `resolution_unavailable` (for example `sandbox_lifecycle_timing_unavailable`,
`conversation_unavailable`, `model_call_ownership_unavailable`), then `invocation_unavailable` or
`multiple_invocations`, `in_session_unrecorded` and `invocation_<status>` for an invocation that did not
record a completed status. Empty when none of these is named.

```python
nemo_gym.eval.recovery.trials._trajectory_text(
    evidence: collections.abc.Mapping[str, typing.Any]
) -> str
```

`trajectory evidence incomplete: N of M (kinds: kind n, ...)`, reported and never a condition.

```python
nemo_gym.eval.recovery.trials._unaccounted_failures(
    row: collections.abc.Mapping[str, typing.Any],
    recovered: collections.abc.Mapping[str, typing.Any]
) -> bool
```

Whether the row recorded a model failure that no recovery accounts for.

Every model failure event the row saved or its captured calls reported must be a recovered attempt (`recovered`: the
events the coverage join joined, with the row's own `_row_recovered_attempts`), a saved copy must agree with its
call's, and every failed model operation must be one whose failure such a recovery accounts for. A trial that
ended on its failed request is counted separately (`final_zero_trials`) and is never asked this.

```python
nemo_gym.eval.recovery.trials._verification_owner(
    row: collections.abc.Mapping[str, typing.Any],
    servers: collections.abc.Mapping[str, typing.Any]
) -> str | None
```

Resolve verification from saved routing bindings, including taskset results without `agent_ref`.

The environment owns a session's verification. An agent reference on a compatibility result must agree with
a declared primary agent; without one, its explicit reference retains its existing attribution.
An agent that grades its own rollouts names itself when neither configuration binds a resources server.

```python
nemo_gym.eval.recovery.trials._verifier_recorded(
    verifier: typing.Any,
    row: collections.abc.Mapping[str, typing.Any],
    resources_server: typing.Any
) -> bool
```

Whether the row's verifier saved its own grading record for it, in place of a recorded grading operation.

The shared verify path records it only for a verify execution that completed without asking any model. It must
name the row's resources server (or the agent, for an agent that verifies its own rollouts), carry the saved score with a `scored` verdict and bind the saved response and
score. A verifier that recorded a grading operation or owns a model call is judged by those, never by the record.

```python
nemo_gym.eval.recovery.trials._verifying_servers(
    collected: nemo_gym.eval.recovery.evidence.CollectedEvidence
) -> str
```

The resources servers the run binds, from its provenance, for an unobserved-trial reason.

```python
nemo_gym.eval.recovery.trials.decision_classification(
    decision: nemo_gym.eval.recovery.decision.RecoveryDecision,
    explicit: nemo_gym.core.failure_kinds.FailureClassificationConfig | None = None
) -> nemo_gym.core.failure_kinds.FailureClassificationConfig
```

The resolved recovery policy snapshot supplies the same mapping to all health readers.

```python
nemo_gym.eval.recovery.trials.fault_report(
    kinds: collections.Counter,
    trials: int,
    policy_timeout_counts_as_model: bool = True,
    attributions: collections.Counter | None = None,
    classification: nemo_gym.core.failure_kinds.FailureClassificationConfig | None = None
) -> dict[str, typing.Any]
```

Report shared classes separately; server is only a subcategory of evaluation.

```python
nemo_gym.eval.recovery.trials.final_zero_trials(
    decision: nemo_gym.eval.recovery.decision.RecoveryDecision,
    collected: nemo_gym.eval.recovery.evidence.CollectedEvidence | None,
    classification: nemo_gym.core.failure_kinds.FailureClassificationConfig | None = None
) -> dict[nemo_gym.eval.recovery.evidence.Identity, str]
```

Scored zeros with a saved policy decision or a failed final policy request, with their failure kinds.

A consistent saved counted-zero decision retains its own registered failure kind. Otherwise, the last
policy call of the row's one invocation ends in a failed model operation whose failure event the call
recorded, and the row's reward is zero. Such a trial is settled as a failure under the benchmark protocol: it
has no selectable response, and it is counted once, in the trial family. An agent that records no call of its own
(it has no invocation, or its invocations reference none) ended on a failed request when its last captured call (`_last_captured_call`)
failed that way; an earlier failed call is no end and stays unaccounted. The kind is keyed on the failure's HTTP
status (`status_fault_kind`): a 5xx reply is the server's fault, and another model's 4xx rejection is
evaluation-side. A policy whose own last call was rejected on length (`_length_refusal_kind`) ended there,
whatever its event's kind: `model_context_limit` for the context window, `invalid_output_budget` for an output
budget the window cannot hold. The policy's last call is its last captured one when its invocations reference
none, since another model's capture may follow.

```python
nemo_gym.eval.recovery.trials.infrastructure_timeouts(
    decision: nemo_gym.eval.recovery.decision.RecoveryDecision,
    collected: nemo_gym.eval.recovery.evidence.CollectedEvidence | None,
    classification: nemo_gym.core.failure_kinds.FailureClassificationConfig | None = None
) -> set[nemo_gym.eval.recovery.evidence.Identity]
```

Scored identities whose graded result is an agent timeout with a server-side fault in its window.

A Harbor graded zero a run without a Store cannot reopen (`is_infrastructure_agent_timeout`): it stays in the
results, and the trial gate counts it once, as `AGENT_TIMEOUT_INFRASTRUCTURE`, an evaluation-side fault.

```python
nemo_gym.eval.recovery.trials.recovered_attempt_joins(
    decision: nemo_gym.eval.recovery.decision.RecoveryDecision,
    collected: nemo_gym.eval.recovery.evidence.CollectedEvidence | None
) -> dict[str, int]
```

Failed model attempts each joined to the completed operation that recovered them, in a scored trial's own call.

Counted like the run's observed upstream recovery (`observed_upstream_recovery`): a model-request failure event
whose operation completed in an answered call. Here the event must be in the failure report of the same call
whose one completed operation it names, as a failed attempt other than the answer, and the operation's attempts
must account for every failed attempt of it (`http_attempts - 1 + transport_failures`, the HTTP-less ones being
the transport failures). A copy the row saved must agree, and no event, attempt or operation joins twice. The
join holds for any model the trial called (its policy, a judge or a simulated user). It says nothing about how
the trial was graded: whether the final answer is bound to its grading is the coverage join's question
(`_observed_trial_operations`).

```python
nemo_gym.eval.recovery.trials.status_fault_kind(
    kind: typing.Any,
    http_status: typing.Any,
    policy: bool = True,
    classification: nemo_gym.core.failure_kinds.FailureClassificationConfig | None = None
) -> typing.Any
```

The shared reporting label for observed HTTP evidence; saved events remain unchanged.

```python
nemo_gym.eval.recovery.trials.trajectory_evidence(
    incomplete: collections.abc.Mapping[nemo_gym.eval.recovery.evidence.Identity, list[str]],
    trials: int
) -> dict[str, typing.Any]
```

Trials whose trajectory evidence is incomplete, over the trials, with the count of each missing kind.

```python
nemo_gym.eval.recovery.trials.trial_boundary(
    quality: collections.abc.Mapping[str, typing.Any] | None
) -> str
```

Why trial failures outside the sidecar cannot be ruled out, from the run's own persisted facts.

Every persisted value is type-checked: a malformed report is named as malformed, never read as a fact.

```python
nemo_gym.eval.recovery.trials.trial_counts(
    decision: nemo_gym.eval.recovery.decision.RecoveryDecision,
    collected: nemo_gym.eval.recovery.evidence.CollectedEvidence | None,
    unobserved_identities: set[nemo_gym.eval.recovery.evidence.Identity] | None = None
) -> dict[str, typing.Any] | None
```

Acceptance's trial counts: observed healthy, unhealthy and unobserved, over the expected trials.

A saved grading artifact or the model's own failed answer (`grading_artifact`, `_model_failure`) resolves a
trial too. `unhealthy` are the failures the trial gate counts (`trial_faults`, model and evaluation-side). When the
gate's observation join covers every other trial, its observed gradings without a valid verdict are unhealthy
too and nothing is unobserved. Otherwise `unobserved` are the other scored trials whose verifier recorded
neither completed grading operations bound to the saved row nor its own grading record: nothing resolved their
grading, so they are unverified, never healthy. `observed_healthy` are the rest; `verifier_recorded` of them
are observed only through the verifier's grading record. None until the run is complete. A caller that passes
`unobserved_identities` gets the identities counted as unobserved added to it.

```python
nemo_gym.eval.recovery.trials.trial_faults(
    decision: nemo_gym.eval.recovery.decision.RecoveryDecision,
    collected: nemo_gym.eval.recovery.evidence.CollectedEvidence | None,
    attributions: collections.Counter | None = None,
    classification: nemo_gym.core.failure_kinds.FailureClassificationConfig | None = None
) -> collections.Counter
```

Count final trial failures once; classify selected outcomes, never recovered observations.

```python
nemo_gym.eval.recovery.trials.trial_population(
    decision: nemo_gym.eval.recovery.decision.RecoveryDecision
) -> int
```

The trial denominator: every expected trial, including repeats, failed and missing ones, less the rows an
accepted partial stage omitted (no trial: reported separately, never in a numerator or denominator).

```python
nemo_gym.eval.recovery.trials.CUTOFF_REASONS = frozenset({'length', 'max_output_tokens', 'max_tokens'})
```

```python
nemo_gym.eval.recovery.trials.LENGTH_REFUSAL_KINDS = {'context_length_exceeded': MODEL_CONTEXT_LIMIT, 'invalid_output_token_parameter...
```

```python
nemo_gym.eval.recovery.trials.UNVERIFIED_NOTE = 'unverified: this withholds a pass and is no health failure'
```

```python
nemo_gym.eval.recovery.trials._COMPLETED_GRADE_OUTCOME = 'completed'
```

```python
nemo_gym.eval.recovery.trials._FAILED_OPERATION_OUTCOME = 'failed'
```

```python
nemo_gym.eval.recovery.trials._HTTP_ERROR_STATUS_MAX = 599
```

```python
nemo_gym.eval.recovery.trials._HTTP_ERROR_STATUS_MIN = 400
```

```python
nemo_gym.eval.recovery.trials._MODEL_REQUEST_OPERATION = 'model_request'
```

```python
nemo_gym.eval.recovery.trials._SANDBOX_FAILURES = frozenset({'failed', 'timeout', 'oom', 'sandbox_error', 'cancelled'})
```