> This page is for version 0.5.1.
> For other versions, use one of these documentation indexes:
> - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt
> - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt
> - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt
> - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt
> - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt
> - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt
> - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt
> - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.

# nemo_gym.eval.recovery.gate

The acceptance gate: response and trial health, metric coverage, MLflow delivery and the gate summary.

## Module Contents

### Classes

| Name                                                        | Description                                                                                               |
| ----------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- |
| [`GateCriteria`](#nemo_gym-eval-recovery-gate-GateCriteria) | The response and trial failure fractions a caller may still pass. They are validated (an invalid value is |

### Functions

| Name                                                                                          | Description                                                                                                                     |
| --------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| [`_bounded`](#nemo_gym-eval-recovery-gate-_bounded)                                           | -                                                                                                                               |
| [`_choices`](#nemo_gym-eval-recovery-gate-_choices)                                           | What in a printed command is the operator's to choose, and that its options go in one call.                                     |
| [`_command`](#nemo_gym-eval-recovery-gate-_command)                                           | The one recovery command the operator verdict prints: `gym eval resubmit &lt;job id&gt; &lt;benchmark&gt; --recover`,           |
| [`_continue_with`](#nemo_gym-eval-recovery-gate-_continue_with)                               | The recovery command that continues the run, after the typed stop's fix when it stopped on one, with what in                    |
| [`_deciding_rows`](#nemo_gym-eval-recovery-gate-_deciding_rows)                               | The saved result row that decides each scored trial: the last one saved for its identity, so an attempt a later                 |
| [`_delivery_action`](#nemo_gym-eval-recovery-gate-_delivery_action)                           | `(what, action)` for a complete run whose MLflow delivery is not closed: the results stand, and the delivery                    |
| [`_flagged_everything`](#nemo_gym-eval-recovery-gate-_flagged_everything)                     | `(what, flagged)` when rollout quality flags every rollout or every task, else None; `flagged` counts in                        |
| [`_flags`](#nemo_gym-eval-recovery-gate-_flags)                                               | The flags one option spells (`--max-walltime T --walltime-reason TEXT` is two).                                                 |
| [`_partial_rerun_note`](#nemo_gym-eval-recovery-gate-_partial_rerun_note)                     | What a rerun in `options` leaves masked: it runs only the results without a gradeable artifact.                                 |
| [`_refusal_verdict`](#nemo_gym-eval-recovery-gate-_refusal_verdict)                           | The verdict of a resubmit the scheduler refused: no job is known to be queued, or it is unknown whether one is.                 |
| [`_rejudged_after_judge_failure`](#nemo_gym-eval-recovery-gate-_rejudged_after_judge_failure) | The scored identities among `only` with a saved judge failure (their answer was judged again), and the HTTP                     |
| [`_stop_kind`](#nemo_gym-eval-recovery-gate-_stop_kind)                                       | The typed failure with a known remedy the run stopped on: the chain guard's typed stop, else the one failure                    |
| [`_stop_verdict`](#nemo_gym-eval-recovery-gate-_stop_verdict)                                 | The verdict of a STOP: the continuation its own cause allows, after any guard and typed fix, else the most                      |
| [`_verdict`](#nemo_gym-eval-recovery-gate-_verdict)                                           | -                                                                                                                               |
| [`_verifier_owned`](#nemo_gym-eval-recovery-gate-_verifier_owned)                             | Whether one of the row's verifier invocations owns the captured call (a judge's call), matched by its call id,                  |
| [`agent_adaptations`](#nemo_gym-eval-recovery-gate-agent_adaptations)                         | The scored rollouts' own `agent_adaptations` counts, summed by name: `&#123;name: &#123;"count": total, "rows": N&#125;&#125;`, |
| [`failure_facts`](#nemo_gym-eval-recovery-gate-failure_facts)                                 | Response and trial failures by cause, each with a count and a rate over the expected rollouts.                                  |
| [`gate_summary_from_collected`](#nemo_gym-eval-recovery-gate-gate_summary_from_collected)     | The acceptance gate task entry from the collected run: decision counts, the named score and the quality summary.                |
| [`gate_task_summary`](#nemo_gym-eval-recovery-gate-gate_task_summary)                         | One task entry of the acceptance gate's run summary, filled from the decision and the existing validators.                      |
| [`health_check_facts`](#nemo_gym-eval-recovery-gate-health_check_facts)                       | Named health-check facts from `quality_summary.json` for the gate warnings, kept even below any criterion.                      |
| [`health_verdict`](#nemo_gym-eval-recovery-gate-health_verdict)                               | \`\`evaluation health: PASS; response failures: 4.70% (truncated 41, empty 6 of 1000); trial failures: ...;                     |
| [`judge_job_note`](#nemo_gym-eval-recovery-gate-judge_job_note)                               | One line on a judge-only job of the run (`CollectedEvidence.judge_jobs`): queued, live, ended without a                         |
| [`length_cutoffs`](#nemo_gym-eval-recovery-gate-length_cutoffs)                               | How many of the policy's model calls in the rows that decide the scored trials ended on a length cutoff, and in                 |
| [`metric_coverage`](#nemo_gym-eval-recovery-gate-metric_coverage)                             | How many expected trials the benchmark's metric covered: `&#123;"trials": N, "expected": M, "source": ...&#125;`.               |
| [`mlflow_delivery`](#nemo_gym-eval-recovery-gate-mlflow_delivery)                             | The MLflow delivery fact of a run: `not_required` by design, `verified` when `mlflow_errors` found nothing,                     |
| [`mlflow_errors`](#nemo_gym-eval-recovery-gate-mlflow_errors)                                 | MLflow delivery errors from the collected exporter files and the server status the caller observed.                             |
| [`mlflow_notes`](#nemo_gym-eval-recovery-gate-mlflow_notes)                                   | What the gate states about a passing MLflow part: a current post-run export beside a live run wins.                             |
| [`mlflow_tracking_required`](#nemo_gym-eval-recovery-gate-mlflow_tracking_required)           | False only when the run's saved config snapshot sets `mlflow_tracking_required` to false; absent, unread or                     |
| [`operator_verdict`](#nemo_gym-eval-recovery-gate-operator_verdict)                           | The one verdict an operator acts on, derived from the gate and the decision only.                                               |
| [`operator_verdict_line`](#nemo_gym-eval-recovery-gate-operator_verdict_line)                 | -                                                                                                                               |
| [`output_budget_refits`](#nemo_gym-eval-recovery-gate-output_budget_refits)                   | How many model calls in the rows that decide the scored trials were sent again with an output budget lowered                    |
| [`quality_inputs`](#nemo_gym-eval-recovery-gate-quality_inputs)                               | Summary validity and per-rollout health from the run's `quality_summary.json`, kept separate.                                   |
| [`query_mlflow_status`](#nemo_gym-eval-recovery-gate-query_mlflow_status)                     | The status of `run_id` on the tracking server, as `(status, None)`, or `(None, error)` when it could not                        |
| [`repetition_stops`](#nemo_gym-eval-recovery-gate-repetition_stops)                           | How many of the policy's model calls in the rows that decide the scored trials the provider stopped for                         |
| [`response_health`](#nemo_gym-eval-recovery-gate-response_health)                             | The response gate from `run.stats.responses` of the health report: faulted over selected responses.                             |
| [`rollout_quality_facts`](#nemo_gym-eval-recovery-gate-rollout_quality_facts)                 | The rollout-quality signals of `quality_summary.json` the operator verdict reads: verdict counts and the                        |
| [`subset_note`](#nemo_gym-eval-recovery-gate-subset_note)                                     | The label of a run that selected its tasks by id: its score is the subset's, never the benchmark's.                             |
| [`tracking_export_due`](#nemo_gym-eval-recovery-gate-tracking_export_due)                     | Whether a complete run lacks the MLflow export receipt its saved config requires, so that                                       |
| [`trial_failure_cause`](#nemo_gym-eval-recovery-gate-trial_failure_cause)                     | A trial failure's cause from its failure kind: request, judge, sandbox, harness, timeout, allocation, or other.                 |
| [`trial_health`](#nemo_gym-eval-recovery-gate-trial_health)                                   | Evaluation health from the trial gate (`trial_verdict`): passed, failed or None, with its reason.                               |
| [`trial_verdict`](#nemo_gym-eval-recovery-gate-trial_verdict)                                 | Evaluation health: evaluation-side trial faults over the expected trials, including repeats.                                    |
| [`unverified_cause`](#nemo_gym-eval-recovery-gate-unverified_cause)                           | What made a complete run's health UNVERIFIED, as `&#123;"kind", "reason"&#125;`: one cause-first sentence with counts.          |
| [`with_operator_verdict`](#nemo_gym-eval-recovery-gate-with_operator_verdict)                 | The outcome with `operator_verdict` set on it and, when there is a gate, inside `gate_task`.                                    |

### Data

[`GATE_FAILURE_LIMIT`](#nemo_gym-eval-recovery-gate-GATE_FAILURE_LIMIT)

[`MLFLOW_NOT_REQUIRED_NOTE`](#nemo_gym-eval-recovery-gate-MLFLOW_NOT_REQUIRED_NOTE)

[`MLFLOW_STATUS_NOT_OBSERVED`](#nemo_gym-eval-recovery-gate-MLFLOW_STATUS_NOT_OBSERVED)

[`MLFLOW_STATUS_QUERY_RETRIES`](#nemo_gym-eval-recovery-gate-MLFLOW_STATUS_QUERY_RETRIES)

[`MLFLOW_STATUS_QUERY_TIMEOUT_S`](#nemo_gym-eval-recovery-gate-MLFLOW_STATUS_QUERY_TIMEOUT_S)

[`RESPONSE_FAILURE_CAUSES`](#nemo_gym-eval-recovery-gate-RESPONSE_FAILURE_CAUSES)

[`TRIAL_FAILURE_CAUSES`](#nemo_gym-eval-recovery-gate-TRIAL_FAILURE_CAUSES)

[`UNREADABLE_ARTIFACTS_ERROR`](#nemo_gym-eval-recovery-gate-UNREADABLE_ARTIFACTS_ERROR)

[`_CONTINUATION_GUARDS`](#nemo_gym-eval-recovery-gate-_CONTINUATION_GUARDS)

[`_JUDGE_PLACEMENT_REMEDIES`](#nemo_gym-eval-recovery-gate-_JUDGE_PLACEMENT_REMEDIES)

[`_MAX_FACTS`](#nemo_gym-eval-recovery-gate-_MAX_FACTS)

[`_NO_KNOWN_COMMAND`](#nemo_gym-eval-recovery-gate-_NO_KNOWN_COMMAND)

[`_OPEN_STATES`](#nemo_gym-eval-recovery-gate-_OPEN_STATES)

[`_PROTOCOL_VIOLATION_REMEDY`](#nemo_gym-eval-recovery-gate-_PROTOCOL_VIOLATION_REMEDY)

[`_RECOVER_ONLY_FLAGS`](#nemo_gym-eval-recovery-gate-_RECOVER_ONLY_FLAGS)

[`_REPIN_STOPS`](#nemo_gym-eval-recovery-gate-_REPIN_STOPS)

[`_RESTART_FIXES`](#nemo_gym-eval-recovery-gate-_RESTART_FIXES)

[`_SHELL_QUOTED`](#nemo_gym-eval-recovery-gate-_SHELL_QUOTED)

[`_TRIAL_FAILURE_STATES`](#nemo_gym-eval-recovery-gate-_TRIAL_FAILURE_STATES)

[`_TYPED_REMEDIES`](#nemo_gym-eval-recovery-gate-_TYPED_REMEDIES)

[`_UNKNOWN_EVIDENCE_STOPS`](#nemo_gym-eval-recovery-gate-_UNKNOWN_EVIDENCE_STOPS)

[`_UNSAFE_RESUME`](#nemo_gym-eval-recovery-gate-_UNSAFE_RESUME)

[`_UNVERIFIED_PRODUCERS`](#nemo_gym-eval-recovery-gate-_UNVERIFIED_PRODUCERS)

### API

```python
class nemo_gym.eval.recovery.gate.GateCriteria()
```

**Bases:** `BaseModel`

The response and trial failure fractions a caller may still pass. They are validated (an invalid value is
refused, never clamped) and reported, but no longer applied: the gate's only health verdict is evaluation health
on the bounded failure policy's limit (`trial_verdict`), and response and trial failures are reported as facts
(`failure_facts`) for consumers to judge.

**`max_response_failure_fraction`** `float = GATE_FAILURE_LIMIT`

---

**`max_trial_failure_fraction`** `float = GATE_FAILURE_LIMIT`

---

**`model_config`** `= ConfigDict(extra='forbid', frozen=True)`

---

**`policy_timeout_counts_as_model`** `bool = Field(default=True, strict=True)`

---

```python
nemo_gym.eval.recovery.gate.GateCriteria._fraction(
    value: typing.Any
) -> float
```

classmethod

```python
nemo_gym.eval.recovery.gate.GateCriteria.report(
    classification: nemo_gym.core.failure_kinds.FailureClassificationConfig | None = None
) -> dict[str, typing.Any]
```

The effective criterion and the accepted, unapplied fractions, as reported beside the gate results.

```python
nemo_gym.eval.recovery.gate._bounded(
    items: list[str]
) -> str
```

```python
nemo_gym.eval.recovery.gate._choices(
    options: collections.abc.Sequence[str]
) -> str
```

What in a printed command is the operator's to choose, and that its options go in one call.

```python
nemo_gym.eval.recovery.gate._command(
    decision: nemo_gym.eval.recovery.decision.RecoveryDecision,
    options: str = (),
    remedy: collections.abc.Sequence[str] = (),
    drop: collections.abc.Sequence[str] = (),
    inspect: bool = False
) -> str
```

The one recovery command the operator verdict prints: `gym eval resubmit <job id> <benchmark> --recover`,
then every option the call asked for (`decision.requested`), then the typed remedy's options, then `options`
(what the stop itself needs, such as a budget extension). An option is named once: `options` replaces the
call's own value, the call's own value replaces a remedy's placeholder, and `drop` names options the run
refused. `inspect` makes it the inspection, without the options an inspection refuses.

```python
nemo_gym.eval.recovery.gate._continue_with(
    decision: nemo_gym.eval.recovery.decision.RecoveryDecision,
    options: str = (),
    drop: collections.abc.Sequence[str] = (),
    kind_from_rows: bool = True
) -> str
```

The recovery command that continues the run, after the typed stop's fix when it stopped on one, with what in
it is the operator's to choose. A typed stop whose own text says a recovery cannot continue (a refused resume,
a protocol violation) names that instead.

```python
nemo_gym.eval.recovery.gate._deciding_rows(
    decision: nemo_gym.eval.recovery.decision.RecoveryDecision,
    collected: nemo_gym.eval.recovery.evidence.CollectedEvidence | None
) -> list[collections.abc.Mapping[str, typing.Any]]
```

The saved result row that decides each scored trial: the last one saved for its identity, so an attempt a later
one superseded is not counted again.

```python
nemo_gym.eval.recovery.gate._delivery_action(
    decision: nemo_gym.eval.recovery.decision.RecoveryDecision,
    delivery: collections.abc.Mapping[str, typing.Any],
    export_due: bool = False
) -> tuple[str, str] | None
```

`(what, action)` for a complete run whose MLflow delivery is not closed: the results stand, and the delivery
is a fact beside them. Unverified by configuration (the export did not upload the rollouts) is reported, as no
export changes it.

```python
nemo_gym.eval.recovery.gate._flagged_everything(
    gate_task: collections.abc.Mapping[str, typing.Any],
    decision: nemo_gym.eval.recovery.decision.RecoveryDecision,
    expected: int
) -> tuple[str, str] | None
```

`(what, flagged)` when rollout quality flags every rollout or every task, else None; `flagged` counts in
the unit of the check.

```python
nemo_gym.eval.recovery.gate._flags(
    option: str
) -> set[str]
```

The flags one option spells (`--max-walltime T --walltime-reason TEXT` is two).

```python
nemo_gym.eval.recovery.gate._partial_rerun_note(
    decision: nemo_gym.eval.recovery.decision.RecoveryDecision,
    options: list[str]
) -> str
```

What a rerun in `options` leaves masked: it runs only the results without a gradeable artifact.

```python
nemo_gym.eval.recovery.gate._refusal_verdict(
    refusal: collections.abc.Mapping[str, typing.Any]
) -> dict[str, str]
```

The verdict of a resubmit the scheduler refused: no job is known to be queued, or it is unknown whether one is.

```python
nemo_gym.eval.recovery.gate._rejudged_after_judge_failure(
    decision: nemo_gym.eval.recovery.decision.RecoveryDecision,
    collected: nemo_gym.eval.recovery.evidence.CollectedEvidence,
    only: set[nemo_gym.eval.recovery.evidence.Identity]
) -> tuple[set[nemo_gym.eval.recovery.evidence.Identity], collections.Counter]
```

The scored identities among `only` with a saved judge failure (their answer was judged again), and the HTTP
statuses of it.

```python
nemo_gym.eval.recovery.gate._stop_kind(
    decision: nemo_gym.eval.recovery.decision.RecoveryDecision,
    from_rows: bool = True
) -> str | None
```

The typed failure with a known remedy the run stopped on: the chain guard's typed stop, else the one failure
kind every terminal identity of the decision records (Gym's own typed stop, such as the grading quota). Read from
typed fields only, never from the reason text. `from_rows=False` reads the chain guard's stop alone: the row
kind explains a STOP, and a decision that continues the saved run (RESUBMIT, REVERIFY) is not one.

```python
nemo_gym.eval.recovery.gate._stop_verdict(
    decision: nemo_gym.eval.recovery.decision.RecoveryDecision,
    expected: int,
    open_text: str
) -> dict[str, str]
```

The verdict of a STOP: the continuation its own cause allows, after any guard and typed fix, else the most
specific known step, else no command.

```python
nemo_gym.eval.recovery.gate._verdict(
    state: str,
    reason: str,
    action: str
) -> dict[str, str]
```

```python
nemo_gym.eval.recovery.gate._verifier_owned(
    row: collections.abc.Mapping[str, typing.Any],
    call: collections.abc.Mapping[str, typing.Any]
) -> bool
```

Whether one of the row's verifier invocations owns the captured call (a judge's call), matched by its call id,
else by its model and response id: a judge served by the policy's own server is still the judge.

```python
nemo_gym.eval.recovery.gate.agent_adaptations(
    decision: nemo_gym.eval.recovery.decision.RecoveryDecision,
    collected: nemo_gym.eval.recovery.evidence.CollectedEvidence | None
) -> dict[str, dict[str, int]]
```

The scored rollouts' own `agent_adaptations` counts, summed by name: `{name: {"count": total, "rows": N}}`,
`rows` being the rollouts with a nonzero count. An agent records what it adapted on its own before a result,
such as a request the endpoint rejected for its input size; a reported fact, never a failure.

```python
nemo_gym.eval.recovery.gate.failure_facts(
    decision: nemo_gym.eval.recovery.decision.RecoveryDecision,
    collected: nemo_gym.eval.recovery.evidence.CollectedEvidence | None
) -> dict[str, typing.Any] | None
```

Response and trial failures by cause, each with a count and a rate over the expected rollouts.

Reported facts, never a verdict: no limit applies to them here. The denominator is the trial gate's
(`trial_population`): every expected rollout, failed and missing ones included; rows an accepted partial stage
omitted are no rollout of the run and are reported separately as `omitted`.

A response failure is a scored rollout without a recorded grade (`_recorded_grade`) whose saved answer is
truncated (incomplete with a cutoff signal, `_cut_off`), empty (no assistant text, `_empty_answer`) or malformed (no response
with an output list, an answer the verifier marked invalid, or one left incomplete without a cutoff signal). A trial failure is a rollout in a failed, masked or
unowned state (`_TRIAL_FAILURE_STATES`), a scored rollout that ended on a failed policy request, a scored
rollout that records its own failure kind or whose judge recorded no verdict, or an observed grading without a
valid verdict, by the cause its failure kind names (`trial_failure_cause`).

```python
nemo_gym.eval.recovery.gate.gate_summary_from_collected(
    decision: nemo_gym.eval.recovery.decision.RecoveryDecision,
    collected: nemo_gym.eval.recovery.evidence.CollectedEvidence,
    max_unhealthy_fraction: float | None,
    mlflow_status: str | None = None,
    mlflow_run_id: str | None = None,
    criteria: nemo_gym.eval.recovery.gate.GateCriteria | None = None,
    status_client_factory: collections.abc.Callable[[str], typing.Any] | None = None
) -> dict[str, typing.Any]
```

The acceptance gate task entry from the collected run: decision counts, the named score and the quality summary.

`mlflow_status` and `mlflow_run_id` are the tracking run's status and id as the caller read them together.
Without a status, the receipt's run is asked of the tracking server the saved config names (`query_mlflow_status`;
`status_client_factory` replaces its client).
`criteria` are accepted and reported under `criteria` but not applied (`GateCriteria`). The only health
verdict is `evaluation_health`; `failures` reports response and trial failures by cause, never a verdict.
`status` follows `gate_task_summary`: an UNVERIFIED evaluation health with nothing failed is UNVERIFIED, and
a chain error turns SUCCESS or UNVERIFIED into FAILED. MLflow delivery is reported as
`mlflow_delivery`, a fact that never changes the status.

```python
nemo_gym.eval.recovery.gate.gate_task_summary(
    decision: nemo_gym.eval.recovery.decision.RecoveryDecision,
    score: float | None,
    summary_valid: bool | None,
    summary_reasons: list[str],
    health_passed: bool | None,
    health_reasons: list[str]
) -> dict[str, typing.Any]
```

One task entry of the acceptance gate's run summary, filled from the decision and the existing validators.

`expected_samples` is always the full materialized inventory; `scored_samples` counts only scored and
final-model identities, so a judge, infra, terminal or missing identity can never make them equal. A settled
final outcome the metric does not score (`IdentityEvidence.unscored`) is stated, not counted as scored. Summary validity
and response/trial health stay separate. `False` is a failure and an error; `None` (unknown or missing) is no
failure but never a pass: its reason is a warning and it withholds SUCCESS.

`status`, first match wins: RUNNING (a job is live); PLANNED (recovery work is eligible; only a caller that
dispatches it reports RESUMING); FAILED (anything in `errors`, or the run is not a complete NOOP); UNVERIFIED
(nothing failed, but summary validity or response/trial health is unknown, so acceptance is not proven); SUCCESS.

```python
nemo_gym.eval.recovery.gate.health_check_facts(
    quality: collections.abc.Mapping[str, typing.Any] | None
) -> list[str]
```

Named health-check facts from `quality_summary.json` for the gate warnings, kept even below any criterion.

Nonzero `run.issues` counts, checks `run.artifacts.coverage` did not fully evaluate (any unobserved or
ignored), and `run.ignored_checks`. A field that is absent or malformed is reported as unknown, never as clean.

```python
nemo_gym.eval.recovery.gate.health_verdict(
    evaluation: dict[str, typing.Any] | None,
    facts: dict[str, typing.Any] | None
) -> str
```

`evaluation health: PASS; response failures: 4.70% (truncated 41, empty 6 of 1000); trial failures: ...;
metric over 1000 of 1000 trials`.

```python
nemo_gym.eval.recovery.gate.judge_job_note(
    job: collections.abc.Mapping[str, typing.Any]
) -> str
```

One line on a judge-only job of the run (`CollectedEvidence.judge_jobs`): queued, live, ended without a
receipt, or what its receipt says it settled.

```python
nemo_gym.eval.recovery.gate.length_cutoffs(
    decision: nemo_gym.eval.recovery.decision.RecoveryDecision,
    collected: nemo_gym.eval.recovery.evidence.CollectedEvidence | None
) -> dict[str, int]
```

How many of the policy's model calls in the rows that decide the scored trials ended on a length cutoff, and in
how many of those rows: a `length` finish reason, or its `max_output_tokens` and `max_tokens` spellings, as
the evidence collector records each call.

A fact, never a failure: the model server returned the shortened answer and the rollout carried on from it, so
a run that lost output to the context window or the output budget does not read as clean. The finish reason does
not tell the two limits apart. A call a verifier owns (a judge's, also when it shares the policy's server) or
another model's call is not counted, nor is an attempt a later one superseded.

```python
nemo_gym.eval.recovery.gate.metric_coverage(
    decision: nemo_gym.eval.recovery.decision.RecoveryDecision,
    collected: nemo_gym.eval.recovery.evidence.CollectedEvidence | None
) -> dict[str, typing.Any]
```

How many expected trials the benchmark's metric covered: `{"trials": N, "expected": M, "source": ...}`.

A benchmark that declares the coverage of its headline metric in its aggregate (`metric_coverage/trials`,
`metric_coverage/expected` and an optional `metric_coverage/name`) is the source (`benchmark`), with its
own denominator and the metric's name. Otherwise the source is collection's own count when its coverage report
has one for this inventory (`coverage_report`):
the rollouts whose score entered the metrics, a failure counted as zero by configuration included. Otherwise it is
read from the saved rows (`saved_rows`): the scored results and the failures their producer marked as counted
zeros, which is what collection hands the metrics unless it counts more by configuration. A failure the benchmark
leaves out of its metric, such as an agent deadline that kept its answer, is a trial the metric did not cover.
The denominator is the trial gate's: every expected trial, less the rows an accepted partial stage omitted.

```python
nemo_gym.eval.recovery.gate.mlflow_delivery(
    problems: collections.abc.Sequence[str],
    tracking_required: bool,
    status_error: str | None = None
) -> dict[str, typing.Any]
```

The MLflow delivery fact of a run: `not_required` by design, `verified` when `mlflow_errors` found nothing,
`pending` while there is no receipt and no live run to judge or only the run's status was not queried, else
`unverified` with its reasons. A fact for consumers to read, never a verdict: it does not touch evaluation
health or the gate's status. A status that was neither given nor queried reads "status not queried"; a query that
failed reads "status query failed: ...": never a silent "not observed".

```python
nemo_gym.eval.recovery.gate.mlflow_errors(
    local: collections.abc.Mapping[str, typing.Any] | None,
    observed_status: str | None,
    configured: bool | None = None,
    observed_run_id: str | None = None
) -> list[str]
```

MLflow delivery errors from the collected exporter files and the server status the caller observed.

`local` holds the live exporter's run id (`rollouts_mlflow_run_id`) and the post-run export receipt
(`rollouts_mlflow_export_done`, checked against the current artifact digests). Only a current post-run receipt
with an observed FINISHED passes. A live run is not accepted here: live uploads log failures and continue, so
neither its run id nor FINISHED shows the configured payloads arrived. A live run beside a post-run export leaves
the observed status unbound. `configured` is the health report's `run.provenance.mlflow.configured`, a
names-only fact about the saved config snapshot (None when not reported); it only qualifies the message. The
observed status is bound to a run: with a post-run receipt, `observed_run_id` must name the receipt's run.
`BaseExporter.export` writes the receipt only after every upload returned and the artifacts did not change, but
its teardown still sets FINISHED on a partial export (`run_failed` is the process's run, not the export's), so
a FINISHED from any other run proves nothing. No
evidence is always an error, never a run that needed none: exporter activation and whether tracking was required
or disabled are not recorded.

```python
nemo_gym.eval.recovery.gate.mlflow_notes(
    local: collections.abc.Mapping[str, typing.Any] | None
) -> list[str]
```

What the gate states about a passing MLflow part: a current post-run export beside a live run wins.

```python
nemo_gym.eval.recovery.gate.mlflow_tracking_required(
    collected: nemo_gym.eval.recovery.evidence.CollectedEvidence | None
) -> bool
```

False only when the run's saved config snapshot sets `mlflow_tracking_required` to false; absent, unread or
any other value means tracking is required. The health report's provenance states it when its pin did; for a run
whose report does not, the collector's read of the saved `rollouts_resolved_config.yaml` does.

```python
nemo_gym.eval.recovery.gate.operator_verdict(
    gate_task: collections.abc.Mapping[str, typing.Any] | None,
    decision: nemo_gym.eval.recovery.decision.RecoveryDecision | None,
    scheduler_refusal: collections.abc.Mapping[str, typing.Any] | None = None,
    walltime_plan: collections.abc.Mapping[str, typing.Any] | None = None
) -> dict[str, str]
```

The one verdict an operator acts on, derived from the gate and the decision only.

`state` is `accept` (gate SUCCESS, coverage complete, run health PASS, decision noop), `needs_action` (a
STOP, FAILED, incomplete coverage or failed run health) or `unknown` (UNVERIFIED, which no recovery changes: by
configuration, by a producer that records no grading, after a re-judgment whose grading was not recorded or
without a known cause; a live job; evidence missing or unreadable). `reason` is one sentence naming the
dominant cause with counts; `action` is the one thing to do. Every command it prints comes from `_command`:
`gym eval resubmit <job id> <benchmark> --recover` with every option the call asked for, the typed stop's fix
and the options the stop needs. It never adds a fresh submit that Gym's own decision or stop text does not name,
and where Gym decided nothing is left to run it says what to report and check. An accepted run whose MLflow
delivery is not closed names the export, or that it is unverified by configuration. Rollout quality never changes
the state: an accepted run whose every rollout is flagged says so in its reason and points at the quality read. A
scheduler refusal of the resubmit and a walltime raise the partition cannot take (refused before any evidence is
read) are known dispatch failures: they need the operator's correction, never a plain recover-now or a reader
retry.

```python
nemo_gym.eval.recovery.gate.operator_verdict_line(
    verdict: collections.abc.Mapping[str, str]
) -> str
```

```python
nemo_gym.eval.recovery.gate.output_budget_refits(
    decision: nemo_gym.eval.recovery.decision.RecoveryDecision,
    collected: nemo_gym.eval.recovery.evidence.CollectedEvidence | None
) -> dict[str, int]
```

How many model calls in the rows that decide the scored trials were sent again with an output budget lowered
to fit the context window (`output_budget_refit` in a call's response metadata), and in how many of those rows.

A count, never a failure: the call generated within the window it was served at. A reader flags a cell whose
requests were budget-capped from it.

```python
nemo_gym.eval.recovery.gate.quality_inputs(
    decision: nemo_gym.eval.recovery.decision.RecoveryDecision,
    collected: nemo_gym.eval.recovery.evidence.CollectedEvidence,
    max_unhealthy_fraction: float | None
) -> tuple[bool | None, list[str], bool | None, list[str]]
```

Summary validity and per-rollout health from the run's `quality_summary.json`, kept separate.

Validity is the health check's own accounting verdict (`run.artifacts.healthy`, null for a partial summary),
and must count the same expected identities as the inventory. Per-rollout health is the rollout verdict count
(`run.verdicts`), judged only against an explicit `max_unhealthy_fraction`; it is not separately validated response or
trial health, which this adapter has no separate evidence for. Any missing piece is `None`.

```python
nemo_gym.eval.recovery.gate.query_mlflow_status(
    tracking_uri: str,
    run_id: str,
    client_factory: collections.abc.Callable[[str], typing.Any] | None = None
) -> tuple[str | None, str | None]
```

The status of `run_id` on the tracking server, as `(status, None)`, or `(None, error)` when it could not
be read: a missing client, an unreachable or refusing server, a missing run. Short timeout and one retry;
`client_factory` builds the client from the URI (default `mlflow.MlflowClient`).

```python
nemo_gym.eval.recovery.gate.repetition_stops(
    decision: nemo_gym.eval.recovery.decision.RecoveryDecision,
    collected: nemo_gym.eval.recovery.evidence.CollectedEvidence | None
) -> dict[str, int]
```

How many of the policy's model calls in the rows that decide the scored trials the provider stopped for
repetition, and in how many of those rows: a `repetition` finish reason or a `repetition_detected` stop reason
(vLLM's repetition detection), as the evidence collector flags each call.

The model's own ending, never a failure: the answer it produced is scored as it is and nothing is retried, so the
count only makes loops visible. A call a verifier owns (a judge's, also when it shares the policy's server) or
another model's call is not counted, nor is an attempt a later one superseded.

```python
nemo_gym.eval.recovery.gate.response_health(
    decision: nemo_gym.eval.recovery.decision.RecoveryDecision,
    quality: collections.abc.Mapping[str, typing.Any] | None,
    limit: float = GATE_FAILURE_LIMIT
) -> tuple[bool | None, str]
```

The response gate from `run.stats.responses` of the health report: faulted over selected responses.

Evaluated only when every scored rollout has a verified response population and every selected call id is
unique across the run (a report that predates it, an unverified call scope, a response selected twice, or zero
selected responses is unknown, neither pass nor fail). Failed trials have no response row and are left out, as
the policy leaves failed attempts out of the response denominator.

```python
nemo_gym.eval.recovery.gate.rollout_quality_facts(
    quality: collections.abc.Mapping[str, typing.Any] | None
) -> dict[str, typing.Any] | None
```

The rollout-quality signals of `quality_summary.json` the operator verdict reads: verdict counts and the
nonzero issue counts. A signal, never a gate; None when the summary has neither.

```python
nemo_gym.eval.recovery.gate.subset_note(
    selection: collections.abc.Mapping[str, typing.Any]
) -> str
```

The label of a run that selected its tasks by id: its score is the subset's, never the benchmark's.

```python
nemo_gym.eval.recovery.gate.tracking_export_due(
    decision: nemo_gym.eval.recovery.decision.RecoveryDecision,
    collected: nemo_gym.eval.recovery.evidence.CollectedEvidence | None,
    tracking_required: bool
) -> bool
```

Whether a complete run lacks the MLflow export receipt its saved config requires, so that
`gym eval resubmit --recover` exports it.

Only a run whose saved config configures MLflow (`provenance.mlflow.configured` is true: tracking URI,
experiment and run name) and does not set `mlflow_tracking_required` to false, with no post-run receipt at all
and no live job, is exported. A run that configured no exporter, or whose provenance does not say, is never
exported, and a receipt, current or not, is never replaced.

```python
nemo_gym.eval.recovery.gate.trial_failure_cause(
    kind: typing.Any
) -> str
```

A trial failure's cause from its failure kind: request, judge, sandbox, harness, timeout, allocation, or other.

`allocation`: the rollout was in flight at repeated ends of its allocation (`ROLLOUT_ALLOCATION_ENDED`).

```python
nemo_gym.eval.recovery.gate.trial_health(
    decision: nemo_gym.eval.recovery.decision.RecoveryDecision,
    collected: nemo_gym.eval.recovery.evidence.CollectedEvidence | None = None,
    limit: float = GATE_FAILURE_LIMIT,
    policy_timeout_counts_as_model: bool = True,
    classification: nemo_gym.core.failure_kinds.FailureClassificationConfig | None = None
) -> tuple[bool | None, str]
```

Evaluation health from the trial gate (`trial_verdict`): passed, failed or None, with its reason.

```python
nemo_gym.eval.recovery.gate.trial_verdict(
    decision: nemo_gym.eval.recovery.decision.RecoveryDecision,
    collected: nemo_gym.eval.recovery.evidence.CollectedEvidence | None = None,
    limit: float = GATE_FAILURE_LIMIT,
    policy_timeout_counts_as_model: bool = True,
    classification: nemo_gym.core.failure_kinds.FailureClassificationConfig | None = None
) -> tuple[bool | None, str, dict[str, typing.Any] | None]
```

Evaluation health: evaluation-side trial faults over the expected trials, including repeats.

The counted failures are `trial_faults`; a model fault (`classify_failure`) is reported with its kind and never
fails. Only the evaluation-side share is judged against the limit: at or above it the gate fails. Below it, it
passes only with an exact count, when actual per-identity operation outcomes cover the observed model/tool and
grading path of every trial that is not a recorded failure (`_observed_trial_operations`); an observed grading
without a valid verdict is an evaluation-side fault. Otherwise the run is UNVERIFIED: a judge, verifier or harness
error scored as a zero in the main output carries no failure record and cannot be ruled out, so the known count is
a lower bound. That withholds a pass, never fails health. The denominator is the full materialized inventory less
accepted omissions. A run that is not complete has unaccounted trials, so the gate is not evaluated.

The report is `status` (PASS, FAIL or UNVERIFIED), the evaluation-side rate, `exact` and the fault counts.

```python
nemo_gym.eval.recovery.gate.unverified_cause(
    decision: nemo_gym.eval.recovery.decision.RecoveryDecision,
    collected: nemo_gym.eval.recovery.evidence.CollectedEvidence,
    evaluation: collections.abc.Mapping[str, typing.Any] | None,
    trial_reason: str
) -> dict[str, str]
```

What made a complete run's health UNVERIFIED, as `{"kind", "reason"}`: one cause-first sentence with counts.

`kind` is `judge_rejudged`, `grading_unobserved`, `unclassified_failures`, `recovered_not_joined` or
`zero_rewards_unprovable`. The last has no cause beyond the missing proof, so the operator verdict treats it as
unknown. The full reason of the trial gate stays in `evaluation_health.detail`.

```python
nemo_gym.eval.recovery.gate.with_operator_verdict(
    outcome: nemo_gym.eval.recovery.decision.RecoveryOutcome
) -> nemo_gym.eval.recovery.decision.RecoveryOutcome
```

The outcome with `operator_verdict` set on it and, when there is a gate, inside `gate_task`.

```python
nemo_gym.eval.recovery.gate.GATE_FAILURE_LIMIT = 0.02
```

```python
nemo_gym.eval.recovery.gate.MLFLOW_NOT_REQUIRED_NOTE = 'no MLflow export by design (mlflow_tracking_required is false in the saved conf...
```

```python
nemo_gym.eval.recovery.gate.MLFLOW_STATUS_NOT_OBSERVED = 'MLflow terminal status not observed'
```

```python
nemo_gym.eval.recovery.gate.MLFLOW_STATUS_QUERY_RETRIES = '1'
```

```python
nemo_gym.eval.recovery.gate.MLFLOW_STATUS_QUERY_TIMEOUT_S = '10'
```

```python
nemo_gym.eval.recovery.gate.RESPONSE_FAILURE_CAUSES = ('truncated', 'empty', 'malformed')
```

```python
nemo_gym.eval.recovery.gate.TRIAL_FAILURE_CAUSES = ('request', 'judge', 'sandbox', 'harness', 'timeout', 'allocation', 'other')
```

```python
nemo_gym.eval.recovery.gate.UNREADABLE_ARTIFACTS_ERROR = 'unreadable artifacts, read as unknown'
```

```python
nemo_gym.eval.recovery.gate._CONTINUATION_GUARDS = {'scheduler_unreachable': ('unknown', 'the scheduler did not answer, so a live o...
```

```python
nemo_gym.eval.recovery.gate._JUDGE_PLACEMENT_REMEDIES = {'cpu_partition_unknown': (('--judge-partition PARTITION',), 'name a CPU partiti...
```

```python
nemo_gym.eval.recovery.gate._MAX_FACTS = 20
```

```python
nemo_gym.eval.recovery.gate._NO_KNOWN_COMMAND = 'read the stop reason; no recovery command is known for it'
```

```python
nemo_gym.eval.recovery.gate._OPEN_STATES = (IdentityState.JUDGE_RECOVERABLE, IdentityState.RETRYABLE, IdentityState.MISSING...
```

```python
nemo_gym.eval.recovery.gate._PROTOCOL_VIOLATION_REMEDY = 'fix the server or the benchmark pairing (a producer bug), then `gym eval submit...
```

```python
nemo_gym.eval.recovery.gate._RECOVER_ONLY_FLAGS = frozenset({'--max-walltime', '--allow-unsafe-resume', '--server-spinup-timeout-s...
```

```python
nemo_gym.eval.recovery.gate._REPIN_STOPS = (re.compile("--repin <this launcher's full commit> --repin-reason TEXT runs the ...
```

```python
nemo_gym.eval.recovery.gate._RESTART_FIXES = (PROVIDER_CREDENTIALS_REJECTED, DRIVER_INSTALL_FAILED, ENVIRONMENT_PREPARE_FAILE...
```

```python
nemo_gym.eval.recovery.gate._SHELL_QUOTED = re.compile('\'(?:[^\']|\'\\\\\'\')*\'|"(?:[^"\\\\]|\\\\.)*\\"')
```

```python
nemo_gym.eval.recovery.gate._TRIAL_FAILURE_STATES = (IdentityState.MODEL_FINAL, IdentityState.JUDGE_RECOVERABLE, IdentityState.RETRY...
```

```python
nemo_gym.eval.recovery.gate._TYPED_REMEDIES: dict[str, tuple[str, tuple[str, ...]]] = {PROVIDER_CREDENTIALS_REJECTED: ('fix the credentials, then run', ()), DRIVER_IN...
```

```python
nemo_gym.eval.recovery.gate._UNKNOWN_EVIDENCE_STOPS = (('the scheduler query failed', 'wait until the scheduler answers and inspect ag...
```

```python
nemo_gym.eval.recovery.gate._UNSAFE_RESUME = ('--allow-unsafe-resume', '--unsafe-resume-reason TEXT')
```

```python
nemo_gym.eval.recovery.gate._UNVERIFIED_PRODUCERS = {'grading_unobserved': "the verifier's grading record", 'unclassified_failures':...
```