> This page is for version 0.5.0.
> For other versions, use one of these documentation indexes:
> - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt
> - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt
> - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt
> - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt
> - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt
> - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt
> - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt
> - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server.

# nemo_gym.rollouts.rollout_recovery

Identity checks for resuming saved evaluation results.

The manifest stores digests, not configuration values or credentials. Recovery reuses
completed rollouts; it does not checkpoint an agent's conversation or remote sandbox.

## Module Contents

### Classes

| Name                                                                                         | Description                                                                                   |
| -------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- |
| [`IncompleteEvaluationError`](#nemo_gym-rollouts-rollout_recovery-IncompleteEvaluationError) | Saved partial results are valid, but the evaluation still has unfinished work.                |
| [`PendingRejudge`](#nemo_gym-rollouts-rollout_recovery-PendingRejudge)                       | A judge-only re-judgment admitted before its judge call; it closes when its outcome is saved. |
| [`ResumeRefusedError`](#nemo_gym-rollouts-rollout_recovery-ResumeRefusedError)               | Saved rollouts whose identity does not verify against the current run are not resumed.        |
| [`RunManifest`](#nemo_gym-rollouts-rollout_recovery-RunManifest)                             | -                                                                                             |

### Functions

| Name                                                                                                           | Description                                                                                                       |
| -------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
| [`_configuration_drift`](#nemo_gym-rollouts-rollout_recovery-_configuration_drift)                             | The semantic and operational keys that differ from the saved identity, or None when it was not saved.             |
| [`_configuration_identity`](#nemo_gym-rollouts-rollout_recovery-_configuration_identity)                       | The configuration a run's results depend on, compared on resume.                                                  |
| [`_credential_safe`](#nemo_gym-rollouts-rollout_recovery-_credential_safe)                                     | The value with every credential replaced by a fixed marker; it is what a run persists for its identity.           |
| [`_differing_keys`](#nemo_gym-rollouts-rollout_recovery-_differing_keys)                                       | Dotted keys whose values differ between two identities; a list is one value.                                      |
| [`_digest`](#nemo_gym-rollouts-rollout_recovery-_digest)                                                       | -                                                                                                                 |
| [`_earlier_collection_forms`](#nemo_gym-rollouts-rollout_recovery-_earlier_collection_forms)                   | The identity as the Gyms before and after each added collection setting recorded it.                              |
| [`_evaluation_event`](#nemo_gym-rollouts-rollout_recovery-_evaluation_event)                                   | -                                                                                                                 |
| [`_file_digest`](#nemo_gym-rollouts-rollout_recovery-_file_digest)                                             | -                                                                                                                 |
| [`_get_max_rollout_attempts`](#nemo_gym-rollouts-rollout_recovery-_get_max_rollout_attempts)                   | Read `NEMO_GYM_MAX_ROLLOUT_ATTEMPTS` (positive int) or default to 3.                                              |
| [`_is_credential_name`](#nemo_gym-rollouts-rollout_recovery-_is_credential_name)                               | -                                                                                                                 |
| [`_is_semantic`](#nemo_gym-rollouts-rollout_recovery-_is_semantic)                                             | -                                                                                                                 |
| [`_keeps_saved_answer`](#nemo_gym-rollouts-rollout_recovery-_keeps_saved_answer)                               | Whether the masked result holds a model answer a rerun would replace: never sampled again.                        |
| [`_legacy_config_digests`](#nemo_gym-rollouts-rollout_recovery-_legacy_config_digests)                         | Digests an earlier Gym recorded for this configuration, with the settings each restored as null defaults.         |
| [`_masked_evaluation_side_kind`](#nemo_gym-rollouts-rollout_recovery-_masked_evaluation_side_kind)             | The evaluation-side kind of a masked result row that is not set aside for another owner; else None.               |
| [`_material_model_call_faults`](#nemo_gym-rollouts-rollout_recovery-_material_model_call_faults)               | Whether server-faulted model calls own the window: the agent's last call faulted, or faulted calls took at        |
| [`_normalize_runtime_references`](#nemo_gym-rollouts-rollout_recovery-_normalize_runtime_references)           | Keep recognized connection references stable wherever clients nest them.                                          |
| [`_null_default_digests`](#nemo_gym-rollouts-rollout_recovery-_null_default_digests)                           | -                                                                                                                 |
| [`_number`](#nemo_gym-rollouts-rollout_recovery-_number)                                                       | -                                                                                                                 |
| [`_plain`](#nemo_gym-rollouts-rollout_recovery-_plain)                                                         | Pydantic's Python dump may still contain nested Hydra containers.                                                 |
| [`_record_resume_drift`](#nemo_gym-rollouts-rollout_recovery-_record_resume_drift)                             | -                                                                                                                 |
| [`_row_failure_kind`](#nemo_gym-rollouts-rollout_recovery-_row_failure_kind)                                   | The selected failure event's kind when the row selects one (None when that event is missing), else the row's.     |
| [`_sandbox_or_tool_fault`](#nemo_gym-rollouts-rollout_recovery-_sandbox_or_tool_fault)                         | Eligible sandbox/session or tool-server evidence classified by the shared mapping.                                |
| [`_server_settings`](#nemo_gym-rollouts-rollout_recovery-_server_settings)                                     | Each server block's settings with its concrete path: server name, kind and implementation.                        |
| [`_without_null_defaults`](#nemo_gym-rollouts-rollout_recovery-_without_null_defaults)                         | The identity without null settings whose default is null: there a null setting and an absent one are the same.    |
| [`_without_runtime_options`](#nemo_gym-rollouts-rollout_recovery-_without_runtime_options)                     | -                                                                                                                 |
| [`applied_failure_class`](#nemo_gym-rollouts-rollout_recovery-applied_failure_class)                           | Read an applied result class, preserving explicit unknown over later configuration.                               |
| [`atomic_output_file`](#nemo_gym-rollouts-rollout_recovery-atomic_output_file)                                 | Replace a file after a successful write, retaining its sharing permissions unless `mode` sets them.               |
| [`atomic_write_json`](#nemo_gym-rollouts-rollout_recovery-atomic_write_json)                                   | Publish JSON metadata without exposing a partial rewrite to readers.                                              |
| [`captured_model_calls`](#nemo_gym-rollouts-rollout_recovery-captured_model_calls)                             | The row's captured model calls in capture order, each reduced to the facts a server fault is read from.           |
| [`config_identity_path_for`](#nemo_gym-rollouts-rollout_recovery-config_identity_path_for)                     | The normalized configuration identity a manifest's `config_digest` was taken from, saved beside it.               |
| [`failure_stage`](#nemo_gym-rollouts-rollout_recovery-failure_stage)                                           | Read selected final context; recovered observations never supply the final stage.                                 |
| [`has_only_intentional_gaps`](#nemo_gym-rollouts-rollout_recovery-has_only_intentional_gaps)                   | Accept omissions only alongside completed tasks, without inventing scores.                                        |
| [`has_recorded_grade`](#nemo_gym-rollouts-rollout_recovery-has_recorded_grade)                                 | Whether the verifier finished grading this failed attempt, so it is a graded result and a rerun would replace it. |
| [`has_saved_generation`](#nemo_gym-rollouts-rollout_recovery-has_saved_generation)                             | Recognize retained response data, including a completed response with empty output.                               |
| [`has_unclassified_execution_failure`](#nemo_gym-rollouts-rollout_recovery-has_unclassified_execution_failure) | A selected route failure names no owned cause that permits running the agent again.                               |
| [`holds_response`](#nemo_gym-rollouts-rollout_recovery-holds_response)                                         | Whether the row keeps a full or partial response object, usable as an answer or not.                              |
| [`is_answered_agent_deadline`](#nemo_gym-rollouts-rollout_recovery-is_answered_agent_deadline)                 | Whether the agent ran out of its own deadline and kept its answer: the evaluated model's final result.            |
| [`is_infrastructure_agent_timeout`](#nemo_gym-rollouts-rollout_recovery-is_infrastructure_agent_timeout)       | Whether an `agent_timeout` row's window held a recorded server-side fault: the evaluation's failure.              |
| [`is_server_fault_call`](#nemo_gym-rollouts-rollout_recovery-is_server_fault_call)                             | Whether eligible recorded model-call failures classify as evaluation-side.                                        |
| [`is_terminal_failure`](#nemo_gym-rollouts-rollout_recovery-is_terminal_failure)                               | Whether a persisted failure must be gated on resume.                                                              |
| [`keeps_partial_response`](#nemo_gym-rollouts-rollout_recovery-keeps_partial_response)                         | Whether the row keeps the agent's partial response from a native cleanup, even one without output.                |
| [`load_attempt_extension`](#nemo_gym-rollouts-rollout_recovery-load_attempt_extension)                         | Attempts an explicit recovery added to every rollout's saved attempt cap (`--extend-attempts`); 0 when none.      |
| [`load_rerun_masked_approvals`](#nemo_gym-rollouts-rollout_recovery-load_rerun_masked_approvals)               | The masked results an operator approved to run again (`--rerun-masked-without-artifact`): each approved           |
| [`load_resubmit_offset`](#nemo_gym-rollouts-rollout_recovery-load_resubmit_offset)                             | Read the number of sidecar lines excluded from the current failure budget.                                        |
| [`manifest_path_for`](#nemo_gym-rollouts-rollout_recovery-manifest_path_for)                                   | -                                                                                                                 |
| [`masked_infrastructure_kind`](#nemo_gym-rollouts-rollout_recovery-masked_infrastructure_kind)                 | The kind of a masked result that infrastructure, not the evaluated model, left unmeasured; else None.             |
| [`masked_kept_answer_kind`](#nemo_gym-rollouts-rollout_recovery-masked_kept_answer_kind)                       | The kind of a masked result that `masked_infrastructure_kind` rejects only for its saved answer and that has      |
| [`matches_failure_class`](#nemo_gym-rollouts-rollout_recovery-matches_failure_class)                           | Match shared failure names and retained producer routing names alike.                                             |
| [`observed_elapsed`](#nemo_gym-rollouts-rollout_recovery-observed_elapsed)                                     | Best-effort per-rollout wallclock from a result/failure row.                                                      |
| [`record_resubmit_offset`](#nemo_gym-rollouts-rollout_recovery-record_resubmit_offset)                         | Reset the failure budget without altering outcomes or execution identities.                                       |
| [`resume_drift_path_for`](#nemo_gym-rollouts-rollout_recovery-resume_drift_path_for)                           | One line per resume that proceeded across operational configuration drift, naming the keys.                       |
| [`run_lock`](#nemo_gym-rollouts-rollout_recovery-run_lock)                                                     | Own a run before preparing inputs, restoring state, or writing artifacts.                                         |
| [`run_lock_path_for`](#nemo_gym-rollouts-rollout_recovery-run_lock_path_for)                                   | Stable lock file; never unlink it while another process may hold it open.                                         |
| [`saved_failure_classification`](#nemo_gym-rollouts-rollout_recovery-saved_failure_classification)             | Read only the owning launcher's saved classification in its conventional artifact layout.                         |
| [`validate_resume`](#nemo_gym-rollouts-rollout_recovery-validate_resume)                                       | Reject unknown or incompatible saved work before any output is modified.                                          |

### Data

[`AGENT_TIMEOUT_FAULT_SHARE`](#nemo_gym-rollouts-rollout_recovery-AGENT_TIMEOUT_FAULT_SHARE)

[`COLLECTED_MODEL_CALLS_KEY`](#nemo_gym-rollouts-rollout_recovery-COLLECTED_MODEL_CALLS_KEY)

[`SERVER_FAULT_CATEGORIES`](#nemo_gym-rollouts-rollout_recovery-SERVER_FAULT_CATEGORIES)

[`_ADDED_COLLECTION_DEFAULTS`](#nemo_gym-rollouts-rollout_recovery-_ADDED_COLLECTION_DEFAULTS)

[`_ASSIGNMENT`](#nemo_gym-rollouts-rollout_recovery-_ASSIGNMENT)

[`_COLLECTION_OPTIONS`](#nemo_gym-rollouts-rollout_recovery-_COLLECTION_OPTIONS)

[`_CONFIG_IDENTITIES`](#nemo_gym-rollouts-rollout_recovery-_CONFIG_IDENTITIES)

[`_CONFIG_REFERENCE`](#nemo_gym-rollouts-rollout_recovery-_CONFIG_REFERENCE)

[`_CREDENTIAL_NAME`](#nemo_gym-rollouts-rollout_recovery-_CREDENTIAL_NAME)

[`_DEFAULT_MAX_ROLLOUT_ATTEMPTS`](#nemo_gym-rollouts-rollout_recovery-_DEFAULT_MAX_ROLLOUT_ATTEMPTS)

[`_ENV_REFERENCE`](#nemo_gym-rollouts-rollout_recovery-_ENV_REFERENCE)

[`_HTTP_CREDENTIAL`](#nemo_gym-rollouts-rollout_recovery-_HTTP_CREDENTIAL)

[`_LEGACY_CONFIG_DIGESTS`](#nemo_gym-rollouts-rollout_recovery-_LEGACY_CONFIG_DIGESTS)

[`_LEGACY_SUBSET_LIMIT`](#nemo_gym-rollouts-rollout_recovery-_LEGACY_SUBSET_LIMIT)

[`_MASKED_REGRADE_KINDS`](#nemo_gym-rollouts-rollout_recovery-_MASKED_REGRADE_KINDS)

[`_MATERIAL_CALL_FAILURE_KINDS`](#nemo_gym-rollouts-rollout_recovery-_MATERIAL_CALL_FAILURE_KINDS)

[`_OPERATIONAL_SETTING`](#nemo_gym-rollouts-rollout_recovery-_OPERATIONAL_SETTING)

[`_REDACTED`](#nemo_gym-rollouts-rollout_recovery-_REDACTED)

[`_REMOVED_NULL_DEFAULTS`](#nemo_gym-rollouts-rollout_recovery-_REMOVED_NULL_DEFAULTS)

[`_SEMANTIC_ASSET_PATH`](#nemo_gym-rollouts-rollout_recovery-_SEMANTIC_ASSET_PATH)

[`_SEMANTIC_KEYS`](#nemo_gym-rollouts-rollout_recovery-_SEMANTIC_KEYS)

[`_SEMANTIC_PARTS`](#nemo_gym-rollouts-rollout_recovery-_SEMANTIC_PARTS)

[`_SERVER_KINDS`](#nemo_gym-rollouts-rollout_recovery-_SERVER_KINDS)

[`_SERVER_RUNTIME_OPTIONS`](#nemo_gym-rollouts-rollout_recovery-_SERVER_RUNTIME_OPTIONS)

[`logger`](#nemo_gym-rollouts-rollout_recovery-logger)

### API

```python
class nemo_gym.rollouts.rollout_recovery.IncompleteEvaluationError(
    message: str,
    retryable: bool = True,
    made_progress: bool = True,
    failure_kind: str | None = None
)
```

**Bases:** `RuntimeError`

Saved partial results are valid, but the evaluation still has unfinished work.

**`exit_code`** `= (75 if made_progress else 1) if retryable else 76`

---

```python
class nemo_gym.rollouts.rollout_recovery.PendingRejudge()
```

**Bases:** `BaseModel`

A judge-only re-judgment admitted before its judge call; it closes when its outcome is saved.

**`admitted_at`** `Annotated[float, Field(ge=0, allow_inf_nan=False, strict=True)]`

---

**`model_config`** `= ConfigDict(extra='forbid', frozen=True)`

---

**`operation_id`** `Annotated[str, Field(pattern='^[0-9a-f]{32}$')]`

---

**`predecessor_failure_id`** `Annotated[str, Field(min_length=1)]`

---

**`predecessor_sha256`** `Annotated[str, Field(pattern='^[0-9a-f]{64}$')]`

---

```python
class nemo_gym.rollouts.rollout_recovery.ResumeRefusedError()
```

**Bases:** [ConfigError](/nemo/gym/nemo-gym/nemo_gym/config/config_types#nemo_gym-config-config_types-ConfigError)

Saved rollouts whose identity does not verify against the current run are not resumed.

```python
class nemo_gym.rollouts.rollout_recovery.RunManifest()
```

**Bases:** `BaseModel`

**`config_digest`** `str`

---

**`identity_overridden`** `bool = False`

---

**`legacy_import`** `bool = False`

---

**`materialized_digest`** `str`

---

**`max_rollout_attempts`** `Annotated[int, Field(ge=1, strict=True)] | None = None`

---

**`model_config`** `= ConfigDict(extra='forbid', frozen=True)`

---

**`next_attempt`** `dict[str, Annotated[int, Field(ge=1, strict=True)]] = Field(default_factory=dict)`

---

**`pending_rejudge`** `dict[str, PendingRejudge] = Field(default_factory=dict)`

---

**`retry_terminal_timeouts`** `bool = False`

---

**`run_id`** `str`

---

**`schema_version`** `Literal[2] = 2`

---

**`selection_policy`** `Literal['latest_allocated'] = 'latest_allocated'`

---

**`source_digest`** `str`

---

```python
nemo_gym.rollouts.rollout_recovery.RunManifest.create(
    source: pathlib.Path,
    rows: list[dict],
    config: dict,
    global_config: dict
) -> nemo_gym.rollouts.rollout_recovery.RunManifest
```

classmethod

```python
nemo_gym.rollouts.rollout_recovery.RunManifest.import_legacy(
    rows: list[dict]
) -> nemo_gym.rollouts.rollout_recovery.RunManifest
```

classmethod

```python
nemo_gym.rollouts.rollout_recovery.RunManifest.matches_config(
    saved_digest: str
) -> bool
```

Whether a saved configuration digest names this configuration, in this or an earlier recorded form.

```python
nemo_gym.rollouts.rollout_recovery.RunManifest.write(
    path: pathlib.Path
) -> None
```

```python
nemo_gym.rollouts.rollout_recovery._configuration_drift(
    path: pathlib.Path,
    saved: nemo_gym.rollouts.rollout_recovery.RunManifest,
    current: nemo_gym.rollouts.rollout_recovery.RunManifest
) -> tuple[list[str], list[str]] | None
```

The semantic and operational keys that differ from the saved identity, or None when it was not saved.

The saved identity counts only when it is the one the manifest's `config_digest` was taken from.

```python
nemo_gym.rollouts.rollout_recovery._configuration_identity(
    config: dict,
    global_config: dict,
    rows: list[dict],
    credential_safe: bool = True
) -> dict
```

The configuration a run's results depend on, compared on resume.

Credentials are redacted (see `_credential_safe`): they do not change what a run measures, and the identity is
persisted beside the manifest. `credential_safe=False` is the form an earlier Gym took its digest from; it is only
ever compared in memory, to resume such a run.

```python
nemo_gym.rollouts.rollout_recovery._credential_safe(
    value: typing.Any,
    secret: bool = False
) -> typing.Any
```

The value with every credential replaced by a fixed marker; it is what a run persists for its identity.

A credential is any value below a credential-named key (every value of a `secrets` mapping, an `Authorization`
header), the `value` of a `{name: SOME_TOKEN, value: ...}` entry, the value of a `SOME_TOKEN=...` string, an
HTTP `Bearer`/`Basic` credential, a URL password, and an interpolation of a credential-named environment variable
or setting. Applied before interpolations are resolved, so such a credential is never read from the environment.

```python
nemo_gym.rollouts.rollout_recovery._differing_keys(
    saved: typing.Any,
    current: typing.Any,
    prefix: str = ''
) -> list[str]
```

Dotted keys whose values differ between two identities; a list is one value.

```python
nemo_gym.rollouts.rollout_recovery._digest(
    value: typing.Any
) -> str
```

```python
nemo_gym.rollouts.rollout_recovery._earlier_collection_forms(
    identity: dict,
    config: collections.abc.Mapping[str, typing.Any]
) -> collections.abc.Iterator[dict]
```

The identity as the Gyms before and after each added collection setting recorded it.

First the identity as it is (a Gym before the setting existed), then with the setting restored at its default and at
the value this configuration sets. A run saved with another value is still compared on its saved identity, if any.

```python
nemo_gym.rollouts.rollout_recovery._evaluation_event(
    event: collections.abc.Mapping[str, typing.Any],
    classification: nemo_gym.core.failure_kinds.FailureClassificationConfig | None = None
) -> bool
```

```python
nemo_gym.rollouts.rollout_recovery._file_digest(
    path: pathlib.Path
) -> str
```

```python
nemo_gym.rollouts.rollout_recovery._get_max_rollout_attempts() -> int
```

Read `NEMO_GYM_MAX_ROLLOUT_ATTEMPTS` (positive int) or default to 3.

```python
nemo_gym.rollouts.rollout_recovery._is_credential_name(
    name: object
) -> bool
```

```python
nemo_gym.rollouts.rollout_recovery._is_semantic(
    key: str
) -> bool
```

```python
nemo_gym.rollouts.rollout_recovery._keeps_saved_answer(
    row: collections.abc.Mapping[str, typing.Any]
) -> bool
```

Whether the masked result holds a model answer a rerun would replace: never sampled again.

Same facts as recovery's saved-answer test: a collector projection's own flag, else the retained generation of the
response (`has_saved_generation`, which leaves out a no-generation placeholder) or of a native partial response.

```python
nemo_gym.rollouts.rollout_recovery._legacy_config_digests(
    identity: dict,
    config: collections.abc.Mapping[str, typing.Any] | None = None
) -> dict[str, tuple[str, ...]]
```

Digests an earlier Gym recorded for this configuration, with the settings each restored as null defaults.

Restoration is per concrete server path: a removed default is added only to a server that lacks it, never over a
value another server of the same implementation sets. Each form is also tried with the collection settings added
since restored (`_earlier_collection_forms`).

```python
nemo_gym.rollouts.rollout_recovery._masked_evaluation_side_kind(
    row: collections.abc.Mapping[str, typing.Any],
    classification: nemo_gym.core.failure_kinds.FailureClassificationConfig | None = None
) -> str | None
```

The evaluation-side kind of a masked result row that is not set aside for another owner; else None.

The rules `masked_infrastructure_kind` and `masked_kept_answer_kind` share, all but the saved answer: the row
is masked by the Environment Server that graded it, is no failure row, omitted stand-in or re-judgment, and its
`failure_kind` is registered, evaluation-side through the shared classifier, not one whose saved answer is graded
again (`_MASKED_REGRADE_KINDS`) and not a `transport_*` verify call.

```python
nemo_gym.rollouts.rollout_recovery._material_model_call_faults(
    row: collections.abc.Mapping[str, typing.Any],
    classification: nemo_gym.core.failure_kinds.FailureClassificationConfig | None = None
) -> bool
```

Whether server-faulted model calls own the window: the agent's last call faulted, or faulted calls took at
least `AGENT_TIMEOUT_FAULT_SHARE` of it. The window is the attempt's recorded elapsed time, else the span of its
captured calls; an early fault that a retry recovered within a long window is the model's time, not ours.

```python
nemo_gym.rollouts.rollout_recovery._normalize_runtime_references(
    value: typing.Any
) -> typing.Any
```

Keep recognized connection references stable wherever clients nest them.

Literal nested task data is untouched, including fields named port or api_key.
Preserve the reference name and any surrounding URL template in the identity.

```python
nemo_gym.rollouts.rollout_recovery._null_default_digests(
    identity: dict
) -> dict[str, tuple[str, ...]]
```

```python
nemo_gym.rollouts.rollout_recovery._number(
    value: typing.Any
) -> float | None
```

```python
nemo_gym.rollouts.rollout_recovery._plain(
    value: typing.Any
) -> typing.Any
```

Pydantic's Python dump may still contain nested Hydra containers.

```python
nemo_gym.rollouts.rollout_recovery._record_resume_drift(
    path: pathlib.Path,
    saved: nemo_gym.rollouts.rollout_recovery.RunManifest,
    current: nemo_gym.rollouts.rollout_recovery.RunManifest,
    keys: list[str]
) -> None
```

```python
nemo_gym.rollouts.rollout_recovery._row_failure_kind(
    row: collections.abc.Mapping[str, typing.Any]
) -> typing.Any
```

The selected failure event's kind when the row selects one (None when that event is missing), else the row's.

```python
nemo_gym.rollouts.rollout_recovery._sandbox_or_tool_fault(
    row: collections.abc.Mapping[str, typing.Any],
    classification: nemo_gym.core.failure_kinds.FailureClassificationConfig | None = None
) -> bool
```

Eligible sandbox/session or tool-server evidence classified by the shared mapping.

Cleanup and verification lie outside the agent's execution window. Registered provider/session kinds
identify candidate observations; prefixes do not assign responsibility. For example agent OOM remains
model-side while seed OOM is evaluation-side when its originating stage is available.

```python
nemo_gym.rollouts.rollout_recovery._server_settings(
    identity: dict
) -> collections.abc.Iterator[tuple[str, str, str, dict]]
```

Each server block's settings with its concrete path: server name, kind and implementation.

```python
nemo_gym.rollouts.rollout_recovery._without_null_defaults(
    identity: dict
) -> dict
```

The identity without null settings whose default is null: there a null setting and an absent one are the same.

Elsewhere null is a value of its own (claude_code_agent's `max_turns: null` is unlimited, its omission 30
turns), so it stays in the identity.

```python
nemo_gym.rollouts.rollout_recovery._without_runtime_options(
    value: typing.Any
) -> typing.Any
```

```python
nemo_gym.rollouts.rollout_recovery.applied_failure_class(
    row: collections.abc.Mapping[str, typing.Any]
) -> nemo_gym.core.failure_kinds.FailureClass | None
```

Read an applied result class, preserving explicit unknown over later configuration.

```python
nemo_gym.rollouts.rollout_recovery.atomic_output_file(
    path: pathlib.Path,
    mode: int | None = None
) -> collections.abc.Iterator[typing.BinaryIO]
```

Replace a file after a successful write, retaining its sharing permissions unless `mode` sets them.

```python
nemo_gym.rollouts.rollout_recovery.atomic_write_json(
    path: pathlib.Path,
    value: dict,
    mode: int | None = None
) -> None
```

Publish JSON metadata without exposing a partial rewrite to readers.

```python
nemo_gym.rollouts.rollout_recovery.captured_model_calls(
    row: collections.abc.Mapping[str, typing.Any]
) -> list[dict[str, typing.Any]]
```

The row's captured model calls in capture order, each reduced to the facts a server fault is read from.

`status_code`, `upstream_status_code`, `error_category`, `failure_events` (each failure-report event's
`failure_kind` and `http_status`), `started_at` and `duration_ms`. Read from the collector's projection
when the row is one, else from `ng_model_call_capture.calls`, else from `ng_trajectory.model_calls`.

```python
nemo_gym.rollouts.rollout_recovery.config_identity_path_for(
    manifest_path: pathlib.Path
) -> pathlib.Path
```

The normalized configuration identity a manifest's `config_digest` was taken from, saved beside it.

```python
nemo_gym.rollouts.rollout_recovery.failure_stage(
    row: collections.abc.Mapping[str, typing.Any]
) -> nemo_gym.core.failure_kinds.FailureStage | None
```

Read selected final context; recovered observations never supply the final stage.

```python
nemo_gym.rollouts.rollout_recovery.has_only_intentional_gaps(
    expected: int,
    completed: int,
    intentionally_omitted: int
) -> bool
```

Accept omissions only alongside completed tasks, without inventing scores.

```python
nemo_gym.rollouts.rollout_recovery.has_recorded_grade(
    row: collections.abc.Mapping[str, typing.Any]
) -> bool
```

Whether the verifier finished grading this failed attempt, so it is a graded result and a rerun would replace it.

Collection keeps no reward on a failure row (a placeholder score is never a result), so the completed
evaluation is the grade's durable record. Only a row with neither a kept answer nor a recorded grade may run fresh.

```python
nemo_gym.rollouts.rollout_recovery.has_saved_generation(
    row: collections.abc.Mapping[str, typing.Any]
) -> bool
```

Recognize retained response data, including a completed response with empty output.

A diagnostic error object or an empty mapping is not generation. A typed response counts when it has at least
one output item or a status: a producer finished it, even with no assistant text. A response object with no
output and no status is a placeholder, for example from an agent that never ran. So is a response an agent
marked with `NG_NO_GENERATION_METADATA`: its empty assistant message was built without any generation.

```python
nemo_gym.rollouts.rollout_recovery.has_unclassified_execution_failure(
    row: collections.abc.Mapping[str, typing.Any]
) -> bool
```

A selected route failure names no owned cause that permits running the agent again.

A request that was possibly delivered permits a fresh run only while the row keeps no full or partial
response: then no saved answer can be duplicated or replaced.

```python
nemo_gym.rollouts.rollout_recovery.holds_response(
    row: collections.abc.Mapping[str, typing.Any]
) -> bool
```

Whether the row keeps a full or partial response object, usable as an answer or not.

```python
nemo_gym.rollouts.rollout_recovery.is_answered_agent_deadline(
    row: collections.abc.Mapping[str, typing.Any],
    classification: nemo_gym.core.failure_kinds.FailureClassificationConfig | None = None
) -> bool
```

Whether the agent ran out of its own deadline and kept its answer: the evaluated model's final result.

The deadline belongs to the agent, so what it kept is its result, scored by the benchmark's rule (a recorded
grade, else a settled zero). Recovery classifies it as final, and no resume or in-process retry runs it again:
that would replace the model's answer. The kind is the selected failure event's when the row selects one, else
the row's own. A timeout that kept no answer keeps its retry policy, and so does one whose window held a recorded
server-side fault (`is_infrastructure_agent_timeout`): what it kept was cut short by us, not by its deadline.

```python
nemo_gym.rollouts.rollout_recovery.is_infrastructure_agent_timeout(
    row: collections.abc.Mapping[str, typing.Any],
    classification: nemo_gym.core.failure_kinds.FailureClassificationConfig | None = None
) -> bool
```

Whether an `agent_timeout` row's window held a recorded server-side fault: the evaluation's failure.

The row (a failure row, or a graded result such as Harbor's) names `agent_timeout` and either recorded a sandbox
or tool-server failure event (`_sandbox_or_tool_fault`), or its captured model calls show a server fault
(`is_server_fault_call`) that owns the window (`_material_model_call_faults`). A clean agent timeout stays the
model's result under the benchmark's time limit. Read from saved rows alone, so it applies to existing runs.

```python
nemo_gym.rollouts.rollout_recovery.is_server_fault_call(
    call: collections.abc.Mapping[str, typing.Any],
    classification: nemo_gym.core.failure_kinds.FailureClassificationConfig | None = None
) -> bool
```

Whether eligible recorded model-call failures classify as evaluation-side.

Temporal materiality excludes timeout, cancellation and truncated-stream evidence: a slow model or an
agent cancelling its current call does not establish time spent on infrastructure. HTTP evidence and the
eligible transport errors use the shared classifier; classification alone does not establish materiality.

```python
nemo_gym.rollouts.rollout_recovery.is_terminal_failure(
    record: collections.abc.Mapping[str, typing.Any],
    retry_terminal_timeouts: bool = False
) -> bool
```

Whether a persisted failure must be gated on resume.

By default a row is terminal iff it is stamped `_ng_failure_terminal`, so
an agent that marks its timeouts terminal on purpose keeps them gated.

`retry_terminal_timeouts` is for agents whose older builds incorrectly
stamped per-attempt timeouts terminal. A timeout reflects the
load/remaining walltime of that attempt, so it remains retryable (up to the
normal max-attempt cap).

`timeout_exceeded` is an `_ng_failure_class` label that agents already
write to the failures sidecar (pinchbench, nooa). It predates
`nemo_gym.core.failure_kinds` and is not registered there; `agent_timeout` is
the shared name.

As `failure_kinds` requires, this function decides retryability for the
occurrence, under an explicit caller opt-in; the name carries no retry meaning.

```python
nemo_gym.rollouts.rollout_recovery.keeps_partial_response(
    row: collections.abc.Mapping[str, typing.Any]
) -> bool
```

Whether the row keeps the agent's partial response from a native cleanup, even one without output.

The agent ran, so its partial response stays a kept answer and is never regenerated. An agent-stage response
the outcome policy rejected as failed is diagnostic only, as in `has_saved_generation`.

```python
nemo_gym.rollouts.rollout_recovery.load_attempt_extension(
    output_fpath: str | pathlib.Path
) -> int
```

Attempts an explicit recovery added to every rollout's saved attempt cap (`--extend-attempts`); 0 when none.

A rollout's cap is the run's saved `max_rollout_attempts` plus this. An unreadable record stops the run rather
than guess its cap.

```python
nemo_gym.rollouts.rollout_recovery.load_rerun_masked_approvals(
    output_fpath: str | pathlib.Path
) -> frozenset[tuple[int, int, int]]
```

The masked results an operator approved to run again (`--rerun-masked-without-artifact`): each approved
attempt as (task index, rollout index, attempt index); empty when none.

An approval names the attempt it covered, so a later attempt of the same rollout is never reopened by it. A line
or entry that cannot be read approves nothing: the default is that a saved answer is never sampled again.

```python
nemo_gym.rollouts.rollout_recovery.load_resubmit_offset(
    output_fpath: str | pathlib.Path
) -> int
```

Read the number of sidecar lines excluded from the current failure budget.

```python
nemo_gym.rollouts.rollout_recovery.manifest_path_for(
    output: pathlib.Path
) -> pathlib.Path
```

```python
nemo_gym.rollouts.rollout_recovery.masked_infrastructure_kind(
    row: collections.abc.Mapping[str, typing.Any],
    classification: nemo_gym.core.failure_kinds.FailureClassificationConfig | None = None
) -> str | None
```

The kind of a masked result that infrastructure, not the evaluated model, left unmeasured; else None.

A result row is masked (`mask_sample`) by the Environment Server that graded it, so its `failure_kind` was
reported at that boundary: a lost session, a failed provider, an infrastructure error, a verifier that could not
run. When the kind is evaluation-side through the shared classifier the rollout is retried like an infrastructure
failure, within its attempt cap: the masked row counts one failed attempt and a later attempt supersedes it. A
model-class kind, no kind, an unregistered or namespaced kind, and the kinds whose saved answer is graded again
(`_MASKED_REGRADE_KINDS`, including a `transport_*` verify call) keep the masked row as the result. So does a
saved answer (`_keeps_saved_answer`): it is never sampled again, whatever its verifier could grade of it.

```python
nemo_gym.rollouts.rollout_recovery.masked_kept_answer_kind(
    row: collections.abc.Mapping[str, typing.Any],
    classification: nemo_gym.core.failure_kinds.FailureClassificationConfig | None = None
) -> str | None
```

The kind of a masked result that `masked_infrastructure_kind` rejects only for its saved answer and that has
no recorded grade (`has_recorded_grade`); else None.

The part of `masked_without_artifact_kind` that the Store can decide: the planner adds that no re-verification
cause applies. A graded result is a result and stays one.

```python
nemo_gym.rollouts.rollout_recovery.matches_failure_class(
    record: collections.abc.Mapping[str, typing.Any],
    classes: collections.abc.Collection[str]
) -> bool
```

Match shared failure names and retained producer routing names alike.

```python
nemo_gym.rollouts.rollout_recovery.observed_elapsed(
    record: dict[str, typing.Any]
) -> float | None
```

Best-effort per-rollout wallclock from a result/failure row.

```python
nemo_gym.rollouts.rollout_recovery.record_resubmit_offset(
    output_fpath: str | pathlib.Path
) -> int
```

Reset the failure budget without altering outcomes or execution identities.

The caller must hold the run lock and validate/repair history first. The
line-based format is shared with GDPVal's existing resubmit implementation.

```python
nemo_gym.rollouts.rollout_recovery.resume_drift_path_for(
    manifest_path: pathlib.Path
) -> pathlib.Path
```

One line per resume that proceeded across operational configuration drift, naming the keys.

```python
nemo_gym.rollouts.rollout_recovery.run_lock(
    output: pathlib.Path,
    checkpoint_dir: pathlib.Path | None = None
) -> collections.abc.Iterator[None]
```

Own a run before preparing inputs, restoring state, or writing artifacts.

Collection currently acquires its own lock in run_from_config; do not nest
it inside this context. Checkpoint integration must extend that acquisition to
cover restore and supply checkpoint_dir, rather than acquire a second lock.
When supplied, the directory lock prevents two outputs sharing checkpoints.
The filesystem must support flock across all writer nodes. Kernel locks are released
on process death; the presence of an old lock file does not block recovery.

```python
nemo_gym.rollouts.rollout_recovery.run_lock_path_for(
    output: pathlib.Path
) -> pathlib.Path
```

Stable lock file; never unlink it while another process may hold it open.

```python
nemo_gym.rollouts.rollout_recovery.saved_failure_classification(
    output: pathlib.Path
) -> nemo_gym.core.failure_kinds.FailureClassificationConfig
```

Read only the owning launcher's saved classification in its conventional artifact layout.

Standalone outputs have no launcher policy and use shared defaults. A malformed saved policy is never
silently replaced with defaults. Applied classes on individual results still take precedence.

```python
nemo_gym.rollouts.rollout_recovery.validate_resume(
    path: pathlib.Path,
    current: nemo_gym.rollouts.rollout_recovery.RunManifest | None,
    materialized_path: pathlib.Path,
    allow_unsafe: bool = False
) -> nemo_gym.rollouts.rollout_recovery.RunManifest | None
```

Reject unknown or incompatible saved work before any output is modified.

A configuration that differs from the saved one is compared on the identity saved beside the manifest: a
difference only in operational settings (timeouts, retries, transport, ports, concurrency, health, paths,
tracking) resumes, with a warning and a line in the resume-drift journal; a difference in a semantic setting
(dataset, task limits, repeats, agent and model identity, sampling, judge and grading) refuses and names it. A
manifest saved without that identity can only be compared by digest, as before.

`allow_unsafe` is an explicit compatibility escape hatch for legacy artifacts or
deliberate identity overrides. Unsupported/corrupt manifests remain errors.

```python
nemo_gym.rollouts.rollout_recovery.AGENT_TIMEOUT_FAULT_SHARE = 0.25
```

```python
nemo_gym.rollouts.rollout_recovery.COLLECTED_MODEL_CALLS_KEY = '_ng_collected_model_calls'
```

```python
nemo_gym.rollouts.rollout_recovery.SERVER_FAULT_CATEGORIES = frozenset({'upstream_error', 'connection'})
```

```python
nemo_gym.rollouts.rollout_recovery._ADDED_COLLECTION_DEFAULTS: dict[str, float] = {'dispatch_connect_grace_s': 60.0}
```

```python
nemo_gym.rollouts.rollout_recovery._ASSIGNMENT = re.compile('^([A-Za-z_][A-Za-z0-9_]*)=(.+)$', re.DOTALL)
```

```python
nemo_gym.rollouts.rollout_recovery._COLLECTION_OPTIONS = frozenset({'resume_from_cache', 'resubmit', 'checkpoint_dir', 'checkpoint_every_...
```

```python
nemo_gym.rollouts.rollout_recovery._CONFIG_IDENTITIES: dict[str, dict] = {}
```

```python
nemo_gym.rollouts.rollout_recovery._CONFIG_REFERENCE = re.compile('\\$\\{([A-Za-z_][\\w.\\-]*)\\}')
```

```python
nemo_gym.rollouts.rollout_recovery._CREDENTIAL_NAME = re.compile('api_?key|secret|passw(or)?d|credential|private_?key|access_?key|cook...
```

```python
nemo_gym.rollouts.rollout_recovery._DEFAULT_MAX_ROLLOUT_ATTEMPTS = 3
```

```python
nemo_gym.rollouts.rollout_recovery._ENV_REFERENCE = re.compile('\\$\\{oc\\.env:\\s*([A-Za-z_][A-Za-z0-9_]*)[^{}]*\\}')
```

```python
nemo_gym.rollouts.rollout_recovery._HTTP_CREDENTIAL = re.compile('(?i)^(bearer|basic)\\s+\\S+$')
```

```python
nemo_gym.rollouts.rollout_recovery._LEGACY_CONFIG_DIGESTS: dict[str, dict[str, tuple[str, ...]]] = {}
```

```python
nemo_gym.rollouts.rollout_recovery._LEGACY_SUBSET_LIMIT = 10
```

```python
nemo_gym.rollouts.rollout_recovery._MASKED_REGRADE_KINDS = frozenset({JUDGE_FAILED, JUDGE_UNPARSEABLE, AGENT_ROUTE_FAILED})
```

```python
nemo_gym.rollouts.rollout_recovery._MATERIAL_CALL_FAILURE_KINDS = frozenset({MODEL_TRANSPORT_ERROR, MODEL_SERVER_ERROR, TRANSPORT_UNREACHABLE, TRA...
```

```python
nemo_gym.rollouts.rollout_recovery._OPERATIONAL_SETTING = re.compile('(^|_)(timeout|timeouts|deadline|retry|retries|attempts|backoff|wait|...
```

```python
nemo_gym.rollouts.rollout_recovery._REDACTED = '<redacted>'
```

```python
nemo_gym.rollouts.rollout_recovery._REMOVED_NULL_DEFAULTS: dict[tuple[str, str], tuple[str, ...]] = {('responses_api_models', 'vllm_model'): ('chat_template_kwargs', 'sampling_over...
```

```python
nemo_gym.rollouts.rollout_recovery._SEMANTIC_ASSET_PATH = re.compile('(^|_)(prompt|prompts|template|templates|rubric|rubrics)_(.+_)?(dir|p...
```

```python
nemo_gym.rollouts.rollout_recovery._SEMANTIC_KEYS = frozenset({'split', 'dataset', 'datasets', 'limit', 'task_ids', 'num_repeats', '...
```

```python
nemo_gym.rollouts.rollout_recovery._SEMANTIC_PARTS = ('judge', 'grad', 'verif', 'reward', 'scor', 'rubric', 'metric', 'prompt', 'mode...
```

```python
nemo_gym.rollouts.rollout_recovery._SERVER_KINDS = ('responses_api_agents', 'responses_api_models', 'resources_servers', 'environme...
```

```python
nemo_gym.rollouts.rollout_recovery._SERVER_RUNTIME_OPTIONS = frozenset({'host', 'port', 'api_key', 'openai_api_key', 'policy_api_key', 'judge...
```

```python
nemo_gym.rollouts.rollout_recovery.logger = logging.getLogger(__name__)
```