nemo_gym.rollouts.rollout_recovery

Bases: RuntimeError

Saved partial results are valid, but the evaluation still has unfinished work.

Bases: BaseModel

A judge-only re-judgment admitted before its judge call; it closes when its outcome is saved.

Bases: ConfigError

Saved rollouts whose identity does not verify against the current run are not resumed.

Bases: BaseModel

nemo_gym.rollouts.rollout_recovery.RunManifest.create(
    source: pathlib.Path,
    rows: list[dict],
    config: dict,
    global_config: dict
) -> nemo_gym.rollouts.rollout_recovery.RunManifest

nemo_gym.rollouts.rollout_recovery.RunManifest.import_legacy(
    rows: list[dict]
) -> nemo_gym.rollouts.rollout_recovery.RunManifest

nemo_gym.rollouts.rollout_recovery.RunManifest.matches_config(
    saved_digest: str
) -> bool

nemo_gym.rollouts.rollout_recovery.RunManifest.write(
    path: pathlib.Path
) -> None

The semantic and operational keys that differ from the saved identity, or None when it was not saved.

The saved identity counts only when it is the one the manifest’s config_digest was taken from.

The configuration a run’s results depend on, compared on resume.

Credentials are redacted (see _credential_safe): they do not change what a run measures, and the identity is persisted beside the manifest. credential_safe=False is the form an earlier Gym took its digest from; it is only ever compared in memory, to resume such a run.

The value with every credential replaced by a fixed marker; it is what a run persists for its identity.

A credential is any value below a credential-named key (every value of a secrets mapping, an Authorization header), the value of a {name: SOME_TOKEN, value: ...} entry, the value of a SOME_TOKEN=... string, an HTTP Bearer/Basic credential, a URL password, and an interpolation of a credential-named environment variable or setting. Applied before interpolations are resolved, so such a credential is never read from the environment.

Dotted keys whose values differ between two identities; a list is one value.

The identity as the Gyms before and after each added collection setting recorded it.

First the identity as it is (a Gym before the setting existed), then with the setting restored at its default and at the value this configuration sets. A run saved with another value is still compared on its saved identity, if any.

Read NEMO_GYM_MAX_ROLLOUT_ATTEMPTS (positive int) or default to 3.

Whether the masked result holds a model answer a rerun would replace: never sampled again.

Same facts as recovery’s saved-answer test: a collector projection’s own flag, else the retained generation of the response (has_saved_generation, which leaves out a no-generation placeholder) or of a native partial response.

Digests an earlier Gym recorded for this configuration, with the settings each restored as null defaults.

Restoration is per concrete server path: a removed default is added only to a server that lacks it, never over a value another server of the same implementation sets. Each form is also tried with the collection settings added since restored (_earlier_collection_forms).

The evaluation-side kind of a masked result row that is not set aside for another owner; else None.

The rules masked_infrastructure_kind and masked_kept_answer_kind share, all but the saved answer: the row is masked by the Environment Server that graded it, is no failure row, omitted stand-in or re-judgment, and its failure_kind is registered, evaluation-side through the shared classifier, not one whose saved answer is graded again (_MASKED_REGRADE_KINDS) and not a transport_* verify call.

Whether server-faulted model calls own the window: the agent’s last call faulted, or faulted calls took at least AGENT_TIMEOUT_FAULT_SHARE of it. The window is the attempt’s recorded elapsed time, else the span of its captured calls; an early fault that a retry recovered within a long window is the model’s time, not ours.

Keep recognized connection references stable wherever clients nest them.

Literal nested task data is untouched, including fields named port or api_key. Preserve the reference name and any surrounding URL template in the identity.

Pydantic’s Python dump may still contain nested Hydra containers.

The selected failure event’s kind when the row selects one (None when that event is missing), else the row’s.

Eligible sandbox/session or tool-server evidence classified by the shared mapping.

Cleanup and verification lie outside the agent’s execution window. Registered provider/session kinds identify candidate observations; prefixes do not assign responsibility. For example agent OOM remains model-side while seed OOM is evaluation-side when its originating stage is available.

Each server block’s settings with its concrete path: server name, kind and implementation.

The identity without null settings whose default is null: there a null setting and an absent one are the same.

Elsewhere null is a value of its own (claude_code_agent’s max_turns: null is unlimited, its omission 30 turns), so it stays in the identity.

Read an applied result class, preserving explicit unknown over later configuration.

Replace a file after a successful write, retaining its sharing permissions unless mode sets them.

Publish JSON metadata without exposing a partial rewrite to readers.

The row’s captured model calls in capture order, each reduced to the facts a server fault is read from.

status_code, upstream_status_code, error_category, failure_events (each failure-report event’s failure_kind and http_status), started_at and duration_ms. Read from the collector’s projection when the row is one, else from ng_model_call_capture.calls, else from ng_trajectory.model_calls.

The normalized configuration identity a manifest’s config_digest was taken from, saved beside it.

Read selected final context; recovered observations never supply the final stage.

Accept omissions only alongside completed tasks, without inventing scores.

Whether the verifier finished grading this failed attempt, so it is a graded result and a rerun would replace it.

Collection keeps no reward on a failure row (a placeholder score is never a result), so the completed evaluation is the grade’s durable record. Only a row with neither a kept answer nor a recorded grade may run fresh.

Recognize retained response data, including a completed response with empty output.

A diagnostic error object or an empty mapping is not generation. A typed response counts when it has at least one output item or a status: a producer finished it, even with no assistant text. A response object with no output and no status is a placeholder, for example from an agent that never ran. So is a response an agent marked with NG_NO_GENERATION_METADATA: its empty assistant message was built without any generation.

A selected route failure names no owned cause that permits running the agent again.

A request that was possibly delivered permits a fresh run only while the row keeps no full or partial response: then no saved answer can be duplicated or replaced.

Whether the row keeps a full or partial response object, usable as an answer or not.

Whether the agent ran out of its own deadline and kept its answer: the evaluated model’s final result.

The deadline belongs to the agent, so what it kept is its result, scored by the benchmark’s rule (a recorded grade, else a settled zero). Recovery classifies it as final, and no resume or in-process retry runs it again: that would replace the model’s answer. The kind is the selected failure event’s when the row selects one, else the row’s own. A timeout that kept no answer keeps its retry policy, and so does one whose window held a recorded server-side fault (is_infrastructure_agent_timeout): what it kept was cut short by us, not by its deadline.

Whether an agent_timeout row’s window held a recorded server-side fault: the evaluation’s failure.

The row (a failure row, or a graded result such as Harbor’s) names agent_timeout and either recorded a sandbox or tool-server failure event (_sandbox_or_tool_fault), or its captured model calls show a server fault (is_server_fault_call) that owns the window (_material_model_call_faults). A clean agent timeout stays the model’s result under the benchmark’s time limit. Read from saved rows alone, so it applies to existing runs.

Whether eligible recorded model-call failures classify as evaluation-side.

Temporal materiality excludes timeout, cancellation and truncated-stream evidence: a slow model or an agent cancelling its current call does not establish time spent on infrastructure. HTTP evidence and the eligible transport errors use the shared classifier; classification alone does not establish materiality.

Whether a persisted failure must be gated on resume.

By default a row is terminal iff it is stamped _ng_failure_terminal, so an agent that marks its timeouts terminal on purpose keeps them gated.

retry_terminal_timeouts is for agents whose older builds incorrectly stamped per-attempt timeouts terminal. A timeout reflects the load/remaining walltime of that attempt, so it remains retryable (up to the normal max-attempt cap).

timeout_exceeded is an _ng_failure_class label that agents already write to the failures sidecar (pinchbench, nooa). It predates nemo_gym.core.failure_kinds and is not registered there; agent_timeout is the shared name.

As failure_kinds requires, this function decides retryability for the occurrence, under an explicit caller opt-in; the name carries no retry meaning.

Whether the row keeps the agent’s partial response from a native cleanup, even one without output.

The agent ran, so its partial response stays a kept answer and is never regenerated. An agent-stage response the outcome policy rejected as failed is diagnostic only, as in has_saved_generation.

Attempts an explicit recovery added to every rollout’s saved attempt cap (--extend-attempts); 0 when none.

A rollout’s cap is the run’s saved max_rollout_attempts plus this. An unreadable record stops the run rather than guess its cap.

The masked results an operator approved to run again (--rerun-masked-without-artifact): each approved attempt as (task index, rollout index, attempt index); empty when none.

An approval names the attempt it covered, so a later attempt of the same rollout is never reopened by it. A line or entry that cannot be read approves nothing: the default is that a saved answer is never sampled again.

Read the number of sidecar lines excluded from the current failure budget.

The kind of a masked result that infrastructure, not the evaluated model, left unmeasured; else None.

A result row is masked (mask_sample) by the Environment Server that graded it, so its failure_kind was reported at that boundary: a lost session, a failed provider, an infrastructure error, a verifier that could not run. When the kind is evaluation-side through the shared classifier the rollout is retried like an infrastructure failure, within its attempt cap: the masked row counts one failed attempt and a later attempt supersedes it. A model-class kind, no kind, an unregistered or namespaced kind, and the kinds whose saved answer is graded again (_MASKED_REGRADE_KINDS, including a transport_* verify call) keep the masked row as the result. So does a saved answer (_keeps_saved_answer): it is never sampled again, whatever its verifier could grade of it.

The kind of a masked result that masked_infrastructure_kind rejects only for its saved answer and that has no recorded grade (has_recorded_grade); else None.

The part of masked_without_artifact_kind that the Store can decide: the planner adds that no re-verification cause applies. A graded result is a result and stays one.

Match shared failure names and retained producer routing names alike.

Best-effort per-rollout wallclock from a result/failure row.

Reset the failure budget without altering outcomes or execution identities.

The caller must hold the run lock and validate/repair history first. The line-based format is shared with GDPVal’s existing resubmit implementation.

One line per resume that proceeded across operational configuration drift, naming the keys.

Own a run before preparing inputs, restoring state, or writing artifacts.

Collection currently acquires its own lock in run_from_config; do not nest it inside this context. Checkpoint integration must extend that acquisition to cover restore and supply checkpoint_dir, rather than acquire a second lock. When supplied, the directory lock prevents two outputs sharing checkpoints. The filesystem must support flock across all writer nodes. Kernel locks are released on process death; the presence of an old lock file does not block recovery.

Stable lock file; never unlink it while another process may hold it open.

Read only the owning launcher’s saved classification in its conventional artifact layout.

Standalone outputs have no launcher policy and use shared defaults. A malformed saved policy is never silently replaced with defaults. Applied classes on individual results still take precedence.

Reject unknown or incompatible saved work before any output is modified.

A configuration that differs from the saved one is compared on the identity saved beside the manifest: a difference only in operational settings (timeouts, retries, transport, ports, concurrency, health, paths, tracking) resumes, with a warning and a line in the resume-drift journal; a difference in a semantic setting (dataset, task limits, repeats, agent and model identity, sampling, judge and grading) refuses and names it. A manifest saved without that identity can only be compared by digest, as before.

allow_unsafe is an explicit compatibility escape hatch for legacy artifacts or deliberate identity overrides. Unsupported/corrupt manifests remain errors.