nemo_gym.eval.recovery.policy
nemo_gym.eval.recovery.policy
The recovery outcome policy: which saved failure is retried, and the failure kind it is selected on.
Module Contents
Classes
Functions
Data
API
Bases: enum.Enum
Where the recovery outcome policy in effect came from.
Bases: OutcomePolicyConfig
Shared policy for eligible recovery failures. Retry permits another episode within the existing guards; verify re-judges a saved judge answer; fail leaves the failure ungraded. Settled outcomes are immutable. Unsupported context/action combinations stop, and explicit-zero scoring is not supported here.
The event failure_id selects and its explicit cause_id chain; other saved events are observations only.
Legacy rows with no event report retain their explicit terminal classification. Otherwise a selected event and every explicit cause must exist; missing references and cycles remain unknown.
The kind of a failure whose producer recorded none, or an unregistered one, from its recorded cause.
Returns (kind, model_final, reason); the kind is None when the cause cannot be determined, with the reason.
The cause is the event the row selects, else every event it saved, with the collector’s facts of the model call
that recorded each one (COLLECTED_CAUSE):
- a model request that failed with HTTP 5xx or in transport is infrastructure: its own kind, retried under the cap;
- an HTTP 4xx of the policy’s own call is the model’s final result:
model_context_limitwhen the call recorded the context limit, elsemodel_request_error; - an HTTP 4xx of a harness call (a user simulator, a judge, any model that is not the policy) is evaluation-side:
environment_episode_failed, retried under the cap; - with no event, the environment’s own episode deadline is evaluation-side too:
environment_episode_failed.
As for a kind (_retry_kind), the unselected events of a row that keeps a response are attempts, not its cause,
and a cause chain that is broken or omitted names no cause.
A judge failure’s treatment: verify re-judges the saved answer only (none saved: stop), fail leaves it ungraded; any other action would run the policy again, so it stops.
Read the final agent classification without selecting an unbound diagnostic event.
Older environments returned a failed agent response but did not select its report’s cause. Support only a delivered agent-stage outcome with one matching observed model rejection. An internal wrapper’s 500, mixed history, incomplete reports and saved completed answers cannot establish an upstream HTTP rejection.
The failure kind and HTTP status the policy selects on: the event failure_id selects first, then the
row’s own producer fields, then a validated final legacy agent outcome. Unselected events are not causes.
Whether replay needs a saved-answer path or model-budget accounting unavailable to recovery.
The reset time a quota-stopped row names: the collected fact, else its reason (a live run’s row).
Apply recovery rules to shared classification without changing the selected evidence or budgets.
Why replay needs a saved-answer path or model-budget accounting unavailable to recovery; None when it does not.
The kind a retry decision selects on: the policy evidence, else, for a row that selects no event, the one kind every saved event names, else the failure record’s own kind. Final-zero and judge decisions do not use it.
A benchmark’s resumable.recovery_outcome_policy section as a policy, or None when it is not set.
It is a partial override: RecoveryOutcomePolicyConfig merges the entries it names into its defaults, and
validates them exactly as the Python API does. A bad section raises, naming each problem.
The policy recovery applies and its source. Precedence: an explicit Python outcome_policy argument
replaces the run’s saved config section (it is not merged with it); the section overrides the defaults. The
budget options (extend_retries, max_walltime) are separate and never change the policy.