Skip to navigation

nemo_gym.eval.recovery.policy

View as MarkdownOpen in Claude

The recovery outcome policy: which saved failure is retried, and the failure kind it is selected on.

Module Contents

Classes

NameDescription
OutcomePolicySourceWhere the recovery outcome policy in effect came from.
RecoveryOutcomePolicyConfigShared policy for eligible recovery failures. Retry permits another episode within the existing guards;

Functions

NameDescription
_cause_chainThe event failure_id selects and its explicit cause_id chain; other saved events are observations only.
_cause_kindThe kind of a failure whose producer recorded none, or an unregistered one, from its recorded cause.
_judge_stateA judge failure’s treatment: verify re-judges the saved answer only (none saved: stop), fail leaves it
_legacy_agent_outcomeRead the final agent classification without selecting an unbound diagnostic event.
_now-
_policy_evidenceThe failure kind and HTTP status the policy selects on: the event failure_id selects first, then the
_policy_replay_is_unsupportedWhether replay needs a saved-answer path or model-budget accounting unavailable to recovery.
_quota_reset_textThe reset time a quota-stopped row names: the collected fact, else its reason (a live run’s row).
_quota_reset_unix-
_quota_wait_reason-
_recovery_actionApply recovery rules to shared classification without changing the selected evidence or budgets.
_replay_refusalWhy replay needs a saved-answer path or model-budget accounting unavailable to recovery; None when it does not.
_retry_kindThe kind a retry decision selects on: the policy evidence, else, for a row that selects no event, the one kind
_saved_answer-
_sidecar_action-
configured_outcome_policyA benchmark’s resumable.recovery_outcome_policy section as a policy, or None when it is not set.
effective_outcome_policyThe policy recovery applies and its source. Precedence: an explicit Python outcome_policy argument

Data

SAVED_ANSWER_KEY

API

class nemo_gym.eval.recovery.policy.OutcomePolicySource

Bases: enum.Enum

Where the recovery outcome policy in effect came from.

CONFIG
= 'config'
DEFAULT
= 'default'
PYTHON
= 'python'
class nemo_gym.eval.recovery.policy.RecoveryOutcomePolicyConfig()

Bases: OutcomePolicyConfig

Shared policy for eligible recovery failures. Retry permits another episode within the existing guards; verify re-judges a saved judge answer; fail leaves the failure ungraded. Settled outcomes are immutable. Unsupported context/action combinations stop, and explicit-zero scoring is not supported here.

by_failure_kind
dict[str, OutcomeAction]
classification
FailureClassificationConfig = Field(default_factory=FailureClassificationConfig)
default_action
OutcomeAction = OutcomeAction.RETRY
supported_actions
nemo_gym.eval.recovery.policy._cause_chain(
row: collections.abc.Mapping[str, typing.Any]
) -> list[tuple[str | None, ...]] | None

The event failure_id selects and its explicit cause_id chain; other saved events are observations only.

Legacy rows with no event report retain their explicit terminal classification. Otherwise a selected event and every explicit cause must exist; missing references and cycles remain unknown.

nemo_gym.eval.recovery.policy._cause_kind(
row: collections.abc.Mapping[str, typing.Any]
) -> tuple[str | None, bool, str | None]

The kind of a failure whose producer recorded none, or an unregistered one, from its recorded cause.

Returns (kind, model_final, reason); the kind is None when the cause cannot be determined, with the reason. The cause is the event the row selects, else every event it saved, with the collector’s facts of the model call that recorded each one (COLLECTED_CAUSE):

  • a model request that failed with HTTP 5xx or in transport is infrastructure: its own kind, retried under the cap;
  • an HTTP 4xx of the policy’s own call is the model’s final result: model_context_limit when the call recorded the context limit, else model_request_error;
  • an HTTP 4xx of a harness call (a user simulator, a judge, any model that is not the policy) is evaluation-side: environment_episode_failed, retried under the cap;
  • with no event, the environment’s own episode deadline is evaluation-side too: environment_episode_failed.

As for a kind (_retry_kind), the unselected events of a row that keeps a response are attempts, not its cause, and a cause chain that is broken or omitted names no cause.

nemo_gym.eval.recovery.policy._judge_state(
row: collections.abc.Mapping[str, typing.Any],

A judge failure’s treatment: verify re-judges the saved answer only (none saved: stop), fail leaves it ungraded; any other action would run the policy again, so it stops.

nemo_gym.eval.recovery.policy._legacy_agent_outcome(
row: collections.abc.Mapping[str, typing.Any]
) -> tuple[str | None, int | None]

Read the final agent classification without selecting an unbound diagnostic event.

Older environments returned a failed agent response but did not select its report’s cause. Support only a delivered agent-stage outcome with one matching observed model rejection. An internal wrapper’s 500, mixed history, incomplete reports and saved completed answers cannot establish an upstream HTTP rejection.

nemo_gym.eval.recovery.policy._now() -> float
nemo_gym.eval.recovery.policy._policy_evidence(
row: collections.abc.Mapping[str, typing.Any]
) -> tuple[str | None, int | None]

The failure kind and HTTP status the policy selects on: the event failure_id selects first, then the row’s own producer fields, then a validated final legacy agent outcome. Unselected events are not causes.

nemo_gym.eval.recovery.policy._policy_replay_is_unsupported(
row: collections.abc.Mapping[str, typing.Any],
) -> bool

Whether replay needs a saved-answer path or model-budget accounting unavailable to recovery.

nemo_gym.eval.recovery.policy._quota_reset_text(
row: collections.abc.Mapping[str, typing.Any]
) -> str | None

The reset time a quota-stopped row names: the collected fact, else its reason (a live run’s row).

nemo_gym.eval.recovery.policy._quota_reset_unix(
row: collections.abc.Mapping[str, typing.Any]
) -> float | None
nemo_gym.eval.recovery.policy._quota_wait_reason(
row: collections.abc.Mapping[str, typing.Any]
) -> str | None
nemo_gym.eval.recovery.policy._recovery_action(
row: collections.abc.Mapping[str, typing.Any],
kind: str,
http_status: int | None

Apply recovery rules to shared classification without changing the selected evidence or budgets.

nemo_gym.eval.recovery.policy._replay_refusal(
row: collections.abc.Mapping[str, typing.Any],
) -> str | None

Why replay needs a saved-answer path or model-budget accounting unavailable to recovery; None when it does not.

nemo_gym.eval.recovery.policy._retry_kind(
row: collections.abc.Mapping[str, typing.Any]
) -> str | None

The kind a retry decision selects on: the policy evidence, else, for a row that selects no event, the one kind every saved event names, else the failure record’s own kind. Final-zero and judge decisions do not use it.

nemo_gym.eval.recovery.policy._saved_answer(
row: collections.abc.Mapping[str, typing.Any]
) -> bool
nemo_gym.eval.recovery.policy._sidecar_action(
row: collections.abc.Mapping[str, typing.Any],
nemo_gym.eval.recovery.policy.configured_outcome_policy(
overrides: collections.abc.Mapping[str, typing.Any] | None

A benchmark’s resumable.recovery_outcome_policy section as a policy, or None when it is not set.

It is a partial override: RecoveryOutcomePolicyConfig merges the entries it names into its defaults, and validates them exactly as the Python API does. A bad section raises, naming each problem.

nemo_gym.eval.recovery.policy.effective_outcome_policy(
overrides: collections.abc.Mapping[str, typing.Any] | None

The policy recovery applies and its source. Precedence: an explicit Python outcome_policy argument replaces the run’s saved config section (it is not merged with it); the section overrides the defaults. The budget options (extend_retries, max_walltime) are separate and never change the policy.

nemo_gym.eval.recovery.policy.SAVED_ANSWER_KEY = '_ng_collected_saved_answer'