> This page is for version Main (default). > For other versions, use one of these documentation indexes: > - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt > - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt > - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt > - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt > - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt > - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt > - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt > - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server. # nemo_gym.orchestration.recovery Whether `gym eval resubmit` may queue a run again, decided from the run's persisted evidence only. Every expected identity (`_ng_task_index`, `_ng_rollout_index`) of the materialized inventory gets one state: * `scored`: a result row that is not a judge failure. Kept as saved; never re-run. * `model_final`: a failures-sidecar row marked `_ng_failure_counts_as_zero`: the evaluated model's terminal failure, a scored zero that no infra retry may replace; or an `agent_timeout` row that kept the agent's answer (`is_answered_agent_deadline`), the model's result under its own deadline, which no resume runs again. * `judge_recoverable`: a result row of class `judge_failed`. Its saved answer is re-judged by `gym eval reverify --judge-failed-only`; queueing the policy again is never needed for it. * `retryable`: the last sidecar row is non-terminal and under its recorded attempt cap: a Gym resume retries it. * `exhausted`: the last sidecar row is non-terminal and at its attempt cap: only a resubmit gives it a fresh budget. * `terminal`: a terminal sidecar row that is not a scored zero; excluded, and the run cannot become complete. * `missing`: no row at all (interrupted or never started); a resume restarts its input. * `masked`: a result row the producer masked or did not evaluate. Collection keeps it as completed and never runs it again, so it is retained, but it is not measured and the run is not complete. * `omitted`: a planned row that a staged collection's accepted partial stage omitted. It is retained, never planned again and not measured, and the run is complete without it. * `unknown`: a failure whose kind is unregistered or absent, or a row whose identity is invalid; nobody owns it. The run-level action follows: `stop` for a spent or unknown resubmit budget, or for an unknown identity that a resume would run again; `noop` when nothing is left to do, `reverify` when only judge failures remain, else `resubmit`. A run that ended before its inventory was written has nothing to recover: `fresh-restart` queues it again as a fresh run with a longer walltime. An unknown identity that a resume never runs again (the Store or the saved result holds it) does not block the missing and retryable ones: the decision reports it apart from what is retried and what is retained. A decision never reports a run as complete while an identity is terminal, missing or unknown. ## Module Contents ### Classes | Name | Description | | --------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | | [`CollectedEvidence`](#nemo_gym-orchestration-recovery-CollectedEvidence) | - | | [`GateCriteria`](#nemo_gym-orchestration-recovery-GateCriteria) | The response and trial failure fractions a caller may still pass. They are validated (an invalid value is | | [`IdentityEvidence`](#nemo_gym-orchestration-recovery-IdentityEvidence) | - | | [`IdentityState`](#nemo_gym-orchestration-recovery-IdentityState) | - | | [`ManagedRecoveryFact`](#nemo_gym-orchestration-recovery-ManagedRecoveryFact) | Read-only facts from the authoritative Store, not a second retry policy. | | [`OperatorBudget`](#nemo_gym-orchestration-recovery-OperatorBudget) | An operator's explicit budget for a run whose chain accounting is unknown. | | [`OperatorEpoch`](#nemo_gym-orchestration-recovery-OperatorEpoch) | Where the last operator continuation started the chain's accounting: its chain length and walltime baseline. | | [`OutcomePolicySource`](#nemo_gym-orchestration-recovery-OutcomePolicySource) | Where the recovery outcome policy in effect came from. | | [`RecoveryAction`](#nemo_gym-orchestration-recovery-RecoveryAction) | - | | [`RecoveryBudget`](#nemo_gym-orchestration-recovery-RecoveryBudget) | The run's stored infra-retry and walltime budget and what its job chain used, as the chain files record them. | | [`RecoveryDecision`](#nemo_gym-orchestration-recovery-RecoveryDecision) | - | | [`RecoveryOutcome`](#nemo_gym-orchestration-recovery-RecoveryOutcome) | What `recover` did: `dispatched` names the one scheduler or judge action taken, None when nothing ran. | | [`RecoveryOutcomePolicyConfig`](#nemo_gym-orchestration-recovery-RecoveryOutcomePolicyConfig) | Shared policy for eligible recovery failures. Retry permits another episode within the existing guards; | | [`RerunMaskedApproval`](#nemo_gym-orchestration-recovery-RerunMaskedApproval) | One masked result the operator approves to run again: its identity, the attempts counted for it before the | ### Functions | Name | Description | | --------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | | [`_answer_cut_off`](#nemo_gym-orchestration-recovery-_answer_cut_off) | Whether the saved answer itself says it was cut off: its `incomplete_details.reason` is `max_output_tokens` | | [`_answer_fault`](#nemo_gym-orchestration-recovery-_answer_fault) | The model fault of a scored-zero incomplete answer. | | [`_bound_answer`](#nemo_gym-orchestration-recovery-_bound_answer) | Whether the collector bound the saved answer and score together (`_ng_collected_grading_binding`). | | [`_bound_artifact`](#nemo_gym-orchestration-recovery-_bound_artifact) | Whether a saved grading artifact or the model's own failed answer resolves the row, as the gate requires: an | | [`_bound_grading`](#nemo_gym-orchestration-recovery-_bound_grading) | Whether the row is `_graded` and every grading operation binds its saved answer and score, as the gate | | [`_bounded`](#nemo_gym-orchestration-recovery-_bounded) | - | | [`_cause_chain`](#nemo_gym-orchestration-recovery-_cause_chain) | The event `failure_id` selects and its explicit `cause_id` chain; other saved events are observations only. | | [`_cause_kind`](#nemo_gym-orchestration-recovery-_cause_kind) | The kind of a failure whose producer recorded none, or an unregistered one, from its recorded cause. | | [`_clip_text`](#nemo_gym-orchestration-recovery-_clip_text) | - | | [`_collector_function`](#nemo_gym-orchestration-recovery-_collector_function) | `COLLECT_SCRIPT`'s own projection `name`, so an in-process reader projects a saved row as the collector does. | | [`_count`](#nemo_gym-orchestration-recovery-_count) | - | | [`_cut_judge_reply`](#nemo_gym-orchestration-recovery-_cut_judge_reply) | Whether a captured judge reply was cut short: an incomplete Responses reply, or a chat completion that stopped | | [`_cut_off`](#nemo_gym-orchestration-recovery-_cut_off) | Whether the saved answer carries an actual cutoff signal, not just an incomplete status. | | [`_decide_from_collected`](#nemo_gym-orchestration-recovery-_decide_from_collected) | Decide from what `COLLECT_SCRIPT` read; a missing result file stops the decision. | | [`_empty_answer`](#nemo_gym-orchestration-recovery-_empty_answer) | Whether the saved answer holds no assistant message with text: none at all, or only ones the collector saw | | [`_ended_on_failed_request`](#nemo_gym-orchestration-recovery-_ended_on_failed_request) | The failure kind of the request a row's rollout ended on, with that call, or None (`final_zero_trials`). | | [`_environment_reward`](#nemo_gym-orchestration-recovery-_environment_reward) | The environment's own reward record: `result.reward_info.reward` equal to the row's reward. | | [`_events`](#nemo_gym-orchestration-recovery-_events) | - | | [`_extend_attempts_refusal`](#nemo_gym-orchestration-recovery-_extend_attempts_refusal) | Why this run cannot take an attempt extension, or None when it can. | | [`_failed_request`](#nemo_gym-orchestration-recovery-_failed_request) | The failure kind and HTTP status of a captured call whose final model operation failed with its own recorded | | [`_fault_text`](#nemo_gym-orchestration-recovery-_fault_text) | `evaluation-side faults: N of M (x%) is [not] below L (server S: kind n; kind n); model faults: ...`. | | [`_graded`](#nemo_gym-orchestration-recovery-_graded) | Whether the row's one verifier recorded grading operations for it that all completed with a valid verdict. | | [`_grading_facts`](#nemo_gym-orchestration-recovery-_grading_facts) | - | | [`_grading_sandbox_failed`](#nemo_gym-orchestration-recovery-_grading_sandbox_failed) | Whether a verifier's sandbox observations report that a grading sandbox failed, or are malformed. | | [`_grading_unrecorded`](#nemo_gym-orchestration-recovery-_grading_unrecorded) | Whether the row's verifier invocations were observed but recorded no grading: no grading operation, no | | [`_guarded_objective_score`](#nemo_gym-orchestration-recovery-_guarded_objective_score) | A guardrail-gated objective score the agent's own rubric saved (`response.metrics`): the row's reward equals | | [`_health_classification`](#nemo_gym-orchestration-recovery-_health_classification) | Express the existing timeout switch as a shared classification override. | | [`_health_view`](#nemo_gym-orchestration-recovery-_health_view) | `decision` as the trial gate counts it: a scored row planned for a re-judge because its judge gave no verdict | | [`_identity`](#nemo_gym-orchestration-recovery-_identity) | - | | [`_incomplete_answer`](#nemo_gym-orchestration-recovery-_incomplete_answer) | Whether the row's saved answer is a response left incomplete, without an error. An incomplete status alone | | [`_judge_error_marker`](#nemo_gym-orchestration-recovery-_judge_error_marker) | Whether the row names a judge error the server wrote: a judge failure kind, a saved judge error (a label or a | | [`_judge_evaluations`](#nemo_gym-orchestration-recovery-_judge_evaluations) | A non-empty `judge_evaluations` list without a judge error; when every evaluation's verdict label parses | | [`_judge_job_unsettled`](#nemo_gym-orchestration-recovery-_judge_job_unsettled) | - | | [`_judge_record`](#nemo_gym-orchestration-recovery-_judge_record) | A saved judge verdict: a `judge_response`, a non-empty `judgments` list with a `judge_score`, or | | [`_judge_state`](#nemo_gym-orchestration-recovery-_judge_state) | A judge failure's treatment: verify re-judges the saved answer only (none saved: stop), fail leaves it | | [`_judge_unparseable`](#nemo_gym-orchestration-recovery-_judge_unparseable) | Whether a scored row is `judge_unparseable`: its judge gave no verdict (`_unparsed_judge_verdict`) or its | | [`_known_kind`](#nemo_gym-orchestration-recovery-_known_kind) | - | | [`_last_captured_call`](#nemo_gym-orchestration-recovery-_last_captured_call) | The call a rollout ended on, for an agent whose invocations reference none of its calls. | | [`_legacy_agent_outcome`](#nemo_gym-orchestration-recovery-_legacy_agent_outcome) | Read the final agent classification without selecting an unbound diagnostic event. | | [`_length_refusal_kind`](#nemo_gym-orchestration-recovery-_length_refusal_kind) | The kind of a captured call rejected with HTTP 4xx on length (`LENGTH_REFUSAL_KINDS`), or None. | | [`_live_reason`](#nemo_gym-orchestration-recovery-_live_reason) | - | | [`_managed_source_digest`](#nemo_gym-orchestration-recovery-_managed_source_digest) | Hash one stable file with bounded memory, using the exporter's existing identity checks. | | [`_managed_trial_metadata`](#nemo_gym-orchestration-recovery-_managed_trial_metadata) | Session trial metadata from the bound saved config and the Store's own inputs, after Store selection. | | [`_metric_record`](#nemo_gym-orchestration-recovery-_metric_record) | Saved in-process metric scores: numeric `sentence_*` fields (`sentence_chrf`, `sentence_spbleu`). | | [`_model_failure`](#nemo_gym-orchestration-recovery-_model_failure) | The benchmark's rule scored the model's own failed answer without grading it: the verifier marked the answer | | [`_native_scorer_score`](#nemo_gym-orchestration-recovery-_native_scorer_score) | A native scorer's saved score (`native_metrics`): the row's reward equals the saved `weighted_score` in | | [`_node_set_f1`](#nemo_gym-orchestration-recovery-_node_set_f1) | A saved node-set F1 re-graded from the answer: the collector re-parses the saved answer's final | | [`_now`](#nemo_gym-orchestration-recovery-_now) | - | | [`_observed_incomplete_answer`](#nemo_gym-orchestration-recovery-_observed_incomplete_answer) | Whether a counted incomplete answer (`_answer_fault`) is the model's own observed answer. | | [`_observed_trial_operations`](#nemo_gym-orchestration-recovery-_observed_trial_operations) | Whether every expected trial that is not a counted failure is observed: its saved answer is bound to its grading. | | [`_operator_continuation`](#nemo_gym-orchestration-recovery-_operator_continuation) | Decide again under the operator's explicit budget, or keep the stop when continuing could be unsafe. | | [`_percent`](#nemo_gym-orchestration-recovery-_percent) | - | | [`_policy_call`](#nemo_gym-orchestration-recovery-_policy_call) | Whether a captured call is the evaluated policy's: its model is the row's policy server, or none is named. | | [`_policy_evidence`](#nemo_gym-orchestration-recovery-_policy_evidence) | The failure kind and HTTP status the policy selects on: the event `failure_id` selects first, then the | | [`_policy_replay_is_unsupported`](#nemo_gym-orchestration-recovery-_policy_replay_is_unsupported) | Whether replay needs a saved-answer path or model-budget accounting unavailable to recovery. | | [`_policy_zero_kind`](#nemo_gym-orchestration-recovery-_policy_zero_kind) | A consistent applied zero keeps its own kind, without reinterpreting diagnostic requests. | | [`_populations`](#nemo_gym-orchestration-recovery-_populations) | What a resume runs, what nobody owns (by kind, largest first) and the saved rows kept as they are. | | [`_predates_operation_observations`](#nemo_gym-orchestration-recovery-_predates_operation_observations) | Whether every scored row was written by a Gym pin from before operation observations were recorded. | | [`_quota_reset_text`](#nemo_gym-orchestration-recovery-_quota_reset_text) | The reset time a quota-stopped row names: the collected fact, else its reason (a live run's row). | | [`_quota_reset_unix`](#nemo_gym-orchestration-recovery-_quota_reset_unix) | - | | [`_quota_wait_reason`](#nemo_gym-orchestration-recovery-_quota_wait_reason) | - | | [`_recorded_by_verifier`](#nemo_gym-orchestration-recovery-_recorded_by_verifier) | Whether the row's one verifier invocation saved its own grading record for it (`_verifier_recorded`). | | [`_recorded_grade`](#nemo_gym-orchestration-recovery-_recorded_grade) | Whether the row's reward came from a recorded grade: completed grading operations, the verifier's own grading | | [`_recovered_note`](#nemo_gym-orchestration-recovery-_recovered_note) | `N failed model attempt(s) were retried and completed in the same operation (...)`, or empty. | | [`_recovery_action`](#nemo_gym-orchestration-recovery-_recovery_action) | Apply recovery rules to shared classification without changing the selected evidence or budgets. | | [`_replay_refusal`](#nemo_gym-orchestration-recovery-_replay_refusal) | Why replay needs a saved-answer path or model-budget accounting unavailable to recovery; None when it does not. | | [`_repository_test_run`](#nemo_gym-orchestration-recovery-_repository_test_run) | A saved repository test run of the answer's patch: `evaluation_completed` true, the run's `resolved` | | [`_requeue_continuation`](#nemo_gym-orchestration-recovery-_requeue_continuation) | In a sentinel: reconcile a requeued chain from the terminal records in `CHAIN_ACCOUNTING_ENV` and decide. | | [`_rerun_masked_refusal`](#nemo_gym-orchestration-recovery-_rerun_masked_refusal) | Why this run cannot run masked results without an artifact again, or None when it can. | | [`_retry_kind`](#nemo_gym-orchestration-recovery-_retry_kind) | The kind a retry decision selects on: the policy evidence, else, for a row that selects no event, the one kind | | [`_reward`](#nemo_gym-orchestration-recovery-_reward) | - | | [`_row_fault_class`](#nemo_gym-orchestration-recovery-_row_fault_class) | - | | [`_row_recovered_attempts`](#nemo_gym-orchestration-recovery-_row_recovered_attempts) | The row's recovered model attempts, by the completed operation that recovered them (`recovered_attempt_joins`). | | [`_rubric_verdict`](#nemo_gym-orchestration-recovery-_rubric_verdict) | A saved rubric judge's verdict: `verdict` YES or NO, a finite `rubric_score` and `scoring_threshold`, no | | [`_saved_answer`](#nemo_gym-orchestration-recovery-_saved_answer) | - | | [`_saved_attempt_cap`](#nemo_gym-orchestration-recovery-_saved_attempt_cap) | The run's saved per-rollout attempt cap, from the Store's facts; None when they do not say. | | [`_saved_model_patch`](#nemo_gym-orchestration-recovery-_saved_model_patch) | Whether the row saved, intact, the workspace patch its verifier collected and graded: the collector's projected | | [`_scored_fault`](#nemo_gym-orchestration-recovery-_scored_fault) | A scored row's one counted fault: its own recorded kind (`_scored_kind`), else its answer's | | [`_scored_kind`](#nemo_gym-orchestration-recovery-_scored_kind) | The registered failure kind a scored result row records as its own outcome, keyed on its recorded HTTP status | | [`_scored_unmasked`](#nemo_gym-orchestration-recovery-_scored_unmasked) | A scored row: unmasked, with a reward and no failure kind. | | [`_session_reward_file`](#nemo_gym-orchestration-recovery-_session_reward_file) | The verifier's own reward file of the row's session, read by the collector beside the artifacts | | [`_sidecar_action`](#nemo_gym-orchestration-recovery-_sidecar_action) | - | | [`_store_writer_ended`](#nemo_gym-orchestration-recovery-_store_writer_ended) | In the in-job guard: whether the allocation that last wrote the Store has ended. | | [`_substep_tests`](#nemo_gym-orchestration-recovery-_substep_tests) | Saved per-sub-step test results: a non-empty `step_results` list of booleans, one environment result per | | [`_symbolic_match`](#nemo_gym-orchestration-recovery-_symbolic_match) | A saved symbolic answer check: a `library_reward` the row's reward equals, or one beside a judge verdict | | [`_teardown_failure_only`](#nemo_gym-orchestration-recovery-_teardown_failure_only) | Whether a sandbox observation's only failure is its teardown: the producer records a failed close or release | | [`_test_execution`](#nemo_gym-orchestration-recovery-_test_execution) | A saved test run: `evaluation_completed` true with an integer `verifier_exit_code` and no | | [`_trajectory_kinds`](#nemo_gym-orchestration-recovery-_trajectory_kinds) | The kinds of trajectory evidence a trial lacks: its recorded gaps, and its agent invocations' own shape. | | [`_trajectory_text`](#nemo_gym-orchestration-recovery-_trajectory_text) | `trajectory evidence incomplete: N of M (kinds: kind n, ...)`, reported and never a condition. | | [`_transport_kind`](#nemo_gym-orchestration-recovery-_transport_kind) | - | | [`_unaccounted_failures`](#nemo_gym-orchestration-recovery-_unaccounted_failures) | Whether the row recorded a model failure that no recovery accounts for. | | [`_unit_test_results`](#nemo_gym-orchestration-recovery-_unit_test_results) | Saved unit test results of the answer's extracted code: at least one test, and the row's reward 1.0 exactly | | [`_unknown_grading_outcome`](#nemo_gym-orchestration-recovery-_unknown_grading_outcome) | Whether a scored row's verifier record reports a judge grading operation with outcome `unknown`: the grading | | [`_unknown_label`](#nemo_gym-orchestration-recovery-_unknown_label) | The kind of an unknown identity with the rule that refused it, without naming the kind twice. | | [`_unparsed_judge_verdict`](#nemo_gym-orchestration-recovery-_unparsed_judge_verdict) | Whether a scored row's grading is `judge_unparseable` for a verdict the judge did not give: a saved judge | | [`_unparsed_verdict`](#nemo_gym-orchestration-recovery-_unparsed_verdict) | Saved judge evaluations of which one recorded no verdict: a null label, or a judge or parse error label. | | [`_verification_owner`](#nemo_gym-orchestration-recovery-_verification_owner) | Who verifies an agent's rollouts: its resources server, or the agent itself when it names none. | | [`_verifier_model_answered`](#nemo_gym-orchestration-recovery-_verifier_model_answered) | Whether one model call of the row's verifiers completed with a 2xx and a usable response, read from the call | | [`_verifier_recorded`](#nemo_gym-orchestration-recovery-_verifier_recorded) | Whether the row's verifier saved its own grading record for it, in place of a recorded grading operation. | | [`_verify_transport_chain`](#nemo_gym-orchestration-recovery-_verify_transport_chain) | Whether `event` and every explicit cause below it name only a lost verify call: an agent route failure or a | | [`_verify_transport_failure`](#nemo_gym-orchestration-recovery-_verify_transport_failure) | Whether the row was masked because its verify call was lost in transport: its kind is an agent route failure | | [`_verifying_servers`](#nemo_gym-orchestration-recovery-_verifying_servers) | The resources servers the run's agents name, from the run provenance, for an unobserved-trial reason. | | [`_with_attempt_extension`](#nemo_gym-orchestration-recovery-_with_attempt_extension) | The Store's facts as they read once `extend_attempts` more attempts are recorded for every rollout. | | [`accounted_chain_walltime`](#nemo_gym-orchestration-recovery-accounted_chain_walltime) | Elapsed seconds of all recorded allocations, only when each has one terminal accounting row. | | [`accounted_requeued_chain_walltime`](#nemo_gym-orchestration-recovery-accounted_requeued_chain_walltime) | Elapsed seconds of every incarnation of a chain with a requeued allocation, when the records are complete. | | [`agent_adaptations`](#nemo_gym-orchestration-recovery-agent_adaptations) | The scored rollouts' own `agent_adaptations` counts, summed by name: `{name: {"count": total, "rows": N}}`, | | [`allocation_cut_from_accounting`](#nemo_gym-orchestration-recovery-allocation_cut_from_accounting) | Whether the first incarnation of `allocation_id` has ended, by any path (`ALLOCATION_ENDED_STATES`). | | [`chain_accounting`](#nemo_gym-orchestration-recovery-chain_accounting) | Rows of `sacct -D -X -n -P --format=CHAIN_ACCOUNTING_FORMAT` as (job id, state, elapsed s, restarts). | | [`configured_outcome_policy`](#nemo_gym-orchestration-recovery-configured_outcome_policy) | A benchmark's `resumable.recovery_outcome_policy` section as a policy, or None when it is not set. | | [`continuation_admission`](#nemo_gym-orchestration-recovery-continuation_admission) | Read saved state in the driver's installed runtime before a terminal allocation continues. | | [`decide_from_collected`](#nemo_gym-orchestration-recovery-decide_from_collected) | `_decide_from_collected`, with the reason naming any operational drift the run's latest resume crossed (see | | [`decide_recovery`](#nemo_gym-orchestration-recovery-decide_recovery) | Classify every materialized identity and choose the run-level action. | | [`decision_classification`](#nemo_gym-orchestration-recovery-decision_classification) | The resolved recovery policy snapshot supplies the same mapping to all health readers. | | [`effective_outcome_policy`](#nemo_gym-orchestration-recovery-effective_outcome_policy) | The policy recovery applies and its source. Precedence: an explicit Python `outcome_policy` argument | | [`failed_judge_jobs_blocking`](#nemo_gym-orchestration-recovery-failed_judge_jobs_blocking) | The ids of the last `JUDGE_JOB_FAILURE_BOUND` judge-only jobs when all ended without settling a saved answer and no | | [`failed_judge_jobs_stop`](#nemo_gym-orchestration-recovery-failed_judge_jobs_stop) | The stop text of `failed_judge_jobs_blocking`: the jobs, where to read them, and the explicit way on. | | [`failure_facts`](#nemo_gym-orchestration-recovery-failure_facts) | Response and trial failures by cause, each with a count and a rate over the expected rollouts. | | [`fault_report`](#nemo_gym-orchestration-recovery-fault_report) | Report shared classes separately; server is only a subcategory of evaluation. | | [`final_zero_trials`](#nemo_gym-orchestration-recovery-final_zero_trials) | Scored zeros with a saved policy decision or a failed final policy request, with their failure kinds. | | [`gate_summary_from_collected`](#nemo_gym-orchestration-recovery-gate_summary_from_collected) | The acceptance gate task entry from the collected run: decision counts, the named score and the quality summary. | | [`gate_task_summary`](#nemo_gym-orchestration-recovery-gate_task_summary) | One task entry of the acceptance gate's run summary, filled from the decision and the existing validators. | | [`grading_artifact`](#nemo_gym-orchestration-recovery-grading_artifact) | The name of the first saved grading artifact (`GRADING_ARTIFACTS`) that records the row's grading. | | [`health_check_facts`](#nemo_gym-orchestration-recovery-health_check_facts) | Named health-check facts from `quality_summary.json` for the gate warnings, kept even below any criterion. | | [`health_verdict`](#nemo_gym-orchestration-recovery-health_verdict) | \`\`evaluation health: PASS; response failures: 4.70% (truncated 41, empty 6 of 1000); trial failures: ...; | | [`infrastructure_timeouts`](#nemo_gym-orchestration-recovery-infrastructure_timeouts) | Scored identities whose graded result is an agent timeout with a server-side fault in its window. | | [`judge_job_note`](#nemo_gym-orchestration-recovery-judge_job_note) | One line on a judge-only job of the run (`CollectedEvidence.judge_jobs`): queued, live, ended without a | | [`live_chain_job`](#nemo_gym-orchestration-recovery-live_chain_job) | The chain job that still waits or runs, as 'id (STATE)' from its latest incarnation; None when none does. | | [`masked_without_artifact_kind`](#nemo_gym-orchestration-recovery-masked_without_artifact_kind) | The kind of a masked result that is never measured nor run again, though infrastructure left it unmeasured; else None. | | [`metric_coverage`](#nemo_gym-orchestration-recovery-metric_coverage) | How many expected trials the benchmark's metric covered: `{"trials": N, "expected": M, "source": ...}`. | | [`mlflow_delivery`](#nemo_gym-orchestration-recovery-mlflow_delivery) | The MLflow delivery fact of a run: `not_required` by design, `verified` when `mlflow_errors` found nothing, | | [`mlflow_errors`](#nemo_gym-orchestration-recovery-mlflow_errors) | MLflow delivery errors from the collected exporter files and the server status the caller observed. | | [`mlflow_notes`](#nemo_gym-orchestration-recovery-mlflow_notes) | What the gate states about a passing MLflow part: a current post-run export beside a live run wins. | | [`mlflow_tracking_required`](#nemo_gym-orchestration-recovery-mlflow_tracking_required) | False only when the run's saved config snapshot sets `mlflow_tracking_required` to false; absent, unread or | | [`omission_summary`](#nemo_gym-orchestration-recovery-omission_summary) | The omitted identities by stage and policy, e.g. "1 omitted by an accepted partial stage (stage_index 1, tolerate_unresolved)". | | [`output_budget_refits`](#nemo_gym-orchestration-recovery-output_budget_refits) | How many model calls in the saved results were sent again with an output budget lowered to fit the context | | [`quality_inputs`](#nemo_gym-orchestration-recovery-quality_inputs) | Summary validity and per-rollout health from the run's `quality_summary.json`, kept separate. | | [`query_mlflow_status`](#nemo_gym-orchestration-recovery-query_mlflow_status) | The status of `run_id` on the tracking server, as `(status, None)`, or `(None, error)` when it could not | | [`reconcile_requeued_chain`](#nemo_gym-orchestration-recovery-reconcile_requeued_chain) | Know a requeued chain's budget again once every incarnation has a terminal scheduler record. | | [`recovered_attempt_joins`](#nemo_gym-orchestration-recovery-recovered_attempt_joins) | Failed model attempts each joined to the completed operation that recovered them, in a scored trial's own call. | | [`requeue_note`](#nemo_gym-orchestration-recovery-requeue_note) | The last requeued job of a chain and what its allocations account for, as a decision states it. | | [`response_health`](#nemo_gym-orchestration-recovery-response_health) | The response gate from `run.stats.responses` of the health report: faulted over selected responses. | | [`resume_drift_note`](#nemo_gym-orchestration-recovery-resume_drift_note) | The sentence a decision reason carries when the run's latest resume crossed operational drift. | | [`retry_budget`](#nemo_gym-orchestration-recovery-retry_budget) | The run's two retry budgets as the gate reports them, each its configured cap and its recorded extension. | | [`reverification_cause`](#nemo_gym-orchestration-recovery-reverification_cause) | Why a saved result row is re-verified from its saved answer, or None when it is not. | | [`saved_row_evidence`](#nemo_gym-orchestration-recovery-saved_row_evidence) | A full saved row with the evidence fields `COLLECT_SCRIPT` projects for the grading predicates: its scalar | | [`select_managed_evidence`](#nemo_gym-orchestration-recovery-select_managed_evidence) | Read locally accessible Store history in this installed Gym process; never copy its selection rules. | | [`status_fault_kind`](#nemo_gym-orchestration-recovery-status_fault_kind) | The shared reporting label for observed HTTP evidence; saved events remain unchanged. | | [`stopped_followup`](#nemo_gym-orchestration-recovery-stopped_followup) | Why the chain stopped when the follow-up `job_id` that its last allocation claimed is not in the chain. | | [`trajectory_evidence`](#nemo_gym-orchestration-recovery-trajectory_evidence) | Trials whose trajectory evidence is incomplete, over the trials, with the count of each missing kind. | | [`trial_boundary`](#nemo_gym-orchestration-recovery-trial_boundary) | Why trial failures outside the sidecar cannot be ruled out, from the run's own persisted facts. | | [`trial_counts`](#nemo_gym-orchestration-recovery-trial_counts) | Acceptance's trial counts: observed healthy, unhealthy and unobserved, over the expected trials. | | [`trial_failure_cause`](#nemo_gym-orchestration-recovery-trial_failure_cause) | A trial failure's cause from its failure kind: request, judge, sandbox, harness, timeout, allocation, or other. | | [`trial_faults`](#nemo_gym-orchestration-recovery-trial_faults) | Count final trial failures once; classify selected outcomes, never recovered observations. | | [`trial_health`](#nemo_gym-orchestration-recovery-trial_health) | Evaluation health from the trial gate (`trial_verdict`): passed, failed or None, with its reason. | | [`trial_population`](#nemo_gym-orchestration-recovery-trial_population) | The trial denominator: every expected trial, including repeats, failed and missing ones, less the rows an | | [`trial_verdict`](#nemo_gym-orchestration-recovery-trial_verdict) | Evaluation health: evaluation-side trial faults over the expected trials, including repeats. | | [`unsettled_chain_note`](#nemo_gym-orchestration-recovery-unsettled_chain_note) | The chain stopped before the run settled: how many rollouts are unsettled, by state, so the stop reads | ### Data [`ACCUMULATED_WALLTIME_FILE`](#nemo_gym-orchestration-recovery-ACCUMULATED_WALLTIME_FILE) [`ACKNOWLEDGE_JUDGE_JOBS_FLAGS`](#nemo_gym-orchestration-recovery-ACKNOWLEDGE_JUDGE_JOBS_FLAGS) [`ALLOCATION_ENDED_STATES`](#nemo_gym-orchestration-recovery-ALLOCATION_ENDED_STATES) [`COLLECTED_CAUSE`](#nemo_gym-orchestration-recovery-COLLECTED_CAUSE) [`COUNTS_AS_ZERO`](#nemo_gym-orchestration-recovery-COUNTS_AS_ZERO) [`CUTOFF_REASONS`](#nemo_gym-orchestration-recovery-CUTOFF_REASONS) [`EXTEND_ATTEMPTS_HINT`](#nemo_gym-orchestration-recovery-EXTEND_ATTEMPTS_HINT) [`FAILURE_CLASS`](#nemo_gym-orchestration-recovery-FAILURE_CLASS) [`FAILURE_EVENTS`](#nemo_gym-orchestration-recovery-FAILURE_EVENTS) [`FAILURE_ID`](#nemo_gym-orchestration-recovery-FAILURE_ID) [`FOLLOWUP_ACCOUNTING_FORMAT`](#nemo_gym-orchestration-recovery-FOLLOWUP_ACCOUNTING_FORMAT) [`FRESH_RESTART_REASON`](#nemo_gym-orchestration-recovery-FRESH_RESTART_REASON) [`GATE_FAILURE_LIMIT`](#nemo_gym-orchestration-recovery-GATE_FAILURE_LIMIT) [`GRADING_ARTIFACTS`](#nemo_gym-orchestration-recovery-GRADING_ARTIFACTS) [`INFRA_RETRIES_FILE`](#nemo_gym-orchestration-recovery-INFRA_RETRIES_FILE) [`INFRA_RETRY_EXTENSION_FILE`](#nemo_gym-orchestration-recovery-INFRA_RETRY_EXTENSION_FILE) [`Identity`](#nemo_gym-orchestration-recovery-Identity) [`JUDGE_JOB_FAILURE_BOUND`](#nemo_gym-orchestration-recovery-JUDGE_JOB_FAILURE_BOUND) [`LENGTH_REFUSAL_KINDS`](#nemo_gym-orchestration-recovery-LENGTH_REFUSAL_KINDS) [`LIVE_JOB_STATES`](#nemo_gym-orchestration-recovery-LIVE_JOB_STATES) [`MAX_ATTEMPTS`](#nemo_gym-orchestration-recovery-MAX_ATTEMPTS) [`MAX_EXTEND_ATTEMPTS`](#nemo_gym-orchestration-recovery-MAX_EXTEND_ATTEMPTS) [`MLFLOW_NOT_REQUIRED_NOTE`](#nemo_gym-orchestration-recovery-MLFLOW_NOT_REQUIRED_NOTE) [`MLFLOW_STATUS_NOT_OBSERVED`](#nemo_gym-orchestration-recovery-MLFLOW_STATUS_NOT_OBSERVED) [`MLFLOW_STATUS_QUERY_RETRIES`](#nemo_gym-orchestration-recovery-MLFLOW_STATUS_QUERY_RETRIES) [`MLFLOW_STATUS_QUERY_TIMEOUT_S`](#nemo_gym-orchestration-recovery-MLFLOW_STATUS_QUERY_TIMEOUT_S) [`NO_INVENTORY`](#nemo_gym-orchestration-recovery-NO_INVENTORY) [`OPERATOR_CONTINUATION`](#nemo_gym-orchestration-recovery-OPERATOR_CONTINUATION) [`OPERATOR_CONTINUATION_NOTE`](#nemo_gym-orchestration-recovery-OPERATOR_CONTINUATION_NOTE) [`READER_PLACEMENT_FIELDS`](#nemo_gym-orchestration-recovery-READER_PLACEMENT_FIELDS) [`RERUN_MASKED_FLAGS`](#nemo_gym-orchestration-recovery-RERUN_MASKED_FLAGS) [`RERUN_MASKED_REASON`](#nemo_gym-orchestration-recovery-RERUN_MASKED_REASON) [`RESERVATION_WITHOUT_OUTCOME`](#nemo_gym-orchestration-recovery-RESERVATION_WITHOUT_OUTCOME) [`RESPONSE_FAILURE_CAUSES`](#nemo_gym-orchestration-recovery-RESPONSE_FAILURE_CAUSES) [`REVERIFY_JUDGE`](#nemo_gym-orchestration-recovery-REVERIFY_JUDGE) [`REVERIFY_UNGRADED`](#nemo_gym-orchestration-recovery-REVERIFY_UNGRADED) [`REVERIFY_VERIFIER`](#nemo_gym-orchestration-recovery-REVERIFY_VERIFIER) [`REVERIFY_VERIFY_TRANSPORT`](#nemo_gym-orchestration-recovery-REVERIFY_VERIFY_TRANSPORT) [`ROLLOUT_INDEX`](#nemo_gym-orchestration-recovery-ROLLOUT_INDEX) [`SAVED_ANSWER_KEY`](#nemo_gym-orchestration-recovery-SAVED_ANSWER_KEY) [`SAVED_ROW_FILES`](#nemo_gym-orchestration-recovery-SAVED_ROW_FILES) [`SCORED_UNPARSEABLE_REASON`](#nemo_gym-orchestration-recovery-SCORED_UNPARSEABLE_REASON) [`STORE_DOES_NOT_REPLAY`](#nemo_gym-orchestration-recovery-STORE_DOES_NOT_REPLAY) [`STORE_READER_UNAVAILABLE`](#nemo_gym-orchestration-recovery-STORE_READER_UNAVAILABLE) [`STORE_SKIP_KEYS`](#nemo_gym-orchestration-recovery-STORE_SKIP_KEYS) [`TASK_INDEX`](#nemo_gym-orchestration-recovery-TASK_INDEX) [`TERMINAL`](#nemo_gym-orchestration-recovery-TERMINAL) [`TRIAL_FAILURE_CAUSES`](#nemo_gym-orchestration-recovery-TRIAL_FAILURE_CAUSES) [`UNKNOWN_BUDGET_HINT`](#nemo_gym-orchestration-recovery-UNKNOWN_BUDGET_HINT) [`UNKNOWN_BUDGET_REASON`](#nemo_gym-orchestration-recovery-UNKNOWN_BUDGET_REASON) [`UNKNOWN_WALLTIME_REASON`](#nemo_gym-orchestration-recovery-UNKNOWN_WALLTIME_REASON) [`UNVERIFIED_NOTE`](#nemo_gym-orchestration-recovery-UNVERIFIED_NOTE) [`_COLLECTOR_FUNCTIONS`](#nemo_gym-orchestration-recovery-_COLLECTOR_FUNCTIONS) [`_CUT_FINISH`](#nemo_gym-orchestration-recovery-_CUT_FINISH) [`_DRIFT_KEYS_SHOWN`](#nemo_gym-orchestration-recovery-_DRIFT_KEYS_SHOWN) [`_EXTEND_RETRIES_SCOPE`](#nemo_gym-orchestration-recovery-_EXTEND_RETRIES_SCOPE) [`_FAILED_RESPONSE_STATUSES`](#nemo_gym-orchestration-recovery-_FAILED_RESPONSE_STATUSES) [`_FOLLOWUP_ENDED_STATES`](#nemo_gym-orchestration-recovery-_FOLLOWUP_ENDED_STATES) [`_INFRA_BUDGET_USED_PREFIX`](#nemo_gym-orchestration-recovery-_INFRA_BUDGET_USED_PREFIX) [`_MAX_FACTS`](#nemo_gym-orchestration-recovery-_MAX_FACTS) [`_REVERIFY_JUDGE_KINDS`](#nemo_gym-orchestration-recovery-_REVERIFY_JUDGE_KINDS) [`_SANDBOX_FAILURES`](#nemo_gym-orchestration-recovery-_SANDBOX_FAILURES) [`_TRIAL_FAILURE_STATES`](#nemo_gym-orchestration-recovery-_TRIAL_FAILURE_STATES) ### API ```python class nemo_gym.orchestration.recovery.CollectedEvidence() ``` **Bases:** `BaseModel` **`accounted_walltime_s`** `float | None = None` --- **`accounting_reconciled`** `bool = False` --- **`accounting_snapshot`** `dict[str, str | None] | None = None` --- **`accounting_unknown_reason`** `str | None = None` --- **`attempt_extension`** `int | None = None` --- **`attempt_extension_applied`** `int | None = None` --- **`attempt_extension_state`** `str = 'absent'` --- **`benchmark_metric_coverage`** `tuple[int, int, str | None] | None = None` --- **`chain_continuation_note`** `str | None = None` --- **`chain_incomplete`** `bool = False` --- **`checkpoint_enabled`** `bool | None = Field(default=None, strict=True)` --- **`collector_fallback_reason`** `str | None = None` --- **`collector_host`** `str | None = None` --- **`counter_states`** `dict[str, str] | None = None` --- **`coverage_allocation_id`** `str | None = None` --- **`coverage_scored`** `tuple[int, int] | None = None` --- **`failures`** `list[dict[str, Any]]` --- **`followup_claims`** `dict[str, str] = Field(default_factory=dict)` --- **`followup_sentinels`** `dict[str, str] = Field(default_factory=dict)` --- **`infra_retries_configured`** `int | None = None` --- **`infra_retries_state`** `str = 'absent'` --- **`infra_retries_used`** `float | None = None` --- **`infra_retry_extension`** `float | None = None` --- **`infra_retry_extension_state`** `str = 'absent'` --- **`job_chain`** `list[str] = Field(default_factory=list)` --- **`judge_job_acknowledgements`** `list[dict[str, Any]] = Field(default_factory=list)` --- **`judge_jobs`** `list[dict[str, Any]] = Field(default_factory=list)` --- **`managed_reader_error`** `str | None = None` --- **`managed_reader_unavailable`** `bool = False` --- **`managed_selection`** `list[ManagedRecoveryFact] | None = None` --- **`managed_store`** `bool = False` --- **`materialized`** `list[dict[str, Any]] | None` --- **`mlflow`** `dict[str, Any] | None = None` --- **`operator_epoch`** `OperatorEpoch | None = None` --- **`quality`** `dict[str, Any] | None = None` --- **`reader_fallback_reason`** `str | None = None` --- **`reader_host`** `str | None = None` --- **`rerun_masked_applied`** `bool | None = None` --- **`resubmit_offset`** `int = 0` --- **`results`** `list[dict[str, Any]] | None` --- **`resume_drift_keys`** `list[str] | None = None` --- **`score`** `float | None = None` --- **`source_digests`** `dict[str, str | None] | None = None` --- **`source_job_identity`** `tuple[str, int, int] | None = None` --- **`unreadable_files`** `list[str] = Field(default_factory=list)` --- **`unresolved_rejudgments`** `list[str] = Field(default_factory=list)` --- **`walltime_state`** `str = 'absent'` --- **`walltime_used_s`** `float | None = None` --- ```python class nemo_gym.orchestration.recovery.GateCriteria() ``` **Bases:** `BaseModel` The response and trial failure fractions a caller may still pass. They are validated (an invalid value is refused, never clamped) and reported, but no longer applied: the gate's only health verdict is evaluation health on the bounded failure policy's limit (`trial_verdict`), and response and trial failures are reported as facts (`failure_facts`) for consumers to judge. **`max_response_failure_fraction`** `float = GATE_FAILURE_LIMIT` --- **`max_trial_failure_fraction`** `float = GATE_FAILURE_LIMIT` --- **`model_config`** `= ConfigDict(extra='forbid', frozen=True)` --- **`policy_timeout_counts_as_model`** `bool = Field(default=True, strict=True)` --- ```python nemo_gym.orchestration.recovery.GateCriteria._fraction( value: typing.Any ) -> float ``` classmethod ```python nemo_gym.orchestration.recovery.GateCriteria.report( classification: nemo_gym.core.failure_kinds.FailureClassificationConfig | None = None ) -> dict[str, typing.Any] ``` The effective criterion and the accepted, unapplied fractions, as reported beside the gate results. ```python class nemo_gym.orchestration.recovery.IdentityEvidence() ``` **Bases:** `BaseModel` **`attempts`** `int = 0` --- **`events`** `list[tuple[str | None, str | None, str | None, str | None, str | None]] = Field(default_factory=list)` --- **`failure_id`** `str | None = None` --- **`failure_kind`** `str | None = None` --- **`http_status`** `int | None = None` --- **`identity`** `Identity` --- **`max_attempts`** `int | None = None` --- **`reason`** `str | None = None` --- **`stage`** `FailureStage | None = None` --- **`state`** `IdentityState` --- **`unscored`** `bool = False` --- ```python class nemo_gym.orchestration.recovery.IdentityState ``` **Bases:** `enum.Enum` **`EXHAUSTED`** `= 'exhausted'` --- **`JUDGE_RECOVERABLE`** `= 'judge_recoverable'` --- **`MASKED`** `= 'masked'` --- **`MISSING`** `= 'missing'` --- **`MODEL_FINAL`** `= 'model_final'` --- **`OMITTED`** `= 'omitted'` --- **`RETRYABLE`** `= 'retryable'` --- **`SCORED`** `= 'scored'` --- **`SKIPPED`** `= 'skipped'` --- **`TERMINAL`** `= 'terminal'` --- **`UNKNOWN`** `= 'unknown'` --- ```python class nemo_gym.orchestration.recovery.ManagedRecoveryFact() ``` **Bases:** `BaseModel` Read-only facts from the authoritative Store, not a second retry policy. **`attempts`** `int = Field(ge=0, strict=True)` --- **`identity`** `Identity` --- **`max_attempts`** `int = Field(ge=1, strict=True)` --- **`pending`** `bool` --- **`replayable`** `bool | None = None` --- **`rerun_approved`** `bool = False` --- **`reserved`** `bool` --- ```python class nemo_gym.orchestration.recovery.OperatorBudget() ``` **Bases:** `BaseModel` An operator's explicit budget for a run whose chain accounting is unknown. `retries` infra retries remain from now (`--extend-retries`, one by default) and `max_walltime` is the walltime cap (`--max-walltime`, raised as a recovery raises it). It only fills counters that are unknown: a readable counter is never lowered or reset. **`max_walltime`** `str` --- **`model_config`** `= ConfigDict(extra='forbid')` --- **`reason`** `str = Field(min_length=1)` --- **`retries`** `int = Field(default=1, ge=1, le=1000)` --- **`user`** `str` --- ```python class nemo_gym.orchestration.recovery.OperatorEpoch() ``` **Bases:** `BaseModel` Where the last operator continuation started the chain's accounting: its chain length and walltime baseline. **`chain_length`** `int = Field(ge=0)` --- **`walltime_s`** `float = Field(ge=0, allow_inf_nan=False)` --- ```python class nemo_gym.orchestration.recovery.OutcomePolicySource ``` **Bases:** `enum.Enum` Where the recovery outcome policy in effect came from. **`CONFIG`** `= 'config'` --- **`DEFAULT`** `= 'default'` --- **`PYTHON`** `= 'python'` --- ```python class nemo_gym.orchestration.recovery.RecoveryAction ``` **Bases:** `enum.Enum` **`FRESH_RESTART`** `= 'fresh-restart'` --- **`NOOP`** `= 'noop'` --- **`RESUBMIT`** `= 'resubmit'` --- **`REVERIFY`** `= 'reverify'` --- **`STOP`** `= 'stop'` --- **`UNKNOWN`** `= 'unknown'` --- ```python class nemo_gym.orchestration.recovery.RecoveryBudget() ``` **Bases:** `BaseModel` The run's stored infra-retry and walltime budget and what its job chain used, as the chain files record them. `max_infra_retries` and `max_walltime_s` come from the benchmark's saved `resume_config`; `infra_retries_used` from `.gym_infra_retries` and `walltime_used_s` from `.gym_accumulated_walltime` (see `resume_script`). The chain records admitted retries and allows another only while `infra_retries_used < max_infra_retries`. `None` means the evidence is unavailable and stops the decision. **`infra_retries_used`** `int | None = Field(default=None, ge=0)` --- **`job_chain`** `list[str]` --- **`live_job`** `bool | None` --- **`live_job_name`** `str | None` --- **`max_infra_retries`** `int | None = Field(default=None, ge=0)` --- **`max_walltime_s`** `float | None` --- **`walltime_capped`** `bool` --- **`walltime_used_s`** `float | None = Field(default=None, ge=0)` --- ```python class nemo_gym.orchestration.recovery.RecoveryDecision() ``` **Bases:** `BaseModel` **`accounting_unknown`** `Literal['budget', 'walltime'] | None = Field(default=None, exclude=True)` --- **`action`** `RecoveryAction` --- **`complete`** `bool` --- **`counts`** `dict[str, int]` --- **`duplicate_results`** `list[Identity] = Field(default_factory=list)` --- **`evidence`** `list[IdentityEvidence]` --- **`expected`** `int` --- **`identities`** `dict[str, list[Identity]]` --- **`masked_without_artifact`** `list[Identity] = Field(default_factory=list)` --- **`operator_continuation`** `dict[str, Any] | None = None` --- **`outcome_policy`** `dict[str, Any] | None = None` --- **`reason`** `str` --- **`rerun_masked`** `list[RerunMaskedApproval] = Field(default_factory=list)` --- **`reverify_first`** `list[Identity] = Field(default_factory=list)` --- **`scheduler_live`** `bool = False` --- **`unverified`** `bool` Whether results of the run were deliberately not verified (`judge_skipped`), so it is UNVERIFIED. --- ```python class nemo_gym.orchestration.recovery.RecoveryOutcome() ``` **Bases:** `BaseModel` What `recover` did: `dispatched` names the one scheduler or judge action taken, None when nothing ran. **`decision`** `RecoveryDecision` --- **`dispatched`** `str | None` --- **`gate_task`** `dict[str, Any] | None = None` --- **`job_id`** `str | None = None` --- **`judge_job`** `dict[str, Any] | None = None` --- **`mlflow`** `dict[str, Any] | None = None` --- **`ok`** `bool` --- **`pin_plan`** `dict[str, Any] | None = None` --- **`reader_token`** `str | None = None` --- **`reverify`** `str | None = None` --- **`scheduler_refusal`** `dict[str, Any] | None = None` --- **`unsupported`** `str | None = None` --- **`walltime_plan`** `dict[str, Any] | None = None` --- ```python class nemo_gym.orchestration.recovery.RecoveryOutcomePolicyConfig() ``` **Bases:** [OutcomePolicyConfig](/nemo/gym/nemo-gym/nemo_gym/core/outcome_policy#nemo_gym-core-outcome_policy-OutcomePolicyConfig) Shared policy for eligible recovery failures. Retry permits another episode within the existing guards; verify re-judges a saved judge answer; fail leaves the failure ungraded. Settled outcomes are immutable. Unsupported context/action combinations stop, and explicit-zero scoring is not supported here. **`by_failure_kind`** `dict[str, OutcomeAction]` --- **`classification`** `FailureClassificationConfig = Field(default_factory=FailureClassificationConfig)` --- **`default_action`** `OutcomeAction = OutcomeAction.RETRY` --- **`supported_actions`** --- ```python class nemo_gym.orchestration.recovery.RerunMaskedApproval() ``` **Bases:** `BaseModel` One masked result the operator approves to run again: its identity, the attempts counted for it before the approval, and the attempt id of the masked row the approval covers (a later attempt is never covered). **`attempt`** `int = Field(ge=0)` --- **`attempts`** `int = Field(ge=0)` --- **`identity`** `Identity` --- ```python nemo_gym.orchestration.recovery._answer_cut_off( response: collections.abc.Mapping[str, typing.Any] ) -> bool ``` Whether the saved answer itself says it was cut off: its `incomplete_details.reason` is `max_output_tokens` or `max_tokens`, or its output tokens reached its `max_output_tokens`. ```python nemo_gym.orchestration.recovery._answer_fault( row: collections.abc.Mapping[str, typing.Any], collected: nemo_gym.orchestration.recovery.CollectedEvidence | None ) -> str | None ``` The model fault of a scored-zero incomplete answer. `TRUNCATED_ANSWER` when the answer carries a cutoff signal (`_cut_off`), whether the benchmark's rule or a recorded grade scored it zero: the cutoff is why it scored zero. Without a cutoff signal, a row whose reward came from a recorded grade (`_recorded_grade`) has none, as its score is its grade; otherwise `EMPTY_ANSWER` when it holds no assistant message with text (`_empty_answer`) and `MALFORMED_ANSWER` when it does. ```python nemo_gym.orchestration.recovery._bound_answer( row: collections.abc.Mapping[str, typing.Any] ) -> bool ``` Whether the collector bound the saved answer and score together (`_ng_collected_grading_binding`). ```python nemo_gym.orchestration.recovery._bound_artifact( row: collections.abc.Mapping[str, typing.Any] ) -> bool ``` Whether a saved grading artifact or the model's own failed answer resolves the row, as the gate requires: an artifact other than the model's failed answer must bind the saved answer and score. ```python nemo_gym.orchestration.recovery._bound_grading( row: collections.abc.Mapping[str, typing.Any] ) -> bool ``` Whether the row is `_graded` and every grading operation binds its saved answer and score, as the gate requires (`_ng_collected_grading_binding`). ```python nemo_gym.orchestration.recovery._bounded( items: list[str] ) -> str ``` ```python nemo_gym.orchestration.recovery._cause_chain( row: collections.abc.Mapping[str, typing.Any] ) -> list[tuple[str | None, ...]] | None ``` The event `failure_id` selects and its explicit `cause_id` chain; other saved events are observations only. Legacy rows with no event report retain their explicit terminal classification. Otherwise a selected event and every explicit cause must exist; missing references and cycles remain unknown. ```python nemo_gym.orchestration.recovery._cause_kind( row: collections.abc.Mapping[str, typing.Any] ) -> tuple[str | None, bool, str | None] ``` The kind of a failure whose producer recorded none, or an unregistered one, from its recorded cause. Returns `(kind, model_final, reason)`; the kind is None when the cause cannot be determined, with the reason. The cause is the event the row selects, else every event it saved, with the collector's facts of the model call that recorded each one (`COLLECTED_CAUSE`): * a model request that failed with HTTP 5xx or in transport is infrastructure: its own kind, retried under the cap; * an HTTP 4xx of the policy's own call is the model's final result: `model_context_limit` when the call recorded the context limit, else `model_request_error`; * an HTTP 4xx of a harness call (a user simulator, a judge, any model that is not the policy) is evaluation-side: `environment_episode_failed`, retried under the cap; * with no event, the environment's own episode deadline is evaluation-side too: `environment_episode_failed`. As for a kind (`_retry_kind`), the unselected events of a row that keeps a response are attempts, not its cause, and a cause chain that is broken or omitted names no cause. ```python nemo_gym.orchestration.recovery._clip_text( quality: collections.abc.Mapping[str, typing.Any] ) -> str ``` ```python nemo_gym.orchestration.recovery._collector_function( name: str ) ``` `COLLECT_SCRIPT`'s own projection `name`, so an in-process reader projects a saved row as the collector does. ```python nemo_gym.orchestration.recovery._count( value: typing.Any ) -> int | None ``` ```python nemo_gym.orchestration.recovery._cut_judge_reply( call: collections.abc.Mapping[str, typing.Any] ) -> bool ``` Whether a captured judge reply was cut short: an incomplete Responses reply, or a chat completion that stopped on its output cap or a content filter. ```python nemo_gym.orchestration.recovery._cut_off( row: collections.abc.Mapping[str, typing.Any] ) -> bool ``` Whether the saved answer carries an actual cutoff signal, not just an incomplete status. The answer's `incomplete_details.reason` is `max_output_tokens` or `max_tokens`; or the captured call that returned it finished for `length` or one of those reasons; or its output tokens reached its `max_output_tokens`. ```python nemo_gym.orchestration.recovery._decide_from_collected( collected: nemo_gym.orchestration.recovery.CollectedEvidence, max_infra_retries: int | None, max_walltime_s: float | None, live_job: bool | None, outcome_policy: nemo_gym.orchestration.recovery.RecoveryOutcomePolicyConfig | None = None, planned_missing_continuation: bool = False, allocation_cut: bool = False, extend_retries: int = 0, live_job_name: str | None = None, operator_budget: nemo_gym.orchestration.recovery.OperatorBudget | None = None, extend_attempts: int = 0, rerun_masked_without_artifact: bool = False ) -> nemo_gym.orchestration.recovery.RecoveryDecision ``` Decide from what `COLLECT_SCRIPT` read; a missing result file stops the decision. `extend_attempts` is the attempts this recovery asks to add to every rollout's cap (`--extend-attempts`): the Store's facts are read as they would be once it is recorded (see `_with_attempt_extension`). `rerun_masked_without_artifact` asks to run again the masked results that kept a saved answer but have no gradeable artifact and no grade (see `decide_recovery`); the run's Store reader must apply the approval, or the ask is refused. When the chain accounting is unknown, `operator_budget` may continue the run instead (see `_operator_continuation`); without it, the stop names the options that allow that. A run with no inventory and no saved row is a fresh restart once no job of it is live. `allocation_cut` (see `decide_recovery`) applies only with no live job. The infra-retry cap is the configured one plus the run's recorded extension plus `extend_retries` that this recovery asks to record. ```python nemo_gym.orchestration.recovery._empty_answer( row: collections.abc.Mapping[str, typing.Any] ) -> bool ``` Whether the saved answer holds no assistant message with text: none at all, or only ones the collector saw without non-whitespace text (`has_text` false). A message whose content was not saved counts as an answer. ```python nemo_gym.orchestration.recovery._ended_on_failed_request( row: collections.abc.Mapping[str, typing.Any], classification: nemo_gym.core.failure_kinds.FailureClassificationConfig | None = None ) -> tuple[typing.Any, collections.abc.Mapping[str, typing.Any]] | None ``` The failure kind of the request a row's rollout ended on, with that call, or None (`final_zero_trials`). The last policy call of the row's one invocation ended in a failed model operation whose failure event the call recorded, or, for an agent that recorded no call of its own, its last captured call failed that way; a policy whose own last call was rejected on length ended there, whatever its event's kind. ```python nemo_gym.orchestration.recovery._environment_reward( row: collections.abc.Mapping[str, typing.Any] ) -> bool ``` The environment's own reward record: `result.reward_info.reward` equal to the row's reward. ```python nemo_gym.orchestration.recovery._events( row: collections.abc.Mapping[str, typing.Any] ) -> list[tuple[str | None, str | None, str | None, str | None, str | None]] ``` ```python nemo_gym.orchestration.recovery._extend_attempts_refusal( collected: nemo_gym.orchestration.recovery.CollectedEvidence ) -> str | None ``` Why this run cannot take an attempt extension, or None when it can. ```python nemo_gym.orchestration.recovery._failed_request( call: typing.Any ) -> tuple[str, typing.Any] | None ``` The failure kind and HTTP status of a captured call whose final model operation failed with its own recorded request event, or None. ```python nemo_gym.orchestration.recovery._fault_text( report: dict[str, typing.Any], limit: float ) -> str ``` `evaluation-side faults: N of M (x%) is [not] below L (server S: kind n; kind n); model faults: ...`. ```python nemo_gym.orchestration.recovery._graded( row: collections.abc.Mapping[str, typing.Any] ) -> bool ``` Whether the row's one verifier recorded grading operations for it that all completed with a valid verdict. An operation without a valid verdict does not resolve the trial; the gate's join decides whether it is a fault. ```python nemo_gym.orchestration.recovery._grading_facts( row: collections.abc.Mapping[str, typing.Any] ) -> collections.abc.Mapping[str, typing.Any] ``` ```python nemo_gym.orchestration.recovery._grading_sandbox_failed( observations: typing.Any ) -> bool ``` Whether a verifier's sandbox observations report that a grading sandbox failed, or are malformed. They are lifecycle evidence of the sandboxes the verification used, not a grading record: a completed or unreported outcome, an unconfirmed or failed teardown, a failing test command's exit code and the agent's own sandbox outcome do not leave a bound grading unobserved. A verifier sandbox whose outcome is a failure, or that records an error other than its teardown's, does. ```python nemo_gym.orchestration.recovery._grading_unrecorded( row: collections.abc.Mapping[str, typing.Any] ) -> bool ``` Whether the row's verifier invocations were observed but recorded no grading: no grading operation, no verifier grading record, no saved grading artifact (`grading_artifact`) and no model-failure rule (`_model_failure`). A row from a pin without verifier observations proves nothing and is not selected. Nor is one whose verifier got an answer from a model (`_verifier_model_answered`) and wrote no judge-error marker (`_judge_error_marker`): a judge the verifier does not observe (`observe_llm_judge_grading`) records no grading however often it is asked again, and a grade it gave is never replaced. A row whose model calls all failed, or that names a judge error, was not graded. ```python nemo_gym.orchestration.recovery._guarded_objective_score( row: collections.abc.Mapping[str, typing.Any] ) -> bool ``` A guardrail-gated objective score the agent's own rubric saved (`response.metrics`): the row's reward equals the saved `aa_headline`, and that headline follows from the saved counts: 0.0 when a guardrail was violated, else the objectives passed over the objectives (0.0 without objectives). A row without any counted assertion (no world or no assertions also scores 0.0) is no evidence and stays unobserved. The counts are the rubric's own, not a re-run of its assertions. The saved answer and score are bound. ```python nemo_gym.orchestration.recovery._health_classification( classification: nemo_gym.core.failure_kinds.FailureClassificationConfig | None, policy_timeout_counts_as_model: bool ) -> nemo_gym.core.failure_kinds.FailureClassificationConfig ``` Express the existing timeout switch as a shared classification override. ```python nemo_gym.orchestration.recovery._health_view( decision: nemo_gym.orchestration.recovery.RecoveryDecision ) -> nemo_gym.orchestration.recovery.RecoveryDecision ``` `decision` as the trial gate counts it: a scored row planned for a re-judge because its judge gave no verdict (`SCORED_UNPARSEABLE_REASON`) is a scored trial whose fault, `judge_unparseable`, is evaluation-side. The recovery action is unchanged; only health reads the row as it stands until the re-judge settles it. ```python nemo_gym.orchestration.recovery._identity( row: collections.abc.Mapping[str, typing.Any] ) -> nemo_gym.orchestration.recovery.Identity | None ``` ```python nemo_gym.orchestration.recovery._incomplete_answer( row: collections.abc.Mapping[str, typing.Any] ) -> bool ``` Whether the row's saved answer is a response left incomplete, without an error. An incomplete status alone says nothing about why: an agent also marks an answer incomplete that holds no message. ```python nemo_gym.orchestration.recovery._judge_error_marker( row: collections.abc.Mapping[str, typing.Any] ) -> bool ``` Whether the row names a judge error the server wrote: a judge failure kind, a saved judge error (a label or a top-level message) or an invalid judge response (top level or inside the rubric facts). ```python nemo_gym.orchestration.recovery._judge_evaluations( facts: collections.abc.Mapping[str, typing.Any], reward: float | None ) -> bool ``` A non-empty `judge_evaluations` list without a judge error; when every evaluation's verdict label parses (`[[A=B]]`/`[[A!=B]]` or `Judgement: yes/no`), the reward is 1.0 exactly when every verdict is equal/yes. ```python nemo_gym.orchestration.recovery._judge_job_unsettled( entry: collections.abc.Mapping[str, typing.Any] ) -> bool ``` ```python nemo_gym.orchestration.recovery._judge_record( row: collections.abc.Mapping[str, typing.Any] ) -> bool ``` A saved judge verdict: a `judge_response`, a non-empty `judgments` list with a `judge_score`, or non-empty `judge_evaluations` whose parsed verdicts agree with the reward (`_judge_evaluations`). ```python nemo_gym.orchestration.recovery._judge_state( row: collections.abc.Mapping[str, typing.Any], policy: nemo_gym.orchestration.recovery.RecoveryOutcomePolicyConfig ) -> nemo_gym.orchestration.recovery.IdentityState ``` A judge failure's treatment: verify re-judges the saved answer only (none saved: stop), fail leaves it ungraded; any other action would run the policy again, so it stops. ```python nemo_gym.orchestration.recovery._judge_unparseable( row: collections.abc.Mapping[str, typing.Any] ) -> bool ``` Whether a scored row is `judge_unparseable`: its judge gave no verdict (`_unparsed_judge_verdict`) or its verifier's grading did not resolve (`_unknown_grading_outcome`). Judges are stateless: it is re-judged. ```python nemo_gym.orchestration.recovery._known_kind( kind: typing.Any ) -> bool ``` ```python nemo_gym.orchestration.recovery._last_captured_call( calls: list ) -> typing.Any ``` The call a rollout ended on, for an agent whose invocations reference none of its calls. The highest capture position (`call_index`, the durable append order). When every call recorded its start time, that call must also be the latest started; otherwise the end is ambiguous and there is none. ```python nemo_gym.orchestration.recovery._legacy_agent_outcome( row: collections.abc.Mapping[str, typing.Any] ) -> tuple[str | None, int | None] ``` Read the final agent classification without selecting an unbound diagnostic event. Older environments returned a failed agent response but did not select its report's cause. Support only a delivered agent-stage outcome with one matching observed model rejection. An internal wrapper's 500, mixed history, incomplete reports and saved completed answers cannot establish an upstream HTTP rejection. ```python nemo_gym.orchestration.recovery._length_refusal_kind( meta: typing.Any ) -> str | None ``` The kind of a captured call rejected with HTTP 4xx on length (`LENGTH_REFUSAL_KINDS`), or None. The reason is the call's `error_category` or its `local_response_reason`; the status is the call's own, its upstream's, or that of a failure event it reported. A 5xx among them is the server's fault, and an event of another kind than a rejected request (a transport failure) is its own cause, never the length's. ```python nemo_gym.orchestration.recovery._live_reason( live_job_name: str | None ) -> str ``` ```python nemo_gym.orchestration.recovery._managed_source_digest( path: pathlib.Path ) -> str | None ``` Hash one stable file with bounded memory, using the exporter's existing identity checks. Two streaming passes must agree on stat identity and bytes: a same-size in-place rewrite within one timestamp tick changes no stat field. This doubles source reads; it does not make the reader atomic against ABA writes. ```python nemo_gym.orchestration.recovery._managed_trial_metadata( artifacts: pathlib.Path, source_digests: dict, quality: dict | None, results: list[dict], store: object ) -> tuple[list[dict], bool | None] ``` Session trial metadata from the bound saved config and the Store's own inputs, after Store selection. Returns the results with `materialized_task_id` on each session row and the effective checkpoint setting, or the results unchanged and `None` when any fact is missing. Self-contained: the remote Store reader runs this source inside the saved install, on that install's Store, materialization and checkpoint APIs. ```python nemo_gym.orchestration.recovery._metric_record( row: collections.abc.Mapping[str, typing.Any] ) -> bool ``` Saved in-process metric scores: numeric `sentence_*` fields (`sentence_chrf`, `sentence_spbleu`). ```python nemo_gym.orchestration.recovery._model_failure( row: collections.abc.Mapping[str, typing.Any] ) -> bool ``` The benchmark's rule scored the model's own failed answer without grading it: the verifier marked the answer invalid (`invalid_model_response`), or the model left it incomplete, or it is empty and scored zero. ```python nemo_gym.orchestration.recovery._native_scorer_score( row: collections.abc.Mapping[str, typing.Any] ) -> bool ``` A native scorer's saved score (`native_metrics`): the row's reward equals the saved `weighted_score` in \[0, 1], the scorer saved its binary `execution` flag and its diagnostics. The score is the scorer's own, not a re-run of it. The saved answer and score are bound. ```python nemo_gym.orchestration.recovery._node_set_f1( row: collections.abc.Mapping[str, typing.Any] ) -> bool ``` A saved node-set F1 re-graded from the answer: the collector re-parses the saved answer's final `Final Answer: [...]` line, reproduces the saved predicted nodes and parse outcome, and scores them against the saved expected node list as the verifier does; the row's reward equals that re-graded F1 and the saved `f1`. The saved answer and score are bound. ```python nemo_gym.orchestration.recovery._now() -> float ``` ```python nemo_gym.orchestration.recovery._observed_incomplete_answer( row: collections.abc.Mapping[str, typing.Any] ) -> bool ``` Whether a counted incomplete answer (`_answer_fault`) is the model's own observed answer. The trial gate counts such a row as a model failure and reports it as a response failure, so its grading is not joined. Its answer must still be observed: the row recorded no failure, its one agent invocation ended completed or incomplete (an agent marks a cut-off answer incomplete) without an error, its last model call is exactly one captured call that answered in one completed operation, and that call's response is the saved answer, or, for an agent that saves its own answer envelope (its own response id), the envelope says it was cut off (`_answer_cut_off`) and that call itself finished on a cutoff (`CUTOFF_REASONS`): two independent signals that the model's own last reply was cut. A verifier that ran must have completed: a verifier error is never read as the model's fault. ```python nemo_gym.orchestration.recovery._observed_trial_operations( decision: nemo_gym.orchestration.recovery.RecoveryDecision, collected: nemo_gym.orchestration.recovery.CollectedEvidence | None, classification: nemo_gym.core.failure_kinds.FailureClassificationConfig | None = None, judge_recovery: dict[str, int] | None = None, grading_failures: list[str] | None = None, policy_recovery: dict[str, int] | None = None, verifier_recorded: list[nemo_gym.orchestration.recovery.Identity] | None = None, artifact_recorded: list[nemo_gym.orchestration.recovery.Identity] | None = None, trajectory_incomplete: dict[nemo_gym.orchestration.recovery.Identity, list[str]] | None = None, agent_failed_graded: list[nemo_gym.orchestration.recovery.Identity] | None = None ) -> bool ``` Whether every expected trial that is not a counted failure is observed: its saved answer is bound to its grading. Observed means a recorded grading bound to the saved answer and score: one completed verifier's grading operation bound to the row, its own grading record, a saved grading artifact (`GRADING_ARTIFACTS`) with the collector's binding, or the model's own failed answer. Only that decides coverage. Each trial's model/tool trajectory is joined too (its one agent invocation, turns, calls, tools and failure events), but that is observability evidence: a trial whose trajectory does not join is reported in `trajectory_incomplete` with the kinds of its missing evidence (`_trajectory_kinds`), and it is still observed. Join observed model/tool operations and the outer verifier invocation for every expected trial. A trial whose verifier recorded no grading operation and no grading record of its own is observed through a saved grading artifact (`GRADING_ARTIFACTS`) or through the model's own failed answer the benchmark's rule scored (`_model_failure`); those trials go to `artifact_recorded`, and the verifier's captured calls are its own. An environment reward record stands in for a missing verifier invocation. A source-reviewed programmatic grading operation must bind the saved response and score. A verifier that recorded no grading operation and owns no model call is observed through its own grading record instead (`_verifier_recorded`); those trials go to `verifier_recorded`. Empty captured model calls or a reward alone never establish grading coverage; old or uninstrumented verifiers remain unknown. Recovered grading attempts are reported separately and do not count toward the trial-health threshold; every failed attempt must join the same completed, result-bound grading dependency. A policy request retried inside one completed operation is recovered the same way (`policy_recovery` receives its counts): each failed attempt is an event of that operation and of the row, and the answer is the final attempt's. ```python nemo_gym.orchestration.recovery._operator_continuation( stopped: nemo_gym.orchestration.recovery.RecoveryDecision, collected: nemo_gym.orchestration.recovery.CollectedEvidence, inputs: dict[str, typing.Any], operator: nemo_gym.orchestration.recovery.OperatorBudget, configured_retries: int | None, max_walltime_s: float | None, extend_retries: int = 0 ) -> nemo_gym.orchestration.recovery.RecoveryDecision ``` Decide again under the operator's explicit budget, or keep the stop when continuing could be unsafe. The caller has checked that the scheduler answered and shows no live or pending job of the run. It fills only unreadable counters: an unknown spent-retry count so that `operator.retries` retries remain under the cap, an unknown used walltime with zero, so the walltime cap is fresh for this continuation, and an unknown extension with zero. A readable counter keeps its value. When the spent-retry count is readable, an explicit `extend_retries` (`--extend-retries`) is what it is on any recovery: that many more infra retries, recorded as the run's retry extension in the continuation's writes and journal. Saved rows are classified as for any decision and never changed. ```python nemo_gym.orchestration.recovery._percent( limit: float ) -> str ``` ```python nemo_gym.orchestration.recovery._policy_call( row: collections.abc.Mapping[str, typing.Any], call: typing.Any ) -> bool ``` Whether a captured call is the evaluated policy's: its model is the row's policy server, or none is named. ```python nemo_gym.orchestration.recovery._policy_evidence( row: collections.abc.Mapping[str, typing.Any] ) -> tuple[str | None, int | None] ``` The failure kind and HTTP status the policy selects on: the event `failure_id` selects first, then the row's own producer fields, then a validated final legacy agent outcome. Unselected events are not causes. ```python nemo_gym.orchestration.recovery._policy_replay_is_unsupported( row: collections.abc.Mapping[str, typing.Any], classification: nemo_gym.core.failure_kinds.FailureClassificationConfig | None = None ) -> bool ``` Whether replay needs a saved-answer path or model-budget accounting unavailable to recovery. ```python nemo_gym.orchestration.recovery._policy_zero_kind( row: collections.abc.Mapping[str, typing.Any] ) -> str | None ``` A consistent applied zero keeps its own kind, without reinterpreting diagnostic requests. ```python nemo_gym.orchestration.recovery._populations( counts: collections.abc.Mapping[str, int], evidence: list[nemo_gym.orchestration.recovery.IdentityEvidence] ) -> str ``` What a resume runs, what nobody owns (by kind, largest first) and the saved rows kept as they are. ```python nemo_gym.orchestration.recovery._predates_operation_observations( decision: nemo_gym.orchestration.recovery.RecoveryDecision, collected: nemo_gym.orchestration.recovery.CollectedEvidence | None ) -> bool ``` Whether every scored row was written by a Gym pin from before operation observations were recorded. Current pins write `response_metadata.model_operations` on every captured model call (a list, or null), and the collector reports whether the key was there (`model_operations_recorded`). Only an explicit False on every call of every scored row counts. A row without captured calls, or read by a collector that does not report the key, proves nothing, so the strict rule applies to it. ```python nemo_gym.orchestration.recovery._quota_reset_text( row: collections.abc.Mapping[str, typing.Any] ) -> str | None ``` The reset time a quota-stopped row names: the collected fact, else its reason (a live run's row). ```python nemo_gym.orchestration.recovery._quota_reset_unix( row: collections.abc.Mapping[str, typing.Any] ) -> float | None ``` ```python nemo_gym.orchestration.recovery._quota_wait_reason( row: collections.abc.Mapping[str, typing.Any] ) -> str | None ``` ```python nemo_gym.orchestration.recovery._recorded_by_verifier( row: collections.abc.Mapping[str, typing.Any], collected: nemo_gym.orchestration.recovery.CollectedEvidence ) -> bool ``` Whether the row's one verifier invocation saved its own grading record for it (`_verifier_recorded`). ```python nemo_gym.orchestration.recovery._recorded_grade( row: collections.abc.Mapping[str, typing.Any], collected: nemo_gym.orchestration.recovery.CollectedEvidence | None ) -> bool ``` Whether the row's reward came from a recorded grade: completed grading operations, the verifier's own grading record, or a saved grading artifact (`GRADING_ARTIFACTS`). The model's own failed answer is not one. ```python nemo_gym.orchestration.recovery._recovered_note( joined: collections.abc.Mapping[str, int] ) -> str ``` `N failed model attempt(s) were retried and completed in the same operation (...)`, or empty. ```python nemo_gym.orchestration.recovery._recovery_action( row: collections.abc.Mapping[str, typing.Any], kind: str, policy: nemo_gym.orchestration.recovery.RecoveryOutcomePolicyConfig, http_status: int | None ) -> nemo_gym.core.outcome_policy.OutcomeAction ``` Apply recovery rules to shared classification without changing the selected evidence or budgets. ```python nemo_gym.orchestration.recovery._replay_refusal( row: collections.abc.Mapping[str, typing.Any], classification: nemo_gym.core.failure_kinds.FailureClassificationConfig | None = None ) -> str | None ``` Why replay needs a saved-answer path or model-budget accounting unavailable to recovery; None when it does not. ```python nemo_gym.orchestration.recovery._repository_test_run( row: collections.abc.Mapping[str, typing.Any] ) -> bool ``` A saved repository test run of the answer's patch: `evaluation_completed` true, the run's `resolved` verdict with the row's reward equal to it (1.0 resolved, 0.0 not), saved `test_results`, no `error`, no evaluation timeout, and a resolved verdict only for a patch that applied. The saved answer and score are bound. ```python nemo_gym.orchestration.recovery._requeue_continuation( job: pathlib.Path, max_retries: int, max_walltime_s: float | None ) -> str ``` In a sentinel: reconcile a requeued chain from the terminal records in `CHAIN_ACCOUNTING_ENV` and decide. The same reconcile and decision as `gym eval resubmit --recover`: nothing else of the chain runs or waits (the sentinel checks the queue), and every allocation ended. Prints the reconciled walltime when the run continues. ```python nemo_gym.orchestration.recovery._rerun_masked_refusal( collected: nemo_gym.orchestration.recovery.CollectedEvidence ) -> str | None ``` Why this run cannot run masked results without an artifact again, or None when it can. ```python nemo_gym.orchestration.recovery._retry_kind( row: collections.abc.Mapping[str, typing.Any] ) -> str | None ``` The kind a retry decision selects on: the policy evidence, else, for a row that selects no event, the one kind every saved event names, else the failure record's own kind. Final-zero and judge decisions do not use it. ```python nemo_gym.orchestration.recovery._reward( row: collections.abc.Mapping[str, typing.Any] ) -> float | None ``` ```python nemo_gym.orchestration.recovery._row_fault_class( row: collections.abc.Mapping[str, typing.Any], fault: str, classification: nemo_gym.core.failure_kinds.FailureClassificationConfig | None ) -> nemo_gym.core.failure_kinds.FailureClass ``` ```python nemo_gym.orchestration.recovery._row_recovered_attempts( row: collections.abc.Mapping[str, typing.Any] ) -> list[tuple[str, dict[str, dict]]] ``` The row's recovered model attempts, by the completed operation that recovered them (`recovered_attempt_joins`). Each entry is an operation id and its failure events, keyed by event id, from one answered call of the row. ```python nemo_gym.orchestration.recovery._rubric_verdict( row: collections.abc.Mapping[str, typing.Any] ) -> bool ``` A saved rubric judge's verdict: `verdict` YES or NO, a finite `rubric_score` and `scoring_threshold`, no `invalid_judge_response`, and the reward 1.0 exactly when the verdict is YES and the score reaches the threshold (0.0 otherwise). The saved answer and score are bound. ```python nemo_gym.orchestration.recovery._saved_answer( row: collections.abc.Mapping[str, typing.Any] ) -> bool ``` ```python nemo_gym.orchestration.recovery._saved_attempt_cap( collected: nemo_gym.orchestration.recovery.CollectedEvidence ) -> int | None ``` The run's saved per-rollout attempt cap, from the Store's facts; None when they do not say. ```python nemo_gym.orchestration.recovery._saved_model_patch( row: collections.abc.Mapping[str, typing.Any] ) -> bool ``` Whether the row saved, intact, the workspace patch its verifier collected and graded: the collector's projected facts, else the same checks on a full saved row (`saved_model_patch`, `collected_patch`). Only a verifier that collects the patch itself can be given it back; one that reads the agent session at verify time cannot. ```python nemo_gym.orchestration.recovery._scored_fault( row: collections.abc.Mapping[str, typing.Any], collected: nemo_gym.orchestration.recovery.CollectedEvidence | None, classification: nemo_gym.core.failure_kinds.FailureClassificationConfig | None = None ) -> str | None ``` A scored row's one counted fault: its own recorded kind (`_scored_kind`), else its answer's (`_answer_fault`), else `judge_unparseable` when its saved judge evaluation recorded no verdict (`_unparsed_judge_verdict`). ```python nemo_gym.orchestration.recovery._scored_kind( row: collections.abc.Mapping[str, typing.Any], classification: nemo_gym.core.failure_kinds.FailureClassificationConfig | None = None ) -> str | None ``` The registered failure kind a scored result row records as its own outcome, keyed on its recorded HTTP status (`status_fault_kind`), or None. For example Harbor's graded `agent_timeout` is a scored main result. A persisted outcome decision governs metric admission, independently of fault attribution and the existing evaluation-health threshold. Counting a provider failure as zero does not make it a model fault or healthy. ```python nemo_gym.orchestration.recovery._scored_unmasked( row: collections.abc.Mapping[str, typing.Any] ) -> bool ``` A scored row: unmasked, with a reward and no failure kind. ```python nemo_gym.orchestration.recovery._session_reward_file( row: collections.abc.Mapping[str, typing.Any] ) -> bool ``` The verifier's own reward file of the row's session, read by the collector beside the artifacts (`resources//verifier/reward.json`): the evaluation completed with no infrastructure error and the row's reward equals the file's `reward`. Test counts and partial credit are metadata, never the grade. The saved answer and score are bound. ```python nemo_gym.orchestration.recovery._sidecar_action( row: collections.abc.Mapping[str, typing.Any], policy: nemo_gym.orchestration.recovery.RecoveryOutcomePolicyConfig ) -> nemo_gym.core.outcome_policy.OutcomeAction ``` ```python nemo_gym.orchestration.recovery._store_writer_ended( collected: nemo_gym.orchestration.recovery.CollectedEvidence, previous_job_id: str ) -> bool ``` In the in-job guard: whether the allocation that last wrote the Store has ended. The guard runs after its predecessor ended, and a chain runs one allocation at a time: every allocation the chain recorded before this one has ended. The writer is often an earlier allocation than the predecessor, for example when the predecessor timed out before it opened the Store. A writer the chain does not name stays unknown. ```python nemo_gym.orchestration.recovery._substep_tests( row: collections.abc.Mapping[str, typing.Any] ) -> bool ``` Saved per-sub-step test results: a non-empty `step_results` list of booleans, one environment result per step, `num_steps_total` and `num_steps_passed` equal to its length and its passes, `problem_accuracy` true exactly when every step passed, and the row's reward equal to it (1.0 or 0.0). The saved answer and score are bound. ```python nemo_gym.orchestration.recovery._symbolic_match( row: collections.abc.Mapping[str, typing.Any] ) -> bool ``` A saved symbolic answer check: a `library_reward` the row's reward equals, or one beside a judge verdict (the judge fallback). ```python nemo_gym.orchestration.recovery._teardown_failure_only( item: collections.abc.Mapping ) -> bool ``` Whether a sandbox observation's only failure is its teardown: the producer records a failed close or release after the work as outcome `sandbox_error` or `unknown` with that cleanup error as its `error_type`. ```python nemo_gym.orchestration.recovery._test_execution( row: collections.abc.Mapping[str, typing.Any] ) -> bool ``` A saved test run: `evaluation_completed` true with an integer `verifier_exit_code` and no `verifier_error`. ```python nemo_gym.orchestration.recovery._trajectory_kinds( observed: typing.Any ) -> list[str] ``` The kinds of trajectory evidence a trial lacks: its recorded gaps, and its agent invocations' own shape. Recorded gap codes other than `resolution_unavailable` (for example `sandbox_lifecycle_timing_unavailable`, `conversation_unavailable`, `model_call_ownership_unavailable`), then `invocation_unavailable` or `multiple_invocations`, `in_session_unrecorded` and `invocation_` for an invocation that did not record a completed status. Empty when none of these is named. ```python nemo_gym.orchestration.recovery._trajectory_text( evidence: collections.abc.Mapping[str, typing.Any] ) -> str ``` `trajectory evidence incomplete: N of M (kinds: kind n, ...)`, reported and never a condition. ```python nemo_gym.orchestration.recovery._transport_kind( kind: typing.Any ) -> bool ``` ```python nemo_gym.orchestration.recovery._unaccounted_failures( row: collections.abc.Mapping[str, typing.Any], recovered: collections.abc.Mapping[str, typing.Any] ) -> bool ``` Whether the row recorded a model failure that no recovery accounts for. Every model failure event the row saved or its captured calls reported must be a recovered attempt (`recovered`: the events the coverage join joined, with the row's own `_row_recovered_attempts`), a saved copy must agree with its call's, and every failed model operation must be one whose failure such a recovery accounts for. A trial that ended on its failed request is counted separately (`final_zero_trials`) and is never asked this. ```python nemo_gym.orchestration.recovery._unit_test_results( row: collections.abc.Mapping[str, typing.Any] ) -> bool ``` Saved unit test results of the answer's extracted code: at least one test, and the row's reward 1.0 exactly when every test passed (0.0 otherwise), or 0.0, the default penalty, for an answer whose reasoning format violated the rule (`reasoning_format_violation_rate` 1.0). A configured non-zero penalty is not saved on the row, so such a row stays unobserved. The saved answer and score are bound. ```python nemo_gym.orchestration.recovery._unknown_grading_outcome( row: collections.abc.Mapping[str, typing.Any] ) -> bool ``` Whether a scored row's verifier record reports a judge grading operation with outcome `unknown`: the grading did not resolve, so its reward is no grade. Read from the verifier invocations the row already saves. ```python nemo_gym.orchestration.recovery._unknown_label( item: nemo_gym.orchestration.recovery.IdentityEvidence ) -> str ``` The kind of an unknown identity with the rule that refused it, without naming the kind twice. ```python nemo_gym.orchestration.recovery._unparsed_judge_verdict( row: collections.abc.Mapping[str, typing.Any] ) -> bool ``` Whether a scored row's grading is `judge_unparseable` for a verdict the judge did not give: a saved judge evaluation recorded no verdict (`_unparsed_verdict`), or a pairwise game has none (`games_unjudged`). Its reward is no grade. ```python nemo_gym.orchestration.recovery._unparsed_verdict( facts: collections.abc.Mapping[str, typing.Any] ) -> bool ``` Saved judge evaluations of which one recorded no verdict: a null label, or a judge or parse error label. ```python nemo_gym.orchestration.recovery._verification_owner( agent_name: typing.Any, agent: collections.abc.Mapping[str, typing.Any] ) -> typing.Any ``` Who verifies an agent's rollouts: its resources server, or the agent itself when it names none. An agent that runs its own rubric (verifiers_agent) has no resources server; its grading record names the agent. ```python nemo_gym.orchestration.recovery._verifier_model_answered( row: collections.abc.Mapping[str, typing.Any], verifiers: list[typing.Any] ) -> bool ``` Whether one model call of the row's verifiers completed with a 2xx and a usable response, read from the call the row saved for each reference (by call id, else response id). A call the row does not hold has not answered. ```python nemo_gym.orchestration.recovery._verifier_recorded( verifier: typing.Any, row: collections.abc.Mapping[str, typing.Any], resources_server: typing.Any ) -> bool ``` Whether the row's verifier saved its own grading record for it, in place of a recorded grading operation. The shared verify path records it only for a verify execution that completed without asking any model. It must name the row's resources server (or the agent, for an agent that verifies its own rollouts), carry the saved score with a `scored` verdict and bind the saved response and score. A verifier that recorded a grading operation or owns a model call is judged by those, never by the record. ```python nemo_gym.orchestration.recovery._verify_transport_chain( events: collections.abc.Mapping[str, typing.Any], event: collections.abc.Mapping[str, typing.Any] ) -> bool ``` Whether `event` and every explicit cause below it name only a lost verify call: an agent route failure or a transport kind, with no reply status. A verifier's own failure or an HTTP reply anywhere in the chain is not one. ```python nemo_gym.orchestration.recovery._verify_transport_failure( row: collections.abc.Mapping[str, typing.Any] ) -> bool ``` Whether the row was masked because its verify call was lost in transport: its kind is an agent route failure or a transport kind, and a saved failure event of the `verify` operation has a transport-only cause chain. ```python nemo_gym.orchestration.recovery._verifying_servers( collected: nemo_gym.orchestration.recovery.CollectedEvidence ) -> str ``` The resources servers the run's agents name, from the run provenance, for an unobserved-trial reason. ```python nemo_gym.orchestration.recovery._with_attempt_extension( facts: list[nemo_gym.orchestration.recovery.ManagedRecoveryFact] | None, extend_attempts: int ) -> list[nemo_gym.orchestration.recovery.ManagedRecoveryFact] | None ``` The Store's facts as they read once `extend_attempts` more attempts are recorded for every rollout. Only a rollout the Store would run again under some cap (`replayable`) gets the higher cap, and it is pending while it has attempts left. Any other rollout keeps its cap and state: a model failure, a judge failure that kept its saved answer (its re-judgment budget is the saved cap), a saved result. ```python nemo_gym.orchestration.recovery.accounted_chain_walltime( job_chain: list[str], accounting: str, stored_walltime_s: float | None = 0 ) -> float | None ``` Elapsed seconds of all recorded allocations, only when each has one terminal accounting row. `stored_walltime_s` None means no allocation stored a walltime: the prologue that accounts the previous allocation stopped first (after an operator's cancel, for one). The records' sum is the used walltime then. ```python nemo_gym.orchestration.recovery.accounted_requeued_chain_walltime( job_chain: list[str], rows: list[tuple[str, str, int, int]], stored_walltime_s: float = 0 ) -> float | None ``` Elapsed seconds of every incarnation of a chain with a requeued allocation, when the records are complete. Every chain job needs terminal records numbered 0..n by restarts, and some job a requeue (n > 0). The stored counter must equal the elapsed time of every incarnation of the jobs before the last requeued one: an earlier requeue was reconciled and stored before the chain went on, and the last requeued incarnation stops before it adds anything, as does every later allocation. Otherwise the budget epoch is unknown. ```python nemo_gym.orchestration.recovery.agent_adaptations( decision: nemo_gym.orchestration.recovery.RecoveryDecision, collected: nemo_gym.orchestration.recovery.CollectedEvidence | None ) -> dict[str, dict[str, int]] ``` The scored rollouts' own `agent_adaptations` counts, summed by name: `{name: {"count": total, "rows": N}}`, `rows` being the rollouts with a nonzero count. An agent records what it adapted on its own before a result, such as a request the endpoint rejected for its input size; a reported fact, never a failure. ```python nemo_gym.orchestration.recovery.allocation_cut_from_accounting( rows: list[tuple[str, str, int, int]], allocation_id: str | None ) -> bool ``` Whether the first incarnation of `allocation_id` has ended, by any path (`ALLOCATION_ENDED_STATES`). Only the first incarnation (no restarts) runs work: the prologue stops a requeued one before any. A state that is absent, ambiguous or not terminal is not an ending. ```python nemo_gym.orchestration.recovery.chain_accounting( accounting: str ) -> list[tuple[str, str, int, int]] | None ``` Rows of `sacct -D -X -n -P --format=CHAIN_ACCOUNTING_FORMAT` as (job id, state, elapsed s, restarts). The state keeps its first word without a trailing `+` ("CANCELLED by 7" is CANCELLED). None when any row is malformed. ```python nemo_gym.orchestration.recovery.configured_outcome_policy( overrides: collections.abc.Mapping[str, typing.Any] | None ) -> nemo_gym.orchestration.recovery.RecoveryOutcomePolicyConfig | None ``` A benchmark's `resumable.recovery_outcome_policy` section as a policy, or None when it is not set. It is a partial override: `RecoveryOutcomePolicyConfig` merges the entries it names into its defaults, and validates them exactly as the Python API does. A bad section raises, naming each problem. ```python nemo_gym.orchestration.recovery.continuation_admission( job_dir: str, max_retries: int, max_walltime_s: float | None, exit_code: str, previous_job_id: str, previous_state: str = 'FAILED', outcome_policy: str | None = None ) -> str ``` Read saved state in the driver's installed runtime before a terminal allocation continues. The shell has already validated and persisted terminal elapsed accounting. This reader never changes artifacts, starts servers, resets budgets, or treats an exit code as sufficient retry evidence. `outcome_policy` is the run's `recovery_outcome_policy` section as JSON, rendered from its saved config, so an automatic continuation decides with the policy a recovery would apply. ```python nemo_gym.orchestration.recovery.decide_from_collected( collected: nemo_gym.orchestration.recovery.CollectedEvidence, options: typing.Any = {} ) -> nemo_gym.orchestration.recovery.RecoveryDecision ``` `_decide_from_collected`, with the reason naming any operational drift the run's latest resume crossed (see `resume_drift_note`) and a continuation that ended before it queued its follow-up, so an inspection, a recovery and a chain continuation all say it. ```python nemo_gym.orchestration.recovery.decide_recovery( materialized: collections.abc.Iterable[collections.abc.Mapping[str, typing.Any]], results: collections.abc.Iterable[collections.abc.Mapping[str, typing.Any]], failures: collections.abc.Iterable[collections.abc.Mapping[str, typing.Any]], budget: nemo_gym.orchestration.recovery.RecoveryBudget, default_max_attempts: int = 3, resubmit_offset: int = 0, outcome_policy: nemo_gym.orchestration.recovery.RecoveryOutcomePolicyConfig | None = None, managed_selection: list[nemo_gym.orchestration.recovery.ManagedRecoveryFact] | None = None, planned_missing_continuation: bool = False, allocation_cut: bool = False, rerun_masked_without_artifact: bool = False ) -> nemo_gym.orchestration.recovery.RecoveryDecision ``` Classify every materialized identity and choose the run-level action. `failures` are every failures-sidecar row in file order (`_line` is the row's line number when known). The standing row of a rollout is chosen as rollout collection chooses it (`_latest_failure_rows`): rows that `is_non_superseding_rollout_row` names are skipped, and an interrupted omitted row means the rollout is missing. Only rows at or after `resubmit_offset` count towards the attempt cap. The rows are only read. `allocation_cut` says that the allocation that last wrote the Store has ended, by any path (one of `ALLOCATION_ENDED_STATES`), and no allocation of the run is live: a reservation with no saved row was in flight when it ended and saved nothing, so it is not started for recovery purposes and the Store runs it again. A masked result with a saved answer, no gradeable artifact and no recorded grade (`masked_without_artifact_kind`) stays MASKED, named in `masked_without_artifact`, unless the Store reports an approval for it (`ManagedRecoveryFact.rerun_approved`) or `rerun_masked_without_artifact` asks for it now: it is then planned like an infrastructure-masked result, one failed attempt within its cap, and `rerun_masked` lists the retries this call would have journaled. ```python nemo_gym.orchestration.recovery.decision_classification( decision: nemo_gym.orchestration.recovery.RecoveryDecision, explicit: nemo_gym.core.failure_kinds.FailureClassificationConfig | None = None ) -> nemo_gym.core.failure_kinds.FailureClassificationConfig ``` The resolved recovery policy snapshot supplies the same mapping to all health readers. ```python nemo_gym.orchestration.recovery.effective_outcome_policy( explicit: nemo_gym.orchestration.recovery.RecoveryOutcomePolicyConfig | None, overrides: collections.abc.Mapping[str, typing.Any] | None ) -> tuple[nemo_gym.orchestration.recovery.RecoveryOutcomePolicyConfig, nemo_gym.orchestration.recovery.OutcomePolicySource] ``` The policy recovery applies and its source. Precedence: an explicit Python `outcome_policy` argument replaces the run's saved config section (it is not merged with it); the section overrides the defaults. The budget options (`extend_retries`, `max_walltime`) are separate and never change the policy. ```python nemo_gym.orchestration.recovery.failed_judge_jobs_blocking( judge_jobs: collections.abc.Sequence[collections.abc.Mapping[str, typing.Any]], acknowledgements: collections.abc.Sequence[collections.abc.Mapping[str, typing.Any]] = () ) -> list[str | None] | None ``` The ids of the last `JUDGE_JOB_FAILURE_BOUND` judge-only jobs when all ended without settling a saved answer and no acknowledgement names exactly them; else None. An acknowledgement covers the failed jobs it names, so a later pair of failures (the one more job it let queue, and the one before it) needs an acknowledgement of its own. ```python nemo_gym.orchestration.recovery.failed_judge_jobs_stop( ids: collections.abc.Sequence[str | None], where: str ) -> str ``` The stop text of `failed_judge_jobs_blocking`: the jobs, where to read them, and the explicit way on. ```python nemo_gym.orchestration.recovery.failure_facts( decision: nemo_gym.orchestration.recovery.RecoveryDecision, collected: nemo_gym.orchestration.recovery.CollectedEvidence | None ) -> dict[str, typing.Any] | None ``` Response and trial failures by cause, each with a count and a rate over the expected rollouts. Reported facts, never a verdict: no limit applies to them here. The denominator is the trial gate's (`trial_population`): every expected rollout, failed and missing ones included; rows an accepted partial stage omitted are no rollout of the run and are reported separately as `omitted`. A response failure is a scored rollout without a recorded grade (`_recorded_grade`) whose saved answer is truncated (incomplete with a cutoff signal, `_cut_off`), empty (no assistant text, `_empty_answer`) or malformed (no response with an output list, an answer the verifier marked invalid, or one left incomplete without a cutoff signal). A trial failure is a rollout in a failed, masked or unowned state (`_TRIAL_FAILURE_STATES`), a scored rollout that ended on a failed policy request, a scored rollout that records its own failure kind or whose judge recorded no verdict, or an observed grading without a valid verdict, by the cause its failure kind names (`trial_failure_cause`). ```python nemo_gym.orchestration.recovery.fault_report( kinds: collections.Counter, trials: int, policy_timeout_counts_as_model: bool = True, attributions: collections.Counter | None = None, classification: nemo_gym.core.failure_kinds.FailureClassificationConfig | None = None ) -> dict[str, typing.Any] ``` Report shared classes separately; server is only a subcategory of evaluation. ```python nemo_gym.orchestration.recovery.final_zero_trials( decision: nemo_gym.orchestration.recovery.RecoveryDecision, collected: nemo_gym.orchestration.recovery.CollectedEvidence | None, classification: nemo_gym.core.failure_kinds.FailureClassificationConfig | None = None ) -> dict[nemo_gym.orchestration.recovery.Identity, str] ``` Scored zeros with a saved policy decision or a failed final policy request, with their failure kinds. A consistent saved counted-zero decision retains its own registered failure kind. Otherwise, the last policy call of the row's one invocation ends in a failed model operation whose failure event the call recorded, and the row's reward is zero. Such a trial is settled as a failure under the benchmark protocol: it has no selectable response, and it is counted once, in the trial family. An agent that records no call of its own (it has no invocation, or its invocations reference none) ended on a failed request when its last captured call (`_last_captured_call`) failed that way; an earlier failed call is no end and stays unaccounted. The kind is keyed on the failure's HTTP status (`status_fault_kind`): a 5xx reply is the server's fault, and another model's 4xx rejection is evaluation-side. A policy whose own last call was rejected on length (`_length_refusal_kind`) ended there, whatever its event's kind: `model_context_limit` for the context window, `invalid_output_budget` for an output budget the window cannot hold. The policy's last call is its last captured one when its invocations reference none, since another model's capture may follow. ```python nemo_gym.orchestration.recovery.gate_summary_from_collected( decision: nemo_gym.orchestration.recovery.RecoveryDecision, collected: nemo_gym.orchestration.recovery.CollectedEvidence, max_unhealthy_fraction: float | None, mlflow_status: str | None = None, mlflow_run_id: str | None = None, criteria: nemo_gym.orchestration.recovery.GateCriteria | None = None, status_client_factory: collections.abc.Callable[[str], typing.Any] | None = None ) -> dict[str, typing.Any] ``` The acceptance gate task entry from the collected run: decision counts, the named score and the quality summary. `mlflow_status` and `mlflow_run_id` are the tracking run's status and id as the caller read them together. Without a status, the receipt's run is asked of the tracking server the saved config names (`query_mlflow_status`; `status_client_factory` replaces its client). `criteria` are accepted and reported under `criteria` but not applied (`GateCriteria`). The only health verdict is `evaluation_health`; `failures` reports response and trial failures by cause, never a verdict. `status` follows `gate_task_summary`: an UNVERIFIED evaluation health with nothing failed is UNVERIFIED, and a chain error turns SUCCESS or UNVERIFIED into FAILED. MLflow delivery is reported as `mlflow_delivery`, a fact that never changes the status. ```python nemo_gym.orchestration.recovery.gate_task_summary( decision: nemo_gym.orchestration.recovery.RecoveryDecision, score: float | None, summary_valid: bool | None, summary_reasons: list[str], health_passed: bool | None, health_reasons: list[str] ) -> dict[str, typing.Any] ``` One task entry of the acceptance gate's run summary, filled from the decision and the existing validators. `expected_samples` is always the full materialized inventory; `scored_samples` counts only scored and final-model identities, so a judge, infra, terminal or missing identity can never make them equal. A settled final outcome the metric does not score (`IdentityEvidence.unscored`) is stated, not counted as scored. Summary validity and response/trial health stay separate. `False` is a failure and an error; `None` (unknown or missing) is no failure but never a pass: its reason is a warning and it withholds SUCCESS. `status`, first match wins: RUNNING (a job is live); PLANNED (recovery work is eligible; only a caller that dispatches it reports RESUMING); FAILED (anything in `errors`, or the run is not a complete NOOP); UNVERIFIED (nothing failed, but summary validity or response/trial health is unknown, so acceptance is not proven); SUCCESS. ```python nemo_gym.orchestration.recovery.grading_artifact( row: collections.abc.Mapping[str, typing.Any] ) -> str | None ``` The name of the first saved grading artifact (`GRADING_ARTIFACTS`) that records the row's grading. ```python nemo_gym.orchestration.recovery.health_check_facts( quality: collections.abc.Mapping[str, typing.Any] | None ) -> list[str] ``` Named health-check facts from `quality_summary.json` for the gate warnings, kept even below any criterion. Nonzero `run.issues` counts, checks `run.artifacts.coverage` did not fully evaluate (any unobserved or ignored), and `run.ignored_checks`. A field that is absent or malformed is reported as unknown, never as clean. ```python nemo_gym.orchestration.recovery.health_verdict( evaluation: dict[str, typing.Any] | None, facts: dict[str, typing.Any] | None ) -> str ``` `evaluation health: PASS; response failures: 4.70% (truncated 41, empty 6 of 1000); trial failures: ...; metric over 1000 of 1000 trials`. ```python nemo_gym.orchestration.recovery.infrastructure_timeouts( decision: nemo_gym.orchestration.recovery.RecoveryDecision, collected: nemo_gym.orchestration.recovery.CollectedEvidence | None, classification: nemo_gym.core.failure_kinds.FailureClassificationConfig | None = None ) -> set[nemo_gym.orchestration.recovery.Identity] ``` Scored identities whose graded result is an agent timeout with a server-side fault in its window. A Harbor graded zero a run without a Store cannot reopen (`is_infrastructure_agent_timeout`): it stays in the results, and the trial gate counts it once, as `AGENT_TIMEOUT_INFRASTRUCTURE`, an evaluation-side fault. ```python nemo_gym.orchestration.recovery.judge_job_note( job: collections.abc.Mapping[str, typing.Any] ) -> str ``` One line on a judge-only job of the run (`CollectedEvidence.judge_jobs`): queued, live, ended without a receipt, or what its receipt says it settled. ```python nemo_gym.orchestration.recovery.live_chain_job( rows: list[tuple[str, str, int, int]] | None, job_chain: list[str] ) -> str | None ``` The chain job that still waits or runs, as 'id (STATE)' from its latest incarnation; None when none does. ```python nemo_gym.orchestration.recovery.masked_without_artifact_kind( row: collections.abc.Mapping[str, typing.Any], classification: nemo_gym.core.failure_kinds.FailureClassificationConfig | None = None ) -> str | None ``` The kind of a masked result that is never measured nor run again, though infrastructure left it unmeasured; else None. A masked result `masked_infrastructure_kind` rejects only because it kept a saved answer (so every other rule of that predicate applies: the kind must be registered, evaluation-side and not graded again), that has no recorded grade (`has_recorded_grade`) and no re-verification cause (`reverification_cause`): its saved answer has no gradeable artifact, a resume keeps it and re-verification cannot grade it. It runs again only with the explicit `--rerun-masked-without-artifact`. `row` is a collector projection, or a saved row passed through `saved_row_evidence`. ```python nemo_gym.orchestration.recovery.metric_coverage( decision: nemo_gym.orchestration.recovery.RecoveryDecision, collected: nemo_gym.orchestration.recovery.CollectedEvidence | None ) -> dict[str, typing.Any] ``` How many expected trials the benchmark's metric covered: `{"trials": N, "expected": M, "source": ...}`. A benchmark that declares the coverage of its headline metric in its aggregate (`metric_coverage/trials`, `metric_coverage/expected` and an optional `metric_coverage/name`) is the source (`benchmark`), with its own denominator and the metric's name. Otherwise the source is collection's own count when its coverage report has one for this inventory (`coverage_report`): the rollouts whose score entered the metrics, a failure counted as zero by configuration included. Otherwise it is read from the saved rows (`saved_rows`): the scored results and the failures their producer marked as counted zeros, which is what collection hands the metrics unless it counts more by configuration. A failure the benchmark leaves out of its metric, such as an agent deadline that kept its answer, is a trial the metric did not cover. The denominator is the trial gate's: every expected trial, less the rows an accepted partial stage omitted. ```python nemo_gym.orchestration.recovery.mlflow_delivery( problems: collections.abc.Sequence[str], tracking_required: bool, status_error: str | None = None ) -> dict[str, typing.Any] ``` The MLflow delivery fact of a run: `not_required` by design, `verified` when `mlflow_errors` found nothing, `pending` while there is no receipt and no live run to judge or only the run's status was not queried, else `unverified` with its reasons. A fact for consumers to read, never a verdict: it does not touch evaluation health or the gate's status. A status that was neither given nor queried reads "status not queried"; a query that failed reads "status query failed: ...": never a silent "not observed". ```python nemo_gym.orchestration.recovery.mlflow_errors( local: collections.abc.Mapping[str, typing.Any] | None, observed_status: str | None, configured: bool | None = None, observed_run_id: str | None = None ) -> list[str] ``` MLflow delivery errors from the collected exporter files and the server status the caller observed. `local` holds the live exporter's run id (`rollouts_mlflow_run_id`) and the post-run export receipt (`rollouts_mlflow_export_done`, checked against the current artifact digests). Only a current post-run receipt with an observed FINISHED passes. A live run is not accepted here: live uploads log failures and continue, so neither its run id nor FINISHED shows the configured payloads arrived. A live run beside a post-run export leaves the observed status unbound. `configured` is the health report's `run.provenance.mlflow.configured`, a names-only fact about the saved config snapshot (None when not reported); it only qualifies the message. The observed status is bound to a run: with a post-run receipt, `observed_run_id` must name the receipt's run. `BaseExporter.export` writes the receipt only after every upload returned and the artifacts did not change, but its teardown still sets FINISHED on a partial export (`run_failed` is the process's run, not the export's), so a FINISHED from any other run proves nothing. No evidence is always an error, never a run that needed none: exporter activation and whether tracking was required or disabled are not recorded. ```python nemo_gym.orchestration.recovery.mlflow_notes( local: collections.abc.Mapping[str, typing.Any] | None ) -> list[str] ``` What the gate states about a passing MLflow part: a current post-run export beside a live run wins. ```python nemo_gym.orchestration.recovery.mlflow_tracking_required( collected: nemo_gym.orchestration.recovery.CollectedEvidence | None ) -> bool ``` False only when the run's saved config snapshot sets `mlflow_tracking_required` to false; absent, unread or any other value means tracking is required. The health report's provenance states it when its pin did; for a run whose report does not, the collector's read of the saved `rollouts_resolved_config.yaml` does. ```python nemo_gym.orchestration.recovery.omission_summary( evidence: collections.abc.Iterable[nemo_gym.orchestration.recovery.IdentityEvidence] ) -> str ``` The omitted identities by stage and policy, e.g. "1 omitted by an accepted partial stage (stage_index 1, tolerate_unresolved)". ```python nemo_gym.orchestration.recovery.output_budget_refits( collected: nemo_gym.orchestration.recovery.CollectedEvidence | None ) -> dict[str, int] ``` How many model calls in the saved results were sent again with an output budget lowered to fit the context window (`output_budget_refit` in a call's response metadata), and in how many result rows. A count, never a failure: the call generated within the window it was served at. A reader flags a cell whose requests were budget-capped from it. ```python nemo_gym.orchestration.recovery.quality_inputs( decision: nemo_gym.orchestration.recovery.RecoveryDecision, collected: nemo_gym.orchestration.recovery.CollectedEvidence, max_unhealthy_fraction: float | None ) -> tuple[bool | None, list[str], bool | None, list[str]] ``` Summary validity and per-rollout health from the run's `quality_summary.json`, kept separate. Validity is the health check's own accounting verdict (`run.artifacts.healthy`, null for a partial summary), and must count the same expected identities as the inventory. Per-rollout health is the rollout verdict count (`run.verdicts`), judged only against an explicit `max_unhealthy_fraction`; it is not separately validated response or trial health, which this adapter has no separate evidence for. Any missing piece is `None`. ```python nemo_gym.orchestration.recovery.query_mlflow_status( tracking_uri: str, run_id: str, client_factory: collections.abc.Callable[[str], typing.Any] | None = None ) -> tuple[str | None, str | None] ``` The status of `run_id` on the tracking server, as `(status, None)`, or `(None, error)` when it could not be read: a missing client, an unreachable or refusing server, a missing run. Short timeout and one retry; `client_factory` builds the client from the URI (default `mlflow.MlflowClient`). ```python nemo_gym.orchestration.recovery.reconcile_requeued_chain( collected: nemo_gym.orchestration.recovery.CollectedEvidence, rows: list[tuple[str, str, int, int]] ) -> nemo_gym.orchestration.recovery.CollectedEvidence ``` Know a requeued chain's budget again once every incarnation has a terminal scheduler record. Only the requeue marker is reconciled: the counters themselves stayed readable and the missing fact is the elapsed time of the requeued incarnations, which complete terminal records give. Any other unknown reason stays unknown. ```python nemo_gym.orchestration.recovery.recovered_attempt_joins( decision: nemo_gym.orchestration.recovery.RecoveryDecision, collected: nemo_gym.orchestration.recovery.CollectedEvidence | None ) -> dict[str, int] ``` Failed model attempts each joined to the completed operation that recovered them, in a scored trial's own call. Counted like the run's observed upstream recovery (`observed_upstream_recovery`): a model-request failure event whose operation completed in an answered call. Here the event must be in the failure report of the same call whose one completed operation it names, as a failed attempt other than the answer, and the operation's attempts must account for every failed attempt of it (`http_attempts - 1 + transport_failures`, the HTTP-less ones being the transport failures). A copy the row saved must agree, and no event, attempt or operation joins twice. The join holds for any model the trial called (its policy, a judge or a simulated user). It says nothing about how the trial was graded: whether the final answer is bound to its grading is the coverage join's question (`_observed_trial_operations`). ```python nemo_gym.orchestration.recovery.requeue_note( job_chain: list[str], rows: list[tuple[str, str, int, int]] ) -> str | None ``` The last requeued job of a chain and what its allocations account for, as a decision states it. ```python nemo_gym.orchestration.recovery.response_health( decision: nemo_gym.orchestration.recovery.RecoveryDecision, quality: collections.abc.Mapping[str, typing.Any] | None, limit: float = GATE_FAILURE_LIMIT ) -> tuple[bool | None, str] ``` The response gate from `run.stats.responses` of the health report: faulted over selected responses. Evaluated only when every scored rollout has a verified response population and every selected call id is unique across the run (a report that predates it, an unverified call scope, a response selected twice, or zero selected responses is unknown, neither pass nor fail). Failed trials have no response row and are left out, as the policy leaves failed attempts out of the response denominator. ```python nemo_gym.orchestration.recovery.resume_drift_note( keys: list[str] | None ) -> str | None ``` The sentence a decision reason carries when the run's latest resume crossed operational drift. ```python nemo_gym.orchestration.recovery.retry_budget( collected: nemo_gym.orchestration.recovery.CollectedEvidence ) -> dict[str, dict[str, int | None]] ``` The run's two retry budgets as the gate reports them, each its configured cap and its recorded extension. `infra` is the chain's infra-retry cap (`--extend-retries`); `attempts` is every rollout's attempt cap (`--extend-attempts`). A value that could not be read is None. ```python nemo_gym.orchestration.recovery.reverification_cause( row: collections.abc.Mapping[str, typing.Any] ) -> str | None ``` Why a saved result row is re-verified from its saved answer, or None when it is not. `REVERIFY_JUDGE`: masked because its judge failed (`judge_failed`, `judge_unparseable`), or scored while its judge gave no verdict or its grading did not resolve (`_judge_unparseable`). `REVERIFY_VERIFY_TRANSPORT`: masked because its verify call was lost in transport (`_verify_transport_failure`). `REVERIFY_VERIFIER`: masked because its verifier produced no verdict (`verifier_error`), with the workspace patch it graded saved intact (`saved_model_patch`): the verifier runs again on that patch, never the agent. `REVERIFY_UNGRADED`: scored, but no grading was recorded for it (`_grading_unrecorded`). Only a result with a saved answer qualifies; a failures-sidecar row, a result a re-verification already settled (its settlement is its grading) and every other masked result do not. `row` is a collector projection, or a saved row passed through `saved_row_evidence`. ```python nemo_gym.orchestration.recovery.saved_row_evidence( row: collections.abc.Mapping[str, typing.Any] ) -> dict[str, typing.Any] ``` A full saved row with the evidence fields `COLLECT_SCRIPT` projects for the grading predicates: its scalar grading facts, whether each assistant message of its answer has text, and its verifier invocations. ```python nemo_gym.orchestration.recovery.select_managed_evidence( collected: nemo_gym.orchestration.recovery.CollectedEvidence, job_dir: pathlib.Path ) -> nemo_gym.orchestration.recovery.CollectedEvidence ``` Read locally accessible Store history in this installed Gym process; never copy its selection rules. The plain remote collector remains dependency-free. Call only on the same host as the installed caller, after collection. Both snapshots must include the manifest and journal and remain byte-identical. ```python nemo_gym.orchestration.recovery.status_fault_kind( kind: typing.Any, http_status: typing.Any, policy: bool = True, classification: nemo_gym.core.failure_kinds.FailureClassificationConfig | None = None ) -> typing.Any ``` The shared reporting label for observed HTTP evidence; saved events remain unchanged. ```python nemo_gym.orchestration.recovery.stopped_followup( job_id: str, accounting: str ) -> str | None ``` Why the chain stopped when the follow-up `job_id` that its last allocation claimed is not in the chain. `accounting` is `sacct -D -X -n -P --format=FOLLOWUP_ACCOUNTING_FORMAT -j job_id`. The prologue records an allocation in the chain before anything else, so a follow-up that started and ended without that entry ran no work. None when the records do not show that: the follow-up never started, is live, or its record is ambiguous. ```python nemo_gym.orchestration.recovery.trajectory_evidence( incomplete: collections.abc.Mapping[nemo_gym.orchestration.recovery.Identity, list[str]], trials: int ) -> dict[str, typing.Any] ``` Trials whose trajectory evidence is incomplete, over the trials, with the count of each missing kind. ```python nemo_gym.orchestration.recovery.trial_boundary( quality: collections.abc.Mapping[str, typing.Any] | None ) -> str ``` Why trial failures outside the sidecar cannot be ruled out, from the run's own persisted facts. Every persisted value is type-checked: a malformed report is named as malformed, never read as a fact. ```python nemo_gym.orchestration.recovery.trial_counts( decision: nemo_gym.orchestration.recovery.RecoveryDecision, collected: nemo_gym.orchestration.recovery.CollectedEvidence | None ) -> dict[str, typing.Any] | None ``` Acceptance's trial counts: observed healthy, unhealthy and unobserved, over the expected trials. A saved grading artifact or the model's own failed answer (`grading_artifact`, `_model_failure`) resolves a trial too. `unhealthy` are the failures the trial gate counts (`trial_faults`, model and evaluation-side). When the gate's observation join covers every other trial, its observed gradings without a valid verdict are unhealthy too and nothing is unobserved. Otherwise `unobserved` are the other scored trials whose verifier recorded neither completed grading operations bound to the saved row nor its own grading record: nothing resolved their grading, so they are unverified, never healthy. `observed_healthy` are the rest; `verifier_recorded` of them are observed only through the verifier's grading record. None until the run is complete. ```python nemo_gym.orchestration.recovery.trial_failure_cause( kind: typing.Any ) -> str ``` A trial failure's cause from its failure kind: request, judge, sandbox, harness, timeout, allocation, or other. `allocation`: the rollout was in flight at repeated ends of its allocation (`ROLLOUT_ALLOCATION_ENDED`). ```python nemo_gym.orchestration.recovery.trial_faults( decision: nemo_gym.orchestration.recovery.RecoveryDecision, collected: nemo_gym.orchestration.recovery.CollectedEvidence | None, attributions: collections.Counter | None = None, classification: nemo_gym.core.failure_kinds.FailureClassificationConfig | None = None ) -> collections.Counter ``` Count final trial failures once; classify selected outcomes, never recovered observations. ```python nemo_gym.orchestration.recovery.trial_health( decision: nemo_gym.orchestration.recovery.RecoveryDecision, collected: nemo_gym.orchestration.recovery.CollectedEvidence | None = None, limit: float = GATE_FAILURE_LIMIT, policy_timeout_counts_as_model: bool = True, classification: nemo_gym.core.failure_kinds.FailureClassificationConfig | None = None ) -> tuple[bool | None, str] ``` Evaluation health from the trial gate (`trial_verdict`): passed, failed or None, with its reason. ```python nemo_gym.orchestration.recovery.trial_population( decision: nemo_gym.orchestration.recovery.RecoveryDecision ) -> int ``` The trial denominator: every expected trial, including repeats, failed and missing ones, less the rows an accepted partial stage omitted (no trial: reported separately, never in a numerator or denominator). ```python nemo_gym.orchestration.recovery.trial_verdict( decision: nemo_gym.orchestration.recovery.RecoveryDecision, collected: nemo_gym.orchestration.recovery.CollectedEvidence | None = None, limit: float = GATE_FAILURE_LIMIT, policy_timeout_counts_as_model: bool = True, classification: nemo_gym.core.failure_kinds.FailureClassificationConfig | None = None ) -> tuple[bool | None, str, dict[str, typing.Any] | None] ``` Evaluation health: evaluation-side trial faults over the expected trials, including repeats. The counted failures are `trial_faults`; a model fault (`classify_failure`) is reported with its kind and never fails. Only the evaluation-side share is judged against the limit: at or above it the gate fails. Below it, it passes only with an exact count, when actual per-identity operation outcomes cover the observed model/tool and grading path of every trial that is not a recorded failure (`_observed_trial_operations`); an observed grading without a valid verdict is an evaluation-side fault. Otherwise the run is UNVERIFIED: a judge, verifier or harness error scored as a zero in the main output carries no failure record and cannot be ruled out, so the known count is a lower bound. That withholds a pass, never fails health. The denominator is the full materialized inventory less accepted omissions. A run that is not complete has unaccounted trials, so the gate is not evaluated. The report is `status` (PASS, FAIL or UNVERIFIED), the evaluation-side rate, `exact` and the fault counts. ```python nemo_gym.orchestration.recovery.unsettled_chain_note( decision: nemo_gym.orchestration.recovery.RecoveryDecision ) -> str ``` The chain stopped before the run settled: how many rollouts are unsettled, by state, so the stop reads as an incomplete run and never as a finished one. Unsettled is wider than the coverage report's `open` (the count in the chain's own stop line): it also holds final failures that were not counted as zero. ```python nemo_gym.orchestration.recovery.ACCUMULATED_WALLTIME_FILE = '.gym_accumulated_walltime' ``` ```python nemo_gym.orchestration.recovery.ACKNOWLEDGE_JUDGE_JOBS_FLAGS = '--acknowledge-failed-judge-jobs --acknowledge-reason' ``` ```python nemo_gym.orchestration.recovery.ALLOCATION_ENDED_STATES = frozenset({'TIMEOUT', 'PREEMPTED', 'NODE_FAIL', 'BOOT_FAIL', 'CANCELLED', 'FAILE... ``` ```python nemo_gym.orchestration.recovery.COLLECTED_CAUSE = '_ng_collected_cause' ``` ```python nemo_gym.orchestration.recovery.COUNTS_AS_ZERO = '_ng_failure_counts_as_zero' ``` ```python nemo_gym.orchestration.recovery.CUTOFF_REASONS = frozenset({'length', 'max_output_tokens', 'max_tokens'}) ``` ```python nemo_gym.orchestration.recovery.EXTEND_ATTEMPTS_HINT = 'an exhausted identity whose attempts went to infrastructure failures or allocat... ``` ```python nemo_gym.orchestration.recovery.FAILURE_CLASS = '_ng_failure_class' ``` ```python nemo_gym.orchestration.recovery.FAILURE_EVENTS = '_ng_failure_events' ``` ```python nemo_gym.orchestration.recovery.FAILURE_ID = '_ng_failure_id' ``` ```python nemo_gym.orchestration.recovery.FOLLOWUP_ACCOUNTING_FORMAT = 'JobIDRaw,State,ElapsedRaw,ExitCode' ``` ```python nemo_gym.orchestration.recovery.FRESH_RESTART_REASON = f'nothing to recover: {NO_INVENTORY}; resubmit it as a fresh run with a longer w... ``` ```python nemo_gym.orchestration.recovery.GATE_FAILURE_LIMIT = 0.02 ``` ```python nemo_gym.orchestration.recovery.GRADING_ARTIFACTS = (('judge_record', _judge_record), ('rubric_verdict', _rubric_verdict), ('symboli... ``` ```python nemo_gym.orchestration.recovery.INFRA_RETRIES_FILE = '.gym_infra_retries' ``` ```python nemo_gym.orchestration.recovery.INFRA_RETRY_EXTENSION_FILE = '.gym_infra_retry_extension' ``` ```python nemo_gym.orchestration.recovery.Identity = tuple[int, int] ``` ```python nemo_gym.orchestration.recovery.JUDGE_JOB_FAILURE_BOUND = 2 ``` ```python nemo_gym.orchestration.recovery.LENGTH_REFUSAL_KINDS = {'context_length_exceeded': MODEL_CONTEXT_LIMIT, 'invalid_output_token_parameter... ``` ```python nemo_gym.orchestration.recovery.LIVE_JOB_STATES = frozenset({'PENDING', 'RUNNING', 'CONFIGURING', 'COMPLETING', 'SUSPENDED', 'REQU... ``` ```python nemo_gym.orchestration.recovery.MAX_ATTEMPTS = '_ng_failure_max_attempts' ``` ```python nemo_gym.orchestration.recovery.MAX_EXTEND_ATTEMPTS = 100 ``` ```python nemo_gym.orchestration.recovery.MLFLOW_NOT_REQUIRED_NOTE = 'no MLflow export by design (mlflow_tracking_required is false in the saved conf... ``` ```python nemo_gym.orchestration.recovery.MLFLOW_STATUS_NOT_OBSERVED = 'MLflow terminal status not observed' ``` ```python nemo_gym.orchestration.recovery.MLFLOW_STATUS_QUERY_RETRIES = '1' ``` ```python nemo_gym.orchestration.recovery.MLFLOW_STATUS_QUERY_TIMEOUT_S = '10' ``` ```python nemo_gym.orchestration.recovery.NO_INVENTORY = 'the run ended before the task inventory was written' ``` ```python nemo_gym.orchestration.recovery.OPERATOR_CONTINUATION = 'OPERATOR-CONTINUATION' ``` ```python nemo_gym.orchestration.recovery.OPERATOR_CONTINUATION_NOTE = f'Operator continuation: an explicit budget filled the unknown accounting; see {... ``` ```python nemo_gym.orchestration.recovery.READER_PLACEMENT_FIELDS = ('collector_host', 'collector_fallback_reason', 'reader_host', 'reader_fallback_... ``` ```python nemo_gym.orchestration.recovery.RERUN_MASKED_FLAGS = '--rerun-masked-without-artifact --rerun-reason' ``` ```python nemo_gym.orchestration.recovery.RERUN_MASKED_REASON = f'retained, never run again, not measured: it reruns only with {RERUN_MASKED_FLA... ``` ```python nemo_gym.orchestration.recovery.RESERVATION_WITHOUT_OUTCOME = 'reservation_without_outcome' ``` ```python nemo_gym.orchestration.recovery.RESPONSE_FAILURE_CAUSES = ('truncated', 'empty', 'malformed') ``` ```python nemo_gym.orchestration.recovery.REVERIFY_JUDGE = 'judge' ``` ```python nemo_gym.orchestration.recovery.REVERIFY_UNGRADED = 'ungraded' ``` ```python nemo_gym.orchestration.recovery.REVERIFY_VERIFIER = 'verifier' ``` ```python nemo_gym.orchestration.recovery.REVERIFY_VERIFY_TRANSPORT = 'verify_transport' ``` ```python nemo_gym.orchestration.recovery.ROLLOUT_INDEX = '_ng_rollout_index' ``` ```python nemo_gym.orchestration.recovery.SAVED_ANSWER_KEY = '_ng_collected_saved_answer' ``` ```python nemo_gym.orchestration.recovery.SAVED_ROW_FILES = ('rollouts_materialized_inputs.jsonl', 'rollouts.jsonl', 'rollouts_failures.json... ``` ```python nemo_gym.orchestration.recovery.SCORED_UNPARSEABLE_REASON = 'judge gave no verdict (unparseable); re-judge planned' ``` ```python nemo_gym.orchestration.recovery.STORE_DOES_NOT_REPLAY = "the policy retries it but the run's Store does not plan it again" ``` ```python nemo_gym.orchestration.recovery.STORE_READER_UNAVAILABLE = 'the run was written by a rollout store and its Store reader could not run, so n... ``` ```python nemo_gym.orchestration.recovery.STORE_SKIP_KEYS = frozenset({'ng_trajectory', 'ng_agent_observations'}) ``` ```python nemo_gym.orchestration.recovery.TASK_INDEX = '_ng_task_index' ``` ```python nemo_gym.orchestration.recovery.TERMINAL = '_ng_failure_terminal' ``` ```python nemo_gym.orchestration.recovery.TRIAL_FAILURE_CAUSES = ('request', 'judge', 'sandbox', 'harness', 'timeout', 'allocation', 'other') ``` ```python nemo_gym.orchestration.recovery.UNKNOWN_BUDGET_HINT = 'An operator may continue it with an explicit budget: gym eval resubmit --recove... ``` ```python nemo_gym.orchestration.recovery.UNKNOWN_BUDGET_REASON = 'the job chain or scheduler state is unavailable, so the budget is unknown' ``` ```python nemo_gym.orchestration.recovery.UNKNOWN_WALLTIME_REASON = 'the run caps its walltime but the used walltime is unavailable' ``` ```python nemo_gym.orchestration.recovery.UNVERIFIED_NOTE = 'unverified: this withholds a pass and is no health failure' ``` ```python nemo_gym.orchestration.recovery._COLLECTOR_FUNCTIONS: dict[str, Any] = {} ``` ```python nemo_gym.orchestration.recovery._CUT_FINISH = frozenset({'length', 'content_filter', 'max_tokens', 'max_output_tokens'}) ``` ```python nemo_gym.orchestration.recovery._DRIFT_KEYS_SHOWN = 20 ``` ```python nemo_gym.orchestration.recovery._EXTEND_RETRIES_SCOPE = '--extend-retries raises infra retries only; it does not raise the per-identity ... ``` ```python nemo_gym.orchestration.recovery._FAILED_RESPONSE_STATUSES = frozenset({'failed', 'incomplete', 'cancelled'}) ``` ```python nemo_gym.orchestration.recovery._FOLLOWUP_ENDED_STATES = frozenset({'FAILED', 'CANCELLED', 'NODE_FAIL', 'BOOT_FAIL', 'PREEMPTED', 'OUT_OF... ``` ```python nemo_gym.orchestration.recovery._INFRA_BUDGET_USED_PREFIX = 'the stored infra-retry budget is used' ``` ```python nemo_gym.orchestration.recovery._MAX_FACTS = 20 ``` ```python nemo_gym.orchestration.recovery._REVERIFY_JUDGE_KINDS = frozenset({JUDGE_FAILED, JUDGE_UNPARSEABLE}) ``` ```python nemo_gym.orchestration.recovery._SANDBOX_FAILURES = frozenset({'failed', 'timeout', 'oom', 'sandbox_error', 'cancelled'}) ``` ```python nemo_gym.orchestration.recovery._TRIAL_FAILURE_STATES = (IdentityState.MODEL_FINAL, IdentityState.JUDGE_RECOVERABLE, IdentityState.RETRY... ```