nemo_gym.orchestration.recovery
nemo_gym.orchestration.recovery
Whether gym eval resubmit may queue a run again, decided from the run’s persisted evidence only.
Every expected identity (_ng_task_index, _ng_rollout_index) of the materialized inventory gets one state:
scored: a result row that is not a judge failure. Kept as saved; never re-run.model_final: a failures-sidecar row marked_ng_failure_counts_as_zero: the evaluated model’s terminal failure, a scored zero that no infra retry may replace; or anagent_timeoutrow that kept the agent’s answer (is_answered_agent_deadline), the model’s result under its own deadline, which no resume runs again.judge_recoverable: a result row of classjudge_failed. Its saved answer is re-judged bygym eval reverify --judge-failed-only; queueing the policy again is never needed for it.retryable: the last sidecar row is non-terminal and under its recorded attempt cap: a Gym resume retries it.exhausted: the last sidecar row is non-terminal and at its attempt cap: only a resubmit gives it a fresh budget.terminal: a terminal sidecar row that is not a scored zero; excluded, and the run cannot become complete.missing: no row at all (interrupted or never started); a resume restarts its input.masked: a result row the producer masked or did not evaluate. Collection keeps it as completed and never runs it again, so it is retained, but it is not measured and the run is not complete.omitted: a planned row that a staged collection’s accepted partial stage omitted. It is retained, never planned again and not measured, and the run is complete without it.unknown: a failure whose kind is unregistered or absent, or a row whose identity is invalid; nobody owns it.
The run-level action follows: stop for a spent or unknown resubmit budget, or for an unknown identity that a resume
would run again; noop when nothing is left to do, reverify when only judge failures remain, else resubmit.
A run that ended before its inventory was written has nothing to recover: fresh-restart queues it again as a fresh
run with a longer walltime.
An unknown identity that a resume never runs again (the Store or the saved result holds it) does not block the
missing and retryable ones: the decision reports it apart from what is retried and what is retained. A decision never
reports a run as complete while an identity is terminal, missing or unknown.
Module Contents
Classes
Functions
Data
API
Bases: BaseModel
Bases: BaseModel
The response and trial failure fractions a caller may still pass. They are validated (an invalid value is
refused, never clamped) and reported, but no longer applied: the gate’s only health verdict is evaluation health
on the bounded failure policy’s limit (trial_verdict), and response and trial failures are reported as facts
(failure_facts) for consumers to judge.
The effective criterion and the accepted, unapplied fractions, as reported beside the gate results.
Bases: BaseModel
Bases: enum.Enum
Bases: BaseModel
Read-only facts from the authoritative Store, not a second retry policy.
Bases: BaseModel
An operator’s explicit budget for a run whose chain accounting is unknown.
retries infra retries remain from now (--extend-retries, one by default) and max_walltime is the
walltime cap (--max-walltime, raised as a recovery raises it). It only fills counters that are unknown: a
readable counter is never lowered or reset.
Bases: BaseModel
Where the last operator continuation started the chain’s accounting: its chain length and walltime baseline.
Bases: enum.Enum
Where the recovery outcome policy in effect came from.
Bases: enum.Enum
Bases: BaseModel
The run’s stored infra-retry and walltime budget and what its job chain used, as the chain files record them.
max_infra_retries and max_walltime_s come from the benchmark’s saved resume_config;
infra_retries_used from .gym_infra_retries and walltime_used_s from .gym_accumulated_walltime
(see resume_script). The chain records admitted retries and allows another only while
infra_retries_used < max_infra_retries. None means the evidence is unavailable and stops the decision.
Bases: BaseModel
Whether results of the run were deliberately not verified (judge_skipped), so it is UNVERIFIED.
Bases: BaseModel
What recover did: dispatched names the one scheduler or judge action taken, None when nothing ran.
Bases: OutcomePolicyConfig
Shared policy for eligible recovery failures. Retry permits another episode within the existing guards; verify re-judges a saved judge answer; fail leaves the failure ungraded. Settled outcomes are immutable. Unsupported context/action combinations stop, and explicit-zero scoring is not supported here.
Bases: BaseModel
One masked result the operator approves to run again: its identity, the attempts counted for it before the approval, and the attempt id of the masked row the approval covers (a later attempt is never covered).
Whether the saved answer itself says it was cut off: its incomplete_details.reason is max_output_tokens
or max_tokens, or its output tokens reached its max_output_tokens.
The model fault of a scored-zero incomplete answer.
TRUNCATED_ANSWER when the answer carries a cutoff signal (_cut_off), whether the benchmark’s rule or a
recorded grade scored it zero: the cutoff is why it scored zero. Without a cutoff signal, a row whose reward came
from a recorded grade (_recorded_grade) has none, as its score is its grade; otherwise EMPTY_ANSWER when it
holds no assistant message with text (_empty_answer) and MALFORMED_ANSWER when it does.
Whether the collector bound the saved answer and score together (_ng_collected_grading_binding).
Whether a saved grading artifact or the model’s own failed answer resolves the row, as the gate requires: an artifact other than the model’s failed answer must bind the saved answer and score.
Whether the row is _graded and every grading operation binds its saved answer and score, as the gate
requires (_ng_collected_grading_binding).
The event failure_id selects and its explicit cause_id chain; other saved events are observations only.
Legacy rows with no event report retain their explicit terminal classification. Otherwise a selected event and every explicit cause must exist; missing references and cycles remain unknown.
The kind of a failure whose producer recorded none, or an unregistered one, from its recorded cause.
Returns (kind, model_final, reason); the kind is None when the cause cannot be determined, with the reason.
The cause is the event the row selects, else every event it saved, with the collector’s facts of the model call
that recorded each one (COLLECTED_CAUSE):
- a model request that failed with HTTP 5xx or in transport is infrastructure: its own kind, retried under the cap;
- an HTTP 4xx of the policy’s own call is the model’s final result:
model_context_limitwhen the call recorded the context limit, elsemodel_request_error; - an HTTP 4xx of a harness call (a user simulator, a judge, any model that is not the policy) is evaluation-side:
environment_episode_failed, retried under the cap; - with no event, the environment’s own episode deadline is evaluation-side too:
environment_episode_failed.
As for a kind (_retry_kind), the unselected events of a row that keeps a response are attempts, not its cause,
and a cause chain that is broken or omitted names no cause.
COLLECT_SCRIPT’s own projection name, so an in-process reader projects a saved row as the collector does.
Whether a captured judge reply was cut short: an incomplete Responses reply, or a chat completion that stopped on its output cap or a content filter.
Whether the saved answer carries an actual cutoff signal, not just an incomplete status.
The answer’s incomplete_details.reason is max_output_tokens or max_tokens; or the captured call that
returned it finished for length or one of those reasons; or its output tokens reached its max_output_tokens.
Decide from what COLLECT_SCRIPT read; a missing result file stops the decision.
extend_attempts is the attempts this recovery asks to add to every rollout’s cap (--extend-attempts):
the Store’s facts are read as they would be once it is recorded (see _with_attempt_extension).
rerun_masked_without_artifact asks to run again the masked results that kept a saved answer but have no gradeable
artifact and no grade (see decide_recovery); the run’s Store reader must apply the approval, or the ask is refused.
When the chain accounting is unknown, operator_budget may continue the run instead (see
_operator_continuation); without it, the stop names the options that allow that.
A run with no inventory and no saved row is a fresh restart once no job of it is live. allocation_cut (see
decide_recovery) applies only with no live job. The infra-retry cap is the configured one plus the run’s
recorded extension plus extend_retries that this recovery asks to record.
Whether the saved answer holds no assistant message with text: none at all, or only ones the collector saw
without non-whitespace text (has_text false). A message whose content was not saved counts as an answer.
The failure kind of the request a row’s rollout ended on, with that call, or None (final_zero_trials).
The last policy call of the row’s one invocation ended in a failed model operation whose failure event the call recorded, or, for an agent that recorded no call of its own, its last captured call failed that way; a policy whose own last call was rejected on length ended there, whatever its event’s kind.
The environment’s own reward record: result.reward_info.reward equal to the row’s reward.
Why this run cannot take an attempt extension, or None when it can.
The failure kind and HTTP status of a captured call whose final model operation failed with its own recorded request event, or None.
evaluation-side faults: N of M (x%) is [not] below L (server S: kind n; kind n); model faults: ....
Whether the row’s one verifier recorded grading operations for it that all completed with a valid verdict.
An operation without a valid verdict does not resolve the trial; the gate’s join decides whether it is a fault.
Whether a verifier’s sandbox observations report that a grading sandbox failed, or are malformed.
They are lifecycle evidence of the sandboxes the verification used, not a grading record: a completed or unreported outcome, an unconfirmed or failed teardown, a failing test command’s exit code and the agent’s own sandbox outcome do not leave a bound grading unobserved. A verifier sandbox whose outcome is a failure, or that records an error other than its teardown’s, does.
Whether the row’s verifier invocations were observed but recorded no grading: no grading operation, no
verifier grading record, no saved grading artifact (grading_artifact) and no model-failure rule
(_model_failure). A row from a pin without verifier observations proves nothing and is not selected. Nor is
one whose verifier got an answer from a model (_verifier_model_answered) and wrote no judge-error marker
(_judge_error_marker): a judge the verifier does not observe (observe_llm_judge_grading) records no
grading however often it is asked again, and a grade it gave is never replaced. A row whose model calls all
failed, or that names a judge error, was not graded.
A guardrail-gated objective score the agent’s own rubric saved (response.metrics): the row’s reward equals
the saved aa_headline, and that headline follows from the saved counts: 0.0 when a guardrail was violated,
else the objectives passed over the objectives (0.0 without objectives). A row without any counted assertion (no
world or no assertions also scores 0.0) is no evidence and stays unobserved. The counts are the rubric’s own, not
a re-run of its assertions. The saved answer and score are bound.
Express the existing timeout switch as a shared classification override.
decision as the trial gate counts it: a scored row planned for a re-judge because its judge gave no verdict
(SCORED_UNPARSEABLE_REASON) is a scored trial whose fault, judge_unparseable, is evaluation-side. The
recovery action is unchanged; only health reads the row as it stands until the re-judge settles it.
Whether the row’s saved answer is a response left incomplete, without an error. An incomplete status alone says nothing about why: an agent also marks an answer incomplete that holds no message.
Whether the row names a judge error the server wrote: a judge failure kind, a saved judge error (a label or a top-level message) or an invalid judge response (top level or inside the rubric facts).
A non-empty judge_evaluations list without a judge error; when every evaluation’s verdict label parses
([[A=B]]/[[A!=B]] or Judgement: yes/no), the reward is 1.0 exactly when every verdict is equal/yes.
A saved judge verdict: a judge_response, a non-empty judgments list with a judge_score, or
non-empty judge_evaluations whose parsed verdicts agree with the reward (_judge_evaluations).
A judge failure’s treatment: verify re-judges the saved answer only (none saved: stop), fail leaves it ungraded; any other action would run the policy again, so it stops.
Whether a scored row is judge_unparseable: its judge gave no verdict (_unparsed_judge_verdict) or its
verifier’s grading did not resolve (_unknown_grading_outcome). Judges are stateless: it is re-judged.
The call a rollout ended on, for an agent whose invocations reference none of its calls.
The highest capture position (call_index, the durable append order). When every call recorded its start
time, that call must also be the latest started; otherwise the end is ambiguous and there is none.
Read the final agent classification without selecting an unbound diagnostic event.
Older environments returned a failed agent response but did not select its report’s cause. Support only a delivered agent-stage outcome with one matching observed model rejection. An internal wrapper’s 500, mixed history, incomplete reports and saved completed answers cannot establish an upstream HTTP rejection.
The kind of a captured call rejected with HTTP 4xx on length (LENGTH_REFUSAL_KINDS), or None.
The reason is the call’s error_category or its local_response_reason; the status is the call’s own, its
upstream’s, or that of a failure event it reported. A 5xx among them is the server’s fault, and an event of another
kind than a rejected request (a transport failure) is its own cause, never the length’s.
Hash one stable file with bounded memory, using the exporter’s existing identity checks.
Two streaming passes must agree on stat identity and bytes: a same-size in-place rewrite within one timestamp tick changes no stat field. This doubles source reads; it does not make the reader atomic against ABA writes.
Session trial metadata from the bound saved config and the Store’s own inputs, after Store selection.
Returns the results with materialized_task_id on each session row and the effective checkpoint setting, or
the results unchanged and None when any fact is missing. Self-contained: the remote Store reader runs this
source inside the saved install, on that install’s Store, materialization and checkpoint APIs.
Saved in-process metric scores: numeric sentence_* fields (sentence_chrf, sentence_spbleu).
The benchmark’s rule scored the model’s own failed answer without grading it: the verifier marked the answer
invalid (invalid_model_response), or the model left it incomplete, or it is empty and scored zero.
A native scorer’s saved score (native_metrics): the row’s reward equals the saved weighted_score in
[0, 1], the scorer saved its binary execution flag and its diagnostics. The score is the scorer’s own, not a
re-run of it. The saved answer and score are bound.
A saved node-set F1 re-graded from the answer: the collector re-parses the saved answer’s final
Final Answer: [...] line, reproduces the saved predicted nodes and parse outcome, and scores them against the
saved expected node list as the verifier does; the row’s reward equals that re-graded F1 and the saved f1.
The saved answer and score are bound.
Whether a counted incomplete answer (_answer_fault) is the model’s own observed answer.
The trial gate counts such a row as a model failure and reports it as a response failure, so its grading is not
joined. Its answer must still be observed: the row recorded no failure, its one agent
invocation ended completed or incomplete (an agent marks a cut-off answer incomplete) without an error, its last
model call is exactly one captured call that answered in one completed operation, and that call’s response is
the saved answer, or, for an agent that saves its own answer envelope (its own response id), the envelope says it
was cut off (_answer_cut_off) and that call itself finished on a cutoff (CUTOFF_REASONS): two independent
signals that the model’s own last reply was cut. A verifier that ran must have completed: a verifier error is never
read as the model’s fault.
Whether every expected trial that is not a counted failure is observed: its saved answer is bound to its grading.
Observed means a recorded grading bound to the saved answer and score: one completed verifier’s grading operation
bound to the row, its own grading record, a saved grading artifact (GRADING_ARTIFACTS) with the collector’s
binding, or the model’s own failed answer. Only that decides coverage. Each trial’s model/tool trajectory is
joined too (its one agent invocation, turns, calls, tools and failure events), but that is observability
evidence: a trial whose trajectory does not join is reported in trajectory_incomplete with the kinds of its
missing evidence (_trajectory_kinds), and it is still observed.
Join observed model/tool operations and the outer verifier invocation for every expected trial.
A trial whose verifier recorded no grading operation and no grading record of its own is observed through a
saved grading artifact (GRADING_ARTIFACTS) or through the model’s own failed answer the benchmark’s rule
scored (_model_failure); those trials go to artifact_recorded, and the verifier’s captured calls are its
own. An environment reward record stands in for a missing verifier invocation.
A source-reviewed programmatic grading operation must bind the saved response and score. A verifier that
recorded no grading operation and owns no model call is observed through its own grading record instead
(_verifier_recorded); those trials go to verifier_recorded. Empty captured model calls or a reward
alone never establish grading coverage; old or uninstrumented verifiers remain unknown.
Recovered grading attempts are reported separately and do not count toward the trial-health threshold;
every failed attempt must join the same completed, result-bound grading dependency. A policy request retried
inside one completed operation is recovered the same way (policy_recovery receives its counts): each failed
attempt is an event of that operation and of the row, and the answer is the final attempt’s.
Decide again under the operator’s explicit budget, or keep the stop when continuing could be unsafe.
The caller has checked that the scheduler answered and shows no live or pending job of the run. It fills only
unreadable counters: an unknown spent-retry count so that operator.retries retries remain under the cap, an
unknown used walltime with zero, so the walltime cap is fresh for this continuation, and an unknown extension with
zero. A readable counter keeps its value. When the spent-retry count is readable, an explicit extend_retries
(--extend-retries) is what it is on any recovery: that many more infra retries, recorded as the run’s retry
extension in the continuation’s writes and journal. Saved rows are classified as for any decision and never
changed.
Whether a captured call is the evaluated policy’s: its model is the row’s policy server, or none is named.
The failure kind and HTTP status the policy selects on: the event failure_id selects first, then the
row’s own producer fields, then a validated final legacy agent outcome. Unselected events are not causes.
Whether replay needs a saved-answer path or model-budget accounting unavailable to recovery.
A consistent applied zero keeps its own kind, without reinterpreting diagnostic requests.
What a resume runs, what nobody owns (by kind, largest first) and the saved rows kept as they are.
Whether every scored row was written by a Gym pin from before operation observations were recorded.
Current pins write response_metadata.model_operations on every captured model call (a list, or null), and
the collector reports whether the key was there (model_operations_recorded). Only an explicit False on every
call of every scored row counts. A row without captured calls, or read by a collector that does not report the
key, proves nothing, so the strict rule applies to it.
The reset time a quota-stopped row names: the collected fact, else its reason (a live run’s row).
Whether the row’s one verifier invocation saved its own grading record for it (_verifier_recorded).
Whether the row’s reward came from a recorded grade: completed grading operations, the verifier’s own grading
record, or a saved grading artifact (GRADING_ARTIFACTS). The model’s own failed answer is not one.
N failed model attempt(s) were retried and completed in the same operation (...), or empty.
Apply recovery rules to shared classification without changing the selected evidence or budgets.
Why replay needs a saved-answer path or model-budget accounting unavailable to recovery; None when it does not.
A saved repository test run of the answer’s patch: evaluation_completed true, the run’s resolved
verdict with the row’s reward equal to it (1.0 resolved, 0.0 not), saved test_results, no error, no
evaluation timeout, and a resolved verdict only for a patch that applied. The saved answer and score are bound.
In a sentinel: reconcile a requeued chain from the terminal records in CHAIN_ACCOUNTING_ENV and decide.
The same reconcile and decision as gym eval resubmit --recover: nothing else of the chain runs or waits (the
sentinel checks the queue), and every allocation ended. Prints the reconciled walltime when the run continues.
Why this run cannot run masked results without an artifact again, or None when it can.
The kind a retry decision selects on: the policy evidence, else, for a row that selects no event, the one kind every saved event names, else the failure record’s own kind. Final-zero and judge decisions do not use it.
The row’s recovered model attempts, by the completed operation that recovered them (recovered_attempt_joins).
Each entry is an operation id and its failure events, keyed by event id, from one answered call of the row.
A saved rubric judge’s verdict: verdict YES or NO, a finite rubric_score and scoring_threshold, no
invalid_judge_response, and the reward 1.0 exactly when the verdict is YES and the score reaches the threshold
(0.0 otherwise). The saved answer and score are bound.
The run’s saved per-rollout attempt cap, from the Store’s facts; None when they do not say.
Whether the row saved, intact, the workspace patch its verifier collected and graded: the collector’s projected
facts, else the same checks on a full saved row (saved_model_patch, collected_patch). Only a verifier that
collects the patch itself can be given it back; one that reads the agent session at verify time cannot.
A scored row’s one counted fault: its own recorded kind (_scored_kind), else its answer’s
(_answer_fault), else judge_unparseable when its saved judge evaluation recorded no verdict
(_unparsed_judge_verdict).
The registered failure kind a scored result row records as its own outcome, keyed on its recorded HTTP status
(status_fault_kind), or None. For example Harbor’s graded agent_timeout is a scored main result.
A persisted outcome decision governs metric admission, independently of fault attribution and the existing evaluation-health threshold. Counting a provider failure as zero does not make it a model fault or healthy.
A scored row: unmasked, with a reward and no failure kind.
The verifier’s own reward file of the row’s session, read by the collector beside the artifacts
(resources/<session_id>/verifier/reward.json): the evaluation completed with no infrastructure error and the
row’s reward equals the file’s reward. Test counts and partial credit are metadata, never the grade. The saved
answer and score are bound.
In the in-job guard: whether the allocation that last wrote the Store has ended.
The guard runs after its predecessor ended, and a chain runs one allocation at a time: every allocation the chain recorded before this one has ended. The writer is often an earlier allocation than the predecessor, for example when the predecessor timed out before it opened the Store. A writer the chain does not name stays unknown.
Saved per-sub-step test results: a non-empty step_results list of booleans, one environment result per
step, num_steps_total and num_steps_passed equal to its length and its passes, problem_accuracy true
exactly when every step passed, and the row’s reward equal to it (1.0 or 0.0). The saved answer and score are
bound.
A saved symbolic answer check: a library_reward the row’s reward equals, or one beside a judge verdict
(the judge fallback).
Whether a sandbox observation’s only failure is its teardown: the producer records a failed close or release
after the work as outcome sandbox_error or unknown with that cleanup error as its error_type.
A saved test run: evaluation_completed true with an integer verifier_exit_code and no
verifier_error.
The kinds of trajectory evidence a trial lacks: its recorded gaps, and its agent invocations’ own shape.
Recorded gap codes other than resolution_unavailable (for example sandbox_lifecycle_timing_unavailable,
conversation_unavailable, model_call_ownership_unavailable), then invocation_unavailable or
multiple_invocations, in_session_unrecorded and invocation_<status> for an invocation that did not
record a completed status. Empty when none of these is named.
trajectory evidence incomplete: N of M (kinds: kind n, ...), reported and never a condition.
Whether the row recorded a model failure that no recovery accounts for.
Every model failure event the row saved or its captured calls reported must be a recovered attempt (recovered: the
events the coverage join joined, with the row’s own _row_recovered_attempts), a saved copy must agree with its
call’s, and every failed model operation must be one whose failure such a recovery accounts for. A trial that
ended on its failed request is counted separately (final_zero_trials) and is never asked this.
Saved unit test results of the answer’s extracted code: at least one test, and the row’s reward 1.0 exactly
when every test passed (0.0 otherwise), or 0.0, the default penalty, for an answer whose reasoning format
violated the rule (reasoning_format_violation_rate 1.0). A configured non-zero penalty is not saved on the row,
so such a row stays unobserved. The saved answer and score are bound.
Whether a scored row’s verifier record reports a judge grading operation with outcome unknown: the grading
did not resolve, so its reward is no grade. Read from the verifier invocations the row already saves.
The kind of an unknown identity with the rule that refused it, without naming the kind twice.
Whether a scored row’s grading is judge_unparseable for a verdict the judge did not give: a saved judge
evaluation recorded no verdict (_unparsed_verdict), or a pairwise game has none (games_unjudged). Its
reward is no grade.
Saved judge evaluations of which one recorded no verdict: a null label, or a judge or parse error label.
Who verifies an agent’s rollouts: its resources server, or the agent itself when it names none.
An agent that runs its own rubric (verifiers_agent) has no resources server; its grading record names the agent.
Whether one model call of the row’s verifiers completed with a 2xx and a usable response, read from the call the row saved for each reference (by call id, else response id). A call the row does not hold has not answered.
Whether the row’s verifier saved its own grading record for it, in place of a recorded grading operation.
The shared verify path records it only for a verify execution that completed without asking any model. It must
name the row’s resources server (or the agent, for an agent that verifies its own rollouts), carry the saved score with a scored verdict and bind the saved response and
score. A verifier that recorded a grading operation or owns a model call is judged by those, never by the record.
Whether event and every explicit cause below it name only a lost verify call: an agent route failure or a
transport kind, with no reply status. A verifier’s own failure or an HTTP reply anywhere in the chain is not one.
Whether the row was masked because its verify call was lost in transport: its kind is an agent route failure
or a transport kind, and a saved failure event of the verify operation has a transport-only cause chain.
The resources servers the run’s agents name, from the run provenance, for an unobserved-trial reason.
The Store’s facts as they read once extend_attempts more attempts are recorded for every rollout.
Only a rollout the Store would run again under some cap (replayable) gets the higher cap, and it is pending
while it has attempts left. Any other rollout keeps its cap and state: a model failure, a judge failure that kept
its saved answer (its re-judgment budget is the saved cap), a saved result.
Elapsed seconds of all recorded allocations, only when each has one terminal accounting row.
stored_walltime_s None means no allocation stored a walltime: the prologue that accounts the previous
allocation stopped first (after an operator’s cancel, for one). The records’ sum is the used walltime then.
Elapsed seconds of every incarnation of a chain with a requeued allocation, when the records are complete.
Every chain job needs terminal records numbered 0..n by restarts, and some job a requeue (n > 0). The stored counter must equal the elapsed time of every incarnation of the jobs before the last requeued one: an earlier requeue was reconciled and stored before the chain went on, and the last requeued incarnation stops before it adds anything, as does every later allocation. Otherwise the budget epoch is unknown.
The scored rollouts’ own agent_adaptations counts, summed by name: {name: {"count": total, "rows": N}},
rows being the rollouts with a nonzero count. An agent records what it adapted on its own before a result,
such as a request the endpoint rejected for its input size; a reported fact, never a failure.
Whether the first incarnation of allocation_id has ended, by any path (ALLOCATION_ENDED_STATES).
Only the first incarnation (no restarts) runs work: the prologue stops a requeued one before any. A state that is absent, ambiguous or not terminal is not an ending.
Rows of sacct -D -X -n -P --format=CHAIN_ACCOUNTING_FORMAT as (job id, state, elapsed s, restarts).
The state keeps its first word without a trailing + (“CANCELLED by 7” is CANCELLED). None when any row is
malformed.
A benchmark’s resumable.recovery_outcome_policy section as a policy, or None when it is not set.
It is a partial override: RecoveryOutcomePolicyConfig merges the entries it names into its defaults, and
validates them exactly as the Python API does. A bad section raises, naming each problem.
Read saved state in the driver’s installed runtime before a terminal allocation continues.
The shell has already validated and persisted terminal elapsed accounting. This reader never changes
artifacts, starts servers, resets budgets, or treats an exit code as sufficient retry evidence.
outcome_policy is the run’s recovery_outcome_policy section as JSON, rendered from its saved config, so
an automatic continuation decides with the policy a recovery would apply.
_decide_from_collected, with the reason naming any operational drift the run’s latest resume crossed (see
resume_drift_note) and a continuation that ended before it queued its follow-up, so an inspection, a recovery
and a chain continuation all say it.
Classify every materialized identity and choose the run-level action.
failures are every failures-sidecar row in file order (_line is the row’s line number when known). The
standing row of a rollout is chosen as rollout collection chooses it (_latest_failure_rows): rows that
is_non_superseding_rollout_row names are skipped, and an interrupted omitted row means the rollout is missing.
Only rows at or after resubmit_offset count towards the attempt cap. The rows are only read.
allocation_cut says that the allocation that last wrote the Store has ended, by any path (one of
ALLOCATION_ENDED_STATES), and no allocation of the run is live: a reservation with no saved row was in flight
when it ended and saved nothing, so it is not started for recovery purposes and the Store runs it again.
A masked result with a saved answer, no gradeable artifact and no recorded grade (masked_without_artifact_kind)
stays MASKED, named in masked_without_artifact, unless the Store reports an approval for it
(ManagedRecoveryFact.rerun_approved) or rerun_masked_without_artifact asks for it now: it is then planned
like an infrastructure-masked result, one failed attempt within its cap, and rerun_masked lists the retries
this call would have journaled.
The resolved recovery policy snapshot supplies the same mapping to all health readers.
The policy recovery applies and its source. Precedence: an explicit Python outcome_policy argument
replaces the run’s saved config section (it is not merged with it); the section overrides the defaults. The
budget options (extend_retries, max_walltime) are separate and never change the policy.
The ids of the last JUDGE_JOB_FAILURE_BOUND judge-only jobs when all ended without settling a saved answer and no
acknowledgement names exactly them; else None. An acknowledgement covers the failed jobs it names, so a later pair of
failures (the one more job it let queue, and the one before it) needs an acknowledgement of its own.
The stop text of failed_judge_jobs_blocking: the jobs, where to read them, and the explicit way on.
Response and trial failures by cause, each with a count and a rate over the expected rollouts.
Reported facts, never a verdict: no limit applies to them here. The denominator is the trial gate’s
(trial_population): every expected rollout, failed and missing ones included; rows an accepted partial stage
omitted are no rollout of the run and are reported separately as omitted.
A response failure is a scored rollout without a recorded grade (_recorded_grade) whose saved answer is
truncated (incomplete with a cutoff signal, _cut_off), empty (no assistant text, _empty_answer) or malformed (no response
with an output list, an answer the verifier marked invalid, or one left incomplete without a cutoff signal). A trial failure is a rollout in a failed, masked or
unowned state (_TRIAL_FAILURE_STATES), a scored rollout that ended on a failed policy request, a scored
rollout that records its own failure kind or whose judge recorded no verdict, or an observed grading without a
valid verdict, by the cause its failure kind names (trial_failure_cause).
Report shared classes separately; server is only a subcategory of evaluation.
Scored zeros with a saved policy decision or a failed final policy request, with their failure kinds.
A consistent saved counted-zero decision retains its own registered failure kind. Otherwise, the last
policy call of the row’s one invocation ends in a failed model operation whose failure event the call
recorded, and the row’s reward is zero. Such a trial is settled as a failure under the benchmark protocol: it
has no selectable response, and it is counted once, in the trial family. An agent that records no call of its own
(it has no invocation, or its invocations reference none) ended on a failed request when its last captured call (_last_captured_call)
failed that way; an earlier failed call is no end and stays unaccounted. The kind is keyed on the failure’s HTTP
status (status_fault_kind): a 5xx reply is the server’s fault, and another model’s 4xx rejection is
evaluation-side. A policy whose own last call was rejected on length (_length_refusal_kind) ended there,
whatever its event’s kind: model_context_limit for the context window, invalid_output_budget for an output
budget the window cannot hold. The policy’s last call is its last captured one when its invocations reference
none, since another model’s capture may follow.
The acceptance gate task entry from the collected run: decision counts, the named score and the quality summary.
mlflow_status and mlflow_run_id are the tracking run’s status and id as the caller read them together.
Without a status, the receipt’s run is asked of the tracking server the saved config names (query_mlflow_status;
status_client_factory replaces its client).
criteria are accepted and reported under criteria but not applied (GateCriteria). The only health
verdict is evaluation_health; failures reports response and trial failures by cause, never a verdict.
status follows gate_task_summary: an UNVERIFIED evaluation health with nothing failed is UNVERIFIED, and
a chain error turns SUCCESS or UNVERIFIED into FAILED. MLflow delivery is reported as
mlflow_delivery, a fact that never changes the status.
One task entry of the acceptance gate’s run summary, filled from the decision and the existing validators.
expected_samples is always the full materialized inventory; scored_samples counts only scored and
final-model identities, so a judge, infra, terminal or missing identity can never make them equal. A settled
final outcome the metric does not score (IdentityEvidence.unscored) is stated, not counted as scored. Summary validity
and response/trial health stay separate. False is a failure and an error; None (unknown or missing) is no
failure but never a pass: its reason is a warning and it withholds SUCCESS.
status, first match wins: RUNNING (a job is live); PLANNED (recovery work is eligible; only a caller that
dispatches it reports RESUMING); FAILED (anything in errors, or the run is not a complete NOOP); UNVERIFIED
(nothing failed, but summary validity or response/trial health is unknown, so acceptance is not proven); SUCCESS.
The name of the first saved grading artifact (GRADING_ARTIFACTS) that records the row’s grading.
Named health-check facts from quality_summary.json for the gate warnings, kept even below any criterion.
Nonzero run.issues counts, checks run.artifacts.coverage did not fully evaluate (any unobserved or
ignored), and run.ignored_checks. A field that is absent or malformed is reported as unknown, never as clean.
evaluation health: PASS; response failures: 4.70% (truncated 41, empty 6 of 1000); trial failures: ...; metric over 1000 of 1000 trials.
Scored identities whose graded result is an agent timeout with a server-side fault in its window.
A Harbor graded zero a run without a Store cannot reopen (is_infrastructure_agent_timeout): it stays in the
results, and the trial gate counts it once, as AGENT_TIMEOUT_INFRASTRUCTURE, an evaluation-side fault.
One line on a judge-only job of the run (CollectedEvidence.judge_jobs): queued, live, ended without a
receipt, or what its receipt says it settled.
The chain job that still waits or runs, as ‘id (STATE)’ from its latest incarnation; None when none does.
The kind of a masked result that is never measured nor run again, though infrastructure left it unmeasured; else None.
A masked result masked_infrastructure_kind rejects only because it kept a saved answer (so every other rule of
that predicate applies: the kind must be registered, evaluation-side and not graded again), that has no recorded
grade (has_recorded_grade) and no re-verification cause (reverification_cause): its saved answer has no
gradeable artifact, a resume keeps it and re-verification cannot grade it. It runs again only with the explicit
--rerun-masked-without-artifact. row is a collector projection, or a saved row passed through
saved_row_evidence.
How many expected trials the benchmark’s metric covered: {"trials": N, "expected": M, "source": ...}.
A benchmark that declares the coverage of its headline metric in its aggregate (metric_coverage/trials,
metric_coverage/expected and an optional metric_coverage/name) is the source (benchmark), with its
own denominator and the metric’s name. Otherwise the source is collection’s own count when its coverage report
has one for this inventory (coverage_report):
the rollouts whose score entered the metrics, a failure counted as zero by configuration included. Otherwise it is
read from the saved rows (saved_rows): the scored results and the failures their producer marked as counted
zeros, which is what collection hands the metrics unless it counts more by configuration. A failure the benchmark
leaves out of its metric, such as an agent deadline that kept its answer, is a trial the metric did not cover.
The denominator is the trial gate’s: every expected trial, less the rows an accepted partial stage omitted.
The MLflow delivery fact of a run: not_required by design, verified when mlflow_errors found nothing,
pending while there is no receipt and no live run to judge or only the run’s status was not queried, else
unverified with its reasons. A fact for consumers to read, never a verdict: it does not touch evaluation
health or the gate’s status. A status that was neither given nor queried reads “status not queried”; a query that
failed reads “status query failed: …”: never a silent “not observed”.
MLflow delivery errors from the collected exporter files and the server status the caller observed.
local holds the live exporter’s run id (rollouts_mlflow_run_id) and the post-run export receipt
(rollouts_mlflow_export_done, checked against the current artifact digests). Only a current post-run receipt
with an observed FINISHED passes. A live run is not accepted here: live uploads log failures and continue, so
neither its run id nor FINISHED shows the configured payloads arrived. A live run beside a post-run export leaves
the observed status unbound. configured is the health report’s run.provenance.mlflow.configured, a
names-only fact about the saved config snapshot (None when not reported); it only qualifies the message. The
observed status is bound to a run: with a post-run receipt, observed_run_id must name the receipt’s run.
BaseExporter.export writes the receipt only after every upload returned and the artifacts did not change, but
its teardown still sets FINISHED on a partial export (run_failed is the process’s run, not the export’s), so
a FINISHED from any other run proves nothing. No
evidence is always an error, never a run that needed none: exporter activation and whether tracking was required
or disabled are not recorded.
What the gate states about a passing MLflow part: a current post-run export beside a live run wins.
False only when the run’s saved config snapshot sets mlflow_tracking_required to false; absent, unread or
any other value means tracking is required. The health report’s provenance states it when its pin did; for a run
whose report does not, the collector’s read of the saved rollouts_resolved_config.yaml does.
The omitted identities by stage and policy, e.g. “1 omitted by an accepted partial stage (stage_index 1, tolerate_unresolved)”.
How many model calls in the saved results were sent again with an output budget lowered to fit the context
window (output_budget_refit in a call’s response metadata), and in how many result rows.
A count, never a failure: the call generated within the window it was served at. A reader flags a cell whose requests were budget-capped from it.
Summary validity and per-rollout health from the run’s quality_summary.json, kept separate.
Validity is the health check’s own accounting verdict (run.artifacts.healthy, null for a partial summary),
and must count the same expected identities as the inventory. Per-rollout health is the rollout verdict count
(run.verdicts), judged only against an explicit max_unhealthy_fraction; it is not separately validated response or
trial health, which this adapter has no separate evidence for. Any missing piece is None.
The status of run_id on the tracking server, as (status, None), or (None, error) when it could not
be read: a missing client, an unreachable or refusing server, a missing run. Short timeout and one retry;
client_factory builds the client from the URI (default mlflow.MlflowClient).
Know a requeued chain’s budget again once every incarnation has a terminal scheduler record.
Only the requeue marker is reconciled: the counters themselves stayed readable and the missing fact is the elapsed time of the requeued incarnations, which complete terminal records give. Any other unknown reason stays unknown.
Failed model attempts each joined to the completed operation that recovered them, in a scored trial’s own call.
Counted like the run’s observed upstream recovery (observed_upstream_recovery): a model-request failure event
whose operation completed in an answered call. Here the event must be in the failure report of the same call
whose one completed operation it names, as a failed attempt other than the answer, and the operation’s attempts
must account for every failed attempt of it (http_attempts - 1 + transport_failures, the HTTP-less ones being
the transport failures). A copy the row saved must agree, and no event, attempt or operation joins twice. The
join holds for any model the trial called (its policy, a judge or a simulated user). It says nothing about how
the trial was graded: whether the final answer is bound to its grading is the coverage join’s question
(_observed_trial_operations).
The last requeued job of a chain and what its allocations account for, as a decision states it.
The response gate from run.stats.responses of the health report: faulted over selected responses.
Evaluated only when every scored rollout has a verified response population and every selected call id is unique across the run (a report that predates it, an unverified call scope, a response selected twice, or zero selected responses is unknown, neither pass nor fail). Failed trials have no response row and are left out, as the policy leaves failed attempts out of the response denominator.
The sentence a decision reason carries when the run’s latest resume crossed operational drift.
The run’s two retry budgets as the gate reports them, each its configured cap and its recorded extension.
infra is the chain’s infra-retry cap (--extend-retries); attempts is every rollout’s attempt cap
(--extend-attempts). A value that could not be read is None.
Why a saved result row is re-verified from its saved answer, or None when it is not.
REVERIFY_JUDGE: masked because its judge failed (judge_failed, judge_unparseable), or scored while
its judge gave no verdict or its grading did not resolve (_judge_unparseable).
REVERIFY_VERIFY_TRANSPORT: masked because its verify call was lost in transport (_verify_transport_failure).
REVERIFY_VERIFIER: masked because its verifier produced no verdict (verifier_error), with the workspace
patch it graded saved intact (saved_model_patch): the verifier runs again on that patch, never the agent.
REVERIFY_UNGRADED: scored, but no grading was recorded for it (_grading_unrecorded).
Only a result with a saved answer qualifies; a failures-sidecar row, a result a re-verification already settled
(its settlement is its grading) and every other masked result do not. row is a collector projection, or a
saved row passed through saved_row_evidence.
A full saved row with the evidence fields COLLECT_SCRIPT projects for the grading predicates: its scalar
grading facts, whether each assistant message of its answer has text, and its verifier invocations.
Read locally accessible Store history in this installed Gym process; never copy its selection rules.
The plain remote collector remains dependency-free. Call only on the same host as the installed caller, after collection. Both snapshots must include the manifest and journal and remain byte-identical.
The shared reporting label for observed HTTP evidence; saved events remain unchanged.
Why the chain stopped when the follow-up job_id that its last allocation claimed is not in the chain.
accounting is sacct -D -X -n -P --format=FOLLOWUP_ACCOUNTING_FORMAT -j job_id. The prologue records an
allocation in the chain before anything else, so a follow-up that started and ended without that entry ran no
work. None when the records do not show that: the follow-up never started, is live, or its record is ambiguous.
Trials whose trajectory evidence is incomplete, over the trials, with the count of each missing kind.
Why trial failures outside the sidecar cannot be ruled out, from the run’s own persisted facts.
Every persisted value is type-checked: a malformed report is named as malformed, never read as a fact.
Acceptance’s trial counts: observed healthy, unhealthy and unobserved, over the expected trials.
A saved grading artifact or the model’s own failed answer (grading_artifact, _model_failure) resolves a
trial too. unhealthy are the failures the trial gate counts (trial_faults, model and evaluation-side). When the
gate’s observation join covers every other trial, its observed gradings without a valid verdict are unhealthy
too and nothing is unobserved. Otherwise unobserved are the other scored trials whose verifier recorded
neither completed grading operations bound to the saved row nor its own grading record: nothing resolved their
grading, so they are unverified, never healthy. observed_healthy are the rest; verifier_recorded of them
are observed only through the verifier’s grading record. None until the run is complete.
A trial failure’s cause from its failure kind: request, judge, sandbox, harness, timeout, allocation, or other.
allocation: the rollout was in flight at repeated ends of its allocation (ROLLOUT_ALLOCATION_ENDED).
Count final trial failures once; classify selected outcomes, never recovered observations.
Evaluation health from the trial gate (trial_verdict): passed, failed or None, with its reason.
The trial denominator: every expected trial, including repeats, failed and missing ones, less the rows an accepted partial stage omitted (no trial: reported separately, never in a numerator or denominator).
Evaluation health: evaluation-side trial faults over the expected trials, including repeats.
The counted failures are trial_faults; a model fault (classify_failure) is reported with its kind and never
fails. Only the evaluation-side share is judged against the limit: at or above it the gate fails. Below it, it
passes only with an exact count, when actual per-identity operation outcomes cover the observed model/tool and
grading path of every trial that is not a recorded failure (_observed_trial_operations); an observed grading
without a valid verdict is an evaluation-side fault. Otherwise the run is UNVERIFIED: a judge, verifier or harness
error scored as a zero in the main output carries no failure record and cannot be ruled out, so the known count is
a lower bound. That withholds a pass, never fails health. The denominator is the full materialized inventory less
accepted omissions. A run that is not complete has unaccounted trials, so the gate is not evaluated.
The report is status (PASS, FAIL or UNVERIFIED), the evaluation-side rate, exact and the fault counts.
The chain stopped before the run settled: how many rollouts are unsettled, by state, so the stop reads
as an incomplete run and never as a finished one. Unsettled is wider than the coverage report’s open (the
count in the chain’s own stop line): it also holds final failures that were not counted as zero.