nemo_gym.eval.recovery.gate
nemo_gym.eval.recovery.gate
The acceptance gate: response and trial health, metric coverage, MLflow delivery and the gate summary.
Module Contents
Classes
Functions
Data
API
Bases: BaseModel
The response and trial failure fractions a caller may still pass. They are validated (an invalid value is
refused, never clamped) and reported, but no longer applied: the gate’s only health verdict is evaluation health
on the bounded failure policy’s limit (trial_verdict), and response and trial failures are reported as facts
(failure_facts) for consumers to judge.
The effective criterion and the accepted, unapplied fractions, as reported beside the gate results.
What in a printed command is the operator’s to choose, and that its options go in one call.
The one recovery command the operator verdict prints: gym eval resubmit <job id> <benchmark> --recover,
then every option the call asked for (decision.requested), then the typed remedy’s options, then options
(what the stop itself needs, such as a budget extension). An option is named once: options replaces the
call’s own value, the call’s own value replaces a remedy’s placeholder, and drop names options the run
refused. inspect makes it the inspection, without the options an inspection refuses.
The recovery command that continues the run, after the typed stop’s fix when it stopped on one, with what in it is the operator’s to choose. A typed stop whose own text says a recovery cannot continue (a refused resume, a protocol violation) names that instead.
The saved result row that decides each scored trial: the last one saved for its identity, so an attempt a later one superseded is not counted again.
(what, action) for a complete run whose MLflow delivery is not closed: the results stand, and the delivery
is a fact beside them. Unverified by configuration (the export did not upload the rollouts) is reported, as no
export changes it.
(what, flagged) when rollout quality flags every rollout or every task, else None; flagged counts in
the unit of the check.
The flags one option spells (--max-walltime T --walltime-reason TEXT is two).
What a rerun in options leaves masked: it runs only the results without a gradeable artifact.
The verdict of a resubmit the scheduler refused: no job is known to be queued, or it is unknown whether one is.
The scored identities among only with a saved judge failure (their answer was judged again), and the HTTP
statuses of it.
The typed failure with a known remedy the run stopped on: the chain guard’s typed stop, else the one failure
kind every terminal identity of the decision records (Gym’s own typed stop, such as the grading quota). Read from
typed fields only, never from the reason text. from_rows=False reads the chain guard’s stop alone: the row
kind explains a STOP, and a decision that continues the saved run (RESUBMIT, REVERIFY) is not one.
The verdict of a STOP: the continuation its own cause allows, after any guard and typed fix, else the most specific known step, else no command.
Whether one of the row’s verifier invocations owns the captured call (a judge’s call), matched by its call id, else by its model and response id: a judge served by the policy’s own server is still the judge.
The scored rollouts’ own agent_adaptations counts, summed by name: {name: {"count": total, "rows": N}},
rows being the rollouts with a nonzero count. An agent records what it adapted on its own before a result,
such as a request the endpoint rejected for its input size; a reported fact, never a failure.
Response and trial failures by cause, each with a count and a rate over the expected rollouts.
Reported facts, never a verdict: no limit applies to them here. The denominator is the trial gate’s
(trial_population): every expected rollout, failed and missing ones included; rows an accepted partial stage
omitted are no rollout of the run and are reported separately as omitted.
A response failure is a scored rollout without a recorded grade (_recorded_grade) whose saved answer is
truncated (incomplete with a cutoff signal, _cut_off), empty (no assistant text, _empty_answer) or malformed (no response
with an output list, an answer the verifier marked invalid, or one left incomplete without a cutoff signal). A trial failure is a rollout in a failed, masked or
unowned state (_TRIAL_FAILURE_STATES), a scored rollout that ended on a failed policy request, a scored
rollout that records its own failure kind or whose judge recorded no verdict, or an observed grading without a
valid verdict, by the cause its failure kind names (trial_failure_cause).
The acceptance gate task entry from the collected run: decision counts, the named score and the quality summary.
mlflow_status and mlflow_run_id are the tracking run’s status and id as the caller read them together.
Without a status, the receipt’s run is asked of the tracking server the saved config names (query_mlflow_status;
status_client_factory replaces its client).
criteria are accepted and reported under criteria but not applied (GateCriteria). The only health
verdict is evaluation_health; failures reports response and trial failures by cause, never a verdict.
status follows gate_task_summary: an UNVERIFIED evaluation health with nothing failed is UNVERIFIED, and
a chain error turns SUCCESS or UNVERIFIED into FAILED. MLflow delivery is reported as
mlflow_delivery, a fact that never changes the status.
One task entry of the acceptance gate’s run summary, filled from the decision and the existing validators.
expected_samples is always the full materialized inventory; scored_samples counts only scored and
final-model identities, so a judge, infra, terminal or missing identity can never make them equal. A settled
final outcome the metric does not score (IdentityEvidence.unscored) is stated, not counted as scored. Summary validity
and response/trial health stay separate. False is a failure and an error; None (unknown or missing) is no
failure but never a pass: its reason is a warning and it withholds SUCCESS.
status, first match wins: RUNNING (a job is live); PLANNED (recovery work is eligible; only a caller that
dispatches it reports RESUMING); FAILED (anything in errors, or the run is not a complete NOOP); UNVERIFIED
(nothing failed, but summary validity or response/trial health is unknown, so acceptance is not proven); SUCCESS.
Named health-check facts from quality_summary.json for the gate warnings, kept even below any criterion.
Nonzero run.issues counts, checks run.artifacts.coverage did not fully evaluate (any unobserved or
ignored), and run.ignored_checks. A field that is absent or malformed is reported as unknown, never as clean.
evaluation health: PASS; response failures: 4.70% (truncated 41, empty 6 of 1000); trial failures: ...; metric over 1000 of 1000 trials.
One line on a judge-only job of the run (CollectedEvidence.judge_jobs): queued, live, ended without a
receipt, or what its receipt says it settled.
How many of the policy’s model calls in the rows that decide the scored trials ended on a length cutoff, and in
how many of those rows: a length finish reason, or its max_output_tokens and max_tokens spellings, as
the evidence collector records each call.
A fact, never a failure: the model server returned the shortened answer and the rollout carried on from it, so a run that lost output to the context window or the output budget does not read as clean. The finish reason does not tell the two limits apart. A call a verifier owns (a judge’s, also when it shares the policy’s server) or another model’s call is not counted, nor is an attempt a later one superseded.
How many expected trials the benchmark’s metric covered: {"trials": N, "expected": M, "source": ...}.
A benchmark that declares the coverage of its headline metric in its aggregate (metric_coverage/trials,
metric_coverage/expected and an optional metric_coverage/name) is the source (benchmark), with its
own denominator and the metric’s name. Otherwise the source is collection’s own count when its coverage report
has one for this inventory (coverage_report):
the rollouts whose score entered the metrics, a failure counted as zero by configuration included. Otherwise it is
read from the saved rows (saved_rows): the scored results and the failures their producer marked as counted
zeros, which is what collection hands the metrics unless it counts more by configuration. A failure the benchmark
leaves out of its metric, such as an agent deadline that kept its answer, is a trial the metric did not cover.
The denominator is the trial gate’s: every expected trial, less the rows an accepted partial stage omitted.
The MLflow delivery fact of a run: not_required by design, verified when mlflow_errors found nothing,
pending while there is no receipt and no live run to judge or only the run’s status was not queried, else
unverified with its reasons. A fact for consumers to read, never a verdict: it does not touch evaluation
health or the gate’s status. A status that was neither given nor queried reads “status not queried”; a query that
failed reads “status query failed: …”: never a silent “not observed”.
MLflow delivery errors from the collected exporter files and the server status the caller observed.
local holds the live exporter’s run id (rollouts_mlflow_run_id) and the post-run export receipt
(rollouts_mlflow_export_done, checked against the current artifact digests). Only a current post-run receipt
with an observed FINISHED passes. A live run is not accepted here: live uploads log failures and continue, so
neither its run id nor FINISHED shows the configured payloads arrived. A live run beside a post-run export leaves
the observed status unbound. configured is the health report’s run.provenance.mlflow.configured, a
names-only fact about the saved config snapshot (None when not reported); it only qualifies the message. The
observed status is bound to a run: with a post-run receipt, observed_run_id must name the receipt’s run.
BaseExporter.export writes the receipt only after every upload returned and the artifacts did not change, but
its teardown still sets FINISHED on a partial export (run_failed is the process’s run, not the export’s), so
a FINISHED from any other run proves nothing. No
evidence is always an error, never a run that needed none: exporter activation and whether tracking was required
or disabled are not recorded.
What the gate states about a passing MLflow part: a current post-run export beside a live run wins.
False only when the run’s saved config snapshot sets mlflow_tracking_required to false; absent, unread or
any other value means tracking is required. The health report’s provenance states it when its pin did; for a run
whose report does not, the collector’s read of the saved rollouts_resolved_config.yaml does.
The one verdict an operator acts on, derived from the gate and the decision only.
state is accept (gate SUCCESS, coverage complete, run health PASS, decision noop), needs_action (a
STOP, FAILED, incomplete coverage or failed run health) or unknown (UNVERIFIED, which no recovery changes: by
configuration, by a producer that records no grading, after a re-judgment whose grading was not recorded or
without a known cause; a live job; evidence missing or unreadable). reason is one sentence naming the
dominant cause with counts; action is the one thing to do. Every command it prints comes from _command:
gym eval resubmit <job id> <benchmark> --recover with every option the call asked for, the typed stop’s fix
and the options the stop needs. It never adds a fresh submit that Gym’s own decision or stop text does not name,
and where Gym decided nothing is left to run it says what to report and check. An accepted run whose MLflow
delivery is not closed names the export, or that it is unverified by configuration. Rollout quality never changes
the state: an accepted run whose every rollout is flagged says so in its reason and points at the quality read. A
scheduler refusal of the resubmit and a walltime raise the partition cannot take (refused before any evidence is
read) are known dispatch failures: they need the operator’s correction, never a plain recover-now or a reader
retry.
How many model calls in the rows that decide the scored trials were sent again with an output budget lowered
to fit the context window (output_budget_refit in a call’s response metadata), and in how many of those rows.
A count, never a failure: the call generated within the window it was served at. A reader flags a cell whose requests were budget-capped from it.
Summary validity and per-rollout health from the run’s quality_summary.json, kept separate.
Validity is the health check’s own accounting verdict (run.artifacts.healthy, null for a partial summary),
and must count the same expected identities as the inventory. Per-rollout health is the rollout verdict count
(run.verdicts), judged only against an explicit max_unhealthy_fraction; it is not separately validated response or
trial health, which this adapter has no separate evidence for. Any missing piece is None.
The status of run_id on the tracking server, as (status, None), or (None, error) when it could not
be read: a missing client, an unreachable or refusing server, a missing run. Short timeout and one retry;
client_factory builds the client from the URI (default mlflow.MlflowClient).
How many of the policy’s model calls in the rows that decide the scored trials the provider stopped for
repetition, and in how many of those rows: a repetition finish reason or a repetition_detected stop reason
(vLLM’s repetition detection), as the evidence collector flags each call.
The model’s own ending, never a failure: the answer it produced is scored as it is and nothing is retried, so the count only makes loops visible. A call a verifier owns (a judge’s, also when it shares the policy’s server) or another model’s call is not counted, nor is an attempt a later one superseded.
The response gate from run.stats.responses of the health report: faulted over selected responses.
Evaluated only when every scored rollout has a verified response population and every selected call id is unique across the run (a report that predates it, an unverified call scope, a response selected twice, or zero selected responses is unknown, neither pass nor fail). Failed trials have no response row and are left out, as the policy leaves failed attempts out of the response denominator.
The rollout-quality signals of quality_summary.json the operator verdict reads: verdict counts and the
nonzero issue counts. A signal, never a gate; None when the summary has neither.
The label of a run that selected its tasks by id: its score is the subset’s, never the benchmark’s.
Whether a complete run lacks the MLflow export receipt its saved config requires, so that
gym eval resubmit --recover exports it.
Only a run whose saved config configures MLflow (provenance.mlflow.configured is true: tracking URI,
experiment and run name) and does not set mlflow_tracking_required to false, with no post-run receipt at all
and no live job, is exported. A run that configured no exporter, or whose provenance does not say, is never
exported, and a receipt, current or not, is never replaced.
A trial failure’s cause from its failure kind: request, judge, sandbox, harness, timeout, allocation, or other.
allocation: the rollout was in flight at repeated ends of its allocation (ROLLOUT_ALLOCATION_ENDED).
Evaluation health from the trial gate (trial_verdict): passed, failed or None, with its reason.
Evaluation health: evaluation-side trial faults over the expected trials, including repeats.
The counted failures are trial_faults; a model fault (classify_failure) is reported with its kind and never
fails. Only the evaluation-side share is judged against the limit: at or above it the gate fails. Below it, it
passes only with an exact count, when actual per-identity operation outcomes cover the observed model/tool and
grading path of every trial that is not a recorded failure (_observed_trial_operations); an observed grading
without a valid verdict is an evaluation-side fault. Otherwise the run is UNVERIFIED: a judge, verifier or harness
error scored as a zero in the main output carries no failure record and cannot be ruled out, so the known count is
a lower bound. That withholds a pass, never fails health. The denominator is the full materialized inventory less
accepted omissions. A run that is not complete has unaccounted trials, so the gate is not evaluated.
The report is status (PASS, FAIL or UNVERIFIED), the evaluation-side rate, exact and the fault counts.
What made a complete run’s health UNVERIFIED, as {"kind", "reason"}: one cause-first sentence with counts.
kind is judge_rejudged, grading_unobserved, unclassified_failures, recovered_not_joined or
zero_rewards_unprovable. The last has no cause beyond the missing proof, so the operator verdict treats it as
unknown. The full reason of the trial gate stays in evaluation_health.detail.
The outcome with operator_verdict set on it and, when there is a gate, inside gate_task.