Skip to navigation

nemo_gym.eval.recovery.gate

View as MarkdownOpen in Claude

The acceptance gate: response and trial health, metric coverage, MLflow delivery and the gate summary.

Module Contents

Classes

NameDescription
GateCriteriaThe response and trial failure fractions a caller may still pass. They are validated (an invalid value is

Functions

NameDescription
_bounded-
_choicesWhat in a printed command is the operator’s to choose, and that its options go in one call.
_commandThe one recovery command the operator verdict prints: gym eval resubmit <job id> <benchmark> --recover,
_continue_withThe recovery command that continues the run, after the typed stop’s fix when it stopped on one, with what in
_deciding_rowsThe saved result row that decides each scored trial: the last one saved for its identity, so an attempt a later
_delivery_action(what, action) for a complete run whose MLflow delivery is not closed: the results stand, and the delivery
_flagged_everything(what, flagged) when rollout quality flags every rollout or every task, else None; flagged counts in
_flagsThe flags one option spells (--max-walltime T --walltime-reason TEXT is two).
_partial_rerun_noteWhat a rerun in options leaves masked: it runs only the results without a gradeable artifact.
_refusal_verdictThe verdict of a resubmit the scheduler refused: no job is known to be queued, or it is unknown whether one is.
_rejudged_after_judge_failureThe scored identities among only with a saved judge failure (their answer was judged again), and the HTTP
_stop_kindThe typed failure with a known remedy the run stopped on: the chain guard’s typed stop, else the one failure
_stop_verdictThe verdict of a STOP: the continuation its own cause allows, after any guard and typed fix, else the most
_verdict-
_verifier_ownedWhether one of the row’s verifier invocations owns the captured call (a judge’s call), matched by its call id,
agent_adaptationsThe scored rollouts’ own agent_adaptations counts, summed by name: {name: {"count": total, "rows": N}},
failure_factsResponse and trial failures by cause, each with a count and a rate over the expected rollouts.
gate_summary_from_collectedThe acceptance gate task entry from the collected run: decision counts, the named score and the quality summary.
gate_task_summaryOne task entry of the acceptance gate’s run summary, filled from the decision and the existing validators.
health_check_factsNamed health-check facts from quality_summary.json for the gate warnings, kept even below any criterion.
health_verdict“evaluation health: PASS; response failures: 4.70% (truncated 41, empty 6 of 1000); trial failures: …;
judge_job_noteOne line on a judge-only job of the run (CollectedEvidence.judge_jobs): queued, live, ended without a
length_cutoffsHow many of the policy’s model calls in the rows that decide the scored trials ended on a length cutoff, and in
metric_coverageHow many expected trials the benchmark’s metric covered: {"trials": N, "expected": M, "source": ...}.
mlflow_deliveryThe MLflow delivery fact of a run: not_required by design, verified when mlflow_errors found nothing,
mlflow_errorsMLflow delivery errors from the collected exporter files and the server status the caller observed.
mlflow_notesWhat the gate states about a passing MLflow part: a current post-run export beside a live run wins.
mlflow_tracking_requiredFalse only when the run’s saved config snapshot sets mlflow_tracking_required to false; absent, unread or
operator_verdictThe one verdict an operator acts on, derived from the gate and the decision only.
operator_verdict_line-
output_budget_refitsHow many model calls in the rows that decide the scored trials were sent again with an output budget lowered
quality_inputsSummary validity and per-rollout health from the run’s quality_summary.json, kept separate.
query_mlflow_statusThe status of run_id on the tracking server, as (status, None), or (None, error) when it could not
repetition_stopsHow many of the policy’s model calls in the rows that decide the scored trials the provider stopped for
response_healthThe response gate from run.stats.responses of the health report: faulted over selected responses.
rollout_quality_factsThe rollout-quality signals of quality_summary.json the operator verdict reads: verdict counts and the
subset_noteThe label of a run that selected its tasks by id: its score is the subset’s, never the benchmark’s.
tracking_export_dueWhether a complete run lacks the MLflow export receipt its saved config requires, so that
trial_failure_causeA trial failure’s cause from its failure kind: request, judge, sandbox, harness, timeout, allocation, or other.
trial_healthEvaluation health from the trial gate (trial_verdict): passed, failed or None, with its reason.
trial_verdictEvaluation health: evaluation-side trial faults over the expected trials, including repeats.
unverified_causeWhat made a complete run’s health UNVERIFIED, as {"kind", "reason"}: one cause-first sentence with counts.
with_operator_verdictThe outcome with operator_verdict set on it and, when there is a gate, inside gate_task.

Data

GATE_FAILURE_LIMIT

MLFLOW_NOT_REQUIRED_NOTE

MLFLOW_STATUS_NOT_OBSERVED

MLFLOW_STATUS_QUERY_RETRIES

MLFLOW_STATUS_QUERY_TIMEOUT_S

RESPONSE_FAILURE_CAUSES

TRIAL_FAILURE_CAUSES

UNREADABLE_ARTIFACTS_ERROR

_CONTINUATION_GUARDS

_JUDGE_PLACEMENT_REMEDIES

_MAX_FACTS

_NO_KNOWN_COMMAND

_OPEN_STATES

_PROTOCOL_VIOLATION_REMEDY

_RECOVER_ONLY_FLAGS

_REPIN_STOPS

_RESTART_FIXES

_SHELL_QUOTED

_TRIAL_FAILURE_STATES

_TYPED_REMEDIES

_UNKNOWN_EVIDENCE_STOPS

_UNSAFE_RESUME

_UNVERIFIED_PRODUCERS

API

class nemo_gym.eval.recovery.gate.GateCriteria()

Bases: BaseModel

The response and trial failure fractions a caller may still pass. They are validated (an invalid value is refused, never clamped) and reported, but no longer applied: the gate’s only health verdict is evaluation health on the bounded failure policy’s limit (trial_verdict), and response and trial failures are reported as facts (failure_facts) for consumers to judge.

max_response_failure_fraction
float = GATE_FAILURE_LIMIT
max_trial_failure_fraction
float = GATE_FAILURE_LIMIT
model_config
= ConfigDict(extra='forbid', frozen=True)
policy_timeout_counts_as_model
bool = Field(default=True, strict=True)
nemo_gym.eval.recovery.gate.GateCriteria._fraction(
value: typing.Any
) -> float
classmethod
nemo_gym.eval.recovery.gate.GateCriteria.report(
) -> dict[str, typing.Any]

The effective criterion and the accepted, unapplied fractions, as reported beside the gate results.

nemo_gym.eval.recovery.gate._bounded(
items: list[str]
) -> str
nemo_gym.eval.recovery.gate._choices(
options: collections.abc.Sequence[str]
) -> str

What in a printed command is the operator’s to choose, and that its options go in one call.

nemo_gym.eval.recovery.gate._command(
options: str = (),
remedy: collections.abc.Sequence[str] = (),
drop: collections.abc.Sequence[str] = (),
inspect: bool = False
) -> str

The one recovery command the operator verdict prints: gym eval resubmit <job id> <benchmark> --recover, then every option the call asked for (decision.requested), then the typed remedy’s options, then options (what the stop itself needs, such as a budget extension). An option is named once: options replaces the call’s own value, the call’s own value replaces a remedy’s placeholder, and drop names options the run refused. inspect makes it the inspection, without the options an inspection refuses.

nemo_gym.eval.recovery.gate._continue_with(
options: str = (),
drop: collections.abc.Sequence[str] = (),
kind_from_rows: bool = True
) -> str

The recovery command that continues the run, after the typed stop’s fix when it stopped on one, with what in it is the operator’s to choose. A typed stop whose own text says a recovery cannot continue (a refused resume, a protocol violation) names that instead.

nemo_gym.eval.recovery.gate._deciding_rows(
) -> list[collections.abc.Mapping[str, typing.Any]]

The saved result row that decides each scored trial: the last one saved for its identity, so an attempt a later one superseded is not counted again.

nemo_gym.eval.recovery.gate._delivery_action(
delivery: collections.abc.Mapping[str, typing.Any],
export_due: bool = False
) -> tuple[str, str] | None

(what, action) for a complete run whose MLflow delivery is not closed: the results stand, and the delivery is a fact beside them. Unverified by configuration (the export did not upload the rollouts) is reported, as no export changes it.

nemo_gym.eval.recovery.gate._flagged_everything(
gate_task: collections.abc.Mapping[str, typing.Any],
expected: int
) -> tuple[str, str] | None

(what, flagged) when rollout quality flags every rollout or every task, else None; flagged counts in the unit of the check.

nemo_gym.eval.recovery.gate._flags(
option: str
) -> set[str]

The flags one option spells (--max-walltime T --walltime-reason TEXT is two).

nemo_gym.eval.recovery.gate._partial_rerun_note(
options: list[str]
) -> str

What a rerun in options leaves masked: it runs only the results without a gradeable artifact.

nemo_gym.eval.recovery.gate._refusal_verdict(
refusal: collections.abc.Mapping[str, typing.Any]
) -> dict[str, str]

The verdict of a resubmit the scheduler refused: no job is known to be queued, or it is unknown whether one is.

nemo_gym.eval.recovery.gate._rejudged_after_judge_failure(
) -> tuple[set[nemo_gym.eval.recovery.evidence.Identity], collections.Counter]

The scored identities among only with a saved judge failure (their answer was judged again), and the HTTP statuses of it.

nemo_gym.eval.recovery.gate._stop_kind(
from_rows: bool = True
) -> str | None

The typed failure with a known remedy the run stopped on: the chain guard’s typed stop, else the one failure kind every terminal identity of the decision records (Gym’s own typed stop, such as the grading quota). Read from typed fields only, never from the reason text. from_rows=False reads the chain guard’s stop alone: the row kind explains a STOP, and a decision that continues the saved run (RESUBMIT, REVERIFY) is not one.

nemo_gym.eval.recovery.gate._stop_verdict(
expected: int,
open_text: str
) -> dict[str, str]

The verdict of a STOP: the continuation its own cause allows, after any guard and typed fix, else the most specific known step, else no command.

nemo_gym.eval.recovery.gate._verdict(
state: str,
reason: str,
action: str
) -> dict[str, str]
nemo_gym.eval.recovery.gate._verifier_owned(
row: collections.abc.Mapping[str, typing.Any],
call: collections.abc.Mapping[str, typing.Any]
) -> bool

Whether one of the row’s verifier invocations owns the captured call (a judge’s call), matched by its call id, else by its model and response id: a judge served by the policy’s own server is still the judge.

nemo_gym.eval.recovery.gate.agent_adaptations(
) -> dict[str, dict[str, int]]

The scored rollouts’ own agent_adaptations counts, summed by name: {name: {"count": total, "rows": N}}, rows being the rollouts with a nonzero count. An agent records what it adapted on its own before a result, such as a request the endpoint rejected for its input size; a reported fact, never a failure.

nemo_gym.eval.recovery.gate.failure_facts(
) -> dict[str, typing.Any] | None

Response and trial failures by cause, each with a count and a rate over the expected rollouts.

Reported facts, never a verdict: no limit applies to them here. The denominator is the trial gate’s (trial_population): every expected rollout, failed and missing ones included; rows an accepted partial stage omitted are no rollout of the run and are reported separately as omitted.

A response failure is a scored rollout without a recorded grade (_recorded_grade) whose saved answer is truncated (incomplete with a cutoff signal, _cut_off), empty (no assistant text, _empty_answer) or malformed (no response with an output list, an answer the verifier marked invalid, or one left incomplete without a cutoff signal). A trial failure is a rollout in a failed, masked or unowned state (_TRIAL_FAILURE_STATES), a scored rollout that ended on a failed policy request, a scored rollout that records its own failure kind or whose judge recorded no verdict, or an observed grading without a valid verdict, by the cause its failure kind names (trial_failure_cause).

nemo_gym.eval.recovery.gate.gate_summary_from_collected(
max_unhealthy_fraction: float | None,
mlflow_status: str | None = None,
mlflow_run_id: str | None = None,
status_client_factory: collections.abc.Callable[[str], typing.Any] | None = None
) -> dict[str, typing.Any]

The acceptance gate task entry from the collected run: decision counts, the named score and the quality summary.

mlflow_status and mlflow_run_id are the tracking run’s status and id as the caller read them together. Without a status, the receipt’s run is asked of the tracking server the saved config names (query_mlflow_status; status_client_factory replaces its client). criteria are accepted and reported under criteria but not applied (GateCriteria). The only health verdict is evaluation_health; failures reports response and trial failures by cause, never a verdict. status follows gate_task_summary: an UNVERIFIED evaluation health with nothing failed is UNVERIFIED, and a chain error turns SUCCESS or UNVERIFIED into FAILED. MLflow delivery is reported as mlflow_delivery, a fact that never changes the status.

nemo_gym.eval.recovery.gate.gate_task_summary(
score: float | None,
summary_valid: bool | None,
summary_reasons: list[str],
health_passed: bool | None,
health_reasons: list[str]
) -> dict[str, typing.Any]

One task entry of the acceptance gate’s run summary, filled from the decision and the existing validators.

expected_samples is always the full materialized inventory; scored_samples counts only scored and final-model identities, so a judge, infra, terminal or missing identity can never make them equal. A settled final outcome the metric does not score (IdentityEvidence.unscored) is stated, not counted as scored. Summary validity and response/trial health stay separate. False is a failure and an error; None (unknown or missing) is no failure but never a pass: its reason is a warning and it withholds SUCCESS.

status, first match wins: RUNNING (a job is live); PLANNED (recovery work is eligible; only a caller that dispatches it reports RESUMING); FAILED (anything in errors, or the run is not a complete NOOP); UNVERIFIED (nothing failed, but summary validity or response/trial health is unknown, so acceptance is not proven); SUCCESS.

nemo_gym.eval.recovery.gate.health_check_facts(
quality: collections.abc.Mapping[str, typing.Any] | None
) -> list[str]

Named health-check facts from quality_summary.json for the gate warnings, kept even below any criterion.

Nonzero run.issues counts, checks run.artifacts.coverage did not fully evaluate (any unobserved or ignored), and run.ignored_checks. A field that is absent or malformed is reported as unknown, never as clean.

nemo_gym.eval.recovery.gate.health_verdict(
evaluation: dict[str, typing.Any] | None,
facts: dict[str, typing.Any] | None
) -> str

evaluation health: PASS; response failures: 4.70% (truncated 41, empty 6 of 1000); trial failures: ...; metric over 1000 of 1000 trials.

nemo_gym.eval.recovery.gate.judge_job_note(
job: collections.abc.Mapping[str, typing.Any]
) -> str

One line on a judge-only job of the run (CollectedEvidence.judge_jobs): queued, live, ended without a receipt, or what its receipt says it settled.

nemo_gym.eval.recovery.gate.length_cutoffs(
) -> dict[str, int]

How many of the policy’s model calls in the rows that decide the scored trials ended on a length cutoff, and in how many of those rows: a length finish reason, or its max_output_tokens and max_tokens spellings, as the evidence collector records each call.

A fact, never a failure: the model server returned the shortened answer and the rollout carried on from it, so a run that lost output to the context window or the output budget does not read as clean. The finish reason does not tell the two limits apart. A call a verifier owns (a judge’s, also when it shares the policy’s server) or another model’s call is not counted, nor is an attempt a later one superseded.

nemo_gym.eval.recovery.gate.metric_coverage(
) -> dict[str, typing.Any]

How many expected trials the benchmark’s metric covered: {"trials": N, "expected": M, "source": ...}.

A benchmark that declares the coverage of its headline metric in its aggregate (metric_coverage/trials, metric_coverage/expected and an optional metric_coverage/name) is the source (benchmark), with its own denominator and the metric’s name. Otherwise the source is collection’s own count when its coverage report has one for this inventory (coverage_report): the rollouts whose score entered the metrics, a failure counted as zero by configuration included. Otherwise it is read from the saved rows (saved_rows): the scored results and the failures their producer marked as counted zeros, which is what collection hands the metrics unless it counts more by configuration. A failure the benchmark leaves out of its metric, such as an agent deadline that kept its answer, is a trial the metric did not cover. The denominator is the trial gate’s: every expected trial, less the rows an accepted partial stage omitted.

nemo_gym.eval.recovery.gate.mlflow_delivery(
problems: collections.abc.Sequence[str],
tracking_required: bool,
status_error: str | None = None
) -> dict[str, typing.Any]

The MLflow delivery fact of a run: not_required by design, verified when mlflow_errors found nothing, pending while there is no receipt and no live run to judge or only the run’s status was not queried, else unverified with its reasons. A fact for consumers to read, never a verdict: it does not touch evaluation health or the gate’s status. A status that was neither given nor queried reads “status not queried”; a query that failed reads “status query failed: …”: never a silent “not observed”.

nemo_gym.eval.recovery.gate.mlflow_errors(
local: collections.abc.Mapping[str, typing.Any] | None,
observed_status: str | None,
configured: bool | None = None,
observed_run_id: str | None = None
) -> list[str]

MLflow delivery errors from the collected exporter files and the server status the caller observed.

local holds the live exporter’s run id (rollouts_mlflow_run_id) and the post-run export receipt (rollouts_mlflow_export_done, checked against the current artifact digests). Only a current post-run receipt with an observed FINISHED passes. A live run is not accepted here: live uploads log failures and continue, so neither its run id nor FINISHED shows the configured payloads arrived. A live run beside a post-run export leaves the observed status unbound. configured is the health report’s run.provenance.mlflow.configured, a names-only fact about the saved config snapshot (None when not reported); it only qualifies the message. The observed status is bound to a run: with a post-run receipt, observed_run_id must name the receipt’s run. BaseExporter.export writes the receipt only after every upload returned and the artifacts did not change, but its teardown still sets FINISHED on a partial export (run_failed is the process’s run, not the export’s), so a FINISHED from any other run proves nothing. No evidence is always an error, never a run that needed none: exporter activation and whether tracking was required or disabled are not recorded.

nemo_gym.eval.recovery.gate.mlflow_notes(
local: collections.abc.Mapping[str, typing.Any] | None
) -> list[str]

What the gate states about a passing MLflow part: a current post-run export beside a live run wins.

nemo_gym.eval.recovery.gate.mlflow_tracking_required(
) -> bool

False only when the run’s saved config snapshot sets mlflow_tracking_required to false; absent, unread or any other value means tracking is required. The health report’s provenance states it when its pin did; for a run whose report does not, the collector’s read of the saved rollouts_resolved_config.yaml does.

nemo_gym.eval.recovery.gate.operator_verdict(
gate_task: collections.abc.Mapping[str, typing.Any] | None,
scheduler_refusal: collections.abc.Mapping[str, typing.Any] | None = None,
walltime_plan: collections.abc.Mapping[str, typing.Any] | None = None
) -> dict[str, str]

The one verdict an operator acts on, derived from the gate and the decision only.

state is accept (gate SUCCESS, coverage complete, run health PASS, decision noop), needs_action (a STOP, FAILED, incomplete coverage or failed run health) or unknown (UNVERIFIED, which no recovery changes: by configuration, by a producer that records no grading, after a re-judgment whose grading was not recorded or without a known cause; a live job; evidence missing or unreadable). reason is one sentence naming the dominant cause with counts; action is the one thing to do. Every command it prints comes from _command: gym eval resubmit <job id> <benchmark> --recover with every option the call asked for, the typed stop’s fix and the options the stop needs. It never adds a fresh submit that Gym’s own decision or stop text does not name, and where Gym decided nothing is left to run it says what to report and check. An accepted run whose MLflow delivery is not closed names the export, or that it is unverified by configuration. Rollout quality never changes the state: an accepted run whose every rollout is flagged says so in its reason and points at the quality read. A scheduler refusal of the resubmit and a walltime raise the partition cannot take (refused before any evidence is read) are known dispatch failures: they need the operator’s correction, never a plain recover-now or a reader retry.

nemo_gym.eval.recovery.gate.operator_verdict_line(
verdict: collections.abc.Mapping[str, str]
) -> str
nemo_gym.eval.recovery.gate.output_budget_refits(
) -> dict[str, int]

How many model calls in the rows that decide the scored trials were sent again with an output budget lowered to fit the context window (output_budget_refit in a call’s response metadata), and in how many of those rows.

A count, never a failure: the call generated within the window it was served at. A reader flags a cell whose requests were budget-capped from it.

nemo_gym.eval.recovery.gate.quality_inputs(
max_unhealthy_fraction: float | None
) -> tuple[bool | None, list[str], bool | None, list[str]]

Summary validity and per-rollout health from the run’s quality_summary.json, kept separate.

Validity is the health check’s own accounting verdict (run.artifacts.healthy, null for a partial summary), and must count the same expected identities as the inventory. Per-rollout health is the rollout verdict count (run.verdicts), judged only against an explicit max_unhealthy_fraction; it is not separately validated response or trial health, which this adapter has no separate evidence for. Any missing piece is None.

nemo_gym.eval.recovery.gate.query_mlflow_status(
tracking_uri: str,
run_id: str,
client_factory: collections.abc.Callable[[str], typing.Any] | None = None
) -> tuple[str | None, str | None]

The status of run_id on the tracking server, as (status, None), or (None, error) when it could not be read: a missing client, an unreachable or refusing server, a missing run. Short timeout and one retry; client_factory builds the client from the URI (default mlflow.MlflowClient).

nemo_gym.eval.recovery.gate.repetition_stops(
) -> dict[str, int]

How many of the policy’s model calls in the rows that decide the scored trials the provider stopped for repetition, and in how many of those rows: a repetition finish reason or a repetition_detected stop reason (vLLM’s repetition detection), as the evidence collector flags each call.

The model’s own ending, never a failure: the answer it produced is scored as it is and nothing is retried, so the count only makes loops visible. A call a verifier owns (a judge’s, also when it shares the policy’s server) or another model’s call is not counted, nor is an attempt a later one superseded.

nemo_gym.eval.recovery.gate.response_health(
quality: collections.abc.Mapping[str, typing.Any] | None,
limit: float = GATE_FAILURE_LIMIT
) -> tuple[bool | None, str]

The response gate from run.stats.responses of the health report: faulted over selected responses.

Evaluated only when every scored rollout has a verified response population and every selected call id is unique across the run (a report that predates it, an unverified call scope, a response selected twice, or zero selected responses is unknown, neither pass nor fail). Failed trials have no response row and are left out, as the policy leaves failed attempts out of the response denominator.

nemo_gym.eval.recovery.gate.rollout_quality_facts(
quality: collections.abc.Mapping[str, typing.Any] | None
) -> dict[str, typing.Any] | None

The rollout-quality signals of quality_summary.json the operator verdict reads: verdict counts and the nonzero issue counts. A signal, never a gate; None when the summary has neither.

nemo_gym.eval.recovery.gate.subset_note(
selection: collections.abc.Mapping[str, typing.Any]
) -> str

The label of a run that selected its tasks by id: its score is the subset’s, never the benchmark’s.

nemo_gym.eval.recovery.gate.tracking_export_due(
tracking_required: bool
) -> bool

Whether a complete run lacks the MLflow export receipt its saved config requires, so that gym eval resubmit --recover exports it.

Only a run whose saved config configures MLflow (provenance.mlflow.configured is true: tracking URI, experiment and run name) and does not set mlflow_tracking_required to false, with no post-run receipt at all and no live job, is exported. A run that configured no exporter, or whose provenance does not say, is never exported, and a receipt, current or not, is never replaced.

nemo_gym.eval.recovery.gate.trial_failure_cause(
kind: typing.Any
) -> str

A trial failure’s cause from its failure kind: request, judge, sandbox, harness, timeout, allocation, or other.

allocation: the rollout was in flight at repeated ends of its allocation (ROLLOUT_ALLOCATION_ENDED).

nemo_gym.eval.recovery.gate.trial_health(
limit: float = GATE_FAILURE_LIMIT,
policy_timeout_counts_as_model: bool = True,
) -> tuple[bool | None, str]

Evaluation health from the trial gate (trial_verdict): passed, failed or None, with its reason.

nemo_gym.eval.recovery.gate.trial_verdict(
limit: float = GATE_FAILURE_LIMIT,
policy_timeout_counts_as_model: bool = True,
) -> tuple[bool | None, str, dict[str, typing.Any] | None]

Evaluation health: evaluation-side trial faults over the expected trials, including repeats.

The counted failures are trial_faults; a model fault (classify_failure) is reported with its kind and never fails. Only the evaluation-side share is judged against the limit: at or above it the gate fails. Below it, it passes only with an exact count, when actual per-identity operation outcomes cover the observed model/tool and grading path of every trial that is not a recorded failure (_observed_trial_operations); an observed grading without a valid verdict is an evaluation-side fault. Otherwise the run is UNVERIFIED: a judge, verifier or harness error scored as a zero in the main output carries no failure record and cannot be ruled out, so the known count is a lower bound. That withholds a pass, never fails health. The denominator is the full materialized inventory less accepted omissions. A run that is not complete has unaccounted trials, so the gate is not evaluated.

The report is status (PASS, FAIL or UNVERIFIED), the evaluation-side rate, exact and the fault counts.

nemo_gym.eval.recovery.gate.unverified_cause(
evaluation: collections.abc.Mapping[str, typing.Any] | None,
trial_reason: str
) -> dict[str, str]

What made a complete run’s health UNVERIFIED, as {"kind", "reason"}: one cause-first sentence with counts.

kind is judge_rejudged, grading_unobserved, unclassified_failures, recovered_not_joined or zero_rewards_unprovable. The last has no cause beyond the missing proof, so the operator verdict treats it as unknown. The full reason of the trial gate stays in evaluation_health.detail.

nemo_gym.eval.recovery.gate.with_operator_verdict(

The outcome with operator_verdict set on it and, when there is a gate, inside gate_task.

nemo_gym.eval.recovery.gate.GATE_FAILURE_LIMIT = 0.02
nemo_gym.eval.recovery.gate.MLFLOW_NOT_REQUIRED_NOTE = 'no MLflow export by design (mlflow_tracking_required is false in the saved conf...
nemo_gym.eval.recovery.gate.MLFLOW_STATUS_NOT_OBSERVED = 'MLflow terminal status not observed'
nemo_gym.eval.recovery.gate.MLFLOW_STATUS_QUERY_RETRIES = '1'
nemo_gym.eval.recovery.gate.MLFLOW_STATUS_QUERY_TIMEOUT_S = '10'
nemo_gym.eval.recovery.gate.RESPONSE_FAILURE_CAUSES = ('truncated', 'empty', 'malformed')
nemo_gym.eval.recovery.gate.TRIAL_FAILURE_CAUSES = ('request', 'judge', 'sandbox', 'harness', 'timeout', 'allocation', 'other')
nemo_gym.eval.recovery.gate.UNREADABLE_ARTIFACTS_ERROR = 'unreadable artifacts, read as unknown'
nemo_gym.eval.recovery.gate._CONTINUATION_GUARDS = {'scheduler_unreachable': ('unknown', 'the scheduler did not answer, so a live o...
nemo_gym.eval.recovery.gate._JUDGE_PLACEMENT_REMEDIES = {'cpu_partition_unknown': (('--judge-partition PARTITION',), 'name a CPU partiti...
nemo_gym.eval.recovery.gate._MAX_FACTS = 20
nemo_gym.eval.recovery.gate._NO_KNOWN_COMMAND = 'read the stop reason; no recovery command is known for it'
nemo_gym.eval.recovery.gate._OPEN_STATES = (IdentityState.JUDGE_RECOVERABLE, IdentityState.RETRYABLE, IdentityState.MISSING...
nemo_gym.eval.recovery.gate._PROTOCOL_VIOLATION_REMEDY = 'fix the server or the benchmark pairing (a producer bug), then `gym eval submit...
nemo_gym.eval.recovery.gate._RECOVER_ONLY_FLAGS = frozenset({'--max-walltime', '--allow-unsafe-resume', '--server-spinup-timeout-s...
nemo_gym.eval.recovery.gate._REPIN_STOPS = (re.compile("--repin <this launcher's full commit> --repin-reason TEXT runs the ...
nemo_gym.eval.recovery.gate._RESTART_FIXES = (PROVIDER_CREDENTIALS_REJECTED, DRIVER_INSTALL_FAILED, ENVIRONMENT_PREPARE_FAILE...
nemo_gym.eval.recovery.gate._SHELL_QUOTED = re.compile('\'(?:[^\']|\'\\\\\'\')*\'|"(?:[^"\\\\]|\\\\.)*\\"')
nemo_gym.eval.recovery.gate._TRIAL_FAILURE_STATES = (IdentityState.MODEL_FINAL, IdentityState.JUDGE_RECOVERABLE, IdentityState.RETRY...
nemo_gym.eval.recovery.gate._TYPED_REMEDIES: dict[str, tuple[str, tuple[str, ...]]] = {PROVIDER_CREDENTIALS_REJECTED: ('fix the credentials, then run', ()), DRIVER_IN...
nemo_gym.eval.recovery.gate._UNKNOWN_EVIDENCE_STOPS = (('the scheduler query failed', 'wait until the scheduler answers and inspect ag...
nemo_gym.eval.recovery.gate._UNSAFE_RESUME = ('--allow-unsafe-resume', '--unsafe-resume-reason TEXT')
nemo_gym.eval.recovery.gate._UNVERIFIED_PRODUCERS = {'grading_unobserved': "the verifier's grading record", 'unclassified_failures':...