nemo_gym.eval.rollout_reverification

Module Contents

Classes

Name	Description
`CacheKeysByStatus`	-
`InputRolloutPair`	-
`OutputPaths`	-
`RolloutReverificationConfig`	-
`RolloutReverificationHelper`	-

Functions

Name	Description
`_agent_to_rs_mapping_from_agent_blocks`	-
`_agent_to_rs_mapping_from_resources_only_config`	-
`_build_agent_to_resources_server_mapping`	-
`_build_verify_payload`	-
`_call_aggregate_metrics`	Call /aggregate_metrics on each resource server after rollouts complete.
`_check_reverify_mode`	Query GET /reverify_mode on each unique resource server referenced by agent_to_rs.
`_drop_cache_from_payloads`	-
`_drop_unsupported_saved_results`	Leave out the bound saved results whose verifier is not stateless: only a `STATELESS` verifier grades a saved
`_get_rs_names`	-
`_guard_atif_preflight`	Validate selected routes before ATIF reverification touches output paths.
`_guard_reverify_mode`	Check reverify_mode for every RS in the config before reverification starts.
`_is_judge_failure`	Whether a failures-sidecar row is a judge failure, the class `--judge-failed-only` recovers.
`_load_cache_keys_by_status`	-
`_load_reverified_results`	Load the full main jsonl (cached + newly re-verified successes), sorted by (task, rollout).
`_parse_output_line`	-
`_parse_output_line_key`	-
`_parse_settle_source_snapshot`	-
`_prepare_atif_payloads`	Build verifier payloads from an explicit ATIF-to-materialized-task manifest.
`_prepare_output_fpaths`	-
`_prepare_payloads`	-
`_project_reverification_row`	Retain source provenance without claiming ownership of its recovery history.
`_recovery_rollout_predicate`	Build the row filter for `--judge-failed-only` recovery.
`_reject_multistage_recovery_source`	Refuse `--judge-failed-only` on multi-stage rows before any file is written.
`_rescoring_source_answer`	Bind a re-score to its saved answer, independently of changes to the source grade.
`_resources_server_exposes_tools_over_mcp`	Read the selected resources server’s MCP exposure flag from Gym config.
`_response_has_function_calls`	-
`_reverifiable_saved_result`	Whether a selected saved result is graded again from its saved answer (`reverification_cause`).
`_rollout_response`	Return the response a rollout’s verifier scored, or explain which result type lacks one.
`_rollout_verify_debug_summary`	-
`_rs_for_row`	The resources server that verifies this row.
`_run_verification_payloads`	-
`_seed_output_with_successes`	Copy the already-successful rollout rows into the output file so the final aggregate covers
`_selected_atif_resources_server_routes`	Resolve each ATIF row through the same routing policy used by `/verify`.
`_selected_rescoring_rows`	Read answers on demand so a bounded re-score does not materialize the whole run.
`_settle_what_a_failed_pass_judged`	An exit callback for a pass that raised in-process after admitting its re-judgments, run under the run lock.
`_settled_count`	How many selected results of the run a re-judgment settled.
`_source_identity_stamps`	Collection’s identity stamps of each source rollout by (task, rollout), and the keys of every source rollout.
`_start_over_without_cache`	A resume that finds neither the output nor its failures sidecar starts the output anew, so the exporters,
`_ungraded_reverification`	Whether a re-verification answered without a usable grade: a failure, a mask or no finite reward.
`_validate_rescored_completion`	Keep a malformed verifier reply out of a managed source’s separate scoring output.
`_validate_rescoring_cache`	Refuse stale successes and failure-budget entries before cache filtering or any dispatch.
`_yield_inputs_and_rollouts_paired`	-
`bind_judge_failures`	Snapshot the source run’s selected judge failures and provenance before any judge call; the caller holds run_lock.
`check_saved_patch_replay`	Refuse to send a saved patch (`saved_model_patch`) to a resources server that does not replay one.
`due_judge_failures`	The rollouts of a managed run whose selected judge failure, or saved result to grade again
`read_settle_receipt`	The receipt at `path`, as `write_settle_receipt` wrote it.
`rejudge_saved_judge_failures`	Re-judge the saved judge failures of the run `output` in place; return how many rollouts it settled.
`settle_rejudgments`	Settle the bound judge failures with the results judged in this run; returns how many were added.
`summarize_cache_usage`	-
`validate_settle_source_snapshot`	Recheck the queued artifact snapshot under the caller’s existing run lock, before judging.
`write_settle_receipt`	Write the judge-only recovery job’s receipt at `path` (`settle_receipt_fpath`) and return it.

Data

ATIF_NO_PERSIST_FAILURE_CLASS

ATIF_PROVENANCE_KEY

JUDGE_FAILED_FAILURE_CLASS

SETTLE_RECEIPT_VERSION

_COLLECTION_ONLY_KEYS

_CONFIG_BOOL_ADAPTER

_IDENTITY_KEYS

_RECOVERY_TWO_SOURCES_WARNING

_SAVED_PATCH_REPLAY_SERVERS

_SETTLEMENT_COUNTS

_SOURCE_ANSWER_KEY

_SOURCE_RUN_ID_KEY

API

class nemo_gym.eval.rollout_reverification.CacheKeysByStatus(
    successful_keys: set[tuple[int, int]],
    terminal_keys: set[tuple[int, int]],
    maxed_out_keys: set[tuple[int, int]]
)

Dataclass

maxed_out_keys

set[tuple[int, int]]

successful_keys

set[tuple[int, int]]

terminal_keys

set[tuple[int, int]]

class nemo_gym.eval.rollout_reverification.InputRolloutPair(
    input: typing.Dict[str, typing.Any],
    rollout: typing.Dict[str, typing.Any]
)

Dataclass

input

Dict[str, Any]

rollout

Dict[str, Any]

class nemo_gym.eval.rollout_reverification.OutputPaths(
    output: pathlib.Path,
    failures: pathlib.Path
)

Dataclass

failures

Path

output

Path

class nemo_gym.eval.rollout_reverification.RolloutReverificationConfig()

Bases: RemovedRolloutOptionsMixin, UploadRolloutsConfigMixin, BaseNeMoGymCLIConfig

append

bool

atif_manifest_jsonl_fpath

Optional[str]

disable_aggregation

bool

force

bool

input_format

Literal['gym', 'atif']

judge_failed_only

bool

limit

Optional[int]

materialized_inputs_jsonl_fpath

str

num_samples_in_parallel

Optional[int]

output_jsonl_fpath

str

overwrite

bool

resume_from_cache

bool

retry_terminal_timeouts

bool

rollouts_jsonl_fpath

Optional[str]

settle_into_source

bool

settle_receipt_fpath

Optional[str]

settle_source_snapshot_json

str | None = None

skip_spent_judge_failures

bool

withdraw_unresolved_admissions

bool

nemo_gym.eval.rollout_reverification.RolloutReverificationConfig._validate_append() -> nemo_gym.eval.rollout_reverification.RolloutReverificationConfig

class nemo_gym.eval.rollout_reverification.RolloutReverificationHelper()

Bases: BaseModel

settlement

Optional[Dict[str, Any]] = None

nemo_gym.eval.rollout_reverification.RolloutReverificationHelper._run_from_config(
    config: nemo_gym.eval.rollout_reverification.RolloutReverificationConfig,
    settle_lock: contextlib.ExitStack
) -> typing.List[typing.Dict]

async

nemo_gym.eval.rollout_reverification.RolloutReverificationHelper.run_from_config(
    config: nemo_gym.eval.rollout_reverification.RolloutReverificationConfig
) -> typing.List[typing.Dict]

async

nemo_gym.eval.rollout_reverification._agent_to_rs_mapping_from_agent_blocks(
    global_config_dict: typing.Union[typing.Dict[str, typing.Any], omegaconf.DictConfig]
) -> typing.Dict[str, str]

nemo_gym.eval.rollout_reverification._agent_to_rs_mapping_from_resources_only_config(
    global_config_dict: typing.Union[typing.Dict[str, typing.Any], omegaconf.DictConfig]
) -> typing.Dict[str, str]

nemo_gym.eval.rollout_reverification._build_agent_to_resources_server_mapping(
    global_config_dict: typing.Union[typing.Dict[str, typing.Any], omegaconf.DictConfig]
) -> typing.Dict[str, str]

nemo_gym.eval.rollout_reverification._build_verify_payload(
    pair: nemo_gym.eval.rollout_reverification.InputRolloutPair
) -> typing.Dict

nemo_gym.eval.rollout_reverification._call_aggregate_metrics(
    results: typing.List[typing.Dict],
    rows: typing.List[typing.Dict],
    output_fpath: pathlib.Path
) -> typing.Optional[pathlib.Path]

async

Call /aggregate_metrics on each resource server after rollouts complete.

Writes a single _aggregate_metrics.json with one entry per agent (same shape as the old _agent_metrics.json). Returns the file path.

nemo_gym.eval.rollout_reverification._check_reverify_mode(
    server_client: nemo_gym.client.server_client.ServerClient,
    agent_to_rs: typing.Dict[str, str]
) -> typing.List[str]

async

Query GET /reverify_mode on each unique resource server referenced by agent_to_rs.

Returns a sorted list of RS names that reported ReverifyMode.UNSUPPORTED or ReverifyMode.UNKNOWN.

nemo_gym.eval.rollout_reverification._drop_cache_from_payloads(
    payloads: typing.List[typing.Dict],
    cache: nemo_gym.eval.rollout_reverification.CacheKeysByStatus
) -> typing.Iterator[typing.Dict]

nemo_gym.eval.rollout_reverification._drop_unsupported_saved_results(
    binding: typing.Dict[str, typing.Any]
) -> None

async

Leave out the bound saved results whose verifier is not stateless: only a STATELESS verifier grades a saved answer again. Judge failures stay bound as before.

nemo_gym.eval.rollout_reverification._get_rs_names(
    agent_to_rs: typing.Dict[str, str]
) -> typing.List[str]

nemo_gym.eval.rollout_reverification._guard_atif_preflight(
    payloads: typing.List[typing.Dict[str, typing.Any]]
) -> None

async

Validate selected routes before ATIF reverification touches output paths.

nemo_gym.eval.rollout_reverification._guard_reverify_mode(
    config: nemo_gym.eval.rollout_reverification.RolloutReverificationConfig
) -> typing.Optional[str]

async

Check reverify_mode for every RS in the config before reverification starts.

Returns a warning string when the user runs full re-verification (not judge-failed-only) and force=True and at least one RS is UNSUPPORTED or UNKNOWN (caller must print it and apply the unsafe_ output prefix). Raises ConfigError when force=False and at least one RS is UNSUPPORTED or UNKNOWN. Returns None when all RS are STATELESS.

nemo_gym.eval.rollout_reverification._is_judge_failure(
    row: typing.Dict[str, typing.Any]
) -> bool

Whether a failures-sidecar row is a judge failure, the class --judge-failed-only recovers.

A record_omitted_rollouts row records a rollout that produced no result, so there is nothing to judge.

nemo_gym.eval.rollout_reverification._load_cache_keys_by_status(
    output_fpaths: nemo_gym.eval.rollout_reverification.OutputPaths,
    retry_terminal_timeouts: bool = False
) -> nemo_gym.eval.rollout_reverification.CacheKeysByStatus

nemo_gym.eval.rollout_reverification._load_reverified_results(
    output_fpath: pathlib.Path,
    identity_sources: typing.Tuple[pathlib.Path, ...] = ()
) -> typing.Tuple[typing.List[typing.Dict], typing.List[typing.Dict]]

Load the full main jsonl (cached + newly re-verified successes), sorted by (task, rollout).

Returns (results, rows): results are the parsed rows — the source of truth used for both the W&B rollouts export and the aggregate-metrics payload; rows is a minimal {agent_ref, task_source} projection used only to route each result to its resources server (with the same resolver as /verify). Read once and reused for both so the file is never read twice.

A row cached by a reverification that did not keep collection’s identity stamps gets the stamps its source rollout (same task and rollout index, from identity_sources) carries, so a resumed output groups as a fresh one does. Only missing stamps are filled, in memory; the output file is not rewritten. A row whose source rollout has no stamp groups as that rollout does. If an unstamped row has no source rollout at all while other rows are stamped, its identity cannot be established and aggregating would split one run, so this raises ConfigError before anything is exported or aggregated; the output file keeps every row.

nemo_gym.eval.rollout_reverification._parse_output_line(
    line: bytes
) -> typing.Dict[str, typing.Any]

nemo_gym.eval.rollout_reverification._parse_output_line_key(
    line: bytes
) -> tuple[int, int] | None

nemo_gym.eval.rollout_reverification._parse_settle_source_snapshot(
    raw: str
) -> dict[str, str | None]

nemo_gym.eval.rollout_reverification._prepare_atif_payloads(
    materialized_inputs_jsonl_fpath: pathlib.Path,
    atif_manifest_jsonl_fpath: pathlib.Path,
    limit: typing.Optional[int] = None
) -> typing.List[typing.Dict]

Build verifier payloads from an explicit ATIF-to-materialized-task manifest.

nemo_gym.eval.rollout_reverification._prepare_output_fpaths(
    output_name_prefix: str,
    output_jsonl_fpath: str,
    resume_from_cache: bool,
    overwrite: bool,
    append: bool,
    protected_sources: tuple[pathlib.Path, ...] = ()
) -> nemo_gym.eval.rollout_reverification.OutputPaths

nemo_gym.eval.rollout_reverification._prepare_payloads(
    materialized_inputs_jsonl_fpath: pathlib.Path,
    rollouts_jsonl_fpath: pathlib.Path,
    output_fpaths: nemo_gym.eval.rollout_reverification.OutputPaths,
    resume_from_cache: bool,
    limit: typing.Optional[int] = None,
    rollout_predicate: typing.Optional[typing.Callable[[Dict[str, Any]], bool]] = None,
    retry_terminal_timeouts: bool = False,
    selected_rollouts: typing.Optional[typing.Iterable[typing.Dict]] = None
) -> typing.List[typing.Dict]

nemo_gym.eval.rollout_reverification._project_reverification_row(
    row: typing.Dict
) -> typing.Dict

Retain source provenance without claiming ownership of its recovery history.

nemo_gym.eval.rollout_reverification._recovery_rollout_predicate(
    skip_keys: typing.Optional[set[tuple[typing.Any, typing.Any]]] = None
) -> typing.Callable[[Dict[str, Any]], bool]

Build the row filter for --judge-failed-only recovery.

Keeps only judge-class failures, skips any whose (task, rollout) key is in skip_keys (the keys already present in the output: seeded successes + rows a prior recovery already appended) so a key-in-both is never re-verified/duplicated and re-runs are idempotent — this dedup is done here, independent of the resume/cache machinery — and dedups on the key so a sidecar with multiple failure attempts for one key re-verifies it exactly once.

nemo_gym.eval.rollout_reverification._reject_multistage_recovery_source(
    rollouts_jsonl_fpath: pathlib.Path
) -> None

Refuse --judge-failed-only on multi-stage rows before any file is written.

Multi-stage rows carry stage_index. Recovering them outside the multi-stage collection would lose stage identity and the adaptive reference set. Recovery reads the failures sidecar.

nemo_gym.eval.rollout_reverification._rescoring_source_answer(
    row: typing.Dict,
    run_id: str
) -> dict[str, str | int]

Bind a re-score to its saved answer, independently of changes to the source grade.

nemo_gym.eval.rollout_reverification._resources_server_exposes_tools_over_mcp(
    global_config_dict: typing.Union[typing.Dict[str, typing.Any], omegaconf.DictConfig],
    resources_server_name: str
) -> bool

Read the selected resources server’s MCP exposure flag from Gym config.

nemo_gym.eval.rollout_reverification._response_has_function_calls(
    row: typing.Dict[str, typing.Any]
) -> bool

nemo_gym.eval.rollout_reverification._reverifiable_saved_result(
    row: typing.Dict[str, typing.Any]
) -> bool

Whether a selected saved result is graded again from its saved answer (reverification_cause).

nemo_gym.eval.rollout_reverification._rollout_response(
    rollout: typing.Dict[str, typing.Any]
) -> typing.Any

Return the response a rollout’s verifier scored, or explain which result type lacks one.

nemo_gym.eval.rollout_reverification._rollout_verify_debug_summary(
    row: typing.Dict[str, typing.Any],
    resources_server_name: str
) -> typing.Dict[str, typing.Any]

nemo_gym.eval.rollout_reverification._rs_for_row(
    row: typing.Dict[str, typing.Any],
    agent_to_rs: typing.Dict[str, str],
    global_config_dict: typing.Union[typing.Dict[str, typing.Any], omegaconf.DictConfig]
) -> str

The resources server that verifies this row.

A task_source naming a resources server is authoritative (it is the declaring instance the dataset was stamped with — no agent indirection needed). Otherwise fall back to the rollout’s agent_ref via the config’s agent->rs edges.

nemo_gym.eval.rollout_reverification._run_verification_payloads(
    payloads: typing.List[typing.Dict],
    semaphore: asyncio.Semaphore | contextlib.nullcontext[None] | None = None
) -> typing.Iterator[asyncio.Future]

nemo_gym.eval.rollout_reverification._seed_output_with_successes(
    successes_fpath: pathlib.Path,
    output_fpath: pathlib.Path,
    exclude: frozenset[tuple[int, int]] = frozenset()
) -> set[tuple[int, int]]

Copy the already-successful rollout rows into the output file so the final aggregate covers successes + recovered rows, and return the set of (task, rollout) keys ALREADY PRESENT in the output afterward. Rows whose key is in exclude (saved results that are re-verified) are not seeded.

nemo_gym.eval.rollout_reverification._selected_atif_resources_server_routes(
    global_config_dict: typing.Union[typing.Dict[str, typing.Any], omegaconf.DictConfig],
    payloads: typing.List[typing.Dict[str, typing.Any]]
) -> typing.List[str]

Resolve each ATIF row through the same routing policy used by /verify.

nemo_gym.eval.rollout_reverification._selected_rescoring_rows(
    store: nemo_gym.rollouts.rollout_store.RolloutStore
) -> typing.Iterator[typing.Dict]

Read answers on demand so a bounded re-score does not materialize the whole run.

nemo_gym.eval.rollout_reverification._settle_what_a_failed_pass_judged(
    source: pathlib.Path,
    identities: typing.List[str],
    judged: typing.List[typing.Dict],
    binding: typing.Dict[str, typing.Any],
    record: typing.Callable[[Dict[str, int]], None]
) -> typing.Callable[..., bool]

An exit callback for a pass that raised in-process after admitting its re-judgments, run under the run lock.

The outcomes the pass already received are settled as usual, then each admission still without a saved outcome is withdrawn: its row stays selected, so a later pass grades it again and recovery decides REVERIFY instead of stopping on an outcome it cannot know. At worst a /verify call that did reach its server is made again for the same saved answer. An interrupted or killed pass leaves its admissions for recovery as before. record gets the pass’s settlement counts, so its failed receipt says what it settled and withdrew.

nemo_gym.eval.rollout_reverification._settled_count(
    output: pathlib.Path
) -> int

How many selected results of the run a re-judgment settled.

nemo_gym.eval.rollout_reverification._source_identity_stamps(
    fpaths: typing.Tuple[pathlib.Path, ...]
) -> typing.Tuple[typing.Dict[typing.Tuple[int, int], typing.Dict[str, typing.Any]], set[typing.Tuple[int, int]]]

Collection’s identity stamps of each source rollout by (task, rollout), and the keys of every source rollout.

Rollouts without stamps have a key but no stamps entry.

nemo_gym.eval.rollout_reverification._start_over_without_cache(
    config: nemo_gym.eval.rollout_reverification.RolloutReverificationConfig,
    output_fpaths: nemo_gym.eval.rollout_reverification.OutputPaths
) -> None

A resume that finds neither the output nor its failures sidecar starts the output anew, so the exporters, which reopened the run recorded next to it, start a new run too. Append recovery continues its output.

nemo_gym.eval.rollout_reverification._ungraded_reverification(
    result: typing.Dict[str, typing.Any]
) -> bool

Whether a re-verification answered without a usable grade: a failure, a mask or no finite reward.

nemo_gym.eval.rollout_reverification._validate_rescored_completion(
    result: object
) -> None

Keep a malformed verifier reply out of a managed source’s separate scoring output.

nemo_gym.eval.rollout_reverification._validate_rescoring_cache(
    store: nemo_gym.rollouts.rollout_store.RolloutStore,
    output_fpaths: nemo_gym.eval.rollout_reverification.OutputPaths
) -> None

Refuse stale successes and failure-budget entries before cache filtering or any dispatch.

nemo_gym.eval.rollout_reverification._yield_inputs_and_rollouts_paired(
    materialized_inputs_jsonl_fpath: pathlib.Path,
    rollouts_jsonl_fpath: pathlib.Path,
    limit: typing.Optional[int] = None,
    rollout_predicate: typing.Optional[typing.Callable[[Dict[str, Any]], bool]] = None,
    selected_rollouts: typing.Optional[typing.Iterable[typing.Dict]] = None
) -> typing.Iterator[nemo_gym.eval.rollout_reverification.InputRolloutPair]

nemo_gym.eval.rollout_reverification.bind_judge_failures(
    source: pathlib.Path,
    skip_spent: bool = False,
    withdraw_unresolved: bool = False
) -> typing.Dict[str, typing.Any]

Snapshot the source run’s selected judge failures and provenance before any judge call; the caller holds run_lock.

A failed re-judgment spends the run’s failure budget (attempt_counts against the saved max_rollout_attempts); a rollout that has spent it, or a run without a saved cap, is refused before judging. With skip_spent a rollout that has spent it is left out of the binding, and the others are bound. An admitted re-judgment with no saved outcome is refused, unless withdraw_unresolved: the caller ruled out a live job of the run, so the pass that admitted it died, and it is withdrawn and its row bound again.

The selected saved results whose grading failed for a judge or verify-transport cause, or was never recorded (_reverifiable_saved_result), are bound too, under saved_results: a failed re-verification of one saves nothing, so it spends no failure attempt.

nemo_gym.eval.rollout_reverification.check_saved_patch_replay(
    row: typing.Dict[str, typing.Any],
    resources_server_name: str,
    global_config_dict: typing.Union[typing.Dict[str, typing.Any], omegaconf.DictConfig]
) -> None

Refuse to send a saved patch (saved_model_patch) to a resources server that does not replay one.

nemo_gym.eval.rollout_reverification.due_judge_failures(
    output: pathlib.Path
) -> typing.List[str]

The rollouts of a managed run whose selected judge failure, or saved result to grade again (_reverifiable_saved_result), a re-judgment can still settle.

A judge failure whose saved failure attempts are spent, a run without a manifest and a run without a saved failure cap have none.

nemo_gym.eval.rollout_reverification.read_settle_receipt(
    path: pathlib.Path
) -> typing.Dict[str, typing.Any]

The receipt at path, as write_settle_receipt wrote it.

nemo_gym.eval.rollout_reverification.rejudge_saved_judge_failures(
    run_config: typing.Dict[str, typing.Any],
    output: pathlib.Path
) -> int

async

Re-judge the saved judge failures of the run output in place; return how many rollouts it settled.

The saved results whose grading failed for a judge or verify-transport cause, or was never recorded, are graded again from their saved answers too (bind_judge_failures); only stateless verifiers grade them.

A run’s driver calls it at the end of collection, and so again at the end of each resumed collection, while the run’s servers are up. Only the saved answers go to /verify again: no policy call, no new model attempt. It is the gym eval reverify --judge-failed-only --settle-into-source path: each re-judgment is admitted in the run manifest before its judge call and spends one of the rollout’s saved failure attempts, so an interrupted pass is not judged twice, and a rollout that has spent them keeps its masked row. The judged rows are kept under reverification/<pass>/ beside the run, and each settled row names its admission and the failure it settles. A run with no due judge failure returns 0 and judges nothing.

nemo_gym.eval.rollout_reverification.settle_rejudgments(
    source: pathlib.Path,
    judged: typing.List[typing.Dict],
    binding: typing.Dict[str, typing.Any]
) -> int

Settle the bound judge failures with the results judged in this run; returns how many were added.

The caller holds the run lock taken for the binding, so no other writer ran in between. The source must still match the binding. A judged identity that the binding does not hold, a changed manifest or materialized input, or a failure that is no longer the selected outcome is refused. A bound saved result replaces its grade only with a result that binds the same saved answer and attempt (is_rejudgment_of); one whose re-verification answered without a grade keeps its saved row, and its admission is withdrawn.

Each settlement is recorded in binding["settled_operations"] (this pass’s admission operation ids) as soon as it is fsynced, and each completed withdrawal in binding["withdrawn_operations"], so a pass that raises afterwards, for example on the Store’s exit-time coverage write, still counts them, each once. Settling again after such an error (the failed pass’s exit callback) skips what this pass already settled or withdrew, closes a settled admission that is still open, and expects only the pass’s other admissions to be open.

nemo_gym.eval.rollout_reverification.summarize_cache_usage(
    cache: nemo_gym.eval.rollout_reverification.CacheKeysByStatus,
    all_payloads: typing.List[typing.Dict],
    filtered_payloads: typing.List[typing.Dict]
) -> None

nemo_gym.eval.rollout_reverification.validate_settle_source_snapshot(
    source: pathlib.Path,
    raw: str
) -> None

Recheck the queued artifact snapshot under the caller’s existing run lock, before judging.

nemo_gym.eval.rollout_reverification.write_settle_receipt(
    path: pathlib.Path,
    state: str,
    source: typing.Optional[str],
    job_id: typing.Optional[str],
    settlement: typing.Optional[typing.Dict[str, typing.Any]],
    started_at: str,
    coverage: typing.Optional[typing.Dict[str, typing.Any]] = None,
    due_after: typing.Optional[int] = None,
    refreshed: typing.Optional[typing.Dict[str, bool]] = None,
    error: typing.Optional[str] = None
) -> typing.Dict[str, typing.Any]

Write the judge-only recovery job’s receipt at path (settle_receipt_fpath) and return it.

state is completed or failed. The counts are the pass’s settlement record (zeros when the pass ended before it settled anything): the saved answers selected for grading again, those judged, those settled with a grade bound to the same saved answer, run and attempt, those re-verified without a grade and left as they were, and those not re-verified because their verifier is not stateless. coverage is the settled run’s RolloutStore.coverage() (its counts only), due_after how many saved answers a later pass would still grade again, and refreshed whether the run’s aggregate metrics and quality summary were recomputed. The receipt is written whole and moved into place, so a reader never sees a partial one.

nemo_gym.eval.rollout_reverification.ATIF_NO_PERSIST_FAILURE_CLASS = 'kill_shaped'

nemo_gym.eval.rollout_reverification.ATIF_PROVENANCE_KEY = '_ng_atif_provenance'

nemo_gym.eval.rollout_reverification.JUDGE_FAILED_FAILURE_CLASS = 'judge_failed'

nemo_gym.eval.rollout_reverification.SETTLE_RECEIPT_VERSION = 1

nemo_gym.eval.rollout_reverification._COLLECTION_ONLY_KEYS = ('limit', 'output_jsonl_fpath', 'materialized_inputs_jsonl_fpath', 'resume_from_...

nemo_gym.eval.rollout_reverification._CONFIG_BOOL_ADAPTER = TypeAdapter(bool)

nemo_gym.eval.rollout_reverification._IDENTITY_KEYS = (ENVIRONMENT_SERVER_STAMP_KEY_NAME, NG_RESULT_TYPE_KEY)

nemo_gym.eval.rollout_reverification._RECOVERY_TWO_SOURCES_WARNING = "WARNING: judge-failed-only recovery merges rewards from TWO sources — the succe...

nemo_gym.eval.rollout_reverification._SAVED_PATCH_REPLAY_SERVERS = frozenset({'deepswe'})

nemo_gym.eval.rollout_reverification._SETTLEMENT_COUNTS = ('selected', 'judged', 'settled', 'withdrawn', 'unsupported')

nemo_gym.eval.rollout_reverification._SOURCE_ANSWER_KEY = '_ng_reverification_source_answer'

nemo_gym.eval.rollout_reverification._SOURCE_RUN_ID_KEY = '_ng_reverification_source_run_id'