> This page is for version 0.2.1. > For other versions, use one of these documentation indexes: > - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt > - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt > - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt > - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt > - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt > - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt > - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt > - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server. # nemo_gym.eval.rollout_reverification ## Module Contents ### Classes | Name | Description | | -------------------------------------------------------------------------------------------------- | ----------- | | [`CacheKeysByStatus`](#nemo_gym-eval-rollout_reverification-CacheKeysByStatus) | - | | [`InputRolloutPair`](#nemo_gym-eval-rollout_reverification-InputRolloutPair) | - | | [`OutputPaths`](#nemo_gym-eval-rollout_reverification-OutputPaths) | - | | [`RolloutReverificationConfig`](#nemo_gym-eval-rollout_reverification-RolloutReverificationConfig) | - | | [`RolloutReverificationHelper`](#nemo_gym-eval-rollout_reverification-RolloutReverificationHelper) | - | ### Functions | Name | Description | | ------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------ | | [`_agent_to_rs_mapping_from_agent_blocks`](#nemo_gym-eval-rollout_reverification-_agent_to_rs_mapping_from_agent_blocks) | - | | [`_agent_to_rs_mapping_from_resources_only_config`](#nemo_gym-eval-rollout_reverification-_agent_to_rs_mapping_from_resources_only_config) | - | | [`_build_agent_to_resources_server_mapping`](#nemo_gym-eval-rollout_reverification-_build_agent_to_resources_server_mapping) | - | | [`_build_verify_payload`](#nemo_gym-eval-rollout_reverification-_build_verify_payload) | - | | [`_call_aggregate_metrics`](#nemo_gym-eval-rollout_reverification-_call_aggregate_metrics) | Call /aggregate_metrics on each resource server after rollouts complete. | | [`_check_reverify_mode`](#nemo_gym-eval-rollout_reverification-_check_reverify_mode) | Query GET /reverify_mode on each unique resource server referenced by agent_to_rs. | | [`_drop_cache_from_payloads`](#nemo_gym-eval-rollout_reverification-_drop_cache_from_payloads) | - | | [`_drop_unsupported_saved_results`](#nemo_gym-eval-rollout_reverification-_drop_unsupported_saved_results) | Leave out the bound saved results whose verifier is not stateless: only a `STATELESS` verifier grades a saved | | [`_get_rs_names`](#nemo_gym-eval-rollout_reverification-_get_rs_names) | - | | [`_guard_atif_preflight`](#nemo_gym-eval-rollout_reverification-_guard_atif_preflight) | Validate selected routes before ATIF reverification touches output paths. | | [`_guard_reverify_mode`](#nemo_gym-eval-rollout_reverification-_guard_reverify_mode) | Check reverify_mode for every RS in the config before reverification starts. | | [`_is_judge_failure`](#nemo_gym-eval-rollout_reverification-_is_judge_failure) | Whether a failures-sidecar row is a judge failure, the class `--judge-failed-only` recovers. | | [`_load_cache_keys_by_status`](#nemo_gym-eval-rollout_reverification-_load_cache_keys_by_status) | - | | [`_load_reverified_results`](#nemo_gym-eval-rollout_reverification-_load_reverified_results) | Load the full main jsonl (cached + newly re-verified successes), sorted by (task, rollout). | | [`_parse_output_line`](#nemo_gym-eval-rollout_reverification-_parse_output_line) | - | | [`_parse_output_line_key`](#nemo_gym-eval-rollout_reverification-_parse_output_line_key) | - | | [`_parse_settle_source_snapshot`](#nemo_gym-eval-rollout_reverification-_parse_settle_source_snapshot) | - | | [`_prepare_atif_payloads`](#nemo_gym-eval-rollout_reverification-_prepare_atif_payloads) | Build verifier payloads from an explicit ATIF-to-materialized-task manifest. | | [`_prepare_output_fpaths`](#nemo_gym-eval-rollout_reverification-_prepare_output_fpaths) | - | | [`_prepare_payloads`](#nemo_gym-eval-rollout_reverification-_prepare_payloads) | - | | [`_project_reverification_row`](#nemo_gym-eval-rollout_reverification-_project_reverification_row) | Retain source provenance without claiming ownership of its recovery history. | | [`_recovery_rollout_predicate`](#nemo_gym-eval-rollout_reverification-_recovery_rollout_predicate) | Build the row filter for `--judge-failed-only` recovery. | | [`_reject_multistage_recovery_source`](#nemo_gym-eval-rollout_reverification-_reject_multistage_recovery_source) | Refuse `--judge-failed-only` on multi-stage rows before any file is written. | | [`_rescoring_source_answer`](#nemo_gym-eval-rollout_reverification-_rescoring_source_answer) | Bind a re-score to its saved answer, independently of changes to the source grade. | | [`_resources_server_exposes_tools_over_mcp`](#nemo_gym-eval-rollout_reverification-_resources_server_exposes_tools_over_mcp) | Read the selected resources server's MCP exposure flag from Gym config. | | [`_response_has_function_calls`](#nemo_gym-eval-rollout_reverification-_response_has_function_calls) | - | | [`_reverifiable_saved_result`](#nemo_gym-eval-rollout_reverification-_reverifiable_saved_result) | Whether a selected saved result is graded again from its saved answer (`reverification_cause`). | | [`_rollout_response`](#nemo_gym-eval-rollout_reverification-_rollout_response) | Return the response a rollout's verifier scored, or explain which result type lacks one. | | [`_rollout_verify_debug_summary`](#nemo_gym-eval-rollout_reverification-_rollout_verify_debug_summary) | - | | [`_rs_for_row`](#nemo_gym-eval-rollout_reverification-_rs_for_row) | The resources server that verifies this row. | | [`_run_verification_payloads`](#nemo_gym-eval-rollout_reverification-_run_verification_payloads) | - | | [`_seed_output_with_successes`](#nemo_gym-eval-rollout_reverification-_seed_output_with_successes) | Copy the already-successful rollout rows into the output file so the final aggregate covers | | [`_selected_atif_resources_server_routes`](#nemo_gym-eval-rollout_reverification-_selected_atif_resources_server_routes) | Resolve each ATIF row through the same routing policy used by `/verify`. | | [`_selected_rescoring_rows`](#nemo_gym-eval-rollout_reverification-_selected_rescoring_rows) | Read answers on demand so a bounded re-score does not materialize the whole run. | | [`_settle_what_a_failed_pass_judged`](#nemo_gym-eval-rollout_reverification-_settle_what_a_failed_pass_judged) | An exit callback for a pass that raised in-process after admitting its re-judgments, run under the run lock. | | [`_settled_count`](#nemo_gym-eval-rollout_reverification-_settled_count) | How many selected results of the run a re-judgment settled. | | [`_source_identity_stamps`](#nemo_gym-eval-rollout_reverification-_source_identity_stamps) | Collection's identity stamps of each source rollout by (task, rollout), and the keys of every source rollout. | | [`_start_over_without_cache`](#nemo_gym-eval-rollout_reverification-_start_over_without_cache) | A resume that finds neither the output nor its failures sidecar starts the output anew, so the exporters, | | [`_ungraded_reverification`](#nemo_gym-eval-rollout_reverification-_ungraded_reverification) | Whether a re-verification answered without a usable grade: a failure, a mask or no finite reward. | | [`_validate_rescored_completion`](#nemo_gym-eval-rollout_reverification-_validate_rescored_completion) | Keep a malformed verifier reply out of a managed source's separate scoring output. | | [`_validate_rescoring_cache`](#nemo_gym-eval-rollout_reverification-_validate_rescoring_cache) | Refuse stale successes and failure-budget entries before cache filtering or any dispatch. | | [`_yield_inputs_and_rollouts_paired`](#nemo_gym-eval-rollout_reverification-_yield_inputs_and_rollouts_paired) | - | | [`bind_judge_failures`](#nemo_gym-eval-rollout_reverification-bind_judge_failures) | Snapshot the source run's selected judge failures and provenance before any judge call; the caller holds run_lock. | | [`check_saved_patch_replay`](#nemo_gym-eval-rollout_reverification-check_saved_patch_replay) | Refuse to send a saved patch (`saved_model_patch`) to a resources server that does not replay one. | | [`due_judge_failures`](#nemo_gym-eval-rollout_reverification-due_judge_failures) | The rollouts of a managed run whose selected judge failure, or saved result to grade again | | [`read_settle_receipt`](#nemo_gym-eval-rollout_reverification-read_settle_receipt) | The receipt at `path`, as `write_settle_receipt` wrote it. | | [`rejudge_saved_judge_failures`](#nemo_gym-eval-rollout_reverification-rejudge_saved_judge_failures) | Re-judge the saved judge failures of the run `output` in place; return how many rollouts it settled. | | [`settle_rejudgments`](#nemo_gym-eval-rollout_reverification-settle_rejudgments) | Settle the bound judge failures with the results judged in this run; returns how many were added. | | [`summarize_cache_usage`](#nemo_gym-eval-rollout_reverification-summarize_cache_usage) | - | | [`validate_settle_source_snapshot`](#nemo_gym-eval-rollout_reverification-validate_settle_source_snapshot) | Recheck the queued artifact snapshot under the caller's existing run lock, before judging. | | [`write_settle_receipt`](#nemo_gym-eval-rollout_reverification-write_settle_receipt) | Write the judge-only recovery job's receipt at `path` (`settle_receipt_fpath`) and return it. | ### Data [`ATIF_NO_PERSIST_FAILURE_CLASS`](#nemo_gym-eval-rollout_reverification-ATIF_NO_PERSIST_FAILURE_CLASS) [`ATIF_PROVENANCE_KEY`](#nemo_gym-eval-rollout_reverification-ATIF_PROVENANCE_KEY) [`JUDGE_FAILED_FAILURE_CLASS`](#nemo_gym-eval-rollout_reverification-JUDGE_FAILED_FAILURE_CLASS) [`SETTLE_RECEIPT_VERSION`](#nemo_gym-eval-rollout_reverification-SETTLE_RECEIPT_VERSION) [`_COLLECTION_ONLY_KEYS`](#nemo_gym-eval-rollout_reverification-_COLLECTION_ONLY_KEYS) [`_CONFIG_BOOL_ADAPTER`](#nemo_gym-eval-rollout_reverification-_CONFIG_BOOL_ADAPTER) [`_IDENTITY_KEYS`](#nemo_gym-eval-rollout_reverification-_IDENTITY_KEYS) [`_RECOVERY_TWO_SOURCES_WARNING`](#nemo_gym-eval-rollout_reverification-_RECOVERY_TWO_SOURCES_WARNING) [`_SAVED_PATCH_REPLAY_SERVERS`](#nemo_gym-eval-rollout_reverification-_SAVED_PATCH_REPLAY_SERVERS) [`_SETTLEMENT_COUNTS`](#nemo_gym-eval-rollout_reverification-_SETTLEMENT_COUNTS) [`_SOURCE_ANSWER_KEY`](#nemo_gym-eval-rollout_reverification-_SOURCE_ANSWER_KEY) [`_SOURCE_RUN_ID_KEY`](#nemo_gym-eval-rollout_reverification-_SOURCE_RUN_ID_KEY) ### API ```python class nemo_gym.eval.rollout_reverification.CacheKeysByStatus( successful_keys: set[tuple[int, int]], terminal_keys: set[tuple[int, int]], maxed_out_keys: set[tuple[int, int]] ) ``` Dataclass **`maxed_out_keys`** `set[tuple[int, int]]` --- **`successful_keys`** `set[tuple[int, int]]` --- **`terminal_keys`** `set[tuple[int, int]]` --- ```python class nemo_gym.eval.rollout_reverification.InputRolloutPair( input: typing.Dict[str, typing.Any], rollout: typing.Dict[str, typing.Any] ) ``` Dataclass **`input`** `Dict[str, Any]` --- **`rollout`** `Dict[str, Any]` --- ```python class nemo_gym.eval.rollout_reverification.OutputPaths( output: pathlib.Path, failures: pathlib.Path ) ``` Dataclass **`failures`** `Path` --- **`output`** `Path` --- ```python class nemo_gym.eval.rollout_reverification.RolloutReverificationConfig() ``` **Bases:** [RemovedRolloutOptionsMixin](/nemo/gym/nemo-gym/nemo_gym/config/config_types#nemo_gym-config-config_types-RemovedRolloutOptionsMixin), [UploadRolloutsConfigMixin](/nemo/gym/nemo-gym/nemo_gym/config/config_types#nemo_gym-config-config_types-UploadRolloutsConfigMixin), [BaseNeMoGymCLIConfig](/nemo/gym/nemo-gym/nemo_gym/config/config_types#nemo_gym-config-config_types-BaseNeMoGymCLIConfig) **`append`** `bool` --- **`atif_manifest_jsonl_fpath`** `Optional[str]` --- **`disable_aggregation`** `bool` --- **`force`** `bool` --- **`input_format`** `Literal['gym', 'atif']` --- **`judge_failed_only`** `bool` --- **`limit`** `Optional[int]` --- **`materialized_inputs_jsonl_fpath`** `str` --- **`num_samples_in_parallel`** `Optional[int]` --- **`output_jsonl_fpath`** `str` --- **`overwrite`** `bool` --- **`resume_from_cache`** `bool` --- **`retry_terminal_timeouts`** `bool` --- **`rollouts_jsonl_fpath`** `Optional[str]` --- **`settle_into_source`** `bool` --- **`settle_receipt_fpath`** `Optional[str]` --- **`settle_source_snapshot_json`** `str | None = None` --- **`skip_spent_judge_failures`** `bool` --- **`withdraw_unresolved_admissions`** `bool` --- ```python nemo_gym.eval.rollout_reverification.RolloutReverificationConfig._validate_append() -> nemo_gym.eval.rollout_reverification.RolloutReverificationConfig ``` ```python class nemo_gym.eval.rollout_reverification.RolloutReverificationHelper() ``` **Bases:** `BaseModel` **`settlement`** `Optional[Dict[str, Any]] = None` --- ```python nemo_gym.eval.rollout_reverification.RolloutReverificationHelper._run_from_config( config: nemo_gym.eval.rollout_reverification.RolloutReverificationConfig, settle_lock: contextlib.ExitStack ) -> typing.List[typing.Dict] ``` async ```python nemo_gym.eval.rollout_reverification.RolloutReverificationHelper.run_from_config( config: nemo_gym.eval.rollout_reverification.RolloutReverificationConfig ) -> typing.List[typing.Dict] ``` async ```python nemo_gym.eval.rollout_reverification._agent_to_rs_mapping_from_agent_blocks( global_config_dict: typing.Union[typing.Dict[str, typing.Any], omegaconf.DictConfig] ) -> typing.Dict[str, str] ``` ```python nemo_gym.eval.rollout_reverification._agent_to_rs_mapping_from_resources_only_config( global_config_dict: typing.Union[typing.Dict[str, typing.Any], omegaconf.DictConfig] ) -> typing.Dict[str, str] ``` ```python nemo_gym.eval.rollout_reverification._build_agent_to_resources_server_mapping( global_config_dict: typing.Union[typing.Dict[str, typing.Any], omegaconf.DictConfig] ) -> typing.Dict[str, str] ``` ```python nemo_gym.eval.rollout_reverification._build_verify_payload( pair: nemo_gym.eval.rollout_reverification.InputRolloutPair ) -> typing.Dict ``` ```python nemo_gym.eval.rollout_reverification._call_aggregate_metrics( results: typing.List[typing.Dict], rows: typing.List[typing.Dict], output_fpath: pathlib.Path ) -> typing.Optional[pathlib.Path] ``` async Call /aggregate_metrics on each resource server after rollouts complete. Writes a single \_aggregate_metrics.json with one entry per agent (same shape as the old \_agent_metrics.json). Returns the file path. ```python nemo_gym.eval.rollout_reverification._check_reverify_mode( server_client: nemo_gym.client.server_client.ServerClient, agent_to_rs: typing.Dict[str, str] ) -> typing.List[str] ``` async Query GET /reverify_mode on each unique resource server referenced by agent_to_rs. Returns a sorted list of RS names that reported ReverifyMode.UNSUPPORTED or ReverifyMode.UNKNOWN. ```python nemo_gym.eval.rollout_reverification._drop_cache_from_payloads( payloads: typing.List[typing.Dict], cache: nemo_gym.eval.rollout_reverification.CacheKeysByStatus ) -> typing.Iterator[typing.Dict] ``` ```python nemo_gym.eval.rollout_reverification._drop_unsupported_saved_results( binding: typing.Dict[str, typing.Any] ) -> None ``` async Leave out the bound saved results whose verifier is not stateless: only a `STATELESS` verifier grades a saved answer again. Judge failures stay bound as before. ```python nemo_gym.eval.rollout_reverification._get_rs_names( agent_to_rs: typing.Dict[str, str] ) -> typing.List[str] ``` ```python nemo_gym.eval.rollout_reverification._guard_atif_preflight( payloads: typing.List[typing.Dict[str, typing.Any]] ) -> None ``` async Validate selected routes before ATIF reverification touches output paths. ```python nemo_gym.eval.rollout_reverification._guard_reverify_mode( config: nemo_gym.eval.rollout_reverification.RolloutReverificationConfig ) -> typing.Optional[str] ``` async Check reverify_mode for every RS in the config before reverification starts. Returns a warning string when the user runs full re-verification (not judge-failed-only) and force=True and at least one RS is UNSUPPORTED or UNKNOWN (caller must print it and apply the unsafe\_ output prefix). Raises ConfigError when force=False and at least one RS is UNSUPPORTED or UNKNOWN. Returns None when all RS are STATELESS. ```python nemo_gym.eval.rollout_reverification._is_judge_failure( row: typing.Dict[str, typing.Any] ) -> bool ``` Whether a failures-sidecar row is a judge failure, the class `--judge-failed-only` recovers. A `record_omitted_rollouts` row records a rollout that produced no result, so there is nothing to judge. ```python nemo_gym.eval.rollout_reverification._load_cache_keys_by_status( output_fpaths: nemo_gym.eval.rollout_reverification.OutputPaths, retry_terminal_timeouts: bool = False ) -> nemo_gym.eval.rollout_reverification.CacheKeysByStatus ``` ```python nemo_gym.eval.rollout_reverification._load_reverified_results( output_fpath: pathlib.Path, identity_sources: typing.Tuple[pathlib.Path, ...] = () ) -> typing.Tuple[typing.List[typing.Dict], typing.List[typing.Dict]] ``` Load the full main jsonl (cached + newly re-verified successes), sorted by (task, rollout). Returns `(results, rows)`: `results` are the parsed rows — the source of truth used for both the W\&B rollouts export and the aggregate-metrics payload; `rows` is a minimal `{agent_ref, task_source}` projection used only to route each result to its resources server (with the same resolver as /verify). Read once and reused for both so the file is never read twice. A row cached by a reverification that did not keep collection's identity stamps gets the stamps its source rollout (same task and rollout index, from `identity_sources`) carries, so a resumed output groups as a fresh one does. Only missing stamps are filled, in memory; the output file is not rewritten. A row whose source rollout has no stamp groups as that rollout does. If an unstamped row has no source rollout at all while other rows are stamped, its identity cannot be established and aggregating would split one run, so this raises `ConfigError` before anything is exported or aggregated; the output file keeps every row. ```python nemo_gym.eval.rollout_reverification._parse_output_line( line: bytes ) -> typing.Dict[str, typing.Any] ``` ```python nemo_gym.eval.rollout_reverification._parse_output_line_key( line: bytes ) -> tuple[int, int] | None ``` ```python nemo_gym.eval.rollout_reverification._parse_settle_source_snapshot( raw: str ) -> dict[str, str | None] ``` ```python nemo_gym.eval.rollout_reverification._prepare_atif_payloads( materialized_inputs_jsonl_fpath: pathlib.Path, atif_manifest_jsonl_fpath: pathlib.Path, limit: typing.Optional[int] = None ) -> typing.List[typing.Dict] ``` Build verifier payloads from an explicit ATIF-to-materialized-task manifest. ```python nemo_gym.eval.rollout_reverification._prepare_output_fpaths( output_name_prefix: str, output_jsonl_fpath: str, resume_from_cache: bool, overwrite: bool, append: bool, protected_sources: tuple[pathlib.Path, ...] = () ) -> nemo_gym.eval.rollout_reverification.OutputPaths ``` ```python nemo_gym.eval.rollout_reverification._prepare_payloads( materialized_inputs_jsonl_fpath: pathlib.Path, rollouts_jsonl_fpath: pathlib.Path, output_fpaths: nemo_gym.eval.rollout_reverification.OutputPaths, resume_from_cache: bool, limit: typing.Optional[int] = None, rollout_predicate: typing.Optional[typing.Callable[[Dict[str, Any]], bool]] = None, retry_terminal_timeouts: bool = False, selected_rollouts: typing.Optional[typing.Iterable[typing.Dict]] = None ) -> typing.List[typing.Dict] ``` ```python nemo_gym.eval.rollout_reverification._project_reverification_row( row: typing.Dict ) -> typing.Dict ``` Retain source provenance without claiming ownership of its recovery history. ```python nemo_gym.eval.rollout_reverification._recovery_rollout_predicate( skip_keys: typing.Optional[set[tuple[typing.Any, typing.Any]]] = None ) -> typing.Callable[[Dict[str, Any]], bool] ``` Build the row filter for `--judge-failed-only` recovery. Keeps only judge-class failures, skips any whose `(task, rollout)` key is in `skip_keys` (the keys already present in the output: seeded successes + rows a prior recovery already appended) so a key-in-both is never re-verified/duplicated and re-runs are idempotent — this dedup is done here, independent of the resume/cache machinery — and dedups on the key so a sidecar with multiple failure attempts for one key re-verifies it exactly once. ```python nemo_gym.eval.rollout_reverification._reject_multistage_recovery_source( rollouts_jsonl_fpath: pathlib.Path ) -> None ``` Refuse `--judge-failed-only` on multi-stage rows before any file is written. Multi-stage rows carry `stage_index`. Recovering them outside the multi-stage collection would lose stage identity and the adaptive reference set. Recovery reads the failures sidecar. ```python nemo_gym.eval.rollout_reverification._rescoring_source_answer( row: typing.Dict, run_id: str ) -> dict[str, str | int] ``` Bind a re-score to its saved answer, independently of changes to the source grade. ```python nemo_gym.eval.rollout_reverification._resources_server_exposes_tools_over_mcp( global_config_dict: typing.Union[typing.Dict[str, typing.Any], omegaconf.DictConfig], resources_server_name: str ) -> bool ``` Read the selected resources server's MCP exposure flag from Gym config. ```python nemo_gym.eval.rollout_reverification._response_has_function_calls( row: typing.Dict[str, typing.Any] ) -> bool ``` ```python nemo_gym.eval.rollout_reverification._reverifiable_saved_result( row: typing.Dict[str, typing.Any] ) -> bool ``` Whether a selected saved result is graded again from its saved answer (`reverification_cause`). ```python nemo_gym.eval.rollout_reverification._rollout_response( rollout: typing.Dict[str, typing.Any] ) -> typing.Any ``` Return the response a rollout's verifier scored, or explain which result type lacks one. ```python nemo_gym.eval.rollout_reverification._rollout_verify_debug_summary( row: typing.Dict[str, typing.Any], resources_server_name: str ) -> typing.Dict[str, typing.Any] ``` ```python nemo_gym.eval.rollout_reverification._rs_for_row( row: typing.Dict[str, typing.Any], agent_to_rs: typing.Dict[str, str], global_config_dict: typing.Union[typing.Dict[str, typing.Any], omegaconf.DictConfig] ) -> str ``` The resources server that verifies this row. A task_source naming a resources server is authoritative (it is the declaring instance the dataset was stamped with — no agent indirection needed). Otherwise fall back to the rollout's agent_ref via the config's agent->rs edges. ```python nemo_gym.eval.rollout_reverification._run_verification_payloads( payloads: typing.List[typing.Dict], semaphore: asyncio.Semaphore | contextlib.nullcontext[None] | None = None ) -> typing.Iterator[asyncio.Future] ``` ```python nemo_gym.eval.rollout_reverification._seed_output_with_successes( successes_fpath: pathlib.Path, output_fpath: pathlib.Path, exclude: frozenset[tuple[int, int]] = frozenset() ) -> set[tuple[int, int]] ``` Copy the already-successful rollout rows into the output file so the final aggregate covers successes + recovered rows, and return the set of (task, rollout) keys ALREADY PRESENT in the output afterward. Rows whose key is in `exclude` (saved results that are re-verified) are not seeded. ```python nemo_gym.eval.rollout_reverification._selected_atif_resources_server_routes( global_config_dict: typing.Union[typing.Dict[str, typing.Any], omegaconf.DictConfig], payloads: typing.List[typing.Dict[str, typing.Any]] ) -> typing.List[str] ``` Resolve each ATIF row through the same routing policy used by `/verify`. ```python nemo_gym.eval.rollout_reverification._selected_rescoring_rows( store: nemo_gym.rollouts.rollout_store.RolloutStore ) -> typing.Iterator[typing.Dict] ``` Read answers on demand so a bounded re-score does not materialize the whole run. ```python nemo_gym.eval.rollout_reverification._settle_what_a_failed_pass_judged( source: pathlib.Path, identities: typing.List[str], judged: typing.List[typing.Dict], binding: typing.Dict[str, typing.Any], record: typing.Callable[[Dict[str, int]], None] ) -> typing.Callable[..., bool] ``` An exit callback for a pass that raised in-process after admitting its re-judgments, run under the run lock. The outcomes the pass already received are settled as usual, then each admission still without a saved outcome is withdrawn: its row stays selected, so a later pass grades it again and recovery decides REVERIFY instead of stopping on an outcome it cannot know. At worst a /verify call that did reach its server is made again for the same saved answer. An interrupted or killed pass leaves its admissions for recovery as before. `record` gets the pass's settlement counts, so its failed receipt says what it settled and withdrew. ```python nemo_gym.eval.rollout_reverification._settled_count( output: pathlib.Path ) -> int ``` How many selected results of the run a re-judgment settled. ```python nemo_gym.eval.rollout_reverification._source_identity_stamps( fpaths: typing.Tuple[pathlib.Path, ...] ) -> typing.Tuple[typing.Dict[typing.Tuple[int, int], typing.Dict[str, typing.Any]], set[typing.Tuple[int, int]]] ``` Collection's identity stamps of each source rollout by (task, rollout), and the keys of every source rollout. Rollouts without stamps have a key but no stamps entry. ```python nemo_gym.eval.rollout_reverification._start_over_without_cache( config: nemo_gym.eval.rollout_reverification.RolloutReverificationConfig, output_fpaths: nemo_gym.eval.rollout_reverification.OutputPaths ) -> None ``` A resume that finds neither the output nor its failures sidecar starts the output anew, so the exporters, which reopened the run recorded next to it, start a new run too. Append recovery continues its output. ```python nemo_gym.eval.rollout_reverification._ungraded_reverification( result: typing.Dict[str, typing.Any] ) -> bool ``` Whether a re-verification answered without a usable grade: a failure, a mask or no finite reward. ```python nemo_gym.eval.rollout_reverification._validate_rescored_completion( result: object ) -> None ``` Keep a malformed verifier reply out of a managed source's separate scoring output. ```python nemo_gym.eval.rollout_reverification._validate_rescoring_cache( store: nemo_gym.rollouts.rollout_store.RolloutStore, output_fpaths: nemo_gym.eval.rollout_reverification.OutputPaths ) -> None ``` Refuse stale successes and failure-budget entries before cache filtering or any dispatch. ```python nemo_gym.eval.rollout_reverification._yield_inputs_and_rollouts_paired( materialized_inputs_jsonl_fpath: pathlib.Path, rollouts_jsonl_fpath: pathlib.Path, limit: typing.Optional[int] = None, rollout_predicate: typing.Optional[typing.Callable[[Dict[str, Any]], bool]] = None, selected_rollouts: typing.Optional[typing.Iterable[typing.Dict]] = None ) -> typing.Iterator[nemo_gym.eval.rollout_reverification.InputRolloutPair] ``` ```python nemo_gym.eval.rollout_reverification.bind_judge_failures( source: pathlib.Path, skip_spent: bool = False, withdraw_unresolved: bool = False ) -> typing.Dict[str, typing.Any] ``` Snapshot the source run's selected judge failures and provenance before any judge call; the caller holds run_lock. A failed re-judgment spends the run's failure budget (`attempt_counts` against the saved `max_rollout_attempts`); a rollout that has spent it, or a run without a saved cap, is refused before judging. With `skip_spent` a rollout that has spent it is left out of the binding, and the others are bound. An admitted re-judgment with no saved outcome is refused, unless `withdraw_unresolved`: the caller ruled out a live job of the run, so the pass that admitted it died, and it is withdrawn and its row bound again. The selected saved results whose grading failed for a judge or verify-transport cause, or was never recorded (`_reverifiable_saved_result`), are bound too, under `saved_results`: a failed re-verification of one saves nothing, so it spends no failure attempt. ```python nemo_gym.eval.rollout_reverification.check_saved_patch_replay( row: typing.Dict[str, typing.Any], resources_server_name: str, global_config_dict: typing.Union[typing.Dict[str, typing.Any], omegaconf.DictConfig] ) -> None ``` Refuse to send a saved patch (`saved_model_patch`) to a resources server that does not replay one. ```python nemo_gym.eval.rollout_reverification.due_judge_failures( output: pathlib.Path ) -> typing.List[str] ``` The rollouts of a managed run whose selected judge failure, or saved result to grade again (`_reverifiable_saved_result`), a re-judgment can still settle. A judge failure whose saved failure attempts are spent, a run without a manifest and a run without a saved failure cap have none. ```python nemo_gym.eval.rollout_reverification.read_settle_receipt( path: pathlib.Path ) -> typing.Dict[str, typing.Any] ``` The receipt at `path`, as `write_settle_receipt` wrote it. ```python nemo_gym.eval.rollout_reverification.rejudge_saved_judge_failures( run_config: typing.Dict[str, typing.Any], output: pathlib.Path ) -> int ``` async Re-judge the saved judge failures of the run `output` in place; return how many rollouts it settled. The saved results whose grading failed for a judge or verify-transport cause, or was never recorded, are graded again from their saved answers too (`bind_judge_failures`); only stateless verifiers grade them. A run's driver calls it at the end of collection, and so again at the end of each resumed collection, while the run's servers are up. Only the saved answers go to /verify again: no policy call, no new model attempt. It is the `gym eval reverify --judge-failed-only --settle-into-source` path: each re-judgment is admitted in the run manifest before its judge call and spends one of the rollout's saved failure attempts, so an interrupted pass is not judged twice, and a rollout that has spent them keeps its masked row. The judged rows are kept under `reverification//` beside the run, and each settled row names its admission and the failure it settles. A run with no due judge failure returns 0 and judges nothing. ```python nemo_gym.eval.rollout_reverification.settle_rejudgments( source: pathlib.Path, judged: typing.List[typing.Dict], binding: typing.Dict[str, typing.Any] ) -> int ``` Settle the bound judge failures with the results judged in this run; returns how many were added. The caller holds the run lock taken for the binding, so no other writer ran in between. The source must still match the binding. A judged identity that the binding does not hold, a changed manifest or materialized input, or a failure that is no longer the selected outcome is refused. A bound saved result replaces its grade only with a result that binds the same saved answer and attempt (`is_rejudgment_of`); one whose re-verification answered without a grade keeps its saved row, and its admission is withdrawn. Each settlement is recorded in `binding["settled_operations"]` (this pass's admission operation ids) as soon as it is fsynced, and each completed withdrawal in `binding["withdrawn_operations"]`, so a pass that raises afterwards, for example on the Store's exit-time coverage write, still counts them, each once. Settling again after such an error (the failed pass's exit callback) skips what this pass already settled or withdrew, closes a settled admission that is still open, and expects only the pass's other admissions to be open. ```python nemo_gym.eval.rollout_reverification.summarize_cache_usage( cache: nemo_gym.eval.rollout_reverification.CacheKeysByStatus, all_payloads: typing.List[typing.Dict], filtered_payloads: typing.List[typing.Dict] ) -> None ``` ```python nemo_gym.eval.rollout_reverification.validate_settle_source_snapshot( source: pathlib.Path, raw: str ) -> None ``` Recheck the queued artifact snapshot under the caller's existing run lock, before judging. ```python nemo_gym.eval.rollout_reverification.write_settle_receipt( path: pathlib.Path, state: str, source: typing.Optional[str], job_id: typing.Optional[str], settlement: typing.Optional[typing.Dict[str, typing.Any]], started_at: str, coverage: typing.Optional[typing.Dict[str, typing.Any]] = None, due_after: typing.Optional[int] = None, refreshed: typing.Optional[typing.Dict[str, bool]] = None, error: typing.Optional[str] = None ) -> typing.Dict[str, typing.Any] ``` Write the judge-only recovery job's receipt at `path` (`settle_receipt_fpath`) and return it. `state` is `completed` or `failed`. The counts are the pass's settlement record (zeros when the pass ended before it settled anything): the saved answers selected for grading again, those judged, those settled with a grade bound to the same saved answer, run and attempt, those re-verified without a grade and left as they were, and those not re-verified because their verifier is not stateless. `coverage` is the settled run's `RolloutStore.coverage()` (its counts only), `due_after` how many saved answers a later pass would still grade again, and `refreshed` whether the run's aggregate metrics and quality summary were recomputed. The receipt is written whole and moved into place, so a reader never sees a partial one. ```python nemo_gym.eval.rollout_reverification.ATIF_NO_PERSIST_FAILURE_CLASS = 'kill_shaped' ``` ```python nemo_gym.eval.rollout_reverification.ATIF_PROVENANCE_KEY = '_ng_atif_provenance' ``` ```python nemo_gym.eval.rollout_reverification.JUDGE_FAILED_FAILURE_CLASS = 'judge_failed' ``` ```python nemo_gym.eval.rollout_reverification.SETTLE_RECEIPT_VERSION = 1 ``` ```python nemo_gym.eval.rollout_reverification._COLLECTION_ONLY_KEYS = ('limit', 'output_jsonl_fpath', 'materialized_inputs_jsonl_fpath', 'resume_from_... ``` ```python nemo_gym.eval.rollout_reverification._CONFIG_BOOL_ADAPTER = TypeAdapter(bool) ``` ```python nemo_gym.eval.rollout_reverification._IDENTITY_KEYS = (ENVIRONMENT_SERVER_STAMP_KEY_NAME, NG_RESULT_TYPE_KEY) ``` ```python nemo_gym.eval.rollout_reverification._RECOVERY_TWO_SOURCES_WARNING = "WARNING: judge-failed-only recovery merges rewards from TWO sources — the succe... ``` ```python nemo_gym.eval.rollout_reverification._SAVED_PATCH_REPLAY_SERVERS = frozenset({'deepswe'}) ``` ```python nemo_gym.eval.rollout_reverification._SETTLEMENT_COUNTS = ('selected', 'judged', 'settled', 'withdrawn', 'unsupported') ``` ```python nemo_gym.eval.rollout_reverification._SOURCE_ANSWER_KEY = '_ng_reverification_source_answer' ``` ```python nemo_gym.eval.rollout_reverification._SOURCE_RUN_ID_KEY = '_ng_reverification_source_run_id' ```