nemo_gym.cli.eval

Bases: PrepareBenchmarkConfig

Prepare one benchmark and write its plan: every task once, with a stable id and a digest of its input.

The plan file’s format is versioned and printed by --schema. Benchmarks whose rows have no task_id/problem_id/instance_id are refused.

Examples:

gym eval plan --benchmark terminal_bench_2_1/terminus_2 --out plan.json
gym eval plan --schema

Bases: RunHealthConfig

The health-check settings of a run config, read for the quality summary of a run a judge-only job settled.

The provider name a top-level sandbox block carries, read from its keys alone (never its values).

Apply the same completion opt-in after successful or interrupted collection.

Distinguish a valid partial run from successful CLI completion.

Remove the collated split metrics an earlier run left under <rollouts stem>/preprocessed_datasets/ (<split>_metrics.json and the <split>_metrics_conflict.json of a refused run), right before a fresh start dispatches the collector.

e2e_rollout_collection collates the rows it dispatches there and checks the metrics it computes against that file, refusing a mismatch (“Found conflicting aggregate metrics”) unless overwrite_metrics_conflicts is set. That protects a committed source dataset; on this path the rows are regenerated from the task folder on every command and the output folder is reused per taskset, so the file is the previous selection’s, and a user must neither see that error nor pass the flag. A resumed run keeps the file: its rows are unchanged (_resume_conflict). So does a run that reuses the prepared input (reuse_existing_data_preparation with the collated <split>.jsonl present): the processor does not run, and the metrics stay with the input they describe. split is the resolved run config’s (_collector_split); without one nothing is known to clear.

Run the built-in collection, then the saved-answer re-judge pass, before a strict run is rejected.

With require_complete, run_from_config raises IncompleteEvaluationError when completed is below expected, and a masked judge failure is not completed: the pass runs first, and the rejection stands unless the saved run is complete afterwards, by the rule check_completion applies. Whether a rejected run is retried is then read from the settled run (_settled_rejection). A non-strict run and a terminated run keep the error as raised. A custom rollout_collection_driver runs neither this pass nor this check.

A resumed run grades its saved answers again first, while its servers are up, and only then runs its policy work: a recovery that resumes a run with both re-verifies the saved answers before the missing and retryable rollouts run.

The split the resolved run config dispatches (validation unless overridden), for the run summary.

A run stopped by SIGTERM exits as a SIGTERM kill would, so it never looks successful.

Export the completed run after the fact when live finalization left it without a receipt.

A live run reopened by a later segment never certifies (finalize_live cannot vouch for rows an earlier segment sent), nor does a live delivery it could not verify. Such a run gets Gym’s post-run export, as gym eval export sends it, from a fresh exporter of the same backend and config; the caller holds the run lock and has prepared the health metrics. An exporter that never certifies a live run keeps no receipt, so it is not exported again, and an existing receipt is never replaced. A failed export is logged; it never changes scores or the exit code.

Verify final persisted coverage and health before certifying an active live exporter.

Report collected health using the CLI’s shutdown state.

Render the gym list benchmarks <name> inspect view for one benchmark.

Make Slurm’s SIGTERM (scancel, time limit, preemption) unwind like Ctrl-C, so finally blocks run.

Python’s default SIGTERM action exits without unwinding, which would skip the quality summary.

Project the existing saved-evidence health calculation under the caller’s run lock.

This is not tracking acceptance. Unknown health exports a known=0 signal, never a healthy rate. Scientific aggregate entries and scores remain unchanged; only the reserved run metadata is replaced.

Recompute the settled run’s aggregate metrics in place and rewrite its quality summary from the settled rows, as the run’s own completion does after its saved-answer pass.

A settled run’s rollouts file holds superseded outcomes (a masked row and the grade that settled it). A health check that selects the Store’s attempts itself reads it as it is; an earlier one reads it only through a selected-result projection: the Store’s selected rows are written under workdir for the check, which is bound to the run’s own artifacts, and removed after it. A step that fails is printed and reported in the receipt, never raised: the settlement stands, and the operator reruns the step the message names.

Re-judge the run’s saved judge failures while its servers are up, then recompute its metrics in place.

Collection calls it at the end of a run and of each resumed run (see rejudge_saved_judge_failures), and at the start of a resumed run, with aggregate off since its collection aggregates afterwards. The saved results to grade again (masked by a judge or verify-transport failure, or never graded) are re-verified in the same pass. A pass that cannot run, or fails, leaves the failures masked as before and never fails the collected run. complete_only recomputes the metrics only when the settled run is complete: a strict run that was rejected as incomplete publishes no score; the saved run’s completeness is then read once, after the pass, and returned (always False without complete_only).

The rollouts path tokens ask for (--output, +output_jsonl_fpath= or a --config overlay), or default when they name none.

Resolved through the same parser dispatch uses, so a quoted --output and a file overlay both count, for the path that dispatches nothing and so has no run config to resolve a collection from.

The collector’s effective configuration for tokens: resolved through the same parser and cache dispatch uses, file overlays included, so what it would resume and run is read from it rather than guessed from the command line.

The saved inputs the collector would resume, or None when it starts fresh. Its own condition: the effective resume_from_cache (flag or config file) and both the results file and the saved inputs.

Why --resume cannot go ahead: the cached rows name other tasks or digests than the rows the collector would run now (expected, after limit).

Run a re-verification; for the judge-only recovery job (settle_receipt_fpath), settle, refresh and write the receipt.

The job grades the selected saved results again through the run’s own settle path (settle_into_source: a grade replaces a saved result only when it binds the same saved answer, run and attempt, and the policy is never called), then recomputes the source run’s aggregate metrics and quality summary from the settled rows, and ends with a receipt: what was selected, judged, settled and left as it was, the run’s coverage, and whether the refresh happened. A pass that fails (the source changed while the job was queued, a verifier that cannot be reached) leaves a failed receipt naming the error and counting what it settled and withdrew before it failed: a grade it settled stays and counts as progress, the other saved answers stay as they were, and the error is raised as before.

The rows file of one agent’s run on a taskset: tasks_<agent>.jsonl next to run_config_<agent>.yaml.

Whether the saved Store selection now meets check_completion: unreadable evidence is not complete.

The rejection of a strict run as its settled state decides it, after the saved-answer pass.

Whether unfinished work remains retryable is read from the Store again, so a pass that settled rows decides the retry, not the state before it. The collection’s own progress still counts. Unreadable evidence keeps the original rejection.

The settled run’s coverage and how many saved answers a later pass would still grade again, for the receipt; None for what cannot be read.

Explicit check (not an assert: user-facing, and must survive python -O).

Fail fast when no config declares a dataset of the requested split’s type.

Data preparation silently produces nothing for such a split, so without this check the run walks the entire preparation sequence (including its success banners) and only dies later trying to read the collated split file.

Write the run summary beside rollouts (nemo_gym.data.tasks.harbor.run_summary) and say where.

Runs after collection whatever it did, so a run that stopped early still leaves a summary of what it wrote; a summary that cannot be written is logged and never displaces the run’s own outcome. expected may be a callable that reads the collector’s selected rows, so a run that did not need them before dispatch reads them here, inside the same guard. before_dispatch (in kwargs) is what a fresh start found under its output names before dispatch, so files it never got to clear are not reported as its own.

Compare baseline and candidate runs across a configured benchmark and model suite.

The environment server’s compose_overlay block in the configuration the command names (COMPOSE_OVERLAY_KEY), validated as the server validates it, or None when the block is absent or null. Prepare merges its files onto every task’s Compose document and records their images, so the document the server starts is the one prepare checked; a block that does not validate is a configuration error here.

The platform a Compose service’s platform must match at prepare: the environment server’s sandbox_provider_options.platform ({os, arch}) in the configuration the command names, else linux/amd64, exactly as the server reads it at seed (deployment_platform).

Only the platform subkey is selected and resolved. The sibling options (volumes, credentials, a ${results_dir} the server’s configuration parser defaults later) are not needed at prepare and are not evaluated; a block that is not a mapping selects no platform and the default stands, the server’s own configuration validation names the malformed block at dispatch.

Print saved progress for a run directory or, through job_artifact_dirs, a Gym job ID.

job_artifact_dirs is required: it maps a target that is not a run directory with status files to each benchmark’s artifacts directory, or to None when it names no submission. Artifacts must be visible on the current filesystem. This command reads saved progress and does not contact a scheduler or a remote host.

The provider name of the block GPU-declaring tasks and GPU sidecars run on: the environment server’s gpu_sandbox_provider in the configuration the command names (GPU_SANDBOX_PROVIDER_KEY), read like sandbox_provider_from_overrides reads the sandbox: block. None when no GPU block is set (every service is then checked against the sandbox: block) or the block does not name one provider.

shared_storage_from_overrides for the block GPU-declaring tasks and GPU sidecars run on (GPU_SANDBOX_PROVIDER_KEY); None when no GPU block is set.

The volume_sources of the block GPU-declaring tasks run on, for checking their Compose volumes at prepare.

The block is the environment server’s gpu_sandbox_provider in the configuration the command names (GPU_SANDBOX_PROVIDER_KEY). None when no GPU block is set: every task is then checked against the sandbox: block. An empty mapping when the block is set but carries no volume_sources, which is what the server resolves such a task against.

Run rollout quality verification for an existing run directory.

By default the run’s persisted limits judge it again. Each limit set explicitly on limits (a RunHealthLimits) overrides only that persisted limit, and None removes it; a limit not set keeps its persisted value.

List available benchmarks, or inspect one by name (gym list benchmarks <name>). Optionally filtered by a query (the gym search entry point).

A benchmark is a specific kind of environment, so it shares gym list environments’ columns (name, domain, description) and reads them through the same read_config_metadata helper. --search-dir adds extra roots to scan on top of the cwd and built-ins.

Whether the run waives the agent admission checks: --allow-unsupported-pairing (which the CLI translates to +allow_unsupported_pairing=true), a configuration file the command names that sets the key, or NEMO_GYM_ALLOW_UNSUPPORTED_PAIRING. The same reading as the collector’s pairing guard at dispatch (pairing_override_enabled), so the early admission check of prepare_target and the guard agree on one explicit override.

CLI command: write a benchmark’s evaluation plan.

CLI command: prepare benchmark data.

The registry credentials prepare records private images with: the sandbox: block’s images.auth (what the provider pulls those images with at seed), turned into the registry client’s prefix-keyed form (credentials_for_image_prefixes), so each image is read with the credentials of its longest matching prefix, as the provider pulls it.

Read from the --sandbox provider file and the configuration the command names, resolved against them (so ${gitlab_registry_password} comes from env.yaml as it does at seed), like volume_sources_from_overrides. Empty when no block names a provider or the block has no images.auth: prepare then reads registries anonymously, as before. A malformed block is a ValueError naming the key.

Entry point for gym eval run <target> ....

The provider name (opensandbox, docker, …) of the sandbox: block the command names.

The block comes from the --sandbox provider file (when given) and the configuration the command line names, merged as _merged_cli_config does; only its keys are read, never its values, so provider credentials are not needed for this (the registry credentials prepare does read are the block’s images.auth, registry_credentials_from_overrides). Prepare checks each Compose task’s tmpfs, ulimits, sysctls and security_opt against this provider’s capabilities. None when no block names a provider: the check then runs at seed, against the live provider.

Whether the sandbox: block’s provider configures shared storage, for checking Compose named volumes and shared logs binds at prepare: OpenSandbox’s shared_storage.host_path under the block’s one provider key, read from the --sandbox provider file and the configuration the command names. None when the block names no single provider (the adapter then decides at seed); otherwise whether the path is set, which is the one setting the server’s provider requires for every such volume.

Entry point for gym dataset validate <target>: run every task’s reference solution and score it.

gym eval run <target> --agent harbor_oracle lands here too, with its agent selection, so the oracle’s results never share a folder with a model’s.

The volume_sources of the sandbox: block the command names, for checking Compose volumes at prepare.

The block comes from the --sandbox provider file (when given) and the configuration the command line names, merged as _block_from_overrides does. None when no volume_sources is configured: prepare then reports every external volume and foreign bind mount as unmapped.