> This page is for version Main (default). > For other versions, use one of these documentation indexes: > - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt > - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt > - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt > - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt > - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt > - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt > - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt > - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server. # Resume an Evaluation > Reuse saved rollout results and check input and configuration identity before resuming. # Resume an Evaluation Gym resumes at completed rollout boundaries. A saved result is reused, including a valid reward of zero. An interrupted rollout starts a new attempt; Gym does not restore an agent's conversation, browser, or sandbox state. ## Resume saved results Restart the evaluation with the same input, agent, output path, and model/environment configuration, adding `+resume_from_cache=true` to the command. Gym compares the current run with the saved `_manifest.json` before reusing results. The manifest contains a run identifier, a format version, and digests of: * The source dataset. * The fully materialized tasks, including prompt and skill expansion. * The collection settings and resolved model/environment configuration that affect the tasks. The manifest does not store credentials or raw configuration values. Changing collection concurrency, reporting options, recognized credential fields, endpoint addresses, or capture directories does not invalidate saved results. This includes the shipped policy/judge/provider API-key fields and authentication headers at known server settings boundaries. Other headers, custom configuration fields, and task parameters remain part of identity; custom integrations should not assume arbitrary credential names are automatically excluded. Gym resolves configuration for the materialized agents, their selected Environment Servers, and referenced servers; unused server configurations need not resolve in the collector. Changing a model, sampling settings, task data paths, prompts, tasks, or Environment Server orchestration does invalidate saved results. Memory retention, resident-task limits, dispatch budgets, longest-first scheduling, and explicit retry controls can change on resume without changing task identity. Keep the manifest, materialized inputs, rollout output, and failure sidecar together. Before dispatch, Gym atomically reserves the next execution number in the manifest. It then appends and flushes each completed result or reported failure to its JSONL file. Intentional omissions are saved in the sidecar with `_ng_omitted=true`. There is no separate dispatch journal or outcome event to reconcile. These guarantees cover collector process failure; they do not promise survival of a machine or storage failure before buffered outcomes reach durable storage. The collector acquires an exclusive run lock before preparation or resume validation and holds it through reporting. A second collector targeting the same output is rejected. The operating system releases the lock after process death; an old lock file can remain safely. The file records the holder's process ID, host and Slurm job ID (when available), which a refused collector includes in its error. The filesystem must support `flock` across every writer node. Lustre `noflock` mounts do not support this lock; `localflock` mounts protect only one node and are insufficient for collectors on different nodes. For parallel jobs, use separate output paths and aggregate afterward. An infrastructure failure is eligible for another attempt according to the configured attempt limit and terminal markers. A completed wrong answer is a result, so it is not retried merely because its reward is zero. ## Environment Server boundary An Environment Server owns one episode attempt: preparing resources, running the agent, verifying its answer, and cleaning up that attempt. The collector owns the saved run history and decides which logical rollouts need another attempt after a restart. Resume sends a new episode attempt through the same Environment Server; it does not restore in-memory environment state or guarantee that cleanup ran after a hard process kill. For example, if two of three episodes finish and the third reports a retryable setup failure, resume keeps the first two results and sends only the third task again with a larger attempt index. A terminal failure is recorded but not retried. Native episode replies must match the dispatched task and attempt identities. Native Environment Servers may return completed results without a `reward` or model `response`. Such results remain saved and reusable; no zero reward is invented. Results without a numeric reward are reported as `unscored`. Older producers' `execute_only=true` results are also completed and reusable without a reward; enabling failure routing does not turn them into failures. `require_complete` checks completed episodes, so an unscored completion satisfies that check without becoming a score. Native episode failures preserve the producer's `failure_kind`, including an unspecified value. The generic `environment_server_failed` label is used only for compatibility routing; it is not invented as the canonical kind. The terminal flag, stage, and partial-response diagnostics are retained. Nonterminal failures can retry the whole episode. Judge-only recovery is a follow-up. Built-in `judge_failsafe` results retain their saved answers without a reward; the collector accepts their two known legacy judge markers while rejecting other reserved result keys. Malformed replies become terminal `environment_protocol_violation` failures when routing is enabled; running the whole task again cannot fix an invalid response format. A reply from a different execution attempt remains retryable because a transport retry can return a stale response. With failure routing disabled (the default), invalid replies stop collection; `+route_failures_to_sidecar=true` records them and lets independent tasks continue. Fix the producer before retrying a task whose response violates the contract. Environment replies establish delivery; gateway/throttling errors (429, 502, 503, 504) leave delivery uncertain because they do not prove the Environment Server received the request. ## Attempts and completion coverage A logical rollout retains its task/rollout identity across restarts. Each execution receives a larger attempt index, reserved by the controller in the run manifest. An execution number is separate from the failure retry budget. `NEMO_GYM_MAX_ROLLOUT_ATTEMPTS` sets the counted-failure limit when a run starts (default: 3). Gym saves that limit as `max_rollout_attempts` in the run manifest. Resume, health reports, and offline aggregation reuse it even when their process has a different environment setting. Older draft manifests without the field use the current setting and record it on their first validated resume. Only a recorded, nonterminal failure whose request was delivered or possibly delivered consumes that budget. A collector interruption, a checkpoint continuation, an admission timeout, a `kill_shaped` interruption, or a failure known to occur before sending does not consume it. The shared record supports `delivery: "not_sent"`; current collector transport failures conservatively use `possibly_delivered`. Terminal failures and intentional omissions stop retrying independently of the budget. The explicit terminal-timeout override makes those otherwise terminal failures eligible and counted under its existing opt-in policy. When the cap is reached, Gym reports the exhausted count and failure sidecar path. Changing the environment variable does not change an existing run. An explicit `gym eval resubmit ` also gives retryable failures a fresh budget. Completed results and terminal failures remain selected, all history is kept, and execution numbers keep increasing. The collector records the reset in `_resubmit_offset` beside the output; keep this companion with the run. Ordinary automatic restarts reuse that boundary and do not reset the budget again. Resubmission still validates that the tasks and model configuration match the saved experiment; it does not enable unsafe resume. Coverage includes `counted_failures`, `failure_budget_exhausted`, and `max_rollout_attempts`. Its `attempts` field counts reserved execution numbers, including reservations made immediately before an interruption; it is not the retry-budget counter. For example, three interruptions can reserve executions 0, 1, and 2 while consuming zero failures. The next resume can still start execution 3 with the original failure budget. Explicit `_ng_rollout_id` values must be valid capture IDs. Preprocessing preserves them for single copies and adds `-r` when repeats or agent fan-out expand a row. Direct preprocessing callers should use the returned IDs and capture keys rather than assuming the original ID survives expansion. The resulting IDs must be unique within the run and follow `EpisodeId` rules; the `-a` suffix for positive attempt numbers is reserved. Invalid or colliding IDs are configuration errors before dispatch. Tasks drained before dispatch by `dispatch_budget_s` consume no attempt and remain available on resume. The opt-in `retry_terminal_timeouts` and `dispatch_longest_first` controls also apply to manifest-backed runs. Incomplete final writes are repaired after history validation; conflicting or foreign payloads still fail validation. Core collection retries unfinished tasks from their inputs. GDPVal manages its own cached-deliverable reuse; core invalid-judge migration options are no longer supported. Managed runs save `retry_terminal_timeouts` in the run manifest. Resume keeps that policy unless the option is explicitly set; changing it updates the manifest before dispatch. Offline aggregation and health checks read the saved policy, so they agree with collection without repeating the option. Older manifests without this field default to `false`; resume with an explicit override to update them. Coverage records the policy and saved `max_rollout_attempts` alongside `failure_budget_exhausted`. These settings do not change selected scores. Reported `kill_shaped` failures are persisted in the failure sidecar, even when the legacy producer also sets `_ng_no_persist`. Nonterminal `kill_shaped` failures consume no failure budget and do not count as progress for automatic restarts. Older collectors discarded these failures; retaining them preserves diagnostic evidence. A suppression without a reported failure is an intentional omission. Existing legacy reverification keeps its previous suppression behavior. The **newest reserved execution number** determines the current outcome. A late result from an older execution remains in the append-only artifacts but cannot replace the newer attempt or count as an extra sample. If the newest execution has no saved outcome, the rollout remains unfinished and can restart from input without spending a failure retry. Collection and offline aggregation share this rule, recorded as `selection_policy: "latest_allocated"`. Gym writes `_coverage.json` with an exact inventory: | Field | Meaning | | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | `expected` | Number of logical rollouts in the materialized inputs. | | `successful` | Rollouts with a completed result, including wrong answers and completed masked results. | | `measured` | Completed results with a numeric reward whose producer did not set `mask_sample`, including valid reward-zero results. | | `masked` | Completed results with a numeric reward whose producer set `mask_sample`. These remain saved and reusable on resume. | | `unscored` | Completed results without a numeric reward, including those also marked `mask_sample`. | | `failed` | Rollouts whose selected attempt reported a no-result failure. | | `failed_final` / `failed_not_final` | Selected failures with no retry available (terminal or budget exhausted), versus selected failures still eligible for retry. | | `failed_by_class` | Selected failure counts by failure class; excludes omissions. | | `intentionally_omitted` | Explicit producer omissions, including terminal `skipped` outcomes. | | `not_started` | Rollouts with no reserved execution. | | `missing` | Rollouts whose newest reserved execution has no saved outcome. | | `retryable` | Rollouts eligible for another attempt under the active retry policy. Aggregate reports expose this per shard. | | `outcomes_recorded` | All outcomes saved during this collector invocation, including interruptions. | | `progress_outcomes_recorded` | Saved successes, omissions, terminal failures or counted failures; distinguishes productive continuation from infrastructure-only retries. | The existing `successful` field remains the completed-result total. The measurement split is additive: ```text successful = measured + masked + unscored expected = measured + masked + unscored + failed + intentionally_omitted + not_started + missing ``` Counts describe the selected attempt for each logical rollout. Older attempts cannot count again or change whether the selected result is masked. Failure metadata on a completed result does not by itself make it a no-result failure. `complete` requires a completed result for every expected rollout; `reconciled` means no reserved execution is missing an outcome. Tasks not yet started do not prevent reconciliation. A fully masked run can be complete while containing no unmasked measurements. Masking alone does not trigger recovery. Completed-result quality scoring follows the verifier/aggregation contract; these counts describe collection coverage. `scored` counts completed results with a numeric reward, plus failures explicitly counted as zero when aggregating. Unscored completions do not enter this count. It is separate from the measurement split. Setting `count_failure_classes_as_zero` does not change a failure into a measured or completed result and cannot make the outcome counts exceed `expected`. An explicit `skipped` entry in `count_failure_classes_as_zero` includes terminal skipped payloads in scoring while their completion disposition remains intentionally omitted. Omissions without a failure payload cannot be scored this way. Collection and offline aggregation export the same counts as `coverage/measured`, `coverage/masked`, `coverage/unscored`, `coverage/failed`, `coverage/omitted`, `coverage/not_started`, and `coverage/missing_outcomes`. For runs with a manifest, `coverage/missing` now means reserved executions with no outcome, matching `missing` in the coverage and health reports. Use `coverage/not_scored` for expected minus scored. This changes the meaning of an existing dashboard metric. For legacy shards without validated history, observed measured/masked counts remain available while the inventory categories cannot be determined. `coverage/failed/` counts selected failures; omissions are excluded. `retries/at_cap` equals `failure_budget_exhausted`, so interruptions and failures before an explicit budget reset do not exhaust the run. `retries/max_attempts` reports the saved limit and is omitted when aggregated runs have different limits. `retries/attempts` remains a historical count of saved failed attempts, including interruption records; it is distinct from the current budget's `counted_failures`. Legacy runs without a manifest retain raw-sidecar accounting and their per-row `_ng_failure_max_attempts` limits. The coverage file is a snapshot written at collection start and cleanup. After an abrupt process kill it may be stale; resume or aggregate to reconstruct it from the manifest and saved outcomes. An incomplete final JSONL write is ignored during reconciliation and its partial bytes are removed before appending. Malformed complete records remain errors. An old materialized inventory alone does not establish validated completion coverage during aggregation. Merged rollout files are selected-result projections; keep the original shard artifacts for further recovery. Automatic health reporting stays inside the collector's run lock. Identity validation must succeed before old reports are removed; a rejected resume leaves them untouched. For runs with a manifest, health accounting uses the same selected history as coverage and exporters. Intentional omissions do not trip failure-rate limits, and exhausted failures count as final. Every newly collected run has recovery history, including its first collection. Health checks work when all rows in the file being checked still represent selected attempts. A superseded failure in the separate failure sidecar does not prevent checking successful retries in the main file. If the checked file contains superseded outcomes, automatic checks print a skip message and standalone checks explain how to export a merged selected-result projection with `gym eval aggregate`. Collection and scoring still finish. Direct health reports for those histories remain a follow-up; they must not count old attempts as extra samples. Known runtime connection references (for example `${policy_base_url}` and `${policy_api_key}` inside agent client settings) do not change experiment identity when their values change. Model names and task parameters still do. Arbitrary nested task fields named `port` or `api_key` are retained in the identity. ### Legacy artifacts and explicit overrides Older runs have no manifest. By default, Gym rejects a resume whose identity cannot be checked. To deliberately reuse legacy artifacts or accept an input/configuration mismatch, set `+allow_unsafe_resume=true` alongside resume. Gym warns and reuses the **saved materialized tasks**. The manifest records that identity was imported or overridden, and coverage reports `identity_verified=false`. Future resumes continue to require the override. Existing legacy payloads are preserved; newly written payloads carry the run identifier. Gym still rejects corrupt or unsupported manifests and materialized inputs that no longer match their saved manifest. With `--no-serve`, Gym obtains the server configuration from the running head server, so a changed model or verifier still prevents verified resume. Missing required server configurations are errors. Nested task parameters and model settings still affect experiment identity. Use a fresh output location when the intention is to evaluate changed tasks or a different model. A non-resume collection at an existing output path replaces the main output, failure sidecar, any retired draft attempt journal, manifest, materialized inputs, coverage, and aggregate metrics. Keep the previous run under a different path to retain it. Both fresh collection and resume resolve an output symlink before taking the run lock; they write the target and its companion files while preserving the symlink. If an explicitly fresh replacement is interrupted during setup, old metadata may remain beside partially replaced files. Resume refuses that inconsistent history, including with `allow_unsafe_resume=true`. Repeat the fresh command without `--resume` to finish replacing that run, or choose a new output path; this cannot recover the overwritten run. If only part of an initial cache exists, there is no manifest or journal, and both outcome files are absent or empty, Gym prints `Skipping resume_from_cache` and starts fresh from the current inputs. This covers interruption during initial preparation before the run identity was published. Non-empty saved results or failures are never erased by this fallback, including older untagged results and incomplete JSONL tails. Restore missing files or choose a new output path. If a manifest or journal exists, missing required artifacts remain an error. The unsafe override permits an explicitly unverified legacy import or a deliberate experiment-identity override. It does not recreate lost execution reservations: if a current-format manifest is missing, restore it from a backup or start at a new output path. Conflicting records, foreign run IDs, and corrupt metadata remain errors. This revision uses manifest version 2. Artifacts from earlier development revisions that used an attempt journal must be read with their original revision or retained separately before starting a new run. Gym reports this format mismatch instead of silently dropping history. Offline aggregation reads legacy append/reverify artifacts line by line when neither a manifest nor a journal exists, even if an old inventory contains repeated explicit rollout IDs. It does not automatically migrate those artifacts or claim known completion coverage. When explicitly importing a legacy run for resume, reused attempt indices are ordered by their recorded arrival, with sidecar failures read before main-file results. This compatibility rule does not relax conflict detection for manifest-backed records. ## Structured failures for direct callers `RolloutCollectionHelper.run_outcomes()` is an opt-in interface for callers that need failure information as data. Each future returns `(input_row, outcome)`, where `outcome` is either the existing completed-result dictionary or a `RolloutFailureRecord`. It associates an `EpisodeId` (logical rollout and attempt) with a required run ID, who observed the failure (`environment` or `collector`), delivery evidence, and the shared `EpisodeFailure`. The nested failure carries `failure_reason`, the terminal flag, an optional failure kind, and an optional execution stage (`admission`, `seed`, `agent`, `verification`, or `cleanup`). Unknown stages remain unset. Delivery evidence does not authorize transport replay: a timeout is `possibly_delivered`, because the server may already have received the request. Saved failure records include `schema_version: 1`; unsupported versions, unknown fields, and inconsistent nested run/attempt identities or routing aliases are rejected. An absent `failure_kind` still represents a failure; only the compatibility routing alias uses a generic source-based category. Structured failures require `failure_reason`; the former `message` name is not accepted. Legacy sidecar routing fields remain supported as described below. Pass `run_id` to associate direct calls with an existing run, overriding existing row stamps in the invocation's copies. Without it, pre-stamped rows retain their IDs and unstamped rows share a new ID for this invocation. The returned rows are copies; caller-owned input rows are not modified. ```python from nemo_gym.rollout_collection import RolloutCollectionHelper from nemo_gym.rollout_outcomes import RolloutFailureRecord async def collect_outcomes(input_rows): helper = RolloutCollectionHelper() rows = helper.preprocess_examples(input_rows) for future in helper.run_outcomes(rows): row, outcome = await future if isinstance(outcome, RolloutFailureRecord): print(outcome.episode_id.capture_key, outcome.failure.failure_kind, outcome.failure.failure_reason) else: print(outcome.get("reward", "Completed without a score")) ``` Configure and start the referenced Gym servers before calling the helper. Callers supplying explicit rollout IDs can skip preprocessing. The direct API does not save files, retry tasks, or make training decisions. Its caller owns those operations. When stopping early, await the returned iterator's `aclose()` to cancel unfinished client requests, or `aclose_and_drain()` to also retrieve completed, unconsumed outcomes. These methods wait for local client tasks, not server or sandbox cleanup. Expected HTTP, JSON, and completed-result validation errors become failure records; programming errors and cancellation still propagate. A malformed completed result must not be interpreted as a reward-zero sample. `run_examples()` retains its existing dictionary/exception behavior. A legacy agent result tagged with `_ng_failure_class` is converted explicitly at the structured API boundary; its old reward/response placeholders are not copied into the failure record. Completed masked results remain completed dictionaries. The evaluation runner validates HTTP result shape and native task/attempt identity regardless of failure routing. Agents cannot return a full collector-owned `RolloutFailureRecord` or choose its source/delivery observations. They report failures through the episode contract, HTTP errors, or supported legacy markers. Its sidecar retains the existing `_ng_failure_*` routing aliases for row-oriented consumers and stores the canonical record under `_ng_failure_record`. `_ng_failure_class` follows the canonical failure kind. If a legacy producer supplied a different routing class, the collector also saves it as `_ng_failure_legacy_class`. Live and offline `count_failure_classes_as_zero` selection matches either name and counts the row once, preserving existing explicit scoring policies. Row-oriented consumers that need the original producer class should prefer `_ng_failure_legacy_class` when present. Those aliases can be removed after the consumers and the producer migration in [#3180](https://github.com/NVIDIA-NeMo/Gym/issues/3180) adopt the new contract. When routing is disabled, exceptions and malformed failure payloads can stop the run. Well-formed explicit failures are still accepted as data. Diagnostic observations, trajectories, and captured model calls are retained in the surrounding sidecar record, outside the canonical failure; they do not turn a failure into a completed training sample. Producer-specific diagnostic fields, such as archived trace paths and grader notes, remain in the sidecar envelope. The canonical explanation comes only from shared `failure_reason` or legacy `_ng_failure_message`; core does not guess from producer-specific fields. The bounded canonical failure record stays separate from those diagnostics. Other producer responses are retained as `_ng_failure_response` for inspection, including any synthetic failure text a producer supplied. This field is diagnostic evidence only: it is neither a scored response nor an eligible saved answer for judge-only recovery. For a legacy `judge_failed` or `judge_invalid` result that includes the actual generated response, the sidecar preserves it outside the canonical failure record, without a reward. Diagnostic token and trajectory evidence is retained too. Evaluation resume restarts unfinished tasks from their inputs. To reuse a saved answer instead, `gym eval reverify --judge-failed-only` can read a manifest-backed run into a **separate output** when every saved outcome is still selected by its recovery history. It carries successful rows through unchanged and retries only saved judge failures that contain an answer; it does not update the source run's recovery state. Writing back into an established run or selecting failed answers from superseded history remains deferred to [the judge-only recovery follow-up](https://github.com/NVIDIA-NeMo/Gym/pull/3879). Ordinary reverification has the same separate-output and selected-source requirements. For ordinary re-scoring of a history containing older attempts, `gym eval aggregate` can export a selected-result projection. That projection contains completed answers, not failed answers for `--judge-failed-only` recovery. Aggregation resolves source file symlinks before locating their companion history, and deduplicates aliases of the same rollout file. Readers also accept older runs whose companions exist only beside the alias, when neither location has a manifest or journal. Competing companion sets remain an error. Rollout collection and its resume require companions beside the resolved output: back up and move an older run's files together before writing it. Legacy reverification `--resume` and `--append` may keep an unambiguous set of companions beside the alias; they do not split or migrate it. Health and reverification preserve the same checks against superseded history through aliases. Reverification protects a manifest- or journal-backed source run's supporting files and main output, even when the supplied source is its `_failures` sidecar. Protection includes aliases and the `unsafe_` output prefix. For a merged output symlink, aggregation replaces the target file and preserves the symlink. It refuses to replace an input or an established recovery history through either spelling. Copied manifest-backed runs are rejected by run identity. Legacy runs have no trustworthy run identifier: independent evaluations may share the same input inventory, so Gym does not identify them by an inventory hash. Supply each original legacy run only once; copies at different locations are treated as separate inputs, and their coverage is relative to those supplied inputs. Metrics-only aggregation (`+merge_shards=false`) writes `_aggregate_coverage.json` so it preserves the collector's `_coverage.json` even when both commands use the same output basename. When `+count_missing_rollouts_as_zero=true` is enabled, a task with no result for its newest attempt contributes a zero to metrics only. It stays unfinished in the recovery coverage and remains eligible for resume. A known failure still requires the separate `count_failure_classes_as_zero` policy, and an intentional unscored completion is not treated as missing. Aggregation reports these added zeros as `imputed`, separately from measured rewards and counted failures. ## Checkpoint-controller integration Restarting unfinished tasks from input is the scope of this change. Continuing an agent from saved execution state belongs to the checkpoint stack ([#3024](https://github.com/NVIDIA-NeMo/Gym/issues/3024)); it is not implemented here. Both paths must use the same controller rules: * The collector currently holds `run_lock(output)` internally. Do not wrap `run_from_config` in another `run_lock`: a second acquisition conflicts even in the same process. The checkpoint integration must extend the existing lock to include `checkpoint_dir` and cover validation, restore, collection and reporting in one acquisition. * Validate the saved experiment first and associate each checkpoint with the manifest's run ID. * Reserve a target with `RolloutStore.allocate_attempt(row)` before restore or a fresh request. Checkpoint code uses that target; it must not independently add one. Failed restore reserves a new target before starting from input. * Continuation and interruption do not consume the failure budget. Only saved eligible failures do. The following incomplete-run exit codes apply when `require_complete=true`, which `gym eval submit` enables by default. Local `gym eval run` and standalone collection default to `require_complete=false` and can return **0** with partial results; use `+require_complete=true` to require completion. With completion required, the CLI exits with **75** if the evaluation is incomplete, tasks remain eligible for resume, and this invocation made progress. Saved successes, omissions, terminal failures and counted failures establish progress. Admission timeouts, nonterminal `kill_shaped` replies and failures known not to have been sent do not. It exits with **1** if work remains retryable but no progress was made, including a dispatch budget that drains every task. Such work can still resume after correcting the cause; automatic restarts spend the configured infrastructure retry limit instead of continuing forever. Increase the dispatch budget or reduce the drain margin if they prevented all work. The CLI exits with **76** if unfinished work includes terminal failures or exhausted retry budgets and nothing is eligible for retry. Slurm marks these evaluations FAILED. Local commands with `require_complete=false` retain their existing partial-result behavior. Intentional omissions do not hide other failed or unfinished tasks. If at least one task completed and the only gaps are intentional omissions, the CLI exits with **0** and warns with the completed, expected and omitted counts. If every task is omitted, it exits with **76**: there are no completed tasks to report, and retrying cannot help. For example, check whether the deliverables folder is mounted if a judge-only run skips every task. After fixing the cause, use a new output path; resume preserves intentional omissions. Omissions keep their separate coverage disposition: they are not completed answers and receive no measured score. GDPVal's explicitly accepted partial-stage omissions follow the same exit policy. None of the nonzero statuses reports a complete evaluation. For external producers, a bare `_ng_no_persist` reply now means an intentional omission in the core collector. Older collection versions wrote nothing for that reply and dispatched the task again on resume. GDPVal multistage still retries bare `_ng_no_persist` replies; its behavior is unchanged. The Slurm auto-resume chain continues after 75 without spending an infrastructure retry and stops after 76. Its accumulated walltime cap applies only when `max_walltime` is configured. Collection coverage exposes eligible tasks as `retryable` and progress as `progress_outcomes_recorded`; aggregate coverage includes `retryable` within each shard. Custom collection drivers calling `check_completion` must explicitly pass `retryable` (whether another attempt can help) and `made_progress` (whether this invocation saved outcomes that establish progress as described above). Both are required; cached outcomes alone do not establish progress. Code 0 does not prove complete coverage when `require_complete=false`. An explicit Ctrl-C in `gym eval run` also retains the existing cancellation behavior: it can return 0 with partial artifacts when `require_complete=false`. To learn how many tasks actually completed, inspect the core collector's coverage file, or GDPVal's stage summaries (multistage does not write that coverage file). Ordinary errors keep their existing failure status. Both the library and CLI respect `require_complete`. Checkpoint-triggered stops must use the same incomplete statuses when that integration lands. End-to-end checkpoint restoration still requires validation with the checkpoint stack.