nemo_gym.rollouts.rollout_health

Bases: enum.Enum

Persisted or derived evidence required to evaluate a check.

Bases: enum.Enum

Bases: BaseModel

Stable, self-describing health-check contract.

nemo_gym.rollouts.rollout_health.CheckSpec._require_one_call_binding_view(
    reads: frozenset[nemo_gym.rollouts.health.types.CheckInput]
) -> frozenset[nemo_gym.rollouts.health.types.CheckInput]

Bases: enum.Enum

Bases: BaseModel

Evidence emitted by a check. Checks never emit verdicts.

Bases: BaseModel

Bases: BaseModel

Bases: BaseModel

The shards gym eval aggregate merged into a rollout file, in its order, each as its path, size and SHA-256, and that file’s size and SHA-256.

The rollouts that failed and never landed, and the rollouts the run owed.

The shards gym eval aggregate merged into run_dir/rollout_name, in its order; None if it merged none.

The merged rows have none of the shards’ failures or inventories, so a recheck reads the shards, as the aggregation did. A merged file that is gone, or not byte for byte the one the list was written for, is not theirs; a shard that is gone cannot be read, and one not byte for byte the one merged holds other rows. Each raises rather than check rows other than the merged ones, or without their failures.

From the row’s own step_usage (written by scicode_agent): a sub-step recorded as context_window_exceeded.

scicode_agent sets that status from a best-effort error-text match or model metadata, so it is a signal of a context-window failure, not proof that the input was clipped.

A config flag as written: a bool is known, anything else (absent, interpolated, malformed) is unknown.

Bind gate facts to saved rollout inputs, not to external captures or observation completeness.

Write the quality summary for whatever a run collected; a run that never started collection has none.

After SIGTERM, Slurm kills the job once KillWait (120 s on some clusters) is over, and the full check can take minutes on a large run. So a stopped run with landed_keys gets the provisional summary: the run accounting only, without reading the rollouts. With disabled, the summary only records that (write_disabled_summary). A check that cannot run writes nothing, so saved evidence stays as it is.

Stream a JSONL file’s objects; an inventory can be hundreds of MB.

Add the run-level check to the accounting: its coverage, and a reason and an unhealthy verdict per finding.

landed keys of one input, keyed as _read_failures keys that input read alone.

The selected failed attempt of every rollout that never landed.

A run with a manifest selects it from its run history, as its accounting does; a legacy output’s sidecar row that stands last is it.

Flag physical records of one input that claim the same logical rollout identity.

Each input numbers its rollouts on its own, as the run accounting counts them: runs merged from different environment servers can reuse task indices.

A names-only fact about the saved config snapshot: whether its three MLflow name fields are concrete.

True when all three are concrete strings, False when any is absent, null or empty, None when a value is interpolated or not a string. It is not MLFlowConfig.is_available (the token, which the snapshot may mask, is not judged) and says nothing about whether an exporter was activated, whether tracking was required or disabled, or whether anything was delivered: overrides, resumes and post-run exports can differ from the snapshot. No value is copied.

Count only observed failed attempts and explicitly completed operations; never certify completeness.

The limits the run’s earlier summary was judged by; none (no limit) when it has no readable summary.

Keep several schedulable chunks per worker without unbounded IPC batches.

Read every input’s failures sidecar and inventory.

The last attempt of a rollout stands, and a rollout that later landed is not failed. With an inventory, only owed rollouts count: an unreadable record or a sidecar row outside the inventory is not one.

The rollouts files a report was computed from, relative to the folder it is written in (run.<QUALITY_SUMMARY_SOURCES_KEY>): the reports keep one name per folder whatever the rollouts file is called, so a reader with several runs’ rollouts in one folder can tell whose they are. Only the directories are resolved, as _aggregated_shards does, so a shard keeps its own name.

The call ids of the rollout’s verified successful raw model responses selected for evaluation, and of the faulted ones among them. IDs only: no response text is kept.

Selected means bound to an agent turn of a complete call scope. A bound call that failed is a trial matter and is left out. Every other bound call must carry an explicit 2xx/3xx status, a raw response and a call id, or the population is unknown. A fault is judged from the raw output itself: a length-limited finish that is not a length refusal (truncated), or an output with nothing usable (_response_has_usable_output: empty, reasoning only or malformed). Token counts alone are never a fault.

Project managed runs from selected history; preserve raw accounting for legacy runs.

Managed missing means reserved attempts with no outcome. Never-dispatched work and intentional omissions have their own categories. Final failures include exhausted budgets as well as terminal outcomes. A stopped collector supplies its in-memory coverage so this path never reloads large rollout records on SIGTERM. In a legacy sidecar, a record_omitted_rollouts row records no result: an interrupted attempt is the rollout’s latest execution and leaves it missing, and a row never started is skipped. scored holds the landed keys whose records are all scored. It is None when the records were not read, which leaves a legacy input’s scored count unknown.

The run_excessive_failures check: one finding per group of failed rollouts that reaches its limit.

It reads only the run accounting, so a stopped run can evaluate it without the rollouts. With no limit configured, or no inventory to count expected rollouts against, it is unobserved.

Run health: failed rollouts against the configured limits, and the coverage reasons.

Collapse duplicate persisted records for task-level repeat semantics.

Failed rollouts’ rows, each with the model calls captured under its rollout id, read with the collector’s key and reader.

Collection joins captures only into a rollout that produced a result, so a row that failed without one (an episode cut by its deadline, say) has none, although the model server captured its calls. Each capture dir is listed once, never created. A row comes back as it is, its time unmeasured, when it carries its own capture (so no call counts twice), when its capture is missing or partial (a write still claimed, a failed write, an unreadable record), or when its identity or its capture cannot be read.

The run’s model-call capture directories for run_health_checks; none when capture is off or misconfigured, so a capture setting never stops the health check.

Evaluate every rollout-level check on one rollout record.

The one code path behind both the post-run report (which reads record from a JSONL line) and live emission (which passes the in-memory result, so line is None).

Recheck a run directory. The run’s own persisted limits judge it, so the verdict of the same artifacts does not change. Each limit set explicitly on limits overrides only that persisted limit and is recorded as an override; set to None it removes the limit. A limit not set keeps its persisted value.

A directory gym eval aggregate merged shards into is rechecked from those shards, with the failures and inventories beside them, as the aggregation checked them. An explicit rollout_file is checked alone.

model_call_capture_dir is the run’s model_call_capture_dir; with it, failed rollouts’ captured calls count in the timeouts section’s time split.

Normalize and validate check IDs supplied by library, CLI, or Hydra config.

Check selected completed attempts while retaining run-wide failure and timeout evidence.

Workers read selected byte ranges from the original histories, which own selection and identity. Aggregation supplies its scoring snapshots so health cannot select a later attempt while retaining older coverage counts. Standalone calls load the current histories; explicit None entries denote loose inputs. History-map keys accept the same relative paths and aliases as rollout_paths. Loose JSONL and explicit failure sidecars retain their line-by-line behavior.

Facts about how the run was configured, read from the resolved config saved beside the rollouts.

Only values the config states are reported: route_failures_to_sidecar and each agent’s skip_verification are booleans when written as booleans and None otherwise, never a default. Each top-level server entry reports its kind (responses_api_agents, resources_servers or environment_servers), its implementation key and, for an agent, the resources server it names. No values beyond these names and flags are copied.

Record the shards merged_fpath is merged from, for a recheck of its directory to read.

The merged file holds only the shards’ rows: their failure sidecars and inventories stay next to them. shards holds each shard’s path with the size and SHA-256 of what the merge read from it, and merged_contents_fpath the rows about to take merged_fpath’s place; the record names both by size and SHA-256, so it never vouches for other contents. Each shard is recorded relative to the record’s directory, so the shards and the merged file can move together, and under its own file name: a shard reached through a file link keeps the sidecars beside the link, as the aggregation read them.

Write the quality summary of a run with disable_health_check set, so a disabled check is recorded as state: disabled rather than by an absent summary. It counts and judges nothing: the counts and healthy are null.

A report written at the end of a run must not stop it, so a write error is logged and returns None.

Write the run accounting of a stopped run fast: rollouts.jsonl is not read, and the per-record checks are skipped.

landed is the (task index, rollout index) of every persisted record, as rollout collection holds it. Duplicates come from the per-record pass, so healthy is false when the accounting already has a reason and null otherwise. gym eval health-check on the run dir writes the full summary.