> This page is for version 0.5.0.
> For other versions, use one of these documentation indexes:
> - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt
> - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt
> - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt
> - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt
> - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt
> - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt
> - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt
> - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server.

# nemo_gym.rollouts.health.timeouts

`run.stats.timeouts`: which rollouts ran out of time, and where their time went.

The section is a report. It changes no row, reward, score or health verdict, and
`gym eval health-check` recomputes it from the saved rows.

A rollout counts as timed out on one of two kinds of evidence:

* `typed`: a `TimeoutEvent` in the row's `timeouts` list, added by the component that enforces
  the limit. The operation records in `ng_trajectory` have no field for it, so the event goes on the row.
* `legacy`: a field that a Gym component writes today when its own deadline fires. Each one is listed in
  `LEGACY_TIMEOUT_SIGNALS` together with the component that writes it.

How long a rollout took never makes it count as timed out.

For the timed-out rollouts, the section also splits their time between the policy model, other model
servers (a judge, a user simulator) and tools, and reports the policy's serving speed. A run that times out
because the deployment is too slow can then be retuned.

## Module Contents

### Classes

| Name                                                                    | Description                                                                                              |
| ----------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
| [`LegacySignal`](#nemo_gym-rollouts-health-timeouts-LegacySignal)       | A row field that a Gym component sets when its own deadline fires; a dotted name reaches a nested field. |
| [`RolloutTiming`](#nemo_gym-rollouts-health-timeouts-RolloutTiming)     | What one rollout contributes to the section, read from its final record.                                 |
| [`TimeSplit`](#nemo_gym-rollouts-health-timeouts-TimeSplit)             | How the elapsed time of the timed-out rollouts with captured model calls divides.                        |
| [`TimedOut`](#nemo_gym-rollouts-health-timeouts-TimedOut)               | -                                                                                                        |
| [`TimeoutEvent`](#nemo_gym-rollouts-health-timeouts-TimeoutEvent)       | One confirmed firing of one time limit, in the `timeouts` list of the rollout's row.                     |
| [`TimeoutsSection`](#nemo_gym-rollouts-health-timeouts-TimeoutsSection) | The `run.stats.timeouts` section of the run health summary.                                              |
| [`_Span`](#nemo_gym-rollouts-health-timeouts-_Span)                     | When a call ran. A call that reports only its duration has no start or end.                              |

### Functions

| Name                                                                                          | Description                                                                                                 |
| --------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
| [`_busy_seconds`](#nemo_gym-rollouts-health-timeouts-_busy_seconds)                           | How long at least one of the calls was running, or None when a call has no timing.                          |
| [`_flatten`](#nemo_gym-rollouts-health-timeouts-_flatten)                                     | -                                                                                                           |
| [`_gap_codes`](#nemo_gym-rollouts-health-timeouts-_gap_codes)                                 | The codes in a JSON list of gaps. A code that is not a string names no gap.                                 |
| [`_is_policy_call`](#nemo_gym-rollouts-health-timeouts-_is_policy_call)                       | -                                                                                                           |
| [`_mapping`](#nemo_gym-rollouts-health-timeouts-_mapping)                                     | A JSON object, or an empty one in place of any other value.                                                 |
| [`_model_call_span`](#nemo_gym-rollouts-health-timeouts-_model_call_span)                     | -                                                                                                           |
| [`_number`](#nemo_gym-rollouts-health-timeouts-_number)                                       | A JSON number, or None in place of any other value. A boolean is not a number here.                         |
| [`_objects`](#nemo_gym-rollouts-health-timeouts-_objects)                                     | The objects in a JSON list. Any other value reads as an empty list, and an item that is not an object       |
| [`_percentiles`](#nemo_gym-rollouts-health-timeouts-_percentiles)                             | -                                                                                                           |
| [`_share`](#nemo_gym-rollouts-health-timeouts-_share)                                         | Summed busy time over summed elapsed time, or None when a rollout does not measure it.                      |
| [`_time_split`](#nemo_gym-rollouts-health-timeouts-_time_split)                               | -                                                                                                           |
| [`_timeout_events`](#nemo_gym-rollouts-health-timeouts-_timeout_events)                       | The valid events in the row's `timeouts` list, once per `event_id`, and how many entries were not           |
| [`_tokens_per_s`](#nemo_gym-rollouts-health-timeouts-_tokens_per_s)                           | End-to-end output speed of one call, so time in the server queue and in prefill counts too. A call that     |
| [`_tool_call_span`](#nemo_gym-rollouts-health-timeouts-_tool_call_span)                       | -                                                                                                           |
| [`_tool_spans`](#nemo_gym-rollouts-health-timeouts-_tool_spans)                               | When the tool calls ran, or None when the trajectory does not measure it.                                   |
| [`classify_timeout`](#nemo_gym-rollouts-health-timeouts-classify_timeout)                     | Why a rollout's record counts as timed out, or None: the one classifier for this section and the run timing |
| [`export_keys`](#nemo_gym-rollouts-health-timeouts-export_keys)                               | The section's numbers as flat exporter keys, such as `health/timeouts/by_evidence/typed`.                   |
| [`export_run_health_timeouts`](#nemo_gym-rollouts-health-timeouts-export_run_health_timeouts) | Send the run's timeouts section to the configured exporters (MLflow, W\&B).                                 |
| [`read_rollout_timing`](#nemo_gym-rollouts-health-timeouts-read_rollout_timing)               | Everything the section needs from one rollout's final record.                                               |
| [`summarize_timeouts`](#nemo_gym-rollouts-health-timeouts-summarize_timeouts)                 | The section for a run, from the timing of every rollout's final record.                                     |

### Data

[`ELAPSED_S_KEY`](#nemo_gym-rollouts-health-timeouts-ELAPSED_S_KEY)

[`Evidence`](#nemo_gym-rollouts-health-timeouts-Evidence)

[`LEGACY_TIMEOUT_SIGNALS`](#nemo_gym-rollouts-health-timeouts-LEGACY_TIMEOUT_SIGNALS)

[`POLICY_MODEL_SERVER`](#nemo_gym-rollouts-health-timeouts-POLICY_MODEL_SERVER)

[`_PARTIAL_CAPTURE_GAPS`](#nemo_gym-rollouts-health-timeouts-_PARTIAL_CAPTURE_GAPS)

[`_TOOL_TIMING_GAPS`](#nemo_gym-rollouts-health-timeouts-_TOOL_TIMING_GAPS)

[`_TRANSCRIPT_GAPS`](#nemo_gym-rollouts-health-timeouts-_TRANSCRIPT_GAPS)

[`logger`](#nemo_gym-rollouts-health-timeouts-logger)

### API

```python
class nemo_gym.rollouts.health.timeouts.LegacySignal()
```

**Bases:** `NamedTuple`

A row field that a Gym component sets when its own deadline fires; a dotted name reaches a nested field.

`value` is the field's value on a timeout, or None for a flag that is set when nonzero. `written_as`
is the source text in `producer` that writes it; the tests check that it is still there.

**`field`** `str`

---

**`name`** `str`

---

**`producer`** `str`

---

**`value`** `Optional[str]`

---

**`written_as`** `str`

---

```python
nemo_gym.rollouts.health.timeouts.LegacySignal.matches(
    row: collections.abc.Mapping[str, typing.Any]
) -> bool
```

```python
class nemo_gym.rollouts.health.timeouts.RolloutTiming(
    failed: bool,
    timed_out: typing.Optional[nemo_gym.rollouts.health.timeouts.TimedOut],
    event_ids: tuple[str, ...],
    invalid_events: int,
    elapsed_s: typing.Optional[float],
    has_model_calls: bool,
    policy_s: typing.Optional[float],
    other_model_s: typing.Optional[float],
    tool_s: typing.Optional[float],
    active_s: typing.Optional[float],
    policy_calls: int,
    policy_tokens_per_s: tuple[float, ...]
)
```

Dataclass

What one rollout contributes to the section, read from its final record.

A busy time is None when the rollout does not measure it: a call without timing, or no tool timing.
`active_s` is how long a call of any kind ran, so a stretch in which two kinds overlap counts once.

**`active_s`** `Optional[float]`

---

**`elapsed_s`** `Optional[float]`

---

**`event_ids`** `tuple[str, ...]`

---

**`failed`** `bool`

---

**`has_model_calls`** `bool`

---

**`invalid_events`** `int`

---

**`other_model_s`** `Optional[float]`

---

**`policy_calls`** `int`

---

**`policy_s`** `Optional[float]`

---

**`policy_tokens_per_s`** `tuple[float, ...]`

---

**`timed_out`** `Optional[TimedOut]`

---

**`tool_s`** `Optional[float]`

---

```python
class nemo_gym.rollouts.health.timeouts.TimeSplit()
```

**Bases:** `BaseModel`

How the elapsed time of the timed-out rollouts with captured model calls divides.

A share is that kind's busy time over the rollouts' summed elapsed time, or None when one of the
rollouts does not measure it. The kinds can overlap, as when a tool calls the policy model, so
`untracked` is the share in which no measured call ran, not one minus the shares.

**`other_model`** `Optional[float] = None`

---

**`policy_calls_per_rollout`** `dict[str, Optional[float]] = {'p50': None, 'p95': None}`

---

**`policy_model`** `Optional[float] = None`

---

**`policy_output_tokens_per_s`** `dict[str, Optional[float]] = {'p50': None, 'p5': None}`

---

**`rollouts_with_captures`** `int = 0`

---

**`tool`** `Optional[float] = None`

---

**`untracked`** `Optional[float] = None`

---

```python
class nemo_gym.rollouts.health.timeouts.TimedOut()
```

**Bases:** `NamedTuple`

**`enforcer`** `Optional[str]`

---

**`evidence`** `Evidence`

---

**`limit`** `Optional[str]`

---

```python
class nemo_gym.rollouts.health.timeouts.TimeoutEvent()
```

**Bases:** `BaseModel`

One confirmed firing of one time limit, in the `timeouts` list of the rollout's row.

An event listed twice keeps its `event_id`, so one firing counts once.

**`configured_s`** `Optional[float] = Field(default=None, ge=0, allow_inf_nan=False)`

---

**`elapsed_s`** `Optional[float] = Field(default=None, ge=0, allow_inf_nan=False)`

---

**`enforcer`** `str = Field(min_length=1)`

---

**`event_id`** `str = Field(min_length=1)`

---

**`evidence`** `Literal['internal', 'external']`

---

**`fired_at`** `Optional[float] = Field(default=None, ge=0, allow_inf_nan=False)`

---

**`limit`** `Optional[str] = Field(default=None, min_length=1)`

---

**`model_config`** `= ConfigDict(extra='forbid')`

---

```python
class nemo_gym.rollouts.health.timeouts.TimeoutsSection()
```

**Bases:** `BaseModel`

The `run.stats.timeouts` section of the run health summary.

**`by_enforcer`** `dict[str, int]`

---

**`by_evidence`** `dict[Evidence, int]`

---

**`by_limit`** `dict[str, int]`

---

**`elapsed_s`** `dict[str, Optional[float]]`

---

**`events`** `int`

---

**`failed_without_signal`** `int`

---

**`invalid_events`** `int`

---

**`rollouts`** `int`

---

**`time_split`** `TimeSplit`

---

**`timed_out`** `int`

---

```python
class nemo_gym.rollouts.health.timeouts._Span(
    start: typing.Optional[float],
    end: typing.Optional[float],
    duration_s: typing.Optional[float]
)
```

Dataclass

When a call ran. A call that reports only its duration has no start or end.

**`duration_s`** `Optional[float]`

---

**`end`** `Optional[float]`

---

**`is_timed`** `bool`

---

**`start`** `Optional[float]`

---

```python
nemo_gym.rollouts.health.timeouts._busy_seconds(
    spans: list[nemo_gym.rollouts.health.timeouts._Span]
) -> typing.Optional[float]
```

How long at least one of the calls was running, or None when a call has no timing.

Overlapping calls, such as parallel subagents, count once. A call that reports only its duration cannot be
placed against other calls, so it is measured alone and leaves a union with other calls unknown.

```python
nemo_gym.rollouts.health.timeouts._flatten(
    values: collections.abc.Mapping[str, typing.Any],
    prefix: str
) -> dict[str, float]
```

```python
nemo_gym.rollouts.health.timeouts._gap_codes(
    value: typing.Any
) -> set[str]
```

The codes in a JSON list of gaps. A code that is not a string names no gap.

```python
nemo_gym.rollouts.health.timeouts._is_policy_call(
    call: collections.abc.Mapping[str, typing.Any],
    policy_model_server: str = POLICY_MODEL_SERVER
) -> bool
```

```python
nemo_gym.rollouts.health.timeouts._mapping(
    value: typing.Any
) -> collections.abc.Mapping[str, typing.Any]
```

A JSON object, or an empty one in place of any other value.

```python
nemo_gym.rollouts.health.timeouts._model_call_span(
    call: collections.abc.Mapping[str, typing.Any]
) -> nemo_gym.rollouts.health.timeouts._Span
```

```python
nemo_gym.rollouts.health.timeouts._number(
    value: typing.Any
) -> typing.Optional[float]
```

A JSON number, or None in place of any other value. A boolean is not a number here.

```python
nemo_gym.rollouts.health.timeouts._objects(
    value: typing.Any
) -> list[collections.abc.Mapping[str, typing.Any]]
```

The objects in a JSON list. Any other value reads as an empty list, and an item that is not an object
is skipped.

```python
nemo_gym.rollouts.health.timeouts._percentiles(
    values: list[float],
    percentiles: int = {}
) -> dict[str, typing.Optional[float]]
```

```python
nemo_gym.rollouts.health.timeouts._share(
    busy_times: list[typing.Optional[float]],
    rollouts: list[nemo_gym.rollouts.health.timeouts.RolloutTiming]
) -> typing.Optional[float]
```

Summed busy time over summed elapsed time, or None when a rollout does not measure it.

A rollout's busy time is capped at its elapsed time, so clock skew between servers cannot push a share
past what the rollout took.

```python
nemo_gym.rollouts.health.timeouts._time_split(
    timed_out: list[nemo_gym.rollouts.health.timeouts.RolloutTiming]
) -> nemo_gym.rollouts.health.timeouts.TimeSplit
```

```python
nemo_gym.rollouts.health.timeouts._timeout_events(
    row: collections.abc.Mapping[str, typing.Any]
) -> tuple[list[nemo_gym.rollouts.health.timeouts.TimeoutEvent], int]
```

The valid events in the row's `timeouts` list, once per `event_id`, and how many entries were not
valid events. Only the row carries events: the trajectory's operation records have no field for them, and
collection drops a trajectory that adds one.

```python
nemo_gym.rollouts.health.timeouts._tokens_per_s(
    call: collections.abc.Mapping[str, typing.Any]
) -> typing.Optional[float]
```

End-to-end output speed of one call, so time in the server queue and in prefill counts too. A call that
produced no output is measured at zero; one without a token count or a positive latency is not measured.

```python
nemo_gym.rollouts.health.timeouts._tool_call_span(
    call: collections.abc.Mapping[str, typing.Any]
) -> nemo_gym.rollouts.health.timeouts._Span
```

```python
nemo_gym.rollouts.health.timeouts._tool_spans(
    trajectory: collections.abc.Mapping[str, typing.Any]
) -> typing.Optional[list[nemo_gym.rollouts.health.timeouts._Span]]
```

When the tool calls ran, or None when the trajectory does not measure it.

An empty `tool_calls` list is a measured zero. A missing or malformed list leaves the time unknown, and so
do a tool call without timing (see `_busy_seconds`) and a gap saying the tool calls were not observed:
`tool_timing_unavailable`, `agent_artifact_unavailable` or `observation_capture_failed`. An
`agent_transcript_unavailable` gap leaves an empty list unknown too, but not tools read from another source.

```python
nemo_gym.rollouts.health.timeouts.classify_timeout(
    row: collections.abc.Mapping[str, typing.Any]
) -> typing.Optional[nemo_gym.rollouts.health.timeouts.TimedOut]
```

Why a rollout's record counts as timed out, or None: the one classifier for this section and the run timing
table. A typed event wins over a legacy signal; of several events, the first one names the limit.

```python
nemo_gym.rollouts.health.timeouts.export_keys(
    section: collections.abc.Mapping[str, typing.Any]
) -> dict[str, float]
```

The section's numbers as flat exporter keys, such as `health/timeouts/by_evidence/typed`.

A null value is left out rather than exported as 0, so an unknown stays unknown on the tracking server.

```python
nemo_gym.rollouts.health.timeouts.export_run_health_timeouts(
    health: nemo_gym.rollouts.health.types.HealthCheckResult
) -> None
```

Send the run's timeouts section to the configured exporters (MLflow, W\&B).

`gym eval run` and `gym eval aggregate` call this wherever they compute the run's health. A failure,
in reading the section or in exporting it, is logged and never stops the run.

```python
nemo_gym.rollouts.health.timeouts.read_rollout_timing(
    row: collections.abc.Mapping[str, typing.Any]
) -> nemo_gym.rollouts.health.timeouts.RolloutTiming
```

Everything the section needs from one rollout's final record.

A malformed record never raises: a value of the wrong type reads as not measured, and the record checks
report the record itself. A capture read only in part, or with a call record that is not an object, does not
measure the model time, so its calls are left out.

```python
nemo_gym.rollouts.health.timeouts.summarize_timeouts(
    timings: collections.abc.Iterable[nemo_gym.rollouts.health.timeouts.RolloutTiming],
    attempts: collections.abc.Iterable[nemo_gym.rollouts.health.timeouts.RolloutTiming] = ()
) -> nemo_gym.rollouts.health.timeouts.TimeoutsSection
```

The section for a run, from the timing of every rollout's final record.

A limit or enforcer named only by `attempts`, the run's earlier failed attempts, is counted 0, as the failure
export counts each failure class the sidecars name: a reopened tracking run keeps a counter's last value until
it is sent.

```python
nemo_gym.rollouts.health.timeouts.ELAPSED_S_KEY = '_ng_elapsed_s'
```

```python
nemo_gym.rollouts.health.timeouts.Evidence = Literal['typed', 'legacy']
```

```python
nemo_gym.rollouts.health.timeouts.LEGACY_TIMEOUT_SIGNALS = (LegacySignal('_ng_failure_class', 'agent_timeout', 'resources_servers/gdpval/ap...
```

```python
nemo_gym.rollouts.health.timeouts.POLICY_MODEL_SERVER = 'policy_model'
```

```python
nemo_gym.rollouts.health.timeouts._PARTIAL_CAPTURE_GAPS = frozenset({'model_call_capture_records_unreadable', 'model_call_capture_pending'...
```

```python
nemo_gym.rollouts.health.timeouts._TOOL_TIMING_GAPS = frozenset({'tool_timing_unavailable', 'agent_artifact_unavailable', 'observation...
```

```python
nemo_gym.rollouts.health.timeouts._TRANSCRIPT_GAPS = frozenset({'agent_transcript_unavailable'})
```

```python
nemo_gym.rollouts.health.timeouts.logger = logging.getLogger(__name__)
```