> This page is for version Main (default).
> For other versions, use one of these documentation indexes:
> - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt
> - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt
> - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt
> - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt
> - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt
> - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt
> - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt
> - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server.

# nemo_gym.rollouts.rollout_perf

Per-rollout performance numbers the collector attaches to each row, and the dispatch latency tracker.

## Module Contents

### Classes

| Name                                                                               | Description                                                                |
| ---------------------------------------------------------------------------------- | -------------------------------------------------------------------------- |
| [`DispatchLatencyTracker`](#nemo_gym-rollouts-rollout_perf-DispatchLatencyTracker) | Per-task latency observed by the dispatcher, and the drain margin from it. |

### Functions

| Name                                                                 | Description                                                      |
| -------------------------------------------------------------------- | ---------------------------------------------------------------- |
| [`_attach_ng_perf`](#nemo_gym-rollouts-rollout_perf-_attach_ng_perf) | -                                                                |
| [`_build_ng_perf`](#nemo_gym-rollouts-rollout_perf-_build_ng_perf)   | Assemble the per-rollout `ng_perf` summary from `ng_trajectory`. |

### Data

[`NG_PERF_KEY`](#nemo_gym-rollouts-rollout_perf-NG_PERF_KEY)

[`logger`](#nemo_gym-rollouts-rollout_perf-logger)

### API

```python
class nemo_gym.rollouts.rollout_perf.DispatchLatencyTracker()
```

Per-task latency observed by the dispatcher, and the drain margin from it.

Rollouts/hr is the number operators watch, and on its own it is misleading:
raising concurrency raises aggregate throughput while making every individual
task slower, so the run looks healthier right up until tasks start breaching
their per-task timeout. Reporting the latency percentiles next to the rate
makes that trade visible while there is still time to react to it.

**`_drained`** `= 0`

---

**`_durations`** `List[float] = []`

---

**`drained`** `int`

---

```python
nemo_gym.rollouts.rollout_perf.DispatchLatencyTracker.drain_margin(
    configured: typing.Optional[float]
) -> typing.Optional[float]
```

Seconds of headroom a task needs before it is worth starting.

An explicit value wins. Otherwise adapt to this run's own p75, once
enough tasks have finished for that to mean anything.

```python
nemo_gym.rollouts.rollout_perf.DispatchLatencyTracker.quantile(
    q: float
) -> typing.Optional[float]
```

```python
nemo_gym.rollouts.rollout_perf.DispatchLatencyTracker.record(
    seconds: float
) -> None
```

```python
nemo_gym.rollouts.rollout_perf.DispatchLatencyTracker.record_drained() -> None
```

```python
nemo_gym.rollouts.rollout_perf.DispatchLatencyTracker.record_failure(
    seconds: float
) -> None
```

Record a failed attempt only when it ran at least as long as the median so far.

A task that timed out or died late shows how long tasks can run, so it can raise the
drain margin. One that failed at once says nothing about duration and would drag the
p75 toward zero, so it is ignored, as is any failure before a real completion exists.

```python
nemo_gym.rollouts.rollout_perf.DispatchLatencyTracker.summary() -> str
```

```python
nemo_gym.rollouts.rollout_perf._attach_ng_perf(
    result: dict[str, typing.Any],
    observability_enabled: bool,
    rollout_latency_ms: typing.Optional[float] = None
) -> None
```

```python
nemo_gym.rollouts.rollout_perf._build_ng_perf(
    result: dict[str, typing.Any],
    rollout_latency_ms: typing.Optional[float]
) -> typing.Optional[dict[str, typing.Any]]
```

Assemble the per-rollout `ng_perf` summary from `ng_trajectory`.

Returns `None` (`ng_perf` stays absent) unless at least one reasoning turn was
observed: per-turn evidence is needed rather than just raw model-call capture,
so a rollout collected with observability disabled produces no `ng_perf` at all.

Token fields are summed over every model call referenced by a reasoning-turn
`AgentInvocation`. This includes compaction calls whenever the harness also lists
them in `AgentInvocation.model_calls`.

`num_turns` counts reasoning turns summed across all invocations (an `AgentInvocation`
is one root-agent or subagent conversation that may span many turns). Each invocation
contributes its explicit `TrajectoryTurn` count when the harness emits turn records,
falling back to its owned model-call count (one assistant response per turn), then to 1
(an invocation that ran had at least one turn) -- so hybrid trajectories where only some
invocations report turns still count every conversation.

`token_observability_coverage` reports what fraction of those turns actually resolved to a
captured call: a turn whose `ModelCallRef` was unmatched or ambiguous silently loses its
tokens from the sums below, and this is the only signal that it happened.

```python
nemo_gym.rollouts.rollout_perf.NG_PERF_KEY = 'ng_perf'
```

```python
nemo_gym.rollouts.rollout_perf.logger = logging.getLogger(__name__)
```