nemo_gym.rollouts.rollout_perf

Per-rollout performance numbers the collector attaches to each row, and the dispatch latency tracker.

Module Contents

Classes

Name	Description
`DispatchLatencyTracker`	Per-task latency observed by the dispatcher, and the drain margin from it.

Functions

Name	Description
`_attach_ng_perf`	-
`_build_ng_perf`	Assemble the per-rollout `ng_perf` summary from `ng_trajectory`.

Data

NG_PERF_KEY

logger

API

class nemo_gym.rollouts.rollout_perf.DispatchLatencyTracker()

Per-task latency observed by the dispatcher, and the drain margin from it.

Rollouts/hr is the number operators watch, and on its own it is misleading: raising concurrency raises aggregate throughput while making every individual task slower, so the run looks healthier right up until tasks start breaching their per-task timeout. Reporting the latency percentiles next to the rate makes that trade visible while there is still time to react to it.

_drained

= 0

_durations

List[float] = []

drained

int

nemo_gym.rollouts.rollout_perf.DispatchLatencyTracker.drain_margin(
    configured: typing.Optional[float]
) -> typing.Optional[float]

Seconds of headroom a task needs before it is worth starting.

An explicit value wins. Otherwise adapt to this run’s own p75, once enough tasks have finished for that to mean anything.

nemo_gym.rollouts.rollout_perf.DispatchLatencyTracker.quantile(
    q: float
) -> typing.Optional[float]

nemo_gym.rollouts.rollout_perf.DispatchLatencyTracker.record(
    seconds: float
) -> None

nemo_gym.rollouts.rollout_perf.DispatchLatencyTracker.record_drained() -> None

nemo_gym.rollouts.rollout_perf.DispatchLatencyTracker.record_failure(
    seconds: float
) -> None

Record a failed attempt only when it ran at least as long as the median so far.

A task that timed out or died late shows how long tasks can run, so it can raise the drain margin. One that failed at once says nothing about duration and would drag the p75 toward zero, so it is ignored, as is any failure before a real completion exists.

nemo_gym.rollouts.rollout_perf.DispatchLatencyTracker.summary() -> str

nemo_gym.rollouts.rollout_perf._attach_ng_perf(
    result: dict[str, typing.Any],
    observability_enabled: bool,
    rollout_latency_ms: typing.Optional[float] = None
) -> None

nemo_gym.rollouts.rollout_perf._build_ng_perf(
    result: dict[str, typing.Any],
    rollout_latency_ms: typing.Optional[float]
) -> typing.Optional[dict[str, typing.Any]]

Assemble the per-rollout ng_perf summary from ng_trajectory.

Returns None (ng_perf stays absent) unless at least one reasoning turn was observed: per-turn evidence is needed rather than just raw model-call capture, so a rollout collected with observability disabled produces no ng_perf at all.

Token fields are summed over every model call referenced by a reasoning-turn AgentInvocation. This includes compaction calls whenever the harness also lists them in AgentInvocation.model_calls.

num_turns counts reasoning turns summed across all invocations (an AgentInvocation is one root-agent or subagent conversation that may span many turns). Each invocation contributes its explicit TrajectoryTurn count when the harness emits turn records, falling back to its owned model-call count (one assistant response per turn), then to 1 (an invocation that ran had at least one turn) — so hybrid trajectories where only some invocations report turns still count every conversation.

token_observability_coverage reports what fraction of those turns actually resolved to a captured call: a turn whose ModelCallRef was unmatched or ambiguous silently loses its tokens from the sums below, and this is the only signal that it happened.

nemo_gym.rollouts.rollout_perf.NG_PERF_KEY = 'ng_perf'

nemo_gym.rollouts.rollout_perf.logger = logging.getLogger(__name__)