nemo_gym.rollouts.health.timeouts

run.stats.timeouts: which rollouts ran out of time, and where their time went.

The section is a report. It changes no row, reward, score or health verdict, and gym eval health-check recomputes it from the saved rows.

A rollout counts as timed out on one of two kinds of evidence:

typed: a TimeoutEvent in the row’s timeouts list, added by the component that enforces the limit. The operation records in ng_trajectory have no field for it, so the event goes on the row.
legacy: a field that a Gym component writes today when its own deadline fires. Each one is listed in LEGACY_TIMEOUT_SIGNALS together with the component that writes it.

How long a rollout took never makes it count as timed out.

For the timed-out rollouts, the section also splits their time between the policy model, other model servers (a judge, a user simulator) and tools, and reports the policy’s serving speed. A run that times out because the deployment is too slow can then be retuned.

Module Contents

Classes

Name	Description
`LegacySignal`	A row field that a Gym component sets when its own deadline fires; a dotted name reaches a nested field.
`RolloutTiming`	What one rollout contributes to the section, read from its final record.
`TimeSplit`	How the elapsed time of the timed-out rollouts with captured model calls divides.
`TimedOut`	-
`TimeoutEvent`	One confirmed firing of one time limit, in the `timeouts` list of the rollout’s row.
`TimeoutsSection`	The `run.stats.timeouts` section of the run health summary.
`_Span`	When a call ran. A call that reports only its duration has no start or end.

Functions

Name	Description
`_busy_seconds`	How long at least one of the calls was running, or None when a call has no timing.
`_flatten`	-
`_gap_codes`	The codes in a JSON list of gaps. A code that is not a string names no gap.
`_is_policy_call`	-
`_mapping`	A JSON object, or an empty one in place of any other value.
`_model_call_span`	-
`_number`	A JSON number, or None in place of any other value. A boolean is not a number here.
`_objects`	The objects in a JSON list. Any other value reads as an empty list, and an item that is not an object
`_percentiles`	-
`_share`	Summed busy time over summed elapsed time, or None when a rollout does not measure it.
`_time_split`	-
`_timeout_events`	The valid events in the row’s `timeouts` list, once per `event_id`, and how many entries were not
`_tokens_per_s`	End-to-end output speed of one call, so time in the server queue and in prefill counts too. A call that
`_tool_call_span`	-
`_tool_spans`	When the tool calls ran, or None when the trajectory does not measure it.
`classify_timeout`	Why a rollout’s record counts as timed out, or None: the one classifier for this section and the run timing
`export_keys`	The section’s numbers as flat exporter keys, such as `health/timeouts/by_evidence/typed`.
`export_run_health_timeouts`	Send the run’s timeouts section to the configured exporters (MLflow, W&B).
`read_rollout_timing`	Everything the section needs from one rollout’s final record.
`summarize_timeouts`	The section for a run, from the timing of every rollout’s final record.

Data

ELAPSED_S_KEY

Evidence

LEGACY_TIMEOUT_SIGNALS

POLICY_MODEL_SERVER

_PARTIAL_CAPTURE_GAPS

_TOOL_TIMING_GAPS

_TRANSCRIPT_GAPS

logger

API

class nemo_gym.rollouts.health.timeouts.LegacySignal()

Bases: NamedTuple

A row field that a Gym component sets when its own deadline fires; a dotted name reaches a nested field.

value is the field’s value on a timeout, or None for a flag that is set when nonzero. written_as is the source text in producer that writes it; the tests check that it is still there.

field

str

name

str

producer

str

value

Optional[str]

written_as

str

nemo_gym.rollouts.health.timeouts.LegacySignal.matches(
    row: collections.abc.Mapping[str, typing.Any]
) -> bool

class nemo_gym.rollouts.health.timeouts.RolloutTiming(
    failed: bool,
    timed_out: typing.Optional[nemo_gym.rollouts.health.timeouts.TimedOut],
    event_ids: tuple[str, ...],
    invalid_events: int,
    elapsed_s: typing.Optional[float],
    has_model_calls: bool,
    policy_s: typing.Optional[float],
    other_model_s: typing.Optional[float],
    tool_s: typing.Optional[float],
    active_s: typing.Optional[float],
    policy_calls: int,
    policy_tokens_per_s: tuple[float, ...]
)

Dataclass

What one rollout contributes to the section, read from its final record.

A busy time is None when the rollout does not measure it: a call without timing, or no tool timing. active_s is how long a call of any kind ran, so a stretch in which two kinds overlap counts once.

active_s

Optional[float]

elapsed_s

Optional[float]

event_ids

tuple[str, ...]

failed

bool

has_model_calls

bool

invalid_events

int

other_model_s

Optional[float]

policy_calls

int

policy_s

Optional[float]

policy_tokens_per_s

tuple[float, ...]

timed_out

Optional[TimedOut]

tool_s

Optional[float]

class nemo_gym.rollouts.health.timeouts.TimeSplit()

Bases: BaseModel

How the elapsed time of the timed-out rollouts with captured model calls divides.

A share is that kind’s busy time over the rollouts’ summed elapsed time, or None when one of the rollouts does not measure it. The kinds can overlap, as when a tool calls the policy model, so untracked is the share in which no measured call ran, not one minus the shares.

other_model

Optional[float] = None

policy_calls_per_rollout

dict[str, Optional[float]] = {'p50': None, 'p95': None}

policy_model

Optional[float] = None

policy_output_tokens_per_s

dict[str, Optional[float]] = {'p50': None, 'p5': None}

rollouts_with_captures

int = 0

tool

Optional[float] = None

untracked

Optional[float] = None

class nemo_gym.rollouts.health.timeouts.TimedOut()

Bases: NamedTuple

enforcer

Optional[str]

evidence

Evidence

limit

Optional[str]

class nemo_gym.rollouts.health.timeouts.TimeoutEvent()

Bases: BaseModel

One confirmed firing of one time limit, in the timeouts list of the rollout’s row.

An event listed twice keeps its event_id, so one firing counts once.

configured_s

Optional[float] = Field(default=None, ge=0, allow_inf_nan=False)

elapsed_s

Optional[float] = Field(default=None, ge=0, allow_inf_nan=False)

enforcer

str = Field(min_length=1)

event_id

str = Field(min_length=1)

evidence

Literal['internal', 'external']

fired_at

Optional[float] = Field(default=None, ge=0, allow_inf_nan=False)

limit

Optional[str] = Field(default=None, min_length=1)

model_config

= ConfigDict(extra='forbid')

class nemo_gym.rollouts.health.timeouts.TimeoutsSection()

Bases: BaseModel

The run.stats.timeouts section of the run health summary.

by_enforcer

dict[str, int]

by_evidence

dict[Evidence, int]

by_limit

dict[str, int]

elapsed_s

dict[str, Optional[float]]

events

int

failed_without_signal

int

invalid_events

int

rollouts

int

time_split

TimeSplit

timed_out

int

class nemo_gym.rollouts.health.timeouts._Span(
    start: typing.Optional[float],
    end: typing.Optional[float],
    duration_s: typing.Optional[float]
)

Dataclass

When a call ran. A call that reports only its duration has no start or end.

duration_s

Optional[float]

end

Optional[float]

is_timed

bool

start

Optional[float]

nemo_gym.rollouts.health.timeouts._busy_seconds(
    spans: list[nemo_gym.rollouts.health.timeouts._Span]
) -> typing.Optional[float]

How long at least one of the calls was running, or None when a call has no timing.

Overlapping calls, such as parallel subagents, count once. A call that reports only its duration cannot be placed against other calls, so it is measured alone and leaves a union with other calls unknown.

nemo_gym.rollouts.health.timeouts._flatten(
    values: collections.abc.Mapping[str, typing.Any],
    prefix: str
) -> dict[str, float]

nemo_gym.rollouts.health.timeouts._gap_codes(
    value: typing.Any
) -> set[str]

The codes in a JSON list of gaps. A code that is not a string names no gap.

nemo_gym.rollouts.health.timeouts._is_policy_call(
    call: collections.abc.Mapping[str, typing.Any],
    policy_model_server: str = POLICY_MODEL_SERVER
) -> bool

nemo_gym.rollouts.health.timeouts._mapping(
    value: typing.Any
) -> collections.abc.Mapping[str, typing.Any]

A JSON object, or an empty one in place of any other value.

nemo_gym.rollouts.health.timeouts._model_call_span(
    call: collections.abc.Mapping[str, typing.Any]
) -> nemo_gym.rollouts.health.timeouts._Span

nemo_gym.rollouts.health.timeouts._number(
    value: typing.Any
) -> typing.Optional[float]

A JSON number, or None in place of any other value. A boolean is not a number here.

nemo_gym.rollouts.health.timeouts._objects(
    value: typing.Any
) -> list[collections.abc.Mapping[str, typing.Any]]

The objects in a JSON list. Any other value reads as an empty list, and an item that is not an object is skipped.

nemo_gym.rollouts.health.timeouts._percentiles(
    values: list[float],
    percentiles: int = {}
) -> dict[str, typing.Optional[float]]

nemo_gym.rollouts.health.timeouts._share(
    busy_times: list[typing.Optional[float]],
    rollouts: list[nemo_gym.rollouts.health.timeouts.RolloutTiming]
) -> typing.Optional[float]

Summed busy time over summed elapsed time, or None when a rollout does not measure it.

A rollout’s busy time is capped at its elapsed time, so clock skew between servers cannot push a share past what the rollout took.

nemo_gym.rollouts.health.timeouts._time_split(
    timed_out: list[nemo_gym.rollouts.health.timeouts.RolloutTiming]
) -> nemo_gym.rollouts.health.timeouts.TimeSplit

nemo_gym.rollouts.health.timeouts._timeout_events(
    row: collections.abc.Mapping[str, typing.Any]
) -> tuple[list[nemo_gym.rollouts.health.timeouts.TimeoutEvent], int]

The valid events in the row’s timeouts list, once per event_id, and how many entries were not valid events. Only the row carries events: the trajectory’s operation records have no field for them, and collection drops a trajectory that adds one.

nemo_gym.rollouts.health.timeouts._tokens_per_s(
    call: collections.abc.Mapping[str, typing.Any]
) -> typing.Optional[float]

End-to-end output speed of one call, so time in the server queue and in prefill counts too. A call that produced no output is measured at zero; one without a token count or a positive latency is not measured.

nemo_gym.rollouts.health.timeouts._tool_call_span(
    call: collections.abc.Mapping[str, typing.Any]
) -> nemo_gym.rollouts.health.timeouts._Span

nemo_gym.rollouts.health.timeouts._tool_spans(
    trajectory: collections.abc.Mapping[str, typing.Any]
) -> typing.Optional[list[nemo_gym.rollouts.health.timeouts._Span]]

When the tool calls ran, or None when the trajectory does not measure it.

An empty tool_calls list is a measured zero. A missing or malformed list leaves the time unknown, and so do a tool call without timing (see _busy_seconds) and a gap saying the tool calls were not observed: tool_timing_unavailable, agent_artifact_unavailable or observation_capture_failed. An agent_transcript_unavailable gap leaves an empty list unknown too, but not tools read from another source.

nemo_gym.rollouts.health.timeouts.classify_timeout(
    row: collections.abc.Mapping[str, typing.Any]
) -> typing.Optional[nemo_gym.rollouts.health.timeouts.TimedOut]

Why a rollout’s record counts as timed out, or None: the one classifier for this section and the run timing table. A typed event wins over a legacy signal; of several events, the first one names the limit.

nemo_gym.rollouts.health.timeouts.export_keys(
    section: collections.abc.Mapping[str, typing.Any]
) -> dict[str, float]

The section’s numbers as flat exporter keys, such as health/timeouts/by_evidence/typed.

A null value is left out rather than exported as 0, so an unknown stays unknown on the tracking server.

nemo_gym.rollouts.health.timeouts.export_run_health_timeouts(
    health: nemo_gym.rollouts.health.types.HealthCheckResult
) -> None

Send the run’s timeouts section to the configured exporters (MLflow, W&B).

gym eval run and gym eval aggregate call this wherever they compute the run’s health. A failure, in reading the section or in exporting it, is logged and never stops the run.

nemo_gym.rollouts.health.timeouts.read_rollout_timing(
    row: collections.abc.Mapping[str, typing.Any]
) -> nemo_gym.rollouts.health.timeouts.RolloutTiming

Everything the section needs from one rollout’s final record.

A malformed record never raises: a value of the wrong type reads as not measured, and the record checks report the record itself. A capture read only in part, or with a call record that is not an object, does not measure the model time, so its calls are left out.

nemo_gym.rollouts.health.timeouts.summarize_timeouts(
    timings: collections.abc.Iterable[nemo_gym.rollouts.health.timeouts.RolloutTiming],
    attempts: collections.abc.Iterable[nemo_gym.rollouts.health.timeouts.RolloutTiming] = ()
) -> nemo_gym.rollouts.health.timeouts.TimeoutsSection

The section for a run, from the timing of every rollout’s final record.

A limit or enforcer named only by attempts, the run’s earlier failed attempts, is counted 0, as the failure export counts each failure class the sidecars name: a reopened tracking run keeps a counter’s last value until it is sent.

nemo_gym.rollouts.health.timeouts.ELAPSED_S_KEY = '_ng_elapsed_s'

nemo_gym.rollouts.health.timeouts.Evidence = Literal['typed', 'legacy']

nemo_gym.rollouts.health.timeouts.LEGACY_TIMEOUT_SIGNALS = (LegacySignal('_ng_failure_class', 'agent_timeout', 'resources_servers/gdpval/ap...

nemo_gym.rollouts.health.timeouts.POLICY_MODEL_SERVER = 'policy_model'

nemo_gym.rollouts.health.timeouts._PARTIAL_CAPTURE_GAPS = frozenset({'model_call_capture_records_unreadable', 'model_call_capture_pending'...

nemo_gym.rollouts.health.timeouts._TOOL_TIMING_GAPS = frozenset({'tool_timing_unavailable', 'agent_artifact_unavailable', 'observation...

nemo_gym.rollouts.health.timeouts._TRANSCRIPT_GAPS = frozenset({'agent_transcript_unavailable'})

nemo_gym.rollouts.health.timeouts.logger = logging.getLogger(__name__)