nemo_gym.rollouts.health.checks

Artifact normalization and single-rollout health checks.

Module Contents

Functions

Name	Description
`_agent_steps`	Normalize canonical TrajectoryTurn records for structural checks.
`_agent_turn_hollow`	-
`_bind_policy_call_views`	Bind turn-only and turn-or-invocation references with shared indexes and resolutions.
`_call_identity`	-
`_call_locator`	-
`_call_ref_key`	-
`_call_reference_signature`	-
`_canonical_trajectory`	-
`_chain_ended_on_failure`	Whether this model’s own last call, in time, failed.
`_chat_has_usable_tool_call`	-
`_conversation_head`	How the conversation a call continues opens: its instructions, system prompt, or first message.
`_deduplicate_reference_items`	-
`_ended_on_failed_call`	Whether any model’s own last call failed.
`_is_context_overflow_rejection`	Exclude only explicit upstream context rejections, never inferred empty generations.
`_is_failed`	-
`_is_failed_attempt`	A failed model call that returned no finished response, so it has no usage to account for.
`_is_length_refusal`	An upstream refusal on length, of either kind: the engine generated nothing, so the token checks have nothing
`_is_successful`	-
`_item_has_tool_call`	-
`_item_model_call_references`	Return usable model-call references from turns or invocations.
`_model_call_missing_token_counts`	-
`_model_call_runaway_generation`	-
`_model_call_zero_completion_tokens`	-
`_model_chains`	Group calls by the model they went to.
`_nonempty`	-
`_normalized_trajectory_calls`	-
`_reasoning_nonempty`	Whether a turn’s reasoning holds anything, Anthropic Messages thinking blocks included.
`_replay_identity`	-
`_response_has_usable_output`	Whether a saved response holds something the agent can act on: answer text, a refusal or a tool call, in the
`_rollout_ended_on_failed_model_call`	Flag a model whose own last call in the rollout failed.
`_rollout_missing_agent_turns`	-
`_rollout_token_count_mismatch`	-
`_rollout_tool_call_failed`	One finding per rollout, counting the tool outputs that open with a harness tool-failure marker.
`_same_conversation`	Whether two calls continue one conversation: their requests open the same way, and no captured session
`_subject`	-
`_token_count`	-
`_tool_output_text`	-
`_trajectory_capture_mismatch`	-
`_trajectory_has_any_gap`	-
`_trajectory_has_gap`	-
`_trajectory_reference_contradictions`	-
`_transcript_tokens`	-
`_turn_ended_on_context_overflow`	Whether a turn’s own model calls end on a context-overflow rejection that another check accounts for.
`_usage_tokens`	-
`normalize_ignored_checks`	Normalize and validate check IDs supplied by library, CLI, or Hydra config.

Data

CHECK_REGISTRY

_AGENT_TOOL_CALL_TYPES

_INCOMPLETE_MODEL_CALL_GAPS

_LENGTH_LIMIT_FINISH_REASONS

_LENGTH_REFUSAL_CATEGORIES

_REFERENCE_CONTRADICTION_GAPS

_ROLLOUT_CHECKS

_ROLLOUT_SPECS

_TASK_SPECS

_TOOL_FAILURE_MARKERS

API

nemo_gym.rollouts.health.checks._agent_steps(
    trajectory: dict[str, typing.Any]
) -> list[nemo_gym.rollouts.health.types._AgentStep]

Normalize canonical TrajectoryTurn records for structural checks.

nemo_gym.rollouts.health.checks._agent_turn_hollow(
    trajectory: dict[str, typing.Any],
    bindings: nemo_gym.rollouts.health.types._CallBindings,
    subject: dict[str, int | str]
) -> list[nemo_gym.rollouts.health.types.Finding]

nemo_gym.rollouts.health.checks._bind_policy_call_views(
    trajectory: dict[str, typing.Any],
    calls: list[dict[str, typing.Any]]
) -> tuple[nemo_gym.rollouts.health.types._CallBindings, nemo_gym.rollouts.health.types._CallBindings]

Bind turn-only and turn-or-invocation references with shared indexes and resolutions.

nemo_gym.rollouts.health.checks._call_identity(
    call: dict[str, typing.Any]
) -> str | None

nemo_gym.rollouts.health.checks._call_locator(
    call: dict[str, typing.Any],
    fallback: int
) -> dict[str, int | str]

nemo_gym.rollouts.health.checks._call_ref_key(
    ref: typing.Any
) -> str | None

nemo_gym.rollouts.health.checks._call_reference_signature(
    reference: dict[str, typing.Any]
) -> tuple[str, str, str, str]

nemo_gym.rollouts.health.checks._canonical_trajectory(
    record: dict[str, typing.Any]
) -> tuple[dict[str, typing.Any] | None, str | None]

nemo_gym.rollouts.health.checks._chain_ended_on_failure(
    chain: collections.abc.Sequence[dict[str, typing.Any]]
) -> bool

Whether this model’s own last call, in time, failed.

Per model rather than across every captured call: a judge or auxiliary request landing after a failed policy call would otherwise read as the rollout recovering, and hide the failure.

Without timestamps a multi-call chain cannot be ordered — call_index is the position in the merged trajectory list, so a producer-supplied retry can precede the attempt it retried. Say nothing rather than call a recovered rollout unhealthy.

A last call the upstream rejected because the context was full is the model’s own end, scored like any other answer, once the model has answered earlier in the same conversation (_same_conversation). A success in another conversation on the same server, such as a user simulator’s or another session’s, says nothing of whether this request fit. Before any answer the request never fit the deployment (a prompt or output budget too large for it), which is a failure like any other. An unordered chain of only failures has no answer and ended on one of them, so it is flagged.

nemo_gym.rollouts.health.checks._chat_has_usable_tool_call(
    calls: typing.Any
) -> bool

nemo_gym.rollouts.health.checks._conversation_head(
    call: dict[str, typing.Any]
) -> typing.Any

How the conversation a call continues opens: its instructions, system prompt, or first message.

Calls of one conversation share it; another conversation on the same model server, such as a user simulator’s, opens differently. A call whose request was not captured has none.

nemo_gym.rollouts.health.checks._deduplicate_reference_items(
    reference_items: collections.abc.Sequence[tuple[str, dict[str, typing.Any]]]
) -> tuple[tuple[str, dict[str, typing.Any]], ...]

nemo_gym.rollouts.health.checks._ended_on_failed_call(
    calls: collections.abc.Sequence[dict[str, typing.Any]]
) -> bool

Whether any model’s own last call failed.

nemo_gym.rollouts.health.checks._is_context_overflow_rejection(
    call: dict[str, typing.Any]
) -> bool

Exclude only explicit upstream context rejections, never inferred empty generations.

nemo_gym.rollouts.health.checks._is_failed(
    call: dict[str, typing.Any]
) -> bool

nemo_gym.rollouts.health.checks._is_failed_attempt(
    call: dict[str, typing.Any]
) -> bool

A failed model call that returned no finished response, so it has no usage to account for.

Covers HTTP error replies and calls cut by a cancellation or timeout. This does not mean nothing was generated: the server may have spent tokens on a cut call, and a cut stream may have delivered part of its output first. Gym cannot tell which from the capture (the first streamed event can arrive before any token), so these calls are skipped rather than guessed at; run.stats.model_call_errors.usage_unknown counts them. Whether the rollout recovered is rollout_ended_on_failed_model_call’s question.

A failed call that did finish (it has a finish reason) keeps the token checks, as does a body capture could not parse, which is a capture defect rather than a failed call.

nemo_gym.rollouts.health.checks._is_length_refusal(
    call: dict[str, typing.Any]
) -> bool

An upstream refusal on length, of either kind: the engine generated nothing, so the token checks have nothing to read. Only an overflow (_is_context_overflow_rejection) can be the model’s own end.

nemo_gym.rollouts.health.checks._is_successful(
    call: dict[str, typing.Any]
) -> bool

nemo_gym.rollouts.health.checks._item_has_tool_call(
    item: typing.Any
) -> bool

nemo_gym.rollouts.health.checks._item_model_call_references(
    items: typing.Any
) -> tuple[tuple[str, dict[str, typing.Any]], ...]

Return usable model-call references from turns or invocations.

nemo_gym.rollouts.health.checks._model_call_missing_token_counts(
    bindings: nemo_gym.rollouts.health.types._CallBindings,
    subject: dict[str, int | str]
) -> list[nemo_gym.rollouts.health.types.Finding]

nemo_gym.rollouts.health.checks._model_call_runaway_generation(
    bindings: nemo_gym.rollouts.health.types._CallBindings,
    subject: dict[str, int | str]
) -> list[nemo_gym.rollouts.health.types.Finding]

nemo_gym.rollouts.health.checks._model_call_zero_completion_tokens(
    bindings: nemo_gym.rollouts.health.types._CallBindings,
    subject: dict[str, int | str]
) -> list[nemo_gym.rollouts.health.types.Finding]

nemo_gym.rollouts.health.checks._model_chains(
    calls: collections.abc.Sequence[dict[str, typing.Any]]
) -> dict[typing.Any, list[dict[str, typing.Any]]]

Group calls by the model they went to.

nemo_gym.rollouts.health.checks._nonempty(
    value: typing.Any
) -> bool

nemo_gym.rollouts.health.checks._normalized_trajectory_calls(
    trajectory: dict[str, typing.Any]
) -> list[dict[str, typing.Any]]

nemo_gym.rollouts.health.checks._reasoning_nonempty(
    value: typing.Any
) -> bool

Whether a turn’s reasoning holds anything, Anthropic Messages thinking blocks included.

Kept out of _nonempty, which also decides whether a response is actionable, so that thinking never counts as an answer there.

nemo_gym.rollouts.health.checks._replay_identity(
    call: dict[str, typing.Any]
) -> str | None

nemo_gym.rollouts.health.checks._response_has_usable_output(
    response: typing.Any
) -> bool

Whether a saved response holds something the agent can act on: answer text, a refusal or a tool call, in the Responses, Chat Completions or Messages shape. Reasoning alone, a summary of it included, is not usable.

nemo_gym.rollouts.health.checks._rollout_ended_on_failed_model_call(
    trajectory: dict[str, typing.Any],
    subject: dict[str, int | str]
) -> list[nemo_gym.rollouts.health.types.Finding]

Flag a model whose own last call in the rollout failed.

The bound-call checks cannot see this: binding resolves a reference by (model_ref, response_id) or model_call_id, and a call that failed came back with none of them, so it is absent from matched_calls no matter what the producer claims. Reading the captured calls directly also covers the agents that publish no trajectory at all.

Judged per model and in time order — see _chain_ended_on_failure. Only the chain’s last call counts, so a failure the client retried successfully stays healthy; the signal is that the rollout ENDED on a failure, which is what makes its reward indistinguishable from a genuine zero. A last call the upstream rejected as a context-length overflow is not flagged once the model has answered earlier in the same conversation: that zero is the model’s own result.

nemo_gym.rollouts.health.checks._rollout_missing_agent_turns(
    record: dict[str, typing.Any],
    trajectory: dict[str, typing.Any],
    subject: dict[str, int | str]
) -> list[nemo_gym.rollouts.health.types.Finding]

nemo_gym.rollouts.health.checks._rollout_token_count_mismatch(
    record: dict[str, typing.Any],
    bindings: nemo_gym.rollouts.health.types._CallBindings,
    subject: dict[str, int | str]
) -> list[nemo_gym.rollouts.health.types.Finding]

nemo_gym.rollouts.health.checks._rollout_tool_call_failed(
    trajectory: dict[str, typing.Any],
    subject: dict[str, int | str]
) -> list[nemo_gym.rollouts.health.types.Finding]

One finding per rollout, counting the tool outputs that open with a harness tool-failure marker.

A marker counts only from a tool that returns it (_TOOL_FAILURE_MARKERS), and a call recorded as completed is not counted: a command that succeeded can print the same text, for example when it reads a log.

nemo_gym.rollouts.health.checks._same_conversation(
    call: dict[str, typing.Any],
    other: dict[str, typing.Any]
) -> bool

Whether two calls continue one conversation: their requests open the same way, and no captured session tells them apart. Two agents can share instructions, so a session only ever separates calls.

nemo_gym.rollouts.health.checks._subject(
    task_index: int | str,
    rollout_index: int | str | None = None
) -> dict[str, int | str]

nemo_gym.rollouts.health.checks._token_count(
    call: dict[str, typing.Any],
    key: str
) -> int

nemo_gym.rollouts.health.checks._tool_output_text(
    output: typing.Any
) -> str | None

nemo_gym.rollouts.health.checks._trajectory_capture_mismatch(
    trajectory: dict[str, typing.Any],
    bindings: nemo_gym.rollouts.health.types._CallBindings,
    subject: dict[str, int | str]
) -> list[nemo_gym.rollouts.health.types.Finding]

nemo_gym.rollouts.health.checks._trajectory_has_any_gap(
    trajectory: dict[str, typing.Any],
    codes: frozenset[str]
) -> bool

nemo_gym.rollouts.health.checks._trajectory_has_gap(
    trajectory: dict[str, typing.Any],
    code: str
) -> bool

nemo_gym.rollouts.health.checks._trajectory_reference_contradictions(
    trajectory: dict[str, typing.Any]
) -> list[dict[str, typing.Any]]

nemo_gym.rollouts.health.checks._transcript_tokens(
    record: dict[str, typing.Any]
) -> tuple[int, int, bool]

nemo_gym.rollouts.health.checks._turn_ended_on_context_overflow(
    turn: dict[str, typing.Any],
    bindings: nemo_gym.rollouts.health.types._CallBindings,
    chains: dict[typing.Any, list[dict[str, typing.Any]]]
) -> bool

Whether a turn’s own model calls end on a context-overflow rejection that another check accounts for.

Each of the turn’s references must resolve to exactly one call: a call that another turn’s reference resolved says nothing about this turn. Of the turn’s calls, the last in time decides, as in _chain_ended_on_failure, so an overflow followed by an empty retry is still hollow. When the calls cannot be ordered, every one must be an overflow.

Each such overflow must be accounted for: by a success of the same model in the same conversation, or by rollout_ended_on_failed_model_call flagging that model’s chain. An unordered chain that mixes in another conversation’s success is flagged by neither, so its empty turn stays hollow.

nemo_gym.rollouts.health.checks._usage_tokens(
    usage: typing.Any
) -> tuple[int | None, int | None]

nemo_gym.rollouts.rollout_health.normalize_ignored_checks(
    checks: collections.abc.Sequence[str] | str | None
) -> tuple[str, ...]

Normalize and validate check IDs supplied by library, CLI, or Hydra config.

nemo_gym.rollouts.health.checks.CHECK_REGISTRY: tuple[CheckSpec, ...] = (CheckSpec(id='check_execution_error', evaluation_scope=(CheckScope.ROLLOUT), su...

nemo_gym.rollouts.health.checks._AGENT_TOOL_CALL_TYPES = frozenset({'function_call', 'tool_call', 'tool_use', 'mcp_call', 'mcp_list_tools...

nemo_gym.rollouts.health.checks._INCOMPLETE_MODEL_CALL_GAPS = frozenset({'model_calls_unavailable', 'model_call_capture_incomplete', 'model_ca...

nemo_gym.rollouts.health.checks._LENGTH_LIMIT_FINISH_REASONS = frozenset({'length', 'max_output_tokens', 'max_tokens'})

nemo_gym.rollouts.health.checks._LENGTH_REFUSAL_CATEGORIES = frozenset({'context_length_exceeded', 'invalid_output_token_parameter'})

nemo_gym.rollouts.health.checks._REFERENCE_CONTRADICTION_GAPS = {'model_call_reference_unmatched': 'missing_captured_call', 'model_call_referenc...

nemo_gym.rollouts.health.checks._ROLLOUT_CHECKS: dict[str, Callable[[dict[str, Any], dict[str, Any], _CallBindings, dict[str, int | str]], list[Finding]]] = {'check_execution_error': lambda record, trajectory, bindings, subject: [], 'rec...

nemo_gym.rollouts.health.checks._ROLLOUT_SPECS = tuple(spec for spec in CHECK_REGISTRY if spec.evaluation_scope == CheckScope.ROL...

nemo_gym.rollouts.health.checks._TASK_SPECS = tuple(spec for spec in CHECK_REGISTRY if spec.evaluation_scope == CheckScope.TAS...

nemo_gym.rollouts.health.checks._TOOL_FAILURE_MARKERS: dict[str, frozenset[str]] = {'ripgrep execution failed': frozenset({'grep', 'glob'})}