> This page is for version 0.7.0. > For other versions, use one of these documentation indexes: > - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt > - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt > - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt > - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt > - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt > - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt > - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt > - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server. # nemo_gym.rollouts.health.checks Artifact normalization and single-rollout health checks. ## Module Contents ### Functions | Name | Description | | --------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | | [`_agent_steps`](#nemo_gym-rollouts-health-checks-_agent_steps) | Normalize canonical TrajectoryTurn records for structural checks. | | [`_agent_turn_hollow`](#nemo_gym-rollouts-health-checks-_agent_turn_hollow) | - | | [`_bind_policy_call_views`](#nemo_gym-rollouts-health-checks-_bind_policy_call_views) | Bind turn-only and turn-or-invocation references with shared indexes and resolutions. | | [`_call_identity`](#nemo_gym-rollouts-health-checks-_call_identity) | - | | [`_call_locator`](#nemo_gym-rollouts-health-checks-_call_locator) | - | | [`_call_ref_key`](#nemo_gym-rollouts-health-checks-_call_ref_key) | - | | [`_call_reference_signature`](#nemo_gym-rollouts-health-checks-_call_reference_signature) | - | | [`_canonical_trajectory`](#nemo_gym-rollouts-health-checks-_canonical_trajectory) | - | | [`_chain_ended_on_failure`](#nemo_gym-rollouts-health-checks-_chain_ended_on_failure) | Whether this model's own last call, in time, failed. | | [`_chat_has_usable_tool_call`](#nemo_gym-rollouts-health-checks-_chat_has_usable_tool_call) | - | | [`_conversation_head`](#nemo_gym-rollouts-health-checks-_conversation_head) | How the conversation a call continues opens: its instructions, system prompt, or first message. | | [`_deduplicate_reference_items`](#nemo_gym-rollouts-health-checks-_deduplicate_reference_items) | - | | [`_ended_on_failed_call`](#nemo_gym-rollouts-health-checks-_ended_on_failed_call) | Whether any model's own last call failed. | | [`_is_context_overflow_rejection`](#nemo_gym-rollouts-health-checks-_is_context_overflow_rejection) | Exclude only explicit upstream context rejections, never inferred empty generations. | | [`_is_failed`](#nemo_gym-rollouts-health-checks-_is_failed) | - | | [`_is_failed_attempt`](#nemo_gym-rollouts-health-checks-_is_failed_attempt) | A failed model call that returned no finished response, so it has no usage to account for. | | [`_is_length_refusal`](#nemo_gym-rollouts-health-checks-_is_length_refusal) | An upstream refusal on length, of either kind: the engine generated nothing, so the token checks have nothing | | [`_is_successful`](#nemo_gym-rollouts-health-checks-_is_successful) | - | | [`_item_has_tool_call`](#nemo_gym-rollouts-health-checks-_item_has_tool_call) | - | | [`_item_model_call_references`](#nemo_gym-rollouts-health-checks-_item_model_call_references) | Return usable model-call references from turns or invocations. | | [`_model_call_missing_token_counts`](#nemo_gym-rollouts-health-checks-_model_call_missing_token_counts) | - | | [`_model_call_runaway_generation`](#nemo_gym-rollouts-health-checks-_model_call_runaway_generation) | - | | [`_model_call_zero_completion_tokens`](#nemo_gym-rollouts-health-checks-_model_call_zero_completion_tokens) | - | | [`_model_chains`](#nemo_gym-rollouts-health-checks-_model_chains) | Group calls by the model they went to. | | [`_nonempty`](#nemo_gym-rollouts-health-checks-_nonempty) | - | | [`_normalized_trajectory_calls`](#nemo_gym-rollouts-health-checks-_normalized_trajectory_calls) | - | | [`_reasoning_nonempty`](#nemo_gym-rollouts-health-checks-_reasoning_nonempty) | Whether a turn's reasoning holds anything, Anthropic Messages thinking blocks included. | | [`_replay_identity`](#nemo_gym-rollouts-health-checks-_replay_identity) | - | | [`_response_has_usable_output`](#nemo_gym-rollouts-health-checks-_response_has_usable_output) | Whether a saved response holds something the agent can act on: answer text, a refusal or a tool call, in the | | [`_rollout_ended_on_failed_model_call`](#nemo_gym-rollouts-health-checks-_rollout_ended_on_failed_model_call) | Flag a model whose own last call in the rollout failed. | | [`_rollout_missing_agent_turns`](#nemo_gym-rollouts-health-checks-_rollout_missing_agent_turns) | - | | [`_rollout_token_count_mismatch`](#nemo_gym-rollouts-health-checks-_rollout_token_count_mismatch) | - | | [`_rollout_tool_call_failed`](#nemo_gym-rollouts-health-checks-_rollout_tool_call_failed) | One finding per rollout, counting the tool outputs that open with a harness tool-failure marker. | | [`_same_conversation`](#nemo_gym-rollouts-health-checks-_same_conversation) | Whether two calls continue one conversation: their requests open the same way, and no captured session | | [`_subject`](#nemo_gym-rollouts-health-checks-_subject) | - | | [`_token_count`](#nemo_gym-rollouts-health-checks-_token_count) | - | | [`_tool_output_text`](#nemo_gym-rollouts-health-checks-_tool_output_text) | - | | [`_trajectory_capture_mismatch`](#nemo_gym-rollouts-health-checks-_trajectory_capture_mismatch) | - | | [`_trajectory_has_any_gap`](#nemo_gym-rollouts-health-checks-_trajectory_has_any_gap) | - | | [`_trajectory_has_gap`](#nemo_gym-rollouts-health-checks-_trajectory_has_gap) | - | | [`_trajectory_reference_contradictions`](#nemo_gym-rollouts-health-checks-_trajectory_reference_contradictions) | - | | [`_transcript_tokens`](#nemo_gym-rollouts-health-checks-_transcript_tokens) | - | | [`_turn_ended_on_context_overflow`](#nemo_gym-rollouts-health-checks-_turn_ended_on_context_overflow) | Whether a turn's own model calls end on a context-overflow rejection that another check accounts for. | | [`_usage_tokens`](#nemo_gym-rollouts-health-checks-_usage_tokens) | - | | [`normalize_ignored_checks`](#nemo_gym-rollouts-health-checks-normalize_ignored_checks) | Normalize and validate check IDs supplied by library, CLI, or Hydra config. | ### Data [`CHECK_REGISTRY`](#nemo_gym-rollouts-health-checks-CHECK_REGISTRY) [`_AGENT_TOOL_CALL_TYPES`](#nemo_gym-rollouts-health-checks-_AGENT_TOOL_CALL_TYPES) [`_INCOMPLETE_MODEL_CALL_GAPS`](#nemo_gym-rollouts-health-checks-_INCOMPLETE_MODEL_CALL_GAPS) [`_LENGTH_LIMIT_FINISH_REASONS`](#nemo_gym-rollouts-health-checks-_LENGTH_LIMIT_FINISH_REASONS) [`_LENGTH_REFUSAL_CATEGORIES`](#nemo_gym-rollouts-health-checks-_LENGTH_REFUSAL_CATEGORIES) [`_REFERENCE_CONTRADICTION_GAPS`](#nemo_gym-rollouts-health-checks-_REFERENCE_CONTRADICTION_GAPS) [`_ROLLOUT_CHECKS`](#nemo_gym-rollouts-health-checks-_ROLLOUT_CHECKS) [`_ROLLOUT_SPECS`](#nemo_gym-rollouts-health-checks-_ROLLOUT_SPECS) [`_TASK_SPECS`](#nemo_gym-rollouts-health-checks-_TASK_SPECS) [`_TOOL_FAILURE_MARKERS`](#nemo_gym-rollouts-health-checks-_TOOL_FAILURE_MARKERS) ### API ```python nemo_gym.rollouts.health.checks._agent_steps( trajectory: dict[str, typing.Any] ) -> list[nemo_gym.rollouts.health.types._AgentStep] ``` Normalize canonical TrajectoryTurn records for structural checks. ```python nemo_gym.rollouts.health.checks._agent_turn_hollow( trajectory: dict[str, typing.Any], bindings: nemo_gym.rollouts.health.types._CallBindings, subject: dict[str, int | str] ) -> list[nemo_gym.rollouts.health.types.Finding] ``` ```python nemo_gym.rollouts.health.checks._bind_policy_call_views( trajectory: dict[str, typing.Any], calls: list[dict[str, typing.Any]] ) -> tuple[nemo_gym.rollouts.health.types._CallBindings, nemo_gym.rollouts.health.types._CallBindings] ``` Bind turn-only and turn-or-invocation references with shared indexes and resolutions. ```python nemo_gym.rollouts.health.checks._call_identity( call: dict[str, typing.Any] ) -> str | None ``` ```python nemo_gym.rollouts.health.checks._call_locator( call: dict[str, typing.Any], fallback: int ) -> dict[str, int | str] ``` ```python nemo_gym.rollouts.health.checks._call_ref_key( ref: typing.Any ) -> str | None ``` ```python nemo_gym.rollouts.health.checks._call_reference_signature( reference: dict[str, typing.Any] ) -> tuple[str, str, str, str] ``` ```python nemo_gym.rollouts.health.checks._canonical_trajectory( record: dict[str, typing.Any] ) -> tuple[dict[str, typing.Any] | None, str | None] ``` ```python nemo_gym.rollouts.health.checks._chain_ended_on_failure( chain: collections.abc.Sequence[dict[str, typing.Any]] ) -> bool ``` Whether this model's own last call, in time, failed. Per model rather than across every captured call: a judge or auxiliary request landing after a failed policy call would otherwise read as the rollout recovering, and hide the failure. Without timestamps a multi-call chain cannot be ordered -- `call_index` is the position in the merged trajectory list, so a producer-supplied retry can precede the attempt it retried. Say nothing rather than call a recovered rollout unhealthy. A last call the upstream rejected because the context was full is the model's own end, scored like any other answer, once the model has answered earlier in the same conversation (`_same_conversation`). A success in another conversation on the same server, such as a user simulator's or another session's, says nothing of whether this request fit. Before any answer the request never fit the deployment (a prompt or output budget too large for it), which is a failure like any other. An unordered chain of only failures has no answer and ended on one of them, so it is flagged. ```python nemo_gym.rollouts.health.checks._chat_has_usable_tool_call( calls: typing.Any ) -> bool ``` ```python nemo_gym.rollouts.health.checks._conversation_head( call: dict[str, typing.Any] ) -> typing.Any ``` How the conversation a call continues opens: its instructions, system prompt, or first message. Calls of one conversation share it; another conversation on the same model server, such as a user simulator's, opens differently. A call whose request was not captured has none. ```python nemo_gym.rollouts.health.checks._deduplicate_reference_items( reference_items: collections.abc.Sequence[tuple[str, dict[str, typing.Any]]] ) -> tuple[tuple[str, dict[str, typing.Any]], ...] ``` ```python nemo_gym.rollouts.health.checks._ended_on_failed_call( calls: collections.abc.Sequence[dict[str, typing.Any]] ) -> bool ``` Whether any model's own last call failed. ```python nemo_gym.rollouts.health.checks._is_context_overflow_rejection( call: dict[str, typing.Any] ) -> bool ``` Exclude only explicit upstream context rejections, never inferred empty generations. ```python nemo_gym.rollouts.health.checks._is_failed( call: dict[str, typing.Any] ) -> bool ``` ```python nemo_gym.rollouts.health.checks._is_failed_attempt( call: dict[str, typing.Any] ) -> bool ``` A failed model call that returned no finished response, so it has no usage to account for. Covers HTTP error replies and calls cut by a cancellation or timeout. This does not mean nothing was generated: the server may have spent tokens on a cut call, and a cut stream may have delivered part of its output first. Gym cannot tell which from the capture (the first streamed event can arrive before any token), so these calls are skipped rather than guessed at; `run.stats.model_call_errors.usage_unknown` counts them. Whether the rollout recovered is `rollout_ended_on_failed_model_call`'s question. A failed call that did finish (it has a finish reason) keeps the token checks, as does a body capture could not parse, which is a capture defect rather than a failed call. ```python nemo_gym.rollouts.health.checks._is_length_refusal( call: dict[str, typing.Any] ) -> bool ``` An upstream refusal on length, of either kind: the engine generated nothing, so the token checks have nothing to read. Only an overflow (`_is_context_overflow_rejection`) can be the model's own end. ```python nemo_gym.rollouts.health.checks._is_successful( call: dict[str, typing.Any] ) -> bool ``` ```python nemo_gym.rollouts.health.checks._item_has_tool_call( item: typing.Any ) -> bool ``` ```python nemo_gym.rollouts.health.checks._item_model_call_references( items: typing.Any ) -> tuple[tuple[str, dict[str, typing.Any]], ...] ``` Return usable model-call references from turns or invocations. ```python nemo_gym.rollouts.health.checks._model_call_missing_token_counts( bindings: nemo_gym.rollouts.health.types._CallBindings, subject: dict[str, int | str] ) -> list[nemo_gym.rollouts.health.types.Finding] ``` ```python nemo_gym.rollouts.health.checks._model_call_runaway_generation( bindings: nemo_gym.rollouts.health.types._CallBindings, subject: dict[str, int | str] ) -> list[nemo_gym.rollouts.health.types.Finding] ``` ```python nemo_gym.rollouts.health.checks._model_call_zero_completion_tokens( bindings: nemo_gym.rollouts.health.types._CallBindings, subject: dict[str, int | str] ) -> list[nemo_gym.rollouts.health.types.Finding] ``` ```python nemo_gym.rollouts.health.checks._model_chains( calls: collections.abc.Sequence[dict[str, typing.Any]] ) -> dict[typing.Any, list[dict[str, typing.Any]]] ``` Group calls by the model they went to. ```python nemo_gym.rollouts.health.checks._nonempty( value: typing.Any ) -> bool ``` ```python nemo_gym.rollouts.health.checks._normalized_trajectory_calls( trajectory: dict[str, typing.Any] ) -> list[dict[str, typing.Any]] ``` ```python nemo_gym.rollouts.health.checks._reasoning_nonempty( value: typing.Any ) -> bool ``` Whether a turn's reasoning holds anything, Anthropic Messages thinking blocks included. Kept out of `_nonempty`, which also decides whether a response is actionable, so that thinking never counts as an answer there. ```python nemo_gym.rollouts.health.checks._replay_identity( call: dict[str, typing.Any] ) -> str | None ``` ```python nemo_gym.rollouts.health.checks._response_has_usable_output( response: typing.Any ) -> bool ``` Whether a saved response holds something the agent can act on: answer text, a refusal or a tool call, in the Responses, Chat Completions or Messages shape. Reasoning alone, a summary of it included, is not usable. ```python nemo_gym.rollouts.health.checks._rollout_ended_on_failed_model_call( trajectory: dict[str, typing.Any], subject: dict[str, int | str] ) -> list[nemo_gym.rollouts.health.types.Finding] ``` Flag a model whose own last call in the rollout failed. The bound-call checks cannot see this: binding resolves a reference by `(model_ref, response_id)` or `model_call_id`, and a call that failed came back with none of them, so it is absent from `matched_calls` no matter what the producer claims. Reading the captured calls directly also covers the agents that publish no trajectory at all. Judged per model and in time order -- see `_chain_ended_on_failure`. Only the chain's last call counts, so a failure the client retried successfully stays healthy; the signal is that the rollout ENDED on a failure, which is what makes its reward indistinguishable from a genuine zero. A last call the upstream rejected as a context-length overflow is not flagged once the model has answered earlier in the same conversation: that zero is the model's own result. ```python nemo_gym.rollouts.health.checks._rollout_missing_agent_turns( record: dict[str, typing.Any], trajectory: dict[str, typing.Any], subject: dict[str, int | str] ) -> list[nemo_gym.rollouts.health.types.Finding] ``` ```python nemo_gym.rollouts.health.checks._rollout_token_count_mismatch( record: dict[str, typing.Any], bindings: nemo_gym.rollouts.health.types._CallBindings, subject: dict[str, int | str] ) -> list[nemo_gym.rollouts.health.types.Finding] ``` ```python nemo_gym.rollouts.health.checks._rollout_tool_call_failed( trajectory: dict[str, typing.Any], subject: dict[str, int | str] ) -> list[nemo_gym.rollouts.health.types.Finding] ``` One finding per rollout, counting the tool outputs that open with a harness tool-failure marker. A marker counts only from a tool that returns it (`_TOOL_FAILURE_MARKERS`), and a call recorded as `completed` is not counted: a command that succeeded can print the same text, for example when it reads a log. ```python nemo_gym.rollouts.health.checks._same_conversation( call: dict[str, typing.Any], other: dict[str, typing.Any] ) -> bool ``` Whether two calls continue one conversation: their requests open the same way, and no captured session tells them apart. Two agents can share instructions, so a session only ever separates calls. ```python nemo_gym.rollouts.health.checks._subject( task_index: int | str, rollout_index: int | str | None = None ) -> dict[str, int | str] ``` ```python nemo_gym.rollouts.health.checks._token_count( call: dict[str, typing.Any], key: str ) -> int ``` ```python nemo_gym.rollouts.health.checks._tool_output_text( output: typing.Any ) -> str | None ``` ```python nemo_gym.rollouts.health.checks._trajectory_capture_mismatch( trajectory: dict[str, typing.Any], bindings: nemo_gym.rollouts.health.types._CallBindings, subject: dict[str, int | str] ) -> list[nemo_gym.rollouts.health.types.Finding] ``` ```python nemo_gym.rollouts.health.checks._trajectory_has_any_gap( trajectory: dict[str, typing.Any], codes: frozenset[str] ) -> bool ``` ```python nemo_gym.rollouts.health.checks._trajectory_has_gap( trajectory: dict[str, typing.Any], code: str ) -> bool ``` ```python nemo_gym.rollouts.health.checks._trajectory_reference_contradictions( trajectory: dict[str, typing.Any] ) -> list[dict[str, typing.Any]] ``` ```python nemo_gym.rollouts.health.checks._transcript_tokens( record: dict[str, typing.Any] ) -> tuple[int, int, bool] ``` ```python nemo_gym.rollouts.health.checks._turn_ended_on_context_overflow( turn: dict[str, typing.Any], bindings: nemo_gym.rollouts.health.types._CallBindings, chains: dict[typing.Any, list[dict[str, typing.Any]]] ) -> bool ``` Whether a turn's own model calls end on a context-overflow rejection that another check accounts for. Each of the turn's references must resolve to exactly one call: a call that another turn's reference resolved says nothing about this turn. Of the turn's calls, the last in time decides, as in `_chain_ended_on_failure`, so an overflow followed by an empty retry is still hollow. When the calls cannot be ordered, every one must be an overflow. Each such overflow must be accounted for: by a success of the same model in the same conversation, or by `rollout_ended_on_failed_model_call` flagging that model's chain. An unordered chain that mixes in another conversation's success is flagged by neither, so its empty turn stays hollow. ```python nemo_gym.rollouts.health.checks._usage_tokens( usage: typing.Any ) -> tuple[int | None, int | None] ``` ```python nemo_gym.rollouts.rollout_health.normalize_ignored_checks( checks: collections.abc.Sequence[str] | str | None ) -> tuple[str, ...] ``` Normalize and validate check IDs supplied by library, CLI, or Hydra config. ```python nemo_gym.rollouts.health.checks.CHECK_REGISTRY: tuple[CheckSpec, ...] = (CheckSpec(id='check_execution_error', evaluation_scope=(CheckScope.ROLLOUT), su... ``` ```python nemo_gym.rollouts.health.checks._AGENT_TOOL_CALL_TYPES = frozenset({'function_call', 'tool_call', 'tool_use', 'mcp_call', 'mcp_list_tools... ``` ```python nemo_gym.rollouts.health.checks._INCOMPLETE_MODEL_CALL_GAPS = frozenset({'model_calls_unavailable', 'model_call_capture_incomplete', 'model_ca... ``` ```python nemo_gym.rollouts.health.checks._LENGTH_LIMIT_FINISH_REASONS = frozenset({'length', 'max_output_tokens', 'max_tokens'}) ``` ```python nemo_gym.rollouts.health.checks._LENGTH_REFUSAL_CATEGORIES = frozenset({'context_length_exceeded', 'invalid_output_token_parameter'}) ``` ```python nemo_gym.rollouts.health.checks._REFERENCE_CONTRADICTION_GAPS = {'model_call_reference_unmatched': 'missing_captured_call', 'model_call_referenc... ``` ```python nemo_gym.rollouts.health.checks._ROLLOUT_CHECKS: dict[str, Callable[[dict[str, Any], dict[str, Any], _CallBindings, dict[str, int | str]], list[Finding]]] = {'check_execution_error': lambda record, trajectory, bindings, subject: [], 'rec... ``` ```python nemo_gym.rollouts.health.checks._ROLLOUT_SPECS = tuple(spec for spec in CHECK_REGISTRY if spec.evaluation_scope == CheckScope.ROL... ``` ```python nemo_gym.rollouts.health.checks._TASK_SPECS = tuple(spec for spec in CHECK_REGISTRY if spec.evaluation_scope == CheckScope.TAS... ``` ```python nemo_gym.rollouts.health.checks._TOOL_FAILURE_MARKERS: dict[str, frozenset[str]] = {'ripgrep execution failed': frozenset({'grep', 'glob'})} ```