Skip to navigation
API ReferenceFull Library ReferenceNemo GymNemo Gym

nemo_gym.rollout_observability

View as Markdown

Shared contracts for rollout observations and trajectories.

Module Contents

Classes

NameDescription
AgentEpisodeAn Agent response and the observations available at its execution boundary.
AgentInvocationOne root Agent or subagent conversation observed by a harness.
AgentObservationBundleNormalized observations returned by one Agent Server for one rollout.
ContextCompactionObservationAn explicit context-compaction event reported by the Agent harness.
ModelCallRefStable identifiers an Agent integration can observe for one model call.
ObservationGapA fact that the selected integration could not observe or join exactly.
ObservationModel-
SandboxObservationOutcome and lifetime resource usage reported by a sandbox-owning harness.
ToolCallObservationTiming observed for one tool call at an Agent-owned boundary.
TrajectoryModelCall-
TrajectoryRecord-
TrajectoryResponseMetadata-
TrajectoryTokenStats-
TrajectoryToolCallTool observation enriched with the model-visible output in a trajectory record.
TrajectoryTurn-

Functions

NameDescription
join_assistant_message_callsJoin persisted messages to HTTP attempts using session, message and model-server IDs.
join_model_call_observationsResolve harness call references against captured model calls without guessing ownership.

Data

AgentObservationRecord

NG_AGENT_OBSERVATIONS_KEY

API

class nemo_gym.rollout_observability.AgentEpisode(
    response: nemo_gym.openai_utils.NeMoGymResponse,
    observations: nemo_gym.rollout_observability.AgentObservationBundle
)
Dataclass

An Agent response and the observations available at its execution boundary.

observations
AgentObservationBundle
response
NeMoGymResponse
class nemo_gym.rollout_observability.AgentInvocation()

Bases: ObservationModel

One root Agent or subagent conversation observed by a harness.

Harness adapters should report an iteration-limit event through an on_iteration_limit_reached(*, invocation_id: str) -> None observer hook. The hook records stop_reason="max_iterations" once on that invocation, leaving parent invocations, budgets, and summary generation alone. Harnesses with native stop events can call it directly; compatibility shims may call it at the point where the harness detects its limit.

conversation
list[NeMoGymResponseInputItem]
duration_ms
Optional[float] = Field(default=None, ge=0)
error_type
Optional[str] = None
invocation_id
str
kind
Literal['agent_invocation'] = 'agent_invocation'
model_calls
list[ModelCallRef] = Field(default_factory=list)
parent_invocation_id
Optional[str] = None
spawned_by_tool_call_id
Optional[str] = None
status
Literal['completed', 'failed', 'incomplete', 'unknown']
stop_reason
Optional[str]
class nemo_gym.rollout_observability.AgentObservationBundle()

Bases: ObservationModel

Normalized observations returned by one Agent Server for one rollout.

gaps
list[ObservationGap] = Field(default_factory=list)
records
list[AgentObservationRecord]
source
str
nemo_gym.rollout_observability.AgentObservationBundle.validate_identity() -> 'AgentObservationBundle'

Require unique invocation IDs and reject cycles in the observed parent graph.

class nemo_gym.rollout_observability.ContextCompactionObservation()

Bases: ObservationModel

An explicit context-compaction event reported by the Agent harness.

after_model_call
Optional[ModelCallRef]
before_model_call
Optional[ModelCallRef]
first_kept_item_id
Optional[str] = None
invocation_id
str
kind
Literal['context_compaction'] = 'context_compaction'
model_calls
list[ModelCallRef]
observed_at
Optional[float] = None
outcome
Literal['completed', 'failed', 'aborted', 'unknown'] = 'unknown'
source_message_ids
list[str]
source_model_ref
Optional[ModelServerRef] = None
summary
Optional[str] = None
tokens_after
Optional[int]
tokens_before
Optional[int]
trigger
Optional[str] = None
class nemo_gym.rollout_observability.ModelCallRef()

Bases: ObservationModel

Stable identifiers an Agent integration can observe for one model call.

model_call_id
Optional[str] = None
model_ref
Optional[ModelServerRef] = None
response_id
Optional[str] = None
nemo_gym.rollout_observability.ModelCallRef.validate_join_key() -> 'ModelCallRef'
class nemo_gym.rollout_observability.ObservationGap()

Bases: ObservationModel

A fact that the selected integration could not observe or join exactly.

code
str
detail
Optional[str] = None
invocation_id
Optional[str] = None
class nemo_gym.rollout_observability.ObservationModel()

Bases: BaseModel

model_config
class nemo_gym.rollout_observability.SandboxObservation()

Bases: ObservationModel

Outcome and lifetime resource usage reported by a sandbox-owning harness.

cpu_time_s
Optional[float]
error_type
Optional[str] = None
exit_code
Optional[int] = None
kind
Literal['sandbox'] = 'sandbox'
outcome
Literal['completed', 'failed', 'timeout', 'oom', 'sandbox_error', 'cancelled', 'unknown'] = 'unknown'
peak_memory_mib
Optional[float]
provider
Optional[str] = None
resource_usage_source
Optional[str] = None
role
Literal['agent', 'verifier', 'environment']
sandbox_id
Optional[str] = None
wall_time_s
Optional[float] = Field(default=None, ge=0)
class nemo_gym.rollout_observability.ToolCallObservation()

Bases: ObservationModel

Timing observed for one tool call at an Agent-owned boundary.

completed_at
Optional[float] = None
duration_ms
Optional[float] = Field(default=None, ge=0)
error_type
Optional[str] = None
invocation_id
str
kind
Literal['tool_call'] = 'tool_call'
sandbox_id
Optional[str]
started_at
Optional[float] = None
status
Literal['completed', 'failed', 'timeout', 'cancelled', 'incomplete', 'unknown'] = 'unknown'
timing_source
Optional[Literal['executor', 'artifact', 'harness']] = None
tool_call_id
str
tool_name
Optional[str] = None
nemo_gym.rollout_observability.ToolCallObservation.validate_timing() -> 'ToolCallObservation'
class nemo_gym.rollout_observability.TrajectoryModelCall()

Bases: ObservationModel

client_assistant_message_id
Optional[str] = None
client_session_id
Optional[str] = None
completed_at
Optional[float] = None
duration_ms
Optional[float] = Field(default=None, ge=0)
model_call_id
Optional[str] = None
request
Optional[Any] = None
response
Optional[Any] = None
response_metadata
TrajectoryResponseMetadata = Field(default_factory=TrajectoryResponseMetadata)
started_at
Optional[float] = None
token_stats
TrajectoryTokenStats = Field(default_factory=TrajectoryTokenStats)
class nemo_gym.rollout_observability.TrajectoryRecord()

Bases: ObservationModel

gaps
list[ObservationGap] = Field(default_factory=list)
invocations
list[AgentInvocation] = Field(default_factory=list)
model_calls
list[TrajectoryModelCall] = Field(default_factory=list)
rollout_id
str
schema_version
Literal['1.0'] = '1.0'
task_id
str
tool_calls
list[TrajectoryToolCall] = Field(default_factory=list)
turns
list[TrajectoryTurn] = Field(default_factory=list)
nemo_gym.rollout_observability.TrajectoryRecord.validate_identity() -> 'TrajectoryRecord'
class nemo_gym.rollout_observability.TrajectoryResponseMetadata()

Bases: ObservationModel

dialect
Optional[str] = None
error_category
Optional[str] = None
finish_reason
Optional[str] = None
latency_ttft_ms
Optional[float] = Field(default=None, ge=0)
local_response_reason
Optional[str] = None
model
Optional[str] = None
model_ref
Optional[ModelServerRef] = None
response_id
Optional[str] = None
response_source
Optional[Literal['upstream', 'local']] = None
response_status
Optional[str] = None
status_code
Optional[int] = None
upstream_attempted
Optional[bool] = None
upstream_status_code
Optional[int] = None
class nemo_gym.rollout_observability.TrajectoryTokenStats()

Bases: ObservationModel

cached_tokens
Optional[int] = Field(default=None, ge=0)
completion_tokens
Optional[int] = Field(default=None, ge=0)
prompt_tokens
Optional[int] = Field(default=None, ge=0)
reasoning_tokens
Optional[int] = Field(default=None, ge=0)
total_tokens
Optional[int] = Field(default=None, ge=0)
class nemo_gym.rollout_observability.TrajectoryToolCall()

Bases: ToolCallObservation

Tool observation enriched with the model-visible output in a trajectory record.

output
Optional[Any] = None
class nemo_gym.rollout_observability.TrajectoryTurn()

Bases: ObservationModel

answer
Optional[Any] = None
invocation_id
str
model_calls
list[ModelCallRef] = Field(default_factory=list)
question
Optional[Any] = None
reasoning_content
Optional[Any] = None
resolved
Optional[bool] = None
rollout_id
str
source_message_id
Optional[str]
source_model_ref
Optional[ModelServerRef]
step_count
int
task_id
str
timestamp
float
turn_no
int
nemo_gym.rollout_observability.join_assistant_message_calls(
    bundle: nemo_gym.rollout_observability.AgentObservationBundle,
    trajectory: nemo_gym.rollout_observability.TrajectoryRecord | None,
    calls: list['ModelCallRecord']
) -> tuple[nemo_gym.rollout_observability.AgentObservationBundle, nemo_gym.rollout_observability.TrajectoryRecord | None]

Join persisted messages to HTTP attempts using session, message and model-server IDs.

Run after invocation ownership resolution. Summary attempts belong to compactions; calls without a saved message (for example title generation) stay invocation-owned. Producers declare source_message_id(s), source_model_ref and invocation IDs. Source labels are not dispatch keys. No content or timing fallback is used.

nemo_gym.rollout_observability.join_model_call_observations(
    bundle: nemo_gym.rollout_observability.AgentObservationBundle,
    calls: collections.abc.Iterable[nemo_gym.base_responses_api_model.ModelCallRecord]
) -> nemo_gym.rollout_observability.AgentObservationBundle

Resolve harness call references against captured model calls without guessing ownership.

nemo_gym.rollout_observability.AgentObservationRecord = Annotated[AgentInvocation | ToolCallObservation | ContextCompactionObservation |...
 
nemo_gym.rollout_observability.NG_AGENT_OBSERVATIONS_KEY = 'ng_agent_observations'