API ReferenceFull Library ReferenceNemo GymNemo Gym

nemo_gym.rollout_observability

View as Markdown

Shared contracts for rollout observations and trajectories.

Module Contents

Classes

NameDescription
AgentEpisodeAn Agent response and the observations available at its execution boundary.
AgentInvocationOne root Agent or subagent conversation observed by a harness.
AgentObservationBundleNormalized observations returned by one Agent Server for one rollout.
ContextCompactionObservationAn explicit context-compaction event reported by the Agent harness.
ModelCallRefStable identifiers an Agent integration can observe for one model call.
ObservationGapA fact that the selected integration could not observe or join exactly.
ObservationModel-
SandboxObservationOutcome and lifetime resource usage reported by a sandbox-owning harness.
ToolCallObservationTiming observed for one tool call at an Agent-owned boundary.
TrajectoryModelCall-
TrajectoryRecord-
TrajectoryResponseMetadata-
TrajectoryTokenStats-
TrajectoryToolCallTool observation enriched with the model-visible output in a trajectory record.
TrajectoryTurn-

Functions

NameDescription
join_model_call_observationsResolve harness call references against captured model calls without guessing ownership.

Data

AgentObservationRecord

API

class nemo_gym.rollout_observability.AgentEpisode(
    response: nemo_gym.openai_utils.NeMoGymResponse,
    observations: nemo_gym.rollout_observability.AgentObservationBundle
)
Dataclass

An Agent response and the observations available at its execution boundary.

observations
AgentObservationBundle
response
NeMoGymResponse
class nemo_gym.rollout_observability.AgentInvocation()

Bases: ObservationModel

One root Agent or subagent conversation observed by a harness.

conversation
list[NeMoGymResponseInputItem]
duration_ms
Optional[float] = Field(default=None, ge=0)
error_type
Optional[str] = None
invocation_id
str
kind
Literal['agent_invocation'] = 'agent_invocation'
model_calls
list[ModelCallRef] = Field(default_factory=list)
parent_invocation_id
Optional[str] = None
spawned_by_tool_call_id
Optional[str] = None
status
Literal['completed', 'failed', 'incomplete', 'unknown']
class nemo_gym.rollout_observability.AgentObservationBundle()

Bases: ObservationModel

Normalized observations returned by one Agent Server for one rollout.

gaps
list[ObservationGap] = Field(default_factory=list)
records
list[AgentObservationRecord]
source
str
nemo_gym.rollout_observability.AgentObservationBundle.validate_identity() -> 'AgentObservationBundle'

Require unique invocation IDs and reject cycles in the observed parent graph.

class nemo_gym.rollout_observability.ContextCompactionObservation()

Bases: ObservationModel

An explicit context-compaction event reported by the Agent harness.

after_model_call
Optional[ModelCallRef]
before_model_call
Optional[ModelCallRef]
first_kept_item_id
Optional[str] = None
invocation_id
str
kind
Literal['context_compaction'] = 'context_compaction'
model_calls
list[ModelCallRef]
observed_at
Optional[float] = None
outcome
Literal['completed', 'failed', 'aborted', 'unknown'] = 'unknown'
summary
Optional[str] = None
tokens_after
Optional[int]
tokens_before
Optional[int]
trigger
Optional[str] = None
class nemo_gym.rollout_observability.ModelCallRef()

Bases: ObservationModel

Stable identifiers an Agent integration can observe for one model call.

model_call_id
Optional[str] = None
model_ref
Optional[ModelServerRef] = None
response_id
Optional[str] = None
nemo_gym.rollout_observability.ModelCallRef.validate_join_key() -> 'ModelCallRef'
class nemo_gym.rollout_observability.ObservationGap()

Bases: ObservationModel

A fact that the selected integration could not observe or join exactly.

code
str
detail
Optional[str] = None
invocation_id
Optional[str] = None
class nemo_gym.rollout_observability.ObservationModel()

Bases: BaseModel

model_config
class nemo_gym.rollout_observability.SandboxObservation()

Bases: ObservationModel

Outcome and lifetime resource usage reported by a sandbox-owning harness.

cpu_time_s
Optional[float]
error_type
Optional[str] = None
exit_code
Optional[int] = None
kind
Literal['sandbox'] = 'sandbox'
outcome
Literal['completed', 'failed', 'timeout', 'oom', 'sandbox_error', 'cancelled', 'unknown'] = 'unknown'
peak_memory_mib
Optional[float]
provider
Optional[str] = None
resource_usage_source
Optional[str] = None
role
Literal['agent', 'verifier', 'environment']
sandbox_id
Optional[str] = None
wall_time_s
Optional[float] = Field(default=None, ge=0)
class nemo_gym.rollout_observability.ToolCallObservation()

Bases: ObservationModel

Timing observed for one tool call at an Agent-owned boundary.

completed_at
Optional[float] = None
duration_ms
Optional[float] = Field(default=None, ge=0)
error_type
Optional[str] = None
invocation_id
str
kind
Literal['tool_call'] = 'tool_call'
sandbox_id
Optional[str]
started_at
Optional[float] = None
status
Literal['completed', 'failed', 'timeout', 'cancelled', 'incomplete', 'unknown'] = 'unknown'
timing_source
Optional[Literal['executor', 'artifact', 'harness']] = None
tool_call_id
str
tool_name
Optional[str] = None
nemo_gym.rollout_observability.ToolCallObservation.validate_timing() -> 'ToolCallObservation'
class nemo_gym.rollout_observability.TrajectoryModelCall()

Bases: ObservationModel

completed_at
Optional[float] = None
duration_ms
Optional[float] = Field(default=None, ge=0)
model_call_id
Optional[str] = None
request
Optional[Any] = None
response
Optional[Any] = None
response_metadata
TrajectoryResponseMetadata = Field(default_factory=TrajectoryResponseMetadata)
started_at
Optional[float] = None
token_stats
TrajectoryTokenStats = Field(default_factory=TrajectoryTokenStats)
class nemo_gym.rollout_observability.TrajectoryRecord()

Bases: ObservationModel

gaps
list[ObservationGap] = Field(default_factory=list)
invocations
list[AgentInvocation] = Field(default_factory=list)
model_calls
list[TrajectoryModelCall] = Field(default_factory=list)
rollout_id
str
schema_version
Literal['1.0'] = '1.0'
task_id
str
tool_calls
list[TrajectoryToolCall] = Field(default_factory=list)
turns
list[TrajectoryTurn] = Field(default_factory=list)
nemo_gym.rollout_observability.TrajectoryRecord.validate_identity() -> 'TrajectoryRecord'
class nemo_gym.rollout_observability.TrajectoryResponseMetadata()

Bases: ObservationModel

dialect
Optional[str] = None
error_category
Optional[str] = None
finish_reason
Optional[str] = None
latency_ttft_ms
Optional[float] = Field(default=None, ge=0)
model
Optional[str] = None
model_ref
Optional[ModelServerRef] = None
response_id
Optional[str] = None
response_status
Optional[str] = None
status_code
Optional[int] = None
class nemo_gym.rollout_observability.TrajectoryTokenStats()

Bases: ObservationModel

cached_tokens
Optional[int] = Field(default=None, ge=0)
completion_tokens
Optional[int] = Field(default=None, ge=0)
prompt_tokens
Optional[int] = Field(default=None, ge=0)
reasoning_tokens
Optional[int] = Field(default=None, ge=0)
total_tokens
Optional[int] = Field(default=None, ge=0)
class nemo_gym.rollout_observability.TrajectoryToolCall()

Bases: ToolCallObservation

Tool observation enriched with the model-visible output in a trajectory record.

output
Optional[Any] = None
class nemo_gym.rollout_observability.TrajectoryTurn()

Bases: ObservationModel

answer
Optional[Any] = None
invocation_id
str
model_calls
list[ModelCallRef] = Field(default_factory=list)
question
Optional[Any] = None
reasoning_content
Optional[Any] = None
resolved
Optional[bool] = None
rollout_id
str
step_count
int
task_id
str
timestamp
float
turn_no
int
nemo_gym.rollout_observability.join_model_call_observations(
    bundle: nemo_gym.rollout_observability.AgentObservationBundle,
    calls: collections.abc.Iterable[nemo_gym.base_responses_api_model.ModelCallRecord]
) -> nemo_gym.rollout_observability.AgentObservationBundle

Resolve harness call references against captured model calls without guessing ownership.

nemo_gym.rollout_observability.AgentObservationRecord = Annotated[AgentInvocation | ToolCallObservation | ContextCompactionObservation |...