nemo_gym.telemetry.span_groups

NeMo Gym span groups — the knob that decides which spans exist at all.

A span group is checked at every instrumentation site before any work happens, so a disabled group costs one frozenset membership test. GymSpanGroup extends the shared nemo.lens.groups.SpanGroup with Gym-shaped groups and replaces its training-oriented presets with rollout-oriented ones.

Span groups live downstream, in the consumer, not in nemo-lens — the same call the NeMo-RL integration made (RLSpanGroup), and the direction lens itself is heading.

Presets

default job plus the cross-process spine (server, http_client, rollout). This is deliberately enough on its own to produce one trace per rollout spanning the agent, model, and resources server processes — the whole point of the integration works without tuning. per_rollout The spine plus per-request detail (verify, agent, model_call, tool_call). Omits job so each rollout is its own bounded root trace rather than nesting every rollout under one run-long span — the same reasoning behind NeMo-RL’s per_step. all Every group, including sandbox and the groups inherited from nemo-lens.

Only groups Gym actually emits under appear in default and per_rollout. The training-oriented groups inherited from nemo.lens.groups.SpanGroup (checkpoint, step, optimizer, evaluate, …) remain resolvable, and are reachable through all, but no Gym call site emits under them.

There is deliberately no tool_call or dataset group. A resources-server tool call is already a SERVER span named after its route (POST /get_weather), which answers the same questions without a second layer; and Gym’s dataset code is CLI upload/download helpers, not a runtime path worth tracing. A span group with no call site is a knob that silently does nothing, so neither is declared until something emits under it.

Disabling server or http_client breaks cross-process trace joining: server is the FastAPI ingress side that adopts an inbound traceparent as its parent, and http_client is the egress side that emits one. They are in every preset for that reason.

Module Contents

Classes

Name	Description
`GymSpanGroup`	Span groups for NeMo Gym instrumentation.
`SpanGroup`	Minimal stub used when nemo-lens is not installed.

API

class nemo_gym.telemetry.span_groups.GymSpanGroup()

Bases: SpanGroup

Span groups for NeMo Gym instrumentation.

AGENT

= 'agent'

Agent-server /run and /v1/responses spans.

ALL_GROUPS

Final[frozenset]

CROSS_PROCESS_SPINE

Final[frozenset] = frozenset([SERVER, HTTP_CLIENT, ROLLOUT])

HTTP_CLIENT

= 'http_client'

Outbound spans around nemo_gym.server_utils.request — Gym’s single aiohttp egress point. The egress half of cross-process propagation: injects traceparent into the outgoing headers.

MODEL_CALL

= 'model_call'

Model-server /v1/chat/completions, /v1/responses and /v1/messages spans.

ROLLOUT

= 'rollout'

Rollout collection spans (one per task attempt, driver side).

SANDBOX

= 'sandbox'

Sandbox provider create/exec/delete spans.

SERVER

= 'server'

Inbound FastAPI request spans on every Gym server process. The ingress half of cross-process propagation: adopts an inbound traceparent as the span’s parent.

VERIFY

= 'verify'

Resources-server /verify spans.

_PRESETS

dict

class nemo_gym.telemetry.span_groups.SpanGroup()

Minimal stub used when nemo-lens is not installed.

Mirrors nemo.lens.groups.SpanGroup at commit b85578fc. resolve() raises rather than returning a wrong answer: without lens there is nothing to enable, and silently returning an empty set would make a typo in telemetry.span_groups indistinguishable from a working config.

ALL_GROUPS

Final[frozenset]

CHECKPOINT

= 'checkpoint'

EVALUATE

= 'evaluate'

FORWARD_BACKWARD

= 'forward_backward'

JOB

= 'job'

LOAD_CHECKPOINT

= 'load_checkpoint'

MODEL_INIT

= 'model_init'

OPTIMIZER

= 'optimizer'

STEP

= 'step'

_PRESETS

dict

nemo_gym.telemetry.span_groups.SpanGroup.resolve(
    spec: str
) -> frozenset

classmethod