nemo_gym.telemetry.span_groups
nemo_gym.telemetry.span_groups
NeMo Gym span groups, declared into nemo-lens’s SpanRegistry.
A span group is checked at every instrumentation site before any work happens, so a
disabled group costs one frozenset membership test. nemo-lens ships no group names of its
own: a consuming library registers the groups it emits under its own namespace, and users
select from them with the span_groups spec. Importing this module registers Gym’s groups
and presets, so it must be imported before setup_telemetry; init_telemetry does that.
GymSpanGroup is a bag of str constants rather than a lens subclass, because
managed_span and is_span_group_enabled take the group as a plain string. Call sites
keep reading GymSpanGroup.SANDBOX so the spelling lives in one place, and the constants
stay importable without nemo-lens: the gate is conditional, not the name.
Presets
default
job plus the cross-process spine (server, http_client, rollout). This
is deliberately enough on its own to produce one trace per rollout spanning the
agent, model, and resources server processes — the whole point of the integration
works without tuning.
per_rollout
The spine plus per-request detail (verify, agent, model_call). Omits
job so each rollout is its own bounded root trace rather than nesting every
rollout under one run-long span — the same reasoning behind NeMo-RL’s per_step.
all
Reserved by nemo-lens: every group registered in the process, including sandbox.
Registration is process-global, and presets union across namespaces. When Gym runs
inside a NeMo-RL process, default selects NeMo-RL’s default groups and Gym’s, and the
job and rollout names are shared with NeMo-RL’s groups of the same name.
There is deliberately no tool_call or dataset group. A resources-server tool call
is already a SERVER span named after its route (POST /get_weather), which answers the
same questions without a second layer; and Gym’s dataset code is CLI upload/download
helpers, not a runtime path worth tracing. A span group with no call site is a knob that
silently does nothing, so neither is declared until something emits under it.
Disabling server or http_client breaks cross-process trace joining: server
is the FastAPI ingress side that adopts an inbound traceparent as its parent, and
http_client is the egress side that emits one. They are in every preset for that
reason.
Module Contents
Classes
Functions
Data
API
Span group names for NeMo Gym instrumentation.
Agent-server /run and /v1/responses spans.
Outbound spans around nemo_gym.server_utils.request — Gym’s single aiohttp
egress point. The egress half of cross-process propagation: injects traceparent
into the outgoing headers.
The whole gym eval / rollout-collection run, driver side. Shares its name with
NeMo-RL’s run-level group, so one spec entry selects both.
Model-server /v1/chat/completions, /v1/responses and /v1/messages spans.
Rollout collection spans (one per task attempt, driver side).
Sandbox provider create/exec/delete spans.
Inbound FastAPI request spans on every Gym server process. The ingress half of
cross-process propagation: adopts an inbound traceparent as the span’s parent.
Server startup spans, once per run.
The supervisor emits gym.startup and one gym.startup.server span per server.
Each server process emits gym.server.startup or gym.server.startup.worker with one child span per stage.
Resources-server /verify spans.
Resolve a span_groups spec against every group registered in this process.
Entries that name nothing registered are dropped rather than raised: the library
that owns them may not be imported in this process. nemo-lens logs a warning naming
them when the spec is applied in setup_telemetry.
Raises:
RuntimeError: nemo-lens is not installed, so there is nothing to resolve.
Declare Gym’s groups and presets to nemo-lens. A no-op without nemo-lens.
Called at import. allow_override makes it idempotent, so a re-import or a test that
cleared the registry can call it again, and silences the shared-name warning for the
job and rollout groups NeMo-RL also registers.