nemo_gym.telemetry.span_groups
nemo_gym.telemetry.span_groups
NeMo Gym span groups, declared into nemo-lens’s SpanRegistry.
A span group is checked at every instrumentation site before any work happens, so a
disabled group costs one frozenset membership test. nemo-lens ships no group names of its
own: a consuming library registers the groups it emits under its own namespace, and users
select from them with the span_groups spec. Importing this module registers Gym’s groups
and presets, so it must be imported before setup_telemetry; init_telemetry does that.
GymSpanGroup is a bag of str constants rather than a lens subclass, because
managed_span and is_span_group_enabled take the group as a plain string. Call sites
keep reading GymSpanGroup.SANDBOX so the spelling lives in one place, and the constants
stay importable without nemo-lens: the gate is conditional, not the name.
Presets
default
job plus the cross-process spine (server, http_client, rollout). This
is deliberately enough on its own to produce one trace per rollout spanning the
agent, model, and resources server processes — the whole point of the integration
works without tuning.
per_rollout
The spine plus per-request detail (verify, agent, model_call). Omits
job so each rollout is its own bounded root trace rather than nesting every
rollout under one run-long span — the same reasoning behind NeMo-RL’s per_step.
all
Reserved by nemo-lens: every group registered in the process, including sandbox.
Registration is process-global, and presets union across namespaces. When Gym runs
inside a NeMo-RL process, default selects NeMo-RL’s default groups and Gym’s, and the
job and rollout names are shared with NeMo-RL’s groups of the same name.
There is deliberately no tool_call or dataset group. A resources-server tool call
is already a SERVER span named after its route (POST /get_weather), which answers the
same questions without a second layer; and Gym’s dataset code is CLI upload/download
helpers, not a runtime path worth tracing. A span group with no call site is a knob that
silently does nothing, so neither is declared until something emits under it.
Disabling server or http_client breaks cross-process trace joining: server
is the FastAPI ingress side that adopts an inbound traceparent as its parent, and
http_client is the egress side that emits one. They are in every preset for that
reason.
Module Contents
Classes
Functions
Data
API
Span group names for NeMo Gym instrumentation.
Agent-server /run and /v1/responses spans.
Outbound spans around nemo_gym.server_utils.request — Gym’s single aiohttp
egress point. The egress half of cross-process propagation: injects traceparent
into the outgoing headers.
The whole gym eval / rollout-collection run, driver side. Shares its name with
NeMo-RL’s run-level group, so one spec entry selects both.
Model-server /v1/chat/completions, /v1/responses and /v1/messages spans.
Rollout collection spans (one per task attempt, driver side).
Sandbox provider create/exec/delete spans.
Inbound FastAPI request spans on every Gym server process. The ingress half of
cross-process propagation: adopts an inbound traceparent as the span’s parent.
Resources-server /verify spans.
Resolve a span_groups spec against every group registered in this process.
Entries that name nothing registered are dropped rather than raised: the library
that owns them may not be imported in this process. nemo-lens logs a warning naming
them when the spec is applied in setup_telemetry.
Raises:
RuntimeError: nemo-lens is not installed, so there is nothing to resolve.
Declare Gym’s groups and presets to nemo-lens. A no-op without nemo-lens.
Called at import. allow_override makes it idempotent, so a re-import or a test that
cleared the registry can call it again, and silences the shared-name warning for the
job and rollout groups NeMo-RL also registers.