nemo_gym.telemetry.span_groups

View as Markdown

NeMo Gym span groups, declared into nemo-lens’s SpanRegistry.

A span group is checked at every instrumentation site before any work happens, so a disabled group costs one frozenset membership test. nemo-lens ships no group names of its own: a consuming library registers the groups it emits under its own namespace, and users select from them with the span_groups spec. Importing this module registers Gym’s groups and presets, so it must be imported before setup_telemetry; init_telemetry does that.

GymSpanGroup is a bag of str constants rather than a lens subclass, because managed_span and is_span_group_enabled take the group as a plain string. Call sites keep reading GymSpanGroup.SANDBOX so the spelling lives in one place, and the constants stay importable without nemo-lens: the gate is conditional, not the name.

Presets

default job plus the cross-process spine (server, http_client, rollout). This is deliberately enough on its own to produce one trace per rollout spanning the agent, model, and resources server processes — the whole point of the integration works without tuning. per_rollout The spine plus per-request detail (verify, agent, model_call). Omits job so each rollout is its own bounded root trace rather than nesting every rollout under one run-long span — the same reasoning behind NeMo-RL’s per_step. all Reserved by nemo-lens: every group registered in the process, including sandbox.

Registration is process-global, and presets union across namespaces. When Gym runs inside a NeMo-RL process, default selects NeMo-RL’s default groups and Gym’s, and the job and rollout names are shared with NeMo-RL’s groups of the same name.

There is deliberately no tool_call or dataset group. A resources-server tool call is already a SERVER span named after its route (POST /get_weather), which answers the same questions without a second layer; and Gym’s dataset code is CLI upload/download helpers, not a runtime path worth tracing. A span group with no call site is a knob that silently does nothing, so neither is declared until something emits under it.

Disabling server or http_client breaks cross-process trace joining: server is the FastAPI ingress side that adopts an inbound traceparent as its parent, and http_client is the egress side that emits one. They are in every preset for that reason.

Module Contents

Classes

NameDescription
GymSpanGroupSpan group names for NeMo Gym instrumentation.

Functions

NameDescription
register_span_groupsDeclare Gym’s groups and presets to nemo-lens. A no-op without nemo-lens.

Data

NAMESPACE

API

class nemo_gym.telemetry.GymSpanGroup()

Span group names for NeMo Gym instrumentation.

AGENT
= 'agent'

Agent-server /run and /v1/responses spans.

ALL_GROUPS
Final[frozenset]
CROSS_PROCESS_SPINE
Final[frozenset] = frozenset([SERVER, HTTP_CLIENT, ROLLOUT])
HTTP_CLIENT
= 'http_client'

Outbound spans around nemo_gym.server_utils.request — Gym’s single aiohttp egress point. The egress half of cross-process propagation: injects traceparent into the outgoing headers.

JOB
= 'job'

The whole gym eval / rollout-collection run, driver side. Shares its name with NeMo-RL’s run-level group, so one spec entry selects both.

MODEL_CALL
= 'model_call'

Model-server /v1/chat/completions, /v1/responses and /v1/messages spans.

ROLLOUT
= 'rollout'

Rollout collection spans (one per task attempt, driver side).

SANDBOX
= 'sandbox'

Sandbox provider create/exec/delete spans.

SERVER
= 'server'

Inbound FastAPI request spans on every Gym server process. The ingress half of cross-process propagation: adopts an inbound traceparent as the span’s parent.

VERIFY
= 'verify'

Resources-server /verify spans.

_PRESETS
dict
nemo_gym.telemetry.GymSpanGroup.resolve(
spec: str
) -> frozenset
classmethod

Resolve a span_groups spec against every group registered in this process.

Entries that name nothing registered are dropped rather than raised: the library that owns them may not be imported in this process. nemo-lens logs a warning naming them when the spec is applied in setup_telemetry.

Raises:

  • RuntimeError: nemo-lens is not installed, so there is nothing to resolve.
nemo_gym.telemetry.span_groups.register_span_groups() -> None

Declare Gym’s groups and presets to nemo-lens. A no-op without nemo-lens.

Called at import. allow_override makes it idempotent, so a re-import or a test that cleared the registry can call it again, and silences the shared-name warning for the job and rollout groups NeMo-RL also registers.

nemo_gym.telemetry.span_groups.NAMESPACE = 'nemo_gym'