Span Groups

A span group is a named switch that decides whether a set of spans exists at all. Groups let you trade detail against volume without changing code, and they are checked before any instrumentation work happens, so a disabled group costs almost nothing.

Presets

Set span_groups to a preset name:

telemetry:
  enabled: true
  span_groups: per_rollout

default is deliberately enough on its own to produce one trace per rollout across the agent, model, and resources server processes. You do not have to tune anything to get the headline behavior.

per_rollout omits job on purpose. job wraps a whole rollout-collection run, so including it nests every rollout of that run under one span. That is useful for seeing a run’s overall shape and unusable for a run with thousands of rollouts.

Groups

Groups You Should Not Disable

server and http_client are the two halves of cross-process propagation. http_client writes the traceparent header and server reads it. Disabling either breaks trace joining: you still get spans, but each process produces its own disconnected trace. Both are in every preset for this reason.

Custom Selections

span_groups also accepts a comma-separated list, and mixes presets with individual names:

telemetry:
  span_groups: "default,sandbox"

telemetry:
  span_groups: "server,http_client,verify"

Names are case-insensitive and surrounding whitespace is ignored. An unknown name does not raise an error. Each server logs a warning that names the unknown entry and lists the registered groups, then ignores it. If telemetry looks broken, check the server logs for that warning first.

Cost

The gate is a frozenset membership test, and nothing runs above it — not attribute construction, not string formatting, and not imports. A disabled site costs less than entering an empty contextlib.nullcontext, because it does not enter a context manager at all.

An enabled span costs orders of magnitude more than the gate that guards it. That is the trade you make when you widen span_groups, and it is why all is a debugging setting rather than a default.

To measure the disabled-path cost on your own hardware, run pytest tests/unit_tests/telemetry/test_overhead.py -s, which reports the figures it compares.

Groups NeMo Gym Does Not Have

There is no tool_call group. A resources-server tool call already appears as a SERVER span named after its route, such as POST /get_weather, which answers the same questions without a second layer.

There is no dataset group. NeMo Gym’s dataset code is command-line upload and download tooling rather than a runtime path, so there is nothing worth tracing.

Running Inside NeMo RL

When NeMo Gym runs in the same process as NeMo RL, both libraries register their span groups with nemo-lens, and presets combine across libraries. default selects NeMo RL’s default groups and NeMo Gym’s. The job and rollout names belong to both libraries, so enabling either name enables both libraries’ spans.

nemo-lens allows one telemetry setup per process. If NeMo RL sets up telemetry first, NeMo Gym reuses it. NeMo RL’s span-group setting, service name, and exporter then apply, and NeMo Gym’s telemetry: block is ignored in that process.