> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server.

# nemo_gym.telemetry.span_groups

NeMo Gym span groups — the knob that decides which spans exist at all.

A span group is checked at every instrumentation site before any work happens, so a
disabled group costs one frozenset membership test. `GymSpanGroup` extends the shared
`nemo.lens.groups.SpanGroup` with Gym-shaped groups and replaces its training-oriented
presets with rollout-oriented ones.

Span groups live downstream, in the consumer, not in nemo-lens — the same call the
NeMo-RL integration made (`RLSpanGroup`), and the direction lens itself is heading.

## Presets

`default`
`job` plus the cross-process spine (`server`, `http_client`, `rollout`). This
is deliberately enough on its own to produce **one trace per rollout spanning the
agent, model, and resources server processes** — the whole point of the integration
works without tuning.
`per_rollout`
The spine plus per-request detail (`verify`, `agent`, `model_call`,
`tool_call`). Omits `job` so each rollout is its own bounded root trace rather
than nesting every rollout under one run-long span — the same reasoning behind
NeMo-RL's `per_step`.
`all`
Every group, including `sandbox` and the groups inherited from nemo-lens.

Only groups Gym actually emits under appear in `default` and `per_rollout`. The
training-oriented groups inherited from `nemo.lens.groups.SpanGroup` (`checkpoint`,
`step`, `optimizer`, `evaluate`, ...) remain resolvable, and are reachable through
`all`, but no Gym call site emits under them.

There is deliberately no `tool_call` or `dataset` group. A resources-server tool call
is already a SERVER span named after its route (`POST /get_weather`), which answers the
same questions without a second layer; and Gym's dataset code is CLI upload/download
helpers, not a runtime path worth tracing. A span group with no call site is a knob that
silently does nothing, so neither is declared until something emits under it.

Disabling `server` or `http_client` breaks cross-process trace joining: `server`
is the FastAPI ingress side that adopts an inbound `traceparent` as its parent, and
`http_client` is the egress side that emits one. They are in every preset for that
reason.

## Module Contents

### Classes

| Name                                                           | Description                                        |
| -------------------------------------------------------------- | -------------------------------------------------- |
| [`GymSpanGroup`](#nemo_gym-telemetry-span_groups-GymSpanGroup) | Span groups for NeMo Gym instrumentation.          |
| [`SpanGroup`](#nemo_gym-telemetry-span_groups-SpanGroup)       | Minimal stub used when nemo-lens is not installed. |

### API

```python
class nemo_gym.telemetry.span_groups.GymSpanGroup()
```

**Bases:** [SpanGroup](#nemo_gym-telemetry-span_groups-SpanGroup)

Span groups for NeMo Gym instrumentation.

**`AGENT`** `= 'agent'`

Agent-server `/run` and `/v1/responses` spans.

---

**`ALL_GROUPS`** `Final[frozenset]`

---

**`CROSS_PROCESS_SPINE`** `Final[frozenset] = frozenset([SERVER, HTTP_CLIENT, ROLLOUT])`

---

**`HTTP_CLIENT`** `= 'http_client'`

Outbound spans around `nemo_gym.server_utils.request` — Gym's single aiohttp
egress point. The egress half of cross-process propagation: injects `traceparent`
into the outgoing headers.

---

**`MODEL_CALL`** `= 'model_call'`

Model-server `/v1/chat/completions`, `/v1/responses` and `/v1/messages` spans.

---

**`ROLLOUT`** `= 'rollout'`

Rollout collection spans (one per task attempt, driver side).

---

**`SANDBOX`** `= 'sandbox'`

Sandbox provider create/exec/delete spans.

---

**`SERVER`** `= 'server'`

Inbound FastAPI request spans on every Gym server process. The ingress half of
cross-process propagation: adopts an inbound `traceparent` as the span's parent.

---

**`VERIFY`** `= 'verify'`

Resources-server `/verify` spans.

---

**`_PRESETS`** `dict`

---

```python
class nemo_gym.telemetry.span_groups.SpanGroup()
```

Minimal stub used when nemo-lens is not installed.

Mirrors `nemo.lens.groups.SpanGroup` at commit `b85578fc`. `resolve()`
raises rather than returning a wrong answer: without lens there is nothing to
enable, and silently returning an empty set would make a typo in
`telemetry.span_groups` indistinguishable from a working config.

**`ALL_GROUPS`** `Final[frozenset]`

---

**`CHECKPOINT`** `= 'checkpoint'`

---

**`EVALUATE`** `= 'evaluate'`

---

**`FORWARD_BACKWARD`** `= 'forward_backward'`

---

**`JOB`** `= 'job'`

---

**`LOAD_CHECKPOINT`** `= 'load_checkpoint'`

---

**`MODEL_INIT`** `= 'model_init'`

---

**`OPTIMIZER`** `= 'optimizer'`

---

**`STEP`** `= 'step'`

---

**`_PRESETS`** `dict`

---

```python
nemo_gym.telemetry.span_groups.SpanGroup.resolve(
    spec: str
) -> frozenset
```

classmethod