> This page is for version 0.5.1.
> For other versions, use one of these documentation indexes:
> - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt
> - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt
> - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt
> - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt
> - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt
> - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt
> - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt
> - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server.

# nemo_gym.telemetry

NeMo Gym telemetry: optional OpenTelemetry instrumentation via nemo-lens.

Importing this package never requires nemo-lens. Install it with
`uv sync --extra telemetry` and switch it on with a `telemetry:` block or
`NEMO_LENS_ENABLED=1`; with either missing, every instrumentation site in `nemo_gym`
is a \~0-cost no-op.

**Public surface**

* `TelemetryConfig` — the `telemetry:` config block.
* `GymSpanGroup` — Gym span groups and presets.
* `configure_telemetry_env` — supervisor side; hands
  the settings to spawned server processes through the environment.
* `init_telemetry` /
  `get_telemetry` /
  `shutdown_telemetry` — per-process lifecycle.

Instrumentation primitives (`managed_span` / `span_cm` / `trace_fn` /
`is_span_group_enabled` / `safe_set_span_attributes`) come from
`nemo_gym.telemetry._fallbacks`, which resolves to the real nemo-lens
implementations when it is installed and to no-op stubs when it is not.

See `nemo_gym/telemetry/README.md` for the design, and
`fern/versions/latest/pages/observability/` for user documentation.

## Submodules

* **[`nemo_gym.telemetry.config`](/nemo/gym/nemo-gym/nemo_gym/telemetry/config)**
* **[`nemo_gym.telemetry.connection_pool`](/nemo/gym/nemo-gym/nemo_gym/telemetry/connection_pool)**
* **[`nemo_gym.telemetry.contrib`](/nemo/gym/nemo-gym/nemo_gym/telemetry/contrib)**
* **[`nemo_gym.telemetry.endpoints`](/nemo/gym/nemo-gym/nemo_gym/telemetry/endpoints)**
* **[`nemo_gym.telemetry.gym_metrics`](/nemo/gym/nemo-gym/nemo_gym/telemetry/gym_metrics)**
* **[`nemo_gym.telemetry.memory`](/nemo/gym/nemo-gym/nemo_gym/telemetry/memory)**
* **[`nemo_gym.telemetry.metrics`](/nemo/gym/nemo-gym/nemo_gym/telemetry/metrics)**
* **[`nemo_gym.telemetry.observer`](/nemo/gym/nemo-gym/nemo_gym/telemetry/observer)**
* **[`nemo_gym.telemetry.resources_operations`](/nemo/gym/nemo-gym/nemo_gym/telemetry/resources_operations)**
* **[`nemo_gym.telemetry.setup`](/nemo/gym/nemo-gym/nemo_gym/telemetry/setup)**
* **[`nemo_gym.telemetry.span_groups`](/nemo/gym/nemo-gym/nemo_gym/telemetry/span_groups)**
* **[`nemo_gym.telemetry.spans`](/nemo/gym/nemo-gym/nemo_gym/telemetry/spans)**
* **[`nemo_gym.telemetry.startup`](/nemo/gym/nemo-gym/nemo_gym/telemetry/startup)**

## Package Contents

### Classes

| Name                                                                        | Description                                           |
| --------------------------------------------------------------------------- | ----------------------------------------------------- |
| [`GymSpanGroup`](#nemo_gym-telemetry-span_groups-GymSpanGroup)              | Span group names for NeMo Gym instrumentation.        |
| [`MemoryProfilingConfig`](#nemo_gym-telemetry-config-MemoryProfilingConfig) | Periodic host and logical-server memory sampling.     |
| [`TelemetryConfig`](#nemo_gym-telemetry-config-TelemetryConfig)             | OpenTelemetry / nemo-lens configuration for NeMo Gym. |

### Functions

| Name                                                                                                   | Description                                                                  |
| ------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------- |
| [`configure_telemetry_env`](#nemo_gym-telemetry-setup-configure_telemetry_env)                         | Translate the `telemetry:` block into env vars for spawned server processes. |
| [`get_telemetry`](#nemo_gym-telemetry-setup-get_telemetry)                                             | Return this process's telemetry handle, or `None` if uninitialised/disabled. |
| [`init_telemetry`](#nemo_gym-telemetry-setup-init_telemetry)                                           | Initialise this process's telemetry. Call once per process; idempotent.      |
| [`is_span_group_enabled`](#nemo_gym-telemetry-_fallbacks-is_span_group_enabled)                        | Always returns False.                                                        |
| [`is_telemetry_env_enabled`](#nemo_gym-telemetry-setup-is_telemetry_env_enabled)                       | True when telemetry is switched on in this process's environment.            |
| [`is_telemetry_metrics_enabled`](#nemo_gym-telemetry-setup-is_telemetry_metrics_enabled)               | Return the effective metrics switch after Gym/Lens env precedence.           |
| [`managed_span`](#nemo_gym-telemetry-_fallbacks-managed_span)                                          | No-op context manager — yields None.                                         |
| [`memory_profiling_config_from_env`](#nemo_gym-telemetry-setup-memory_profiling_config_from_env)       | Resolve memory profiling after raw environment variables take precedence.    |
| [`safe_set_span_attributes`](#nemo_gym-telemetry-_fallbacks-safe_set_span_attributes)                  | No-op.                                                                       |
| [`shutdown_telemetry`](#nemo_gym-telemetry-setup-shutdown_telemetry)                                   | Flush and shut down this process's telemetry providers.                      |
| [`span_cm`](#nemo_gym-telemetry-_fallbacks-span_cm)                                                    | No-op context manager — yields None.                                         |
| [`telemetry_config_from_global_config`](#nemo_gym-telemetry-setup-telemetry_config_from_global_config) | Build a `TelemetryConfig` from Gym's merged global config dict.              |
| [`trace_fn`](#nemo_gym-telemetry-_fallbacks-trace_fn)                                                  | No-op decorator — returns the function unchanged.                            |

### Data

[`TELEMETRY_KEY_NAME`](#nemo_gym-telemetry-setup-TELEMETRY_KEY_NAME)

### API

```python
class nemo_gym.telemetry.GymSpanGroup()
```

Span group names for NeMo Gym instrumentation.

**`AGENT`** `= 'agent'`

Agent-server `/run` and `/v1/responses` spans.

---

**`ALL_GROUPS`** `Final[frozenset]`

---

**`CROSS_PROCESS_SPINE`** `Final[frozenset] = frozenset([SERVER, HTTP_CLIENT, ROLLOUT])`

---

**`HTTP_CLIENT`** `= 'http_client'`

Outbound spans around `nemo_gym.server_utils.request` — Gym's single aiohttp
egress point. The egress half of cross-process propagation: injects `traceparent`
into the outgoing headers.

---

**`JOB`** `= 'job'`

The whole `gym eval` / rollout-collection run, driver side. Shares its name with
NeMo-RL's run-level group, so one spec entry selects both.

---

**`MODEL_CALL`** `= 'model_call'`

Model-server `/v1/chat/completions`, `/v1/responses` and `/v1/messages` spans.

---

**`ROLLOUT`** `= 'rollout'`

Rollout collection spans (one per task attempt, driver side).

---

**`SANDBOX`** `= 'sandbox'`

Sandbox provider create/exec/delete spans.

---

**`SERVER`** `= 'server'`

Inbound FastAPI request spans on every Gym server process. The ingress half of
cross-process propagation: adopts an inbound `traceparent` as the span's parent.

---

**`STARTUP`** `= 'startup'`

Server startup spans, once per run.

The supervisor emits `gym.startup` and one `gym.startup.server` span per server.
Each server process emits `gym.server.startup` or `gym.server.startup.worker` with one child span per stage.

---

**`VERIFY`** `= 'verify'`

Resources-server `/verify` spans.

---

**`_PRESETS`** `dict`

---

```python
nemo_gym.telemetry.GymSpanGroup.resolve(
    spec: str
) -> frozenset
```

classmethod

Resolve a `span_groups` spec against every group registered in this process.

Entries that name nothing registered are dropped rather than raised: the library
that owns them may not be imported in this process. nemo-lens logs a warning naming
them when the spec is applied in `setup_telemetry`.

**Raises:**

* `RuntimeError`: nemo-lens is not installed, so there is nothing to resolve.

```python
class nemo_gym.telemetry.MemoryProfilingConfig()
```

**Bases:** `BaseModel`

Periodic host and logical-server memory sampling.

**`enabled`** `bool = False`

Collect host memory and every server process tree.

---

**`interval_seconds`** `float = Field(default=1.0, gt=0)`

Seconds between samples.

---

```python
class nemo_gym.telemetry.TelemetryConfig()
```

**Bases:** `BaseModel`

OpenTelemetry / nemo-lens configuration for NeMo Gym.

Telemetry is optional twice over: it activates only when `enabled` is true *and*
nemo-lens is installed (`uv sync --extra telemetry`). When either is missing every
instrumentation site in `nemo_gym` degrades to a \~0-cost no-op and Gym behaves
exactly as it does today.

**`enabled`** `bool = False`

Master switch. When false, all instrumentation is a \~0-cost no-op.

---

**`event_loop_sample_interval_seconds`** `float = Field(default=1.0, gt=0)`

Seconds between samples of each process's event-loop lag, when metrics are on.

---

**`event_loop_task_count`** `bool = False`

Also report how many asyncio tasks exist on each event loop, at every sample.

Off by default because counting tasks walks every task on the loop while holding it.
That takes about a millisecond per sample at 16,000 tasks, which a driver at high concurrency can reach.

---

**`exporter`** `str = 'otlp'`

Exporter backend: `otlp` | `console`. `console` writes one JSON object per
line to stdout, which is what the local validation flow greps.

---

**`logs_enabled`** `bool = False`

Bridge Python logging to OTel logs, exported with trace correlation.

---

**`memory_profiling`** `MemoryProfilingConfig = Field(default_factory=MemoryProfilingConfig)`

Sample each Gym-managed server process tree and emit memory metrics.

---

**`metrics_enabled`** `bool = True`

Emit metric instruments (the `gym.*` histograms/gauges plus the FastAPI
instrumentor's `http.server.*`).

---

**`otlp_endpoint`** `Optional[str] = None`

`OTEL_EXPORTER_OTLP_ENDPOINT` for the `otlp` exporter, e.g.
`https://api.honeycomb.io`. Optional: the standard env var still works and always
wins (`setdefault`), so any OTLP-compatible backend or collector that expects the
raw env var keeps working unchanged. Setting it here just means a run's destination is
committed with the rest of the config instead of exported by hand every time.

---

**`otlp_headers`** `Optional[str] = None`

`OTEL_EXPORTER_OTLP_HEADERS`, e.g. `x-honeycomb-team=...`. Prefer
`${oc.env:MY_API_KEY}` interpolation for the secret portion so the raw credential
does not live in a committed YAML file.

---

**`otlp_protocol`** `Optional[str] = None`

`OTEL_EXPORTER_OTLP_PROTOCOL`: `grpc` (the OTel SDK default) or
`http/protobuf`.

---

**`run_id`** `Optional[str] = None`

Correlates every process of one `gym env start` / `gym env test` invocation.
Generated in the supervisor and inherited by the servers when unset.

---

**`service_name`** `str = 'nemo-gym'`

`service.name` reported to the backend.

Passed through the standard `OTEL_SERVICE_NAME` env var, not through
`instrument_fastapi(service_name=...)` — that parameter is accepted and ignored by
nemo-lens at the pinned commit (`contrib/fastapi.py`).

Each server process appends its own Gym server name, so a run yields
`nemo-gym/example_single_tool_call`, `nemo-gym/policy_model`, and so on rather
than three indistinguishable `nemo-gym` services. Set `service_name_per_server`
to false to opt out.

---

**`service_name_per_server`** `bool = True`

Suffix `service_name` with each server's Gym config name.

On by default because Gym runs several processes that would otherwise share one
`service.name`, which makes a backend's service map useless. The unsuffixed name
stays available as the `nemo.gym.service_group` resource attribute.

---

**`span_groups`** `str = 'default'`

Span-group spec: a preset (`default` | `per_rollout` | `all`) or a
comma-separated list of group names (e.g. `"default,sandbox"`). See
`GymSpanGroup`. An unknown name is logged as a
warning and ignored rather than rejected.

---

**`traces_enabled`** `bool = True`

Emit trace spans.

---

```python
nemo_gym.telemetry.configure_telemetry_env(
    telemetry_config: typing.Union[nemo_gym.telemetry.config.TelemetryConfig, None]
) -> typing.Optional[str]
```

Translate the `telemetry:` block into env vars for spawned server processes.

Call once in the supervisor, **before** spawning any server. `os.environ` is what
`Popen` snapshots into each child, so this is how a YAML setting reaches a server
process that shares nothing else with its parent.

Uses `setdefault` throughout: a raw `NEMO_GYM_OTEL_*` / `NEMO_LENS_*` /
`OTEL_SERVICE_NAME` / `OTEL_EXPORTER_OTLP_*` set by the user always wins over YAML.

Returns the run id shared by every process in this run, or `None` when telemetry is
disabled.

```python
nemo_gym.telemetry.get_telemetry() -> typing.Optional[nemo.lens.TelemetryHandle]
```

Return this process's telemetry handle, or `None` if uninitialised/disabled.

```python
nemo_gym.telemetry.init_telemetry(
    server_name: typing.Optional[str] = None,
    server_type: typing.Optional[str] = None,
    resource_attributes: typing.Optional[dict] = None
) -> typing.Optional[nemo.lens.TelemetryHandle]
```

Initialise this process's telemetry. Call once per process; idempotent.

Reads the `NEMO_GYM_OTEL_*` environment that `configure_telemetry_env` put in
place, so a server process needs no config file access to agree with its siblings.

**Parameters:**

**`server_name`** `Optional[str]` — default: None

This server's Gym config name (e.g. `example_single_tool_call`).
Used to disambiguate `service.name` across the fleet.

---

**`server_type`** `Optional[str]` — default: None

`resources_servers` | `responses_api_agents` |
`responses_api_models`, or `supervisor` for the CLI.

---

**`resource_attributes`** `Optional[dict]` — default: None

Extra process-lifetime attributes to merge.

---

**Returns:** `Optional[TelemetryHandle]`

class:`TelemetryHandle`, or `None` when nemo-lens is absent or telemetry is

```python
nemo_gym.telemetry.is_span_group_enabled(
    group
)
```

Always returns False.

```python
nemo_gym.telemetry.is_telemetry_env_enabled() -> bool
```

True when telemetry is switched on in this process's environment.

Cheap enough to call before doing setup work, and importantly it does not import
nemo-lens — a process with telemetry off never pays for the lens import at all.

```python
nemo_gym.telemetry.is_telemetry_metrics_enabled() -> bool
```

Return the effective metrics switch after Gym/Lens env precedence.

```python
nemo_gym.telemetry.managed_span(
    group,
    name,
    tracer = None,
    attributes = {}
)
```

No-op context manager — yields None.

```python
nemo_gym.telemetry.memory_profiling_config_from_env(
    defaults: nemo_gym.telemetry.config.MemoryProfilingConfig
) -> nemo_gym.telemetry.config.MemoryProfilingConfig
```

Resolve memory profiling after raw environment variables take precedence.

```python
nemo_gym.telemetry.safe_set_span_attributes(
    span,
    attributes,
    redact_keys = None
)
```

No-op.

```python
nemo_gym.telemetry.shutdown_telemetry(
    timeout_ms: int = 5000
) -> None
```

Flush and shut down this process's telemetry providers.

Idempotent — `TelemetryHandle.shutdown` guards against a second call, and Gym
reaches this from more than one terminal path. A no-op when Gym reused providers that
another library set up. Never raises.

```python
nemo_gym.telemetry.span_cm(
    name,
    tracer = None,
    record_exception = True,
    attributes = {}
)
```

No-op context manager — yields None.

```python
nemo_gym.telemetry.telemetry_config_from_global_config(
    global_config_dict: typing.Any
) -> nemo_gym.telemetry.config.TelemetryConfig
```

Build a `TelemetryConfig` from Gym's merged global config dict.

Reads the optional top-level `telemetry:` block. A run that never mentions
telemetry gets the all-defaults config, which is disabled.

```python
nemo_gym.telemetry.trace_fn(
    group,
    name,
    tracer = None
)
```

No-op decorator — returns the function unchanged.

```python
nemo_gym.telemetry.setup.TELEMETRY_KEY_NAME = 'telemetry'
```