> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server.

# nemo_gym.telemetry

NeMo Gym telemetry: optional OpenTelemetry instrumentation via nemo-lens.

Importing this package never requires nemo-lens. Install it with
`uv sync --extra telemetry` and switch it on with a `telemetry:` block or
`NEMO_LENS_ENABLED=1`; with either missing, every instrumentation site in `nemo_gym`
is a \~0-cost no-op.

**Public surface**

* `TelemetryConfig` — the `telemetry:` config block.
* `GymSpanGroup` — Gym span groups and presets.
* `configure_telemetry_env` — orchestrator side; hands
  the settings to spawned server processes through the environment.
* `init_telemetry` /
  `get_telemetry` /
  `shutdown_telemetry` — per-process lifecycle.

Instrumentation primitives (`managed_span` / `span_cm` / `trace_fn` /
`is_span_group_enabled` / `safe_set_span_attributes`) come from
`nemo_gym.telemetry._fallbacks`, which resolves to the real nemo-lens
implementations when it is installed and to no-op stubs when it is not.

See `nemo_gym/telemetry/README.md` for the design, and
`fern/versions/latest/pages/observability/` for user documentation.

## Submodules

* **[`nemo_gym.telemetry.config`](/nemo/gym/nemo-gym/nemo_gym/telemetry/config)**
* **[`nemo_gym.telemetry.connection_pool`](/nemo/gym/nemo-gym/nemo_gym/telemetry/connection_pool)**
* **[`nemo_gym.telemetry.contrib`](/nemo/gym/nemo-gym/nemo_gym/telemetry/contrib)**
* **[`nemo_gym.telemetry.endpoints`](/nemo/gym/nemo-gym/nemo_gym/telemetry/endpoints)**
* **[`nemo_gym.telemetry.gym_metrics`](/nemo/gym/nemo-gym/nemo_gym/telemetry/gym_metrics)**
* **[`nemo_gym.telemetry.memory`](/nemo/gym/nemo-gym/nemo_gym/telemetry/memory)**
* **[`nemo_gym.telemetry.metrics`](/nemo/gym/nemo-gym/nemo_gym/telemetry/metrics)**
* **[`nemo_gym.telemetry.setup`](/nemo/gym/nemo-gym/nemo_gym/telemetry/setup)**
* **[`nemo_gym.telemetry.span_groups`](/nemo/gym/nemo-gym/nemo_gym/telemetry/span_groups)**
* **[`nemo_gym.telemetry.spans`](/nemo/gym/nemo-gym/nemo_gym/telemetry/spans)**

## Package Contents

### Classes

| Name                                                                        | Description                                           |
| --------------------------------------------------------------------------- | ----------------------------------------------------- |
| [`GymSpanGroup`](#nemo_gym-telemetry-span_groups-GymSpanGroup)              | Span group names for NeMo Gym instrumentation.        |
| [`MemoryProfilingConfig`](#nemo_gym-telemetry-config-MemoryProfilingConfig) | Periodic host and logical-server memory sampling.     |
| [`TelemetryConfig`](#nemo_gym-telemetry-config-TelemetryConfig)             | OpenTelemetry / nemo-lens configuration for NeMo Gym. |

### Functions

| Name                                                                                                   | Description                                                                  |
| ------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------- |
| [`configure_telemetry_env`](#nemo_gym-telemetry-setup-configure_telemetry_env)                         | Translate the `telemetry:` block into env vars for spawned server processes. |
| [`get_telemetry`](#nemo_gym-telemetry-setup-get_telemetry)                                             | Return this process's telemetry handle, or `None` if uninitialised/disabled. |
| [`init_telemetry`](#nemo_gym-telemetry-setup-init_telemetry)                                           | Initialise this process's telemetry. Call once per process; idempotent.      |
| [`is_span_group_enabled`](#nemo_gym-telemetry-_fallbacks-is_span_group_enabled)                        | Always returns False.                                                        |
| [`is_telemetry_env_enabled`](#nemo_gym-telemetry-setup-is_telemetry_env_enabled)                       | True when telemetry is switched on in this process's environment.            |
| [`is_telemetry_metrics_enabled`](#nemo_gym-telemetry-setup-is_telemetry_metrics_enabled)               | Return the effective metrics switch after Gym/Lens env precedence.           |
| [`managed_span`](#nemo_gym-telemetry-_fallbacks-managed_span)                                          | No-op context manager — yields None.                                         |
| [`memory_profiling_config_from_env`](#nemo_gym-telemetry-setup-memory_profiling_config_from_env)       | Resolve memory profiling after raw environment variables take precedence.    |
| [`safe_set_span_attributes`](#nemo_gym-telemetry-_fallbacks-safe_set_span_attributes)                  | No-op.                                                                       |
| [`shutdown_telemetry`](#nemo_gym-telemetry-setup-shutdown_telemetry)                                   | Flush and shut down this process's telemetry providers.                      |
| [`span_cm`](#nemo_gym-telemetry-_fallbacks-span_cm)                                                    | No-op context manager — yields None.                                         |
| [`telemetry_config_from_global_config`](#nemo_gym-telemetry-setup-telemetry_config_from_global_config) | Build a `TelemetryConfig` from Gym's merged global config dict.              |
| [`trace_fn`](#nemo_gym-telemetry-_fallbacks-trace_fn)                                                  | No-op decorator — returns the function unchanged.                            |

### Data

[`TELEMETRY_KEY_NAME`](#nemo_gym-telemetry-setup-TELEMETRY_KEY_NAME)

### API

```python
class nemo_gym.telemetry.GymSpanGroup()
```

Span group names for NeMo Gym instrumentation.

**`AGENT`** `= 'agent'`

Agent-server `/run` and `/v1/responses` spans.

---

**`ALL_GROUPS`** `Final[frozenset]`

---

**`CROSS_PROCESS_SPINE`** `Final[frozenset] = frozenset([SERVER, HTTP_CLIENT, ROLLOUT])`

---

**`HTTP_CLIENT`** `= 'http_client'`

Outbound spans around `nemo_gym.server_utils.request` — Gym's single aiohttp
egress point. The egress half of cross-process propagation: injects `traceparent`
into the outgoing headers.

---

**`JOB`** `= 'job'`

The whole `gym eval` / rollout-collection run, driver side. Shares its name with
NeMo-RL's run-level group, so one spec entry selects both.

---

**`MODEL_CALL`** `= 'model_call'`

Model-server `/v1/chat/completions`, `/v1/responses` and `/v1/messages` spans.

---

**`ROLLOUT`** `= 'rollout'`

Rollout collection spans (one per task attempt, driver side).

---

**`SANDBOX`** `= 'sandbox'`

Sandbox provider create/exec/delete spans.

---

**`SERVER`** `= 'server'`

Inbound FastAPI request spans on every Gym server process. The ingress half of
cross-process propagation: adopts an inbound `traceparent` as the span's parent.

---

**`VERIFY`** `= 'verify'`

Resources-server `/verify` spans.

---

**`_PRESETS`** `dict`

---

```python
nemo_gym.telemetry.GymSpanGroup.resolve(
    spec: str
) -> frozenset
```

classmethod

Resolve a `span_groups` spec against every group registered in this process.

Entries that name nothing registered are dropped rather than raised: the library
that owns them may not be imported in this process. nemo-lens logs a warning naming
them when the spec is applied in `setup_telemetry`.

**Raises:**

* `RuntimeError`: nemo-lens is not installed, so there is nothing to resolve.

```python
class nemo_gym.telemetry.MemoryProfilingConfig()
```

**Bases:** `BaseModel`

Periodic host and logical-server memory sampling.

**`enabled`** `bool = False`

Collect host memory and every server process tree.

---

**`interval_seconds`** `float = Field(default=1.0, gt=0)`

Seconds between samples.

---

```python
class nemo_gym.telemetry.TelemetryConfig()
```

**Bases:** `BaseModel`

OpenTelemetry / nemo-lens configuration for NeMo Gym.

Telemetry is optional twice over: it activates only when `enabled` is true *and*
nemo-lens is installed (`uv sync --extra telemetry`). When either is missing every
instrumentation site in `nemo_gym` degrades to a \~0-cost no-op and Gym behaves
exactly as it does today.

**`enabled`** `bool = False`

Master switch. When false, all instrumentation is a \~0-cost no-op.

---

**`exporter`** `str = 'otlp'`

Exporter backend: `otlp` | `console`. `console` writes one JSON object per
line to stdout, which is what the local validation flow greps.

---

**`instrument_aiohttp`** `bool = True`

Use nemo-lens's aiohttp auto-instrumentation for outbound calls, which produces a
CLIENT span per request in addition to injecting `traceparent`.

When false, `nemo_gym.server_utils.request` still injects `traceparent` manually,
so cross-process traces stay joined — you lose the client-side span, not the trace.
Turn it off if the per-request patching cost matters at very high concurrency.

---

**`logs_enabled`** `bool = False`

Bridge Python logging to OTel logs, exported with trace correlation.

---

**`memory_profiling`** `MemoryProfilingConfig = Field(default_factory=MemoryProfilingConfig)`

Sample each Gym-managed server process tree and emit memory metrics.

---

**`metrics_enabled`** `bool = True`

Emit metric instruments (the `gym.*` histograms/gauges plus the FastAPI
instrumentor's `http.server.*`).

---

**`otlp_endpoint`** `Optional[str] = None`

`OTEL_EXPORTER_OTLP_ENDPOINT` for the `otlp` exporter, e.g.
`https://api.honeycomb.io`. Optional: the standard env var still works and always
wins (`setdefault`), so any OTLP-compatible backend or collector that expects the
raw env var keeps working unchanged. Setting it here just means a run's destination is
committed with the rest of the config instead of exported by hand every time.

---

**`otlp_headers`** `Optional[str] = None`

`OTEL_EXPORTER_OTLP_HEADERS`, e.g. `x-honeycomb-team=...`. Prefer
`${oc.env:MY_API_KEY}` interpolation for the secret portion so the raw credential
does not live in a committed YAML file.

---

**`otlp_protocol`** `Optional[str] = None`

`OTEL_EXPORTER_OTLP_PROTOCOL`: `grpc` (the OTel SDK default) or
`http/protobuf`.

---

**`run_id`** `Optional[str] = None`

Correlates every process of one `gym env start` / `gym env test` invocation.
Generated in the orchestrator and inherited by the servers when unset.

---

**`service_name`** `str = 'nemo-gym'`

`service.name` reported to the backend.

Passed through the standard `OTEL_SERVICE_NAME` env var, not through
`instrument_fastapi(service_name=...)` — that parameter is accepted and ignored by
nemo-lens at the pinned commit (`contrib/fastapi.py`).

Each server process appends its own Gym server name, so a run yields
`nemo-gym/example_single_tool_call`, `nemo-gym/policy_model`, and so on rather
than three indistinguishable `nemo-gym` services. Set `service_name_per_server`
to false to opt out.

---

**`service_name_per_server`** `bool = True`

Suffix `service_name` with each server's Gym config name.

On by default because Gym runs several processes that would otherwise share one
`service.name`, which makes a backend's service map useless. The unsuffixed name
stays available as the `nemo.gym.service_group` resource attribute.

---

**`span_groups`** `str = 'default'`

Span-group spec: a preset (`default` | `per_rollout` | `all`) or a
comma-separated list of group names (e.g. `"default,sandbox"`). See
`GymSpanGroup`. An unknown name is logged as a
warning and ignored rather than rejected.

---

**`traces_enabled`** `bool = True`

Emit trace spans.

---

```python
nemo_gym.telemetry.configure_telemetry_env(
    telemetry_config: typing.Union[nemo_gym.telemetry.config.TelemetryConfig, None]
) -> typing.Optional[str]
```

Translate the `telemetry:` block into env vars for spawned server processes.

Call once in the orchestrator, **before** spawning any server. `os.environ` is what
`Popen` snapshots into each child, so this is how a YAML setting reaches a server
process that shares nothing else with its parent.

Uses `setdefault` throughout: a raw `NEMO_GYM_OTEL_*` / `NEMO_LENS_*` /
`OTEL_SERVICE_NAME` / `OTEL_EXPORTER_OTLP_*` set by the user always wins over YAML.

Returns the run id shared by every process in this run, or `None` when telemetry is
disabled.

```python
nemo_gym.telemetry.get_telemetry() -> typing.Optional[nemo.lens.TelemetryHandle]
```

Return this process's telemetry handle, or `None` if uninitialised/disabled.

```python
nemo_gym.telemetry.init_telemetry(
    server_name: typing.Optional[str] = None,
    server_type: typing.Optional[str] = None,
    resource_attributes: typing.Optional[dict] = None
) -> typing.Optional[nemo.lens.TelemetryHandle]
```

Initialise this process's telemetry. Call once per process; idempotent.

Reads the `NEMO_GYM_OTEL_*` environment that `configure_telemetry_env` put in
place, so a server process needs no config file access to agree with its siblings.

**Parameters:**

**`server_name`** `Optional[str]` — default: None

This server's Gym config name (e.g. `example_single_tool_call`).
Used to disambiguate `service.name` across the fleet.

---

**`server_type`** `Optional[str]` — default: None

`resources_servers` | `responses_api_agents` |
`responses_api_models`, or `orchestrator` for the CLI.

---

**`resource_attributes`** `Optional[dict]` — default: None

Extra process-lifetime attributes to merge.

---

**Returns:** `Optional[TelemetryHandle]`

class:`TelemetryHandle`, or `None` when nemo-lens is absent or telemetry is

```python
nemo_gym.telemetry.is_span_group_enabled(
    group
)
```

Always returns False.

```python
nemo_gym.telemetry.is_telemetry_env_enabled() -> bool
```

True when telemetry is switched on in this process's environment.

Cheap enough to call before doing setup work, and importantly it does not import
nemo-lens — a process with telemetry off never pays for the lens import at all.

```python
nemo_gym.telemetry.is_telemetry_metrics_enabled() -> bool
```

Return the effective metrics switch after Gym/Lens env precedence.

```python
nemo_gym.telemetry.managed_span(
    group,
    name,
    tracer = None,
    attributes = {}
)
```

No-op context manager — yields None.

```python
nemo_gym.telemetry.memory_profiling_config_from_env(
    defaults: nemo_gym.telemetry.config.MemoryProfilingConfig
) -> nemo_gym.telemetry.config.MemoryProfilingConfig
```

Resolve memory profiling after raw environment variables take precedence.

```python
nemo_gym.telemetry.safe_set_span_attributes(
    span,
    attributes,
    redact_keys = None
)
```

No-op.

```python
nemo_gym.telemetry.shutdown_telemetry(
    timeout_ms: int = 5000
) -> None
```

Flush and shut down this process's telemetry providers.

Idempotent — `TelemetryHandle.shutdown` guards against a second call, and Gym
reaches this from more than one terminal path. A no-op when Gym reused providers that
another library set up. Never raises.

```python
nemo_gym.telemetry.span_cm(
    name,
    tracer = None,
    record_exception = True,
    attributes = {}
)
```

No-op context manager — yields None.

```python
nemo_gym.telemetry.telemetry_config_from_global_config(
    global_config_dict: typing.Any
) -> nemo_gym.telemetry.config.TelemetryConfig
```

Build a `TelemetryConfig` from Gym's merged global config dict.

Reads the optional top-level `telemetry:` block. A run that never mentions
telemetry gets the all-defaults config, which is disabled.

```python
nemo_gym.telemetry.trace_fn(
    group,
    name,
    tracer = None
)
```

No-op decorator — returns the function unchanged.

```python
nemo_gym.telemetry.setup.TELEMETRY_KEY_NAME = 'telemetry'
```