> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server.

# nemo_gym.telemetry.setup

Process-global nemo-lens telemetry lifecycle for NeMo Gym.

Gym's process model is the thing this module exists to handle. Megatron-LM is one
process tree and NeMo-RL is a Ray driver plus actors, but a Gym run is **N independent
FastAPI processes** — a resources server, a model server, an agent server, sometimes more
— spawned by a CLI orchestrator via `Popen`, each with its own interpreter. There is no
shared memory and no parent handle to inherit: the only thing that crosses the boundary
is the environment.

So there are two entry points:

* `configure_telemetry_env` — called once in the **orchestrator** (`gym env
  start` / `gym env test`) before any server is spawned. It translates the
  `telemetry:` config block into `NEMO_GYM_OTEL_*` env vars with `setdefault`, so
  every server process it spawns resolves the same settings, and any env var the user set
  by hand still wins.
* `init_telemetry` — called once inside **each** server process, from
  `SimpleServer.run_webserver`. It reads that propagated environment and builds this
  process's providers.

nemo-lens allows one `setup_telemetry` per process. When something else in the process
already called it — NeMo-RL, when Gym runs inside an RL job — `init_telemetry`
reuses those providers instead of failing, and leaves their shutdown to their owner.

Importing this module never requires nemo-lens: every lens import is function-local and
guarded. With lens absent, or `enabled: false`, the init functions return `None` and
every instrumentation site stays a no-op through `nemo_gym.telemetry._fallbacks`.

## Module Contents

### Functions

| Name                                                                                                   | Description                                                                           |
| ------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------- |
| [`_build_resource_attributes`](#nemo_gym-telemetry-setup-_build_resource_attributes)                   | Process-lifetime resource attributes.                                                 |
| [`_env_flag`](#nemo_gym-telemetry-setup-_env_flag)                                                     | Read a Gym-owned boolean env var, tolerating an unset or blank value.                 |
| [`_installed_requirement`](#nemo_gym-telemetry-setup-_installed_requirement)                           | Build a requirement string pinning *dist\_name* to the copy installed right here.     |
| [`_reset_for_testing`](#nemo_gym-telemetry-setup-_reset_for_testing)                                   | Drop the process-global handle so a test can initialise again.                        |
| [`_sdk_meter_provider_installed`](#nemo_gym-telemetry-setup-_sdk_meter_provider_installed)             | Whether a real OpenTelemetry SDK meter provider, not a no-op or proxy, is registered. |
| [`configure_telemetry_env`](#nemo_gym-telemetry-setup-configure_telemetry_env)                         | Translate the `telemetry:` block into env vars for spawned server processes.          |
| [`get_telemetry`](#nemo_gym-telemetry-setup-get_telemetry)                                             | Return this process's telemetry handle, or `None` if uninitialised/disabled.          |
| [`init_telemetry`](#nemo_gym-telemetry-setup-init_telemetry)                                           | Initialise this process's telemetry. Call once per process; idempotent.               |
| [`is_metrics_exporter_active`](#nemo_gym-telemetry-setup-is_metrics_exporter_active)                   | Whether this process has an active metrics exporter for Gym to record into.           |
| [`is_telemetry_env_enabled`](#nemo_gym-telemetry-setup-is_telemetry_env_enabled)                       | True when telemetry is switched on in this process's environment.                     |
| [`is_telemetry_metrics_enabled`](#nemo_gym-telemetry-setup-is_telemetry_metrics_enabled)               | Return the effective metrics switch after Gym/Lens env precedence.                    |
| [`memory_profiling_config_from_env`](#nemo_gym-telemetry-setup-memory_profiling_config_from_env)       | Resolve memory profiling after raw environment variables take precedence.             |
| [`server_venv_requirements`](#nemo_gym-telemetry-setup-server_venv_requirements)                       | Extra requirements to install into every per-server venv.                             |
| [`shutdown_telemetry`](#nemo_gym-telemetry-setup-shutdown_telemetry)                                   | Flush and shut down this process's telemetry providers.                               |
| [`telemetry_config_from_global_config`](#nemo_gym-telemetry-setup-telemetry_config_from_global_config) | Build a `TelemetryConfig` from Gym's merged global config dict.                       |

### Data

[`TELEMETRY_KEY_NAME`](#nemo_gym-telemetry-setup-TELEMETRY_KEY_NAME)

[`_ENV_FIELD_MAP`](#nemo_gym-telemetry-setup-_ENV_FIELD_MAP)

[`_INITIALISED`](#nemo_gym-telemetry-setup-_INITIALISED)

[`_INIT_LOCK`](#nemo_gym-telemetry-setup-_INIT_LOCK)

[`_MEMORY_PROFILING_ENV_FIELD_MAP`](#nemo_gym-telemetry-setup-_MEMORY_PROFILING_ENV_FIELD_MAP)

[`_METRICS_EXPORTING`](#nemo_gym-telemetry-setup-_METRICS_EXPORTING)

[`_OTEL_FALLBACK_PREFIX`](#nemo_gym-telemetry-setup-_OTEL_FALLBACK_PREFIX)

[`_OTEL_PREFIX`](#nemo_gym-telemetry-setup-_OTEL_PREFIX)

[`_OTLP_ENV_FIELD_MAP`](#nemo_gym-telemetry-setup-_OTLP_ENV_FIELD_MAP)

[`_OWNS_PROVIDERS`](#nemo_gym-telemetry-setup-_OWNS_PROVIDERS)

[`_SERVER_TELEMETRY_PACKAGES`](#nemo_gym-telemetry-setup-_SERVER_TELEMETRY_PACKAGES)

[`_SERVICE_NAME_ENV`](#nemo_gym-telemetry-setup-_SERVICE_NAME_ENV)

[`_TELEMETRY_HANDLE`](#nemo_gym-telemetry-setup-_TELEMETRY_HANDLE)

[`_TRUTHY`](#nemo_gym-telemetry-setup-_TRUTHY)

[`logger`](#nemo_gym-telemetry-setup-logger)

### API

```python
nemo_gym.telemetry.setup._build_resource_attributes(
    server_name: typing.Optional[str],
    server_type: typing.Optional[str]
) -> dict
```

Process-lifetime resource attributes.

Only values constant for this process's whole life belong here — anything that varies
per request is a span attribute instead
(`kb/knowledge/conventions/telemetry-classification.md`).

```python
nemo_gym.telemetry.setup._env_flag(
    name: str,
    default: bool
) -> bool
```

Read a Gym-owned boolean env var, tolerating an unset or blank value.

```python
nemo_gym.telemetry.setup._installed_requirement(
    dist_name: str,
    requirement_name: str
) -> typing.Optional[str]
```

Build a requirement string pinning *dist\_name* to the copy installed right here.

Prefers the recorded install source over the version number. A git-installed
nemo-lens reports a local version such as `0.2.0+b85578f` that exists on no index,
so `==` would be unsatisfiable; the recorded VCS URL and commit reinstall exactly
what this process is running.

```python
nemo_gym.telemetry.setup._reset_for_testing() -> None
```

Drop the process-global handle so a test can initialise again.

Test-only. Production code has exactly one init per process, which is what
`_INITIALISED` enforces.

```python
nemo_gym.telemetry.setup._sdk_meter_provider_installed() -> bool
```

Whether a real OpenTelemetry SDK meter provider, not a no-op or proxy, is registered.

```python
nemo_gym.telemetry.configure_telemetry_env(
    telemetry_config: typing.Union[nemo_gym.telemetry.config.TelemetryConfig, None]
) -> typing.Optional[str]
```

Translate the `telemetry:` block into env vars for spawned server processes.

Call once in the orchestrator, **before** spawning any server. `os.environ` is what
`Popen` snapshots into each child, so this is how a YAML setting reaches a server
process that shares nothing else with its parent.

Uses `setdefault` throughout: a raw `NEMO_GYM_OTEL_*` / `NEMO_LENS_*` /
`OTEL_SERVICE_NAME` / `OTEL_EXPORTER_OTLP_*` set by the user always wins over YAML.

Returns the run id shared by every process in this run, or `None` when telemetry is
disabled.

```python
nemo_gym.telemetry.get_telemetry() -> typing.Optional[nemo.lens.TelemetryHandle]
```

Return this process's telemetry handle, or `None` if uninitialised/disabled.

```python
nemo_gym.telemetry.init_telemetry(
    server_name: typing.Optional[str] = None,
    server_type: typing.Optional[str] = None,
    resource_attributes: typing.Optional[dict] = None
) -> typing.Optional[nemo.lens.TelemetryHandle]
```

Initialise this process's telemetry. Call once per process; idempotent.

Reads the `NEMO_GYM_OTEL_*` environment that `configure_telemetry_env` put in
place, so a server process needs no config file access to agree with its siblings.

**Parameters:**

**`server_name`** `Optional[str]` — default: None

This server's Gym config name (e.g. `example_single_tool_call`).
Used to disambiguate `service.name` across the fleet.

---

**`server_type`** `Optional[str]` — default: None

`resources_servers` | `responses_api_agents` |
`responses_api_models`, or `orchestrator` for the CLI.

---

**`resource_attributes`** `Optional[dict]` — default: None

Extra process-lifetime attributes to merge.

---

**Returns:** `Optional[TelemetryHandle]`

class:`TelemetryHandle`, or `None` when nemo-lens is absent or telemetry is

```python
nemo_gym.telemetry.setup.is_metrics_exporter_active() -> bool
```

Whether this process has an active metrics exporter for Gym to record into.

True after Gym's own telemetry setup succeeds with metrics enabled, or after Gym reuses
providers that another library set up with metrics enabled. Unlike
`is_telemetry_metrics_enabled`, this stays false until telemetry is set up.

```python
nemo_gym.telemetry.is_telemetry_env_enabled() -> bool
```

True when telemetry is switched on in this process's environment.

Cheap enough to call before doing setup work, and importantly it does not import
nemo-lens — a process with telemetry off never pays for the lens import at all.

```python
nemo_gym.telemetry.is_telemetry_metrics_enabled() -> bool
```

Return the effective metrics switch after Gym/Lens env precedence.

```python
nemo_gym.telemetry.memory_profiling_config_from_env(
    defaults: nemo_gym.telemetry.config.MemoryProfilingConfig
) -> nemo_gym.telemetry.config.MemoryProfilingConfig
```

Resolve memory profiling after raw environment variables take precedence.

```python
nemo_gym.telemetry.setup.server_venv_requirements() -> list
```

Extra requirements to install into every per-server venv.

Gym builds an isolated venv per server, and those venvs install `nemo-gym[dev]` —
not `nemo-gym[telemetry]`. Without this, telemetry would be enabled in the
orchestrator and simply absent in every server process, which is the one
configuration that produces a trace with a hole in the middle of it.

Pinning is derived from what *this* process has installed rather than restated here.
A `[tool.uv.sources]` entry only governs dependencies resolved through the local
project: a bare `nemo-lens[sdk]` passed to `uv pip install` ignores it and
resolves from PyPI, which would put lens 0.1.0 in the servers while the orchestrator
runs the pinned commit. Reading the installed distribution's `direct_url.json`
makes that skew impossible by construction instead of by keeping two pins in sync.

Returns an empty list when telemetry is off or nemo-lens is not installed, so a
normal run's venvs are byte-for-byte what they are today.

```python
nemo_gym.telemetry.shutdown_telemetry(
    timeout_ms: int = 5000
) -> None
```

Flush and shut down this process's telemetry providers.

Idempotent — `TelemetryHandle.shutdown` guards against a second call, and Gym
reaches this from more than one terminal path. A no-op when Gym reused providers that
another library set up. Never raises.

```python
nemo_gym.telemetry.telemetry_config_from_global_config(
    global_config_dict: typing.Any
) -> nemo_gym.telemetry.config.TelemetryConfig
```

Build a `TelemetryConfig` from Gym's merged global config dict.

Reads the optional top-level `telemetry:` block. A run that never mentions
telemetry gets the all-defaults config, which is disabled.

```python
nemo_gym.telemetry.setup.TELEMETRY_KEY_NAME = 'telemetry'
```

```python
nemo_gym.telemetry.setup._ENV_FIELD_MAP = {'enabled': f'{_OTEL_PREFIX}_ENABLED', 'span_groups': f'{_OTEL_PREFIX}_SPAN_GROU...
```

```python
nemo_gym.telemetry.setup._INITIALISED = False
```

```python
nemo_gym.telemetry.setup._INIT_LOCK = threading.Lock()
```

```python
nemo_gym.telemetry.setup._MEMORY_PROFILING_ENV_FIELD_MAP = {'enabled': f'{_OTEL_PREFIX}_MEMORY_PROFILING_ENABLED', 'interval_seconds': f'{_...
```

```python
nemo_gym.telemetry.setup._METRICS_EXPORTING = False
```

```python
nemo_gym.telemetry.setup._OTEL_FALLBACK_PREFIX = 'NEMO_LENS'
```

```python
nemo_gym.telemetry.setup._OTEL_PREFIX = 'NEMO_GYM_OTEL'
```

```python
nemo_gym.telemetry.setup._OTLP_ENV_FIELD_MAP = {'otlp_endpoint': 'OTEL_EXPORTER_OTLP_ENDPOINT', 'otlp_protocol': 'OTEL_EXPORTER...
```

```python
nemo_gym.telemetry.setup._OWNS_PROVIDERS = False
```

```python
nemo_gym.telemetry.setup._SERVER_TELEMETRY_PACKAGES = (('nemo-lens', 'nemo-lens[sdk]'), ('opentelemetry-instrumentation-fastapi', 'ope...
```

```python
nemo_gym.telemetry.setup._SERVICE_NAME_ENV = 'OTEL_SERVICE_NAME'
```

```python
nemo_gym.telemetry.setup._TELEMETRY_HANDLE: Optional[TelemetryHandle] = None
```

```python
nemo_gym.telemetry.setup._TRUTHY = ('1', 'true', 'yes', 'on')
```

```python
nemo_gym.telemetry.setup.logger = logging.getLogger(__name__)
```