> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/gym/v0.2.1/nemo-gym/nemo_gym/telemetry/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server. # nemo_gym.telemetry NeMo Gym telemetry: optional OpenTelemetry instrumentation via nemo-lens. Importing this package never requires nemo-lens. Install it with `uv sync --extra telemetry` and switch it on with a `telemetry:` block or `NEMO_LENS_ENABLED=1`; with either missing, every instrumentation site in `nemo_gym` is a \~0-cost no-op. **Public surface** * `TelemetryConfig` — the `telemetry:` config block. * `GymSpanGroup` — Gym span groups and presets. * `configure_telemetry_env` — supervisor side; hands the settings to spawned server processes through the environment. * `init_telemetry` / `get_telemetry` / `shutdown_telemetry` — per-process lifecycle. Instrumentation primitives (`managed_span` / `span_cm` / `trace_fn` / `is_span_group_enabled` / `safe_set_span_attributes`) come from `nemo_gym.telemetry._fallbacks`, which resolves to the real nemo-lens implementations when it is installed and to no-op stubs when it is not. See `nemo_gym/telemetry/README.md` for the design, and `fern/versions/latest/pages/observability/` for user documentation. ## Submodules * **[`nemo_gym.telemetry.config`](/nemo/gym/nemo-gym/nemo_gym/telemetry/config)** * **[`nemo_gym.telemetry.connection_pool`](/nemo/gym/nemo-gym/nemo_gym/telemetry/connection_pool)** * **[`nemo_gym.telemetry.contrib`](/nemo/gym/nemo-gym/nemo_gym/telemetry/contrib)** * **[`nemo_gym.telemetry.endpoints`](/nemo/gym/nemo-gym/nemo_gym/telemetry/endpoints)** * **[`nemo_gym.telemetry.gym_metrics`](/nemo/gym/nemo-gym/nemo_gym/telemetry/gym_metrics)** * **[`nemo_gym.telemetry.memory`](/nemo/gym/nemo-gym/nemo_gym/telemetry/memory)** * **[`nemo_gym.telemetry.metrics`](/nemo/gym/nemo-gym/nemo_gym/telemetry/metrics)** * **[`nemo_gym.telemetry.observer`](/nemo/gym/nemo-gym/nemo_gym/telemetry/observer)** * **[`nemo_gym.telemetry.resources_operations`](/nemo/gym/nemo-gym/nemo_gym/telemetry/resources_operations)** * **[`nemo_gym.telemetry.setup`](/nemo/gym/nemo-gym/nemo_gym/telemetry/setup)** * **[`nemo_gym.telemetry.span_groups`](/nemo/gym/nemo-gym/nemo_gym/telemetry/span_groups)** * **[`nemo_gym.telemetry.spans`](/nemo/gym/nemo-gym/nemo_gym/telemetry/spans)** * **[`nemo_gym.telemetry.startup`](/nemo/gym/nemo-gym/nemo_gym/telemetry/startup)** ## Package Contents ### Classes | Name | Description | | --------------------------------------------------------------------------- | ----------------------------------------------------- | | [`GymSpanGroup`](#nemo_gym-telemetry-span_groups-GymSpanGroup) | Span group names for NeMo Gym instrumentation. | | [`MemoryProfilingConfig`](#nemo_gym-telemetry-config-MemoryProfilingConfig) | Periodic host and logical-server memory sampling. | | [`TelemetryConfig`](#nemo_gym-telemetry-config-TelemetryConfig) | OpenTelemetry / nemo-lens configuration for NeMo Gym. | ### Functions | Name | Description | | ------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------- | | [`configure_telemetry_env`](#nemo_gym-telemetry-setup-configure_telemetry_env) | Translate the `telemetry:` block into env vars for spawned server processes. | | [`get_telemetry`](#nemo_gym-telemetry-setup-get_telemetry) | Return this process's telemetry handle, or `None` if uninitialised/disabled. | | [`init_telemetry`](#nemo_gym-telemetry-setup-init_telemetry) | Initialise this process's telemetry. Call once per process; idempotent. | | [`is_span_group_enabled`](#nemo_gym-telemetry-_fallbacks-is_span_group_enabled) | Always returns False. | | [`is_telemetry_env_enabled`](#nemo_gym-telemetry-setup-is_telemetry_env_enabled) | True when telemetry is switched on in this process's environment. | | [`is_telemetry_metrics_enabled`](#nemo_gym-telemetry-setup-is_telemetry_metrics_enabled) | Return the effective metrics switch after Gym/Lens env precedence. | | [`managed_span`](#nemo_gym-telemetry-_fallbacks-managed_span) | No-op context manager — yields None. | | [`memory_profiling_config_from_env`](#nemo_gym-telemetry-setup-memory_profiling_config_from_env) | Resolve memory profiling after raw environment variables take precedence. | | [`safe_set_span_attributes`](#nemo_gym-telemetry-_fallbacks-safe_set_span_attributes) | No-op. | | [`shutdown_telemetry`](#nemo_gym-telemetry-setup-shutdown_telemetry) | Flush and shut down this process's telemetry providers. | | [`span_cm`](#nemo_gym-telemetry-_fallbacks-span_cm) | No-op context manager — yields None. | | [`telemetry_config_from_global_config`](#nemo_gym-telemetry-setup-telemetry_config_from_global_config) | Build a `TelemetryConfig` from Gym's merged global config dict. | | [`trace_fn`](#nemo_gym-telemetry-_fallbacks-trace_fn) | No-op decorator — returns the function unchanged. | ### Data [`TELEMETRY_KEY_NAME`](#nemo_gym-telemetry-setup-TELEMETRY_KEY_NAME) ### API ```python class nemo_gym.telemetry.GymSpanGroup() ``` Span group names for NeMo Gym instrumentation. **`AGENT`** `= 'agent'` Agent-server `/run` and `/v1/responses` spans. --- **`ALL_GROUPS`** `Final[frozenset]` --- **`CROSS_PROCESS_SPINE`** `Final[frozenset] = frozenset([SERVER, HTTP_CLIENT, ROLLOUT])` --- **`HTTP_CLIENT`** `= 'http_client'` Outbound spans around `nemo_gym.server_utils.request` — Gym's single aiohttp egress point. The egress half of cross-process propagation: injects `traceparent` into the outgoing headers. --- **`JOB`** `= 'job'` The whole `gym eval` / rollout-collection run, driver side. Shares its name with NeMo-RL's run-level group, so one spec entry selects both. --- **`MODEL_CALL`** `= 'model_call'` Model-server `/v1/chat/completions`, `/v1/responses` and `/v1/messages` spans. --- **`ROLLOUT`** `= 'rollout'` Rollout collection spans (one per task attempt, driver side). --- **`SANDBOX`** `= 'sandbox'` Sandbox provider create/exec/delete spans. --- **`SERVER`** `= 'server'` Inbound FastAPI request spans on every Gym server process. The ingress half of cross-process propagation: adopts an inbound `traceparent` as the span's parent. --- **`STARTUP`** `= 'startup'` Server startup spans, once per run. The supervisor emits `gym.startup` and one `gym.startup.server` span per server. Each server process emits `gym.server.startup` or `gym.server.startup.worker` with one child span per stage. --- **`VERIFY`** `= 'verify'` Resources-server `/verify` spans. --- **`_PRESETS`** `dict` --- ```python nemo_gym.telemetry.GymSpanGroup.resolve( spec: str ) -> frozenset ``` classmethod Resolve a `span_groups` spec against every group registered in this process. Entries that name nothing registered are dropped rather than raised: the library that owns them may not be imported in this process. nemo-lens logs a warning naming them when the spec is applied in `setup_telemetry`. **Raises:** * `RuntimeError`: nemo-lens is not installed, so there is nothing to resolve. ```python class nemo_gym.telemetry.MemoryProfilingConfig() ``` **Bases:** `BaseModel` Periodic host and logical-server memory sampling. **`enabled`** `bool = False` Collect host memory and every server process tree. --- **`interval_seconds`** `float = Field(default=1.0, gt=0)` Seconds between samples. --- ```python class nemo_gym.telemetry.TelemetryConfig() ``` **Bases:** `BaseModel` OpenTelemetry / nemo-lens configuration for NeMo Gym. Telemetry is optional twice over: it activates only when `enabled` is true *and* nemo-lens is installed (`uv sync --extra telemetry`). When either is missing every instrumentation site in `nemo_gym` degrades to a \~0-cost no-op and Gym behaves exactly as it does today. **`enabled`** `bool = False` Master switch. When false, all instrumentation is a \~0-cost no-op. --- **`event_loop_sample_interval_seconds`** `float = Field(default=1.0, gt=0)` Seconds between samples of each process's event-loop lag, when metrics are on. --- **`event_loop_task_count`** `bool = False` Also report how many asyncio tasks exist on each event loop, at every sample. Off by default because counting tasks walks every task on the loop while holding it. That takes about a millisecond per sample at 16,000 tasks, which a driver at high concurrency can reach. --- **`exporter`** `str = 'otlp'` Exporter backend: `otlp` | `console`. `console` writes one JSON object per line to stdout, which is what the local validation flow greps. --- **`logs_enabled`** `bool = False` Bridge Python logging to OTel logs, exported with trace correlation. --- **`memory_profiling`** `MemoryProfilingConfig = Field(default_factory=MemoryProfilingConfig)` Sample each Gym-managed server process tree and emit memory metrics. --- **`metrics_enabled`** `bool = True` Emit metric instruments (the `gym.*` histograms/gauges plus the FastAPI instrumentor's `http.server.*`). --- **`otlp_endpoint`** `Optional[str] = None` `OTEL_EXPORTER_OTLP_ENDPOINT` for the `otlp` exporter, e.g. `https://api.honeycomb.io`. Optional: the standard env var still works and always wins (`setdefault`), so any OTLP-compatible backend or collector that expects the raw env var keeps working unchanged. Setting it here just means a run's destination is committed with the rest of the config instead of exported by hand every time. --- **`otlp_headers`** `Optional[str] = None` `OTEL_EXPORTER_OTLP_HEADERS`, e.g. `x-honeycomb-team=...`. Prefer `${oc.env:MY_API_KEY}` interpolation for the secret portion so the raw credential does not live in a committed YAML file. --- **`otlp_protocol`** `Optional[str] = None` `OTEL_EXPORTER_OTLP_PROTOCOL`: `grpc` (the OTel SDK default) or `http/protobuf`. --- **`run_id`** `Optional[str] = None` Correlates every process of one `gym env start` / `gym env test` invocation. Generated in the supervisor and inherited by the servers when unset. --- **`service_name`** `str = 'nemo-gym'` `service.name` reported to the backend. Passed through the standard `OTEL_SERVICE_NAME` env var, not through `instrument_fastapi(service_name=...)` — that parameter is accepted and ignored by nemo-lens at the pinned commit (`contrib/fastapi.py`). Each server process appends its own Gym server name, so a run yields `nemo-gym/example_single_tool_call`, `nemo-gym/policy_model`, and so on rather than three indistinguishable `nemo-gym` services. Set `service_name_per_server` to false to opt out. --- **`service_name_per_server`** `bool = True` Suffix `service_name` with each server's Gym config name. On by default because Gym runs several processes that would otherwise share one `service.name`, which makes a backend's service map useless. The unsuffixed name stays available as the `nemo.gym.service_group` resource attribute. --- **`span_groups`** `str = 'default'` Span-group spec: a preset (`default` | `per_rollout` | `all`) or a comma-separated list of group names (e.g. `"default,sandbox"`). See `GymSpanGroup`. An unknown name is logged as a warning and ignored rather than rejected. --- **`traces_enabled`** `bool = True` Emit trace spans. --- ```python nemo_gym.telemetry.configure_telemetry_env( telemetry_config: typing.Union[nemo_gym.telemetry.config.TelemetryConfig, None] ) -> typing.Optional[str] ``` Translate the `telemetry:` block into env vars for spawned server processes. Call once in the supervisor, **before** spawning any server. `os.environ` is what `Popen` snapshots into each child, so this is how a YAML setting reaches a server process that shares nothing else with its parent. Uses `setdefault` throughout: a raw `NEMO_GYM_OTEL_*` / `NEMO_LENS_*` / `OTEL_SERVICE_NAME` / `OTEL_EXPORTER_OTLP_*` set by the user always wins over YAML. Returns the run id shared by every process in this run, or `None` when telemetry is disabled. ```python nemo_gym.telemetry.get_telemetry() -> typing.Optional[nemo.lens.TelemetryHandle] ``` Return this process's telemetry handle, or `None` if uninitialised/disabled. ```python nemo_gym.telemetry.init_telemetry( server_name: typing.Optional[str] = None, server_type: typing.Optional[str] = None, resource_attributes: typing.Optional[dict] = None ) -> typing.Optional[nemo.lens.TelemetryHandle] ``` Initialise this process's telemetry. Call once per process; idempotent. Reads the `NEMO_GYM_OTEL_*` environment that `configure_telemetry_env` put in place, so a server process needs no config file access to agree with its siblings. **Parameters:** **`server_name`** `Optional[str]` — default: None This server's Gym config name (e.g. `example_single_tool_call`). Used to disambiguate `service.name` across the fleet. --- **`server_type`** `Optional[str]` — default: None `resources_servers` | `responses_api_agents` | `responses_api_models`, or `supervisor` for the CLI. --- **`resource_attributes`** `Optional[dict]` — default: None Extra process-lifetime attributes to merge. --- **Returns:** `Optional[TelemetryHandle]` class:`TelemetryHandle`, or `None` when nemo-lens is absent or telemetry is ```python nemo_gym.telemetry.is_span_group_enabled( group ) ``` Always returns False. ```python nemo_gym.telemetry.is_telemetry_env_enabled() -> bool ``` True when telemetry is switched on in this process's environment. Cheap enough to call before doing setup work, and importantly it does not import nemo-lens — a process with telemetry off never pays for the lens import at all. ```python nemo_gym.telemetry.is_telemetry_metrics_enabled() -> bool ``` Return the effective metrics switch after Gym/Lens env precedence. ```python nemo_gym.telemetry.managed_span( group, name, tracer = None, attributes = {} ) ``` No-op context manager — yields None. ```python nemo_gym.telemetry.memory_profiling_config_from_env( defaults: nemo_gym.telemetry.config.MemoryProfilingConfig ) -> nemo_gym.telemetry.config.MemoryProfilingConfig ``` Resolve memory profiling after raw environment variables take precedence. ```python nemo_gym.telemetry.safe_set_span_attributes( span, attributes, redact_keys = None ) ``` No-op. ```python nemo_gym.telemetry.shutdown_telemetry( timeout_ms: int = 5000 ) -> None ``` Flush and shut down this process's telemetry providers. Idempotent — `TelemetryHandle.shutdown` guards against a second call, and Gym reaches this from more than one terminal path. A no-op when Gym reused providers that another library set up. Never raises. ```python nemo_gym.telemetry.span_cm( name, tracer = None, record_exception = True, attributes = {} ) ``` No-op context manager — yields None. ```python nemo_gym.telemetry.telemetry_config_from_global_config( global_config_dict: typing.Any ) -> nemo_gym.telemetry.config.TelemetryConfig ``` Build a `TelemetryConfig` from Gym's merged global config dict. Reads the optional top-level `telemetry:` block. A run that never mentions telemetry gets the all-defaults config, which is disabled. ```python nemo_gym.telemetry.trace_fn( group, name, tracer = None ) ``` No-op decorator — returns the function unchanged. ```python nemo_gym.telemetry.setup.TELEMETRY_KEY_NAME = 'telemetry' ``` ## Docs - [nemo_gym.telemetry.config](https://docs.nvidia.com/nemo/gym/v0.2.1/nemo-gym/nemo_gym/telemetry/config.md) - [nemo_gym.telemetry.connection_pool](https://docs.nvidia.com/nemo/gym/v0.2.1/nemo-gym/nemo_gym/telemetry/connection_pool.md) - [nemo_gym.telemetry.contrib](https://docs.nvidia.com/nemo/gym/v0.2.1/nemo-gym/nemo_gym/telemetry/contrib.md) - [nemo_gym.telemetry.endpoints](https://docs.nvidia.com/nemo/gym/v0.2.1/nemo-gym/nemo_gym/telemetry/endpoints.md) - [nemo_gym.telemetry.gym_metrics](https://docs.nvidia.com/nemo/gym/v0.2.1/nemo-gym/nemo_gym/telemetry/gym_metrics.md) - [nemo_gym.telemetry.memory](https://docs.nvidia.com/nemo/gym/v0.2.1/nemo-gym/nemo_gym/telemetry/memory.md) - [nemo_gym.telemetry.metrics](https://docs.nvidia.com/nemo/gym/v0.2.1/nemo-gym/nemo_gym/telemetry/metrics.md) - [nemo_gym.telemetry.observer](https://docs.nvidia.com/nemo/gym/v0.2.1/nemo-gym/nemo_gym/telemetry/observer.md) - [nemo_gym.telemetry.resources_operations](https://docs.nvidia.com/nemo/gym/v0.2.1/nemo-gym/nemo_gym/telemetry/resources_operations.md) - [nemo_gym.telemetry.setup](https://docs.nvidia.com/nemo/gym/v0.2.1/nemo-gym/nemo_gym/telemetry/setup.md) - [nemo_gym.telemetry.span_groups](https://docs.nvidia.com/nemo/gym/v0.2.1/nemo-gym/nemo_gym/telemetry/span_groups.md) - [nemo_gym.telemetry.spans](https://docs.nvidia.com/nemo/gym/v0.2.1/nemo-gym/nemo_gym/telemetry/spans.md) - [nemo_gym.telemetry.startup](https://docs.nvidia.com/nemo/gym/v0.2.1/nemo-gym/nemo_gym/telemetry/startup.md)