> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server. # nemo_gym.telemetry.setup Process-global nemo-lens telemetry lifecycle for NeMo Gym. Gym's process model is the thing this module exists to handle. Megatron-LM is one process tree and NeMo-RL is a Ray driver plus actors, but a Gym run is **N independent FastAPI processes** — a resources server, a model server, an agent server, sometimes more — spawned by a CLI orchestrator via `Popen`, each with its own interpreter. There is no shared memory and no parent handle to inherit: the only thing that crosses the boundary is the environment. So there are two entry points: * :func:`configure_telemetry_env` — called once in the **orchestrator** (`gym env start` / `gym env test`) before any server is spawned. It translates the `telemetry:` config block into `NEMO_GYM_OTEL_*` env vars with `setdefault`, so every server process it spawns resolves the same settings, and any env var the user set by hand still wins. * :func:`init_telemetry` — called once inside **each** server process, from `SimpleServer.run_webserver`. It reads that propagated environment and builds this process's providers. `export_strategy` defaults to `all_ranks` here rather than `single_rank`. Each Gym server is rank 0 of its own world of 1, so a rank-based filter would either silence every process or none; and silencing any one of them puts a hole in the middle of the distributed trace this integration exists to produce. Importing this module never requires nemo-lens: every lens import is function-local and guarded. With lens absent, or `enabled: false`, the init functions return `None` and every instrumentation site stays a no-op through :mod:`nemo_gym.telemetry._fallbacks`. ## Module Contents ### Functions | Name | Description | | ------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------- | | [`_build_resource_attributes`](#nemo_gym-telemetry-setup-_build_resource_attributes) | Process-lifetime resource attributes. | | [`_env_flag`](#nemo_gym-telemetry-setup-_env_flag) | Read a Gym-owned boolean env var, tolerating an unset or blank value. | | [`_installed_requirement`](#nemo_gym-telemetry-setup-_installed_requirement) | Build a requirement string pinning *dist\_name* to the copy installed right here. | | [`_reset_for_testing`](#nemo_gym-telemetry-setup-_reset_for_testing) | Drop the process-global handle so a test can initialise again. | | [`configure_telemetry_env`](#nemo_gym-telemetry-setup-configure_telemetry_env) | Translate the `telemetry:` block into env vars for spawned server processes. | | [`get_telemetry`](#nemo_gym-telemetry-setup-get_telemetry) | Return this process's telemetry handle, or `None` if uninitialised/disabled. | | [`init_telemetry`](#nemo_gym-telemetry-setup-init_telemetry) | Initialise this process's telemetry. Call once per process; idempotent. | | [`is_telemetry_env_enabled`](#nemo_gym-telemetry-setup-is_telemetry_env_enabled) | True when telemetry is switched on in this process's environment. | | [`server_venv_requirements`](#nemo_gym-telemetry-setup-server_venv_requirements) | Extra requirements to install into every per-server venv. | | [`shutdown_telemetry`](#nemo_gym-telemetry-setup-shutdown_telemetry) | Flush and shut down this process's telemetry providers. | | [`telemetry_config_from_global_config`](#nemo_gym-telemetry-setup-telemetry_config_from_global_config) | Build a :class:`TelemetryConfig` from Gym's merged global config dict. | ### Data [`TELEMETRY_KEY_NAME`](#nemo_gym-telemetry-setup-TELEMETRY_KEY_NAME) [`_ENV_FIELD_MAP`](#nemo_gym-telemetry-setup-_ENV_FIELD_MAP) [`_INITIALISED`](#nemo_gym-telemetry-setup-_INITIALISED) [`_INIT_LOCK`](#nemo_gym-telemetry-setup-_INIT_LOCK) [`_OTEL_FALLBACK_PREFIX`](#nemo_gym-telemetry-setup-_OTEL_FALLBACK_PREFIX) [`_OTEL_PREFIX`](#nemo_gym-telemetry-setup-_OTEL_PREFIX) [`_OTLP_ENV_FIELD_MAP`](#nemo_gym-telemetry-setup-_OTLP_ENV_FIELD_MAP) [`_SERVER_TELEMETRY_PACKAGES`](#nemo_gym-telemetry-setup-_SERVER_TELEMETRY_PACKAGES) [`_SERVICE_NAME_ENV`](#nemo_gym-telemetry-setup-_SERVICE_NAME_ENV) [`_TELEMETRY_HANDLE`](#nemo_gym-telemetry-setup-_TELEMETRY_HANDLE) [`_TRUTHY`](#nemo_gym-telemetry-setup-_TRUTHY) [`logger`](#nemo_gym-telemetry-setup-logger) ### API ```python nemo_gym.telemetry.setup._build_resource_attributes( server_name: typing.Optional[str], server_type: typing.Optional[str] ) -> dict ``` Process-lifetime resource attributes. Only values constant for this process's whole life belong here — anything that varies per request is a span attribute instead (`kb/knowledge/conventions/telemetry-classification.md`). ```python nemo_gym.telemetry.setup._env_flag( name: str, default: bool ) -> bool ``` Read a Gym-owned boolean env var, tolerating an unset or blank value. ```python nemo_gym.telemetry.setup._installed_requirement( dist_name: str, requirement_name: str ) -> typing.Optional[str] ``` Build a requirement string pinning *dist\_name* to the copy installed right here. Prefers the recorded install source over the version number. A git-installed nemo-lens reports a local version such as `0.2.0+b85578f` that exists on no index, so `==` would be unsatisfiable; the recorded VCS URL and commit reinstall exactly what this process is running. ```python nemo_gym.telemetry.setup._reset_for_testing() -> None ``` Drop the process-global handle so a test can initialise again. Test-only. Production code has exactly one init per process, which is what `_INITIALISED` enforces. ```python nemo_gym.telemetry.setup.configure_telemetry_env( telemetry_config: typing.Union[nemo_gym.telemetry.config.TelemetryConfig, None] ) -> typing.Optional[str] ``` Translate the `telemetry:` block into env vars for spawned server processes. Call once in the orchestrator, **before** spawning any server. `os.environ` is what `Popen` snapshots into each child, so this is how a YAML setting reaches a server process that shares nothing else with its parent. Uses `setdefault` throughout: a raw `NEMO_GYM_OTEL_*` / `NEMO_LENS_*` / `OTEL_SERVICE_NAME` / `OTEL_EXPORTER_OTLP_*` set by the user always wins over YAML. Returns the run id shared by every process in this run, or `None` when telemetry is disabled. ```python nemo_gym.telemetry.setup.get_telemetry() -> typing.Optional[nemo.lens.TelemetryHandle] ``` Return this process's telemetry handle, or `None` if uninitialised/disabled. ```python nemo_gym.telemetry.setup.init_telemetry( server_name: typing.Optional[str] = None, server_type: typing.Optional[str] = None, resource_attributes: typing.Optional[dict] = None, rank: int = 0, world_size: int = 1 ) -> typing.Optional[nemo.lens.TelemetryHandle] ``` Initialise this process's telemetry. Call once per process; idempotent. Reads the `NEMO_GYM_OTEL_*` environment that :func:`configure_telemetry_env` put in place, so a server process needs no config file access to agree with its siblings. **Parameters:** **`server_name`** `Optional[str]` — default: None This server's Gym config name (e.g. `example_single_tool_call`). Used to disambiguate `service.name` across the fleet. --- **`server_type`** `Optional[str]` — default: None `resources_servers` | `responses_api_agents` | `responses_api_models`, or `orchestrator` for the CLI. --- **`resource_attributes`** `Optional[dict]` — default: None Extra process-lifetime attributes to merge. --- **`rank / world_size`** Passed to the export strategy. Each Gym server process is rank 0 of a world of 1 by default, which is why `all_ranks` is the default strategy. --- **Returns:** `Optional[TelemetryHandle]` class:`TelemetryHandle`, or `None` when nemo-lens is absent or telemetry is ```python nemo_gym.telemetry.setup.is_telemetry_env_enabled() -> bool ``` True when telemetry is switched on in this process's environment. Cheap enough to call before doing setup work, and importantly it does not import nemo-lens — a process with telemetry off never pays for the lens import at all. ```python nemo_gym.telemetry.setup.server_venv_requirements() -> list ``` Extra requirements to install into every per-server venv. Gym builds an isolated venv per server, and those venvs install `nemo-gym[dev]` — not `nemo-gym[telemetry]`. Without this, telemetry would be enabled in the orchestrator and simply absent in every server process, which is the one configuration that produces a trace with a hole in the middle of it. Pinning is derived from what *this* process has installed rather than restated here. A `[tool.uv.sources]` entry only governs dependencies resolved through the local project: a bare `nemo-lens[sdk]` passed to `uv pip install` ignores it and resolves from PyPI, which would put lens 0.1.0 in the servers while the orchestrator runs the pinned commit. Reading the installed distribution's `direct_url.json` makes that skew impossible by construction instead of by keeping two pins in sync. Returns an empty list when telemetry is off or nemo-lens is not installed, so a normal run's venvs are byte-for-byte what they are today. ```python nemo_gym.telemetry.setup.shutdown_telemetry( timeout_ms: int = 5000 ) -> None ``` Flush and shut down this process's telemetry providers. Idempotent — `TelemetryHandle.shutdown` guards against a second call, and Gym reaches this from more than one terminal path. Never raises. ```python nemo_gym.telemetry.setup.telemetry_config_from_global_config( global_config_dict: typing.Any ) -> nemo_gym.telemetry.config.TelemetryConfig ``` Build a :class:`TelemetryConfig` from Gym's merged global config dict. Reads the optional top-level `telemetry:` block. A run that never mentions telemetry gets the all-defaults config, which is disabled. ```python nemo_gym.telemetry.setup.TELEMETRY_KEY_NAME = 'telemetry' ``` ```python nemo_gym.telemetry.setup._ENV_FIELD_MAP = {'enabled': f'{_OTEL_PREFIX}_ENABLED', 'span_groups': f'{_OTEL_PREFIX}_SPAN_GROU... ``` ```python nemo_gym.telemetry.setup._INITIALISED = False ``` ```python nemo_gym.telemetry.setup._INIT_LOCK = threading.Lock() ``` ```python nemo_gym.telemetry.setup._OTEL_FALLBACK_PREFIX = 'NEMO_LENS' ``` ```python nemo_gym.telemetry.setup._OTEL_PREFIX = 'NEMO_GYM_OTEL' ``` ```python nemo_gym.telemetry.setup._OTLP_ENV_FIELD_MAP = {'otlp_endpoint': 'OTEL_EXPORTER_OTLP_ENDPOINT', 'otlp_protocol': 'OTEL_EXPORTER... ``` ```python nemo_gym.telemetry.setup._SERVER_TELEMETRY_PACKAGES = (('nemo-lens', 'nemo-lens[sdk]'), ('opentelemetry-instrumentation-fastapi', 'ope... ``` ```python nemo_gym.telemetry.setup._SERVICE_NAME_ENV = 'OTEL_SERVICE_NAME' ``` ```python nemo_gym.telemetry.setup._TELEMETRY_HANDLE: Optional[TelemetryHandle] = None ``` ```python nemo_gym.telemetry.setup._TRUTHY = ('1', 'true', 'yes', 'on') ``` ```python nemo_gym.telemetry.setup.logger = logging.getLogger(__name__) ```