nemo_gym.telemetry.setup

Process-global nemo-lens telemetry lifecycle for NeMo Gym.

Gym’s process model is the thing this module exists to handle. Megatron-LM is one process tree and NeMo-RL is a Ray driver plus actors, but a Gym run is N independent FastAPI processes — a resources server, a model server, an agent server, sometimes more — spawned by a CLI orchestrator via Popen, each with its own interpreter. There is no shared memory and no parent handle to inherit: the only thing that crosses the boundary is the environment.

So there are two entry points:

:func:configure_telemetry_env — called once in the orchestrator (gym env start / gym env test) before any server is spawned. It translates the telemetry: config block into NEMO_GYM_OTEL_* env vars with setdefault, so every server process it spawns resolves the same settings, and any env var the user set by hand still wins.
:func:init_telemetry — called once inside each server process, from SimpleServer.run_webserver. It reads that propagated environment and builds this process’s providers.

export_strategy defaults to all_ranks here rather than single_rank. Each Gym server is rank 0 of its own world of 1, so a rank-based filter would either silence every process or none; and silencing any one of them puts a hole in the middle of the distributed trace this integration exists to produce.

Importing this module never requires nemo-lens: every lens import is function-local and guarded. With lens absent, or enabled: false, the init functions return None and every instrumentation site stays a no-op through :mod:nemo_gym.telemetry._fallbacks.

Module Contents

Functions

Name	Description
`_build_resource_attributes`	Process-lifetime resource attributes.
`_env_flag`	Read a Gym-owned boolean env var, tolerating an unset or blank value.
`_installed_requirement`	Build a requirement string pinning dist_name to the copy installed right here.
`_reset_for_testing`	Drop the process-global handle so a test can initialise again.
`configure_telemetry_env`	Translate the `telemetry:` block into env vars for spawned server processes.
`get_telemetry`	Return this process’s telemetry handle, or `None` if uninitialised/disabled.
`init_telemetry`	Initialise this process’s telemetry. Call once per process; idempotent.
`is_telemetry_env_enabled`	True when telemetry is switched on in this process’s environment.
`server_venv_requirements`	Extra requirements to install into every per-server venv.
`shutdown_telemetry`	Flush and shut down this process’s telemetry providers.
`telemetry_config_from_global_config`	Build a :class:`TelemetryConfig` from Gym’s merged global config dict.

Data

_OTEL_FALLBACK_PREFIX

_OTEL_PREFIX

_OTLP_ENV_FIELD_MAP

_SERVER_TELEMETRY_PACKAGES

API

nemo_gym.telemetry.setup._build_resource_attributes(
    server_name: typing.Optional[str],
    server_type: typing.Optional[str]
) -> dict

Process-lifetime resource attributes.

Only values constant for this process’s whole life belong here — anything that varies per request is a span attribute instead (kb/knowledge/conventions/telemetry-classification.md).

nemo_gym.telemetry.setup._env_flag(
    name: str,
    default: bool
) -> bool

Read a Gym-owned boolean env var, tolerating an unset or blank value.

nemo_gym.telemetry.setup._installed_requirement(
    dist_name: str,
    requirement_name: str
) -> typing.Optional[str]

Build a requirement string pinning dist_name to the copy installed right here.

Prefers the recorded install source over the version number. A git-installed nemo-lens reports a local version such as 0.2.0+b85578f that exists on no index, so == would be unsatisfiable; the recorded VCS URL and commit reinstall exactly what this process is running.

nemo_gym.telemetry.setup._reset_for_testing() -> None

Drop the process-global handle so a test can initialise again.

Test-only. Production code has exactly one init per process, which is what _INITIALISED enforces.

nemo_gym.telemetry.setup.configure_telemetry_env(
    telemetry_config: typing.Union[nemo_gym.telemetry.config.TelemetryConfig, None]
) -> typing.Optional[str]

Translate the telemetry: block into env vars for spawned server processes.

Call once in the orchestrator, before spawning any server. os.environ is what Popen snapshots into each child, so this is how a YAML setting reaches a server process that shares nothing else with its parent.

Uses setdefault throughout: a raw NEMO_GYM_OTEL_* / NEMO_LENS_* / OTEL_SERVICE_NAME / OTEL_EXPORTER_OTLP_* set by the user always wins over YAML.

Returns the run id shared by every process in this run, or None when telemetry is disabled.

nemo_gym.telemetry.setup.get_telemetry() -> typing.Optional[nemo.lens.TelemetryHandle]

Return this process’s telemetry handle, or None if uninitialised/disabled.

nemo_gym.telemetry.setup.init_telemetry(
    server_name: typing.Optional[str] = None,
    server_type: typing.Optional[str] = None,
    resource_attributes: typing.Optional[dict] = None,
    rank: int = 0,
    world_size: int = 1
) -> typing.Optional[nemo.lens.TelemetryHandle]

Initialise this process’s telemetry. Call once per process; idempotent.

Reads the NEMO_GYM_OTEL_* environment that :func:configure_telemetry_env put in place, so a server process needs no config file access to agree with its siblings.

Parameters:

server_name

Optional[str]Defaults to None

This server’s Gym config name (e.g. example_single_tool_call). Used to disambiguate service.name across the fleet.

server_type

Optional[str]Defaults to None

resources_servers | responses_api_agents | responses_api_models, or orchestrator for the CLI.

resource_attributes

Optional[dict]Defaults to None

Extra process-lifetime attributes to merge.

rank / world_size

Passed to the export strategy. Each Gym server process is rank 0 of a world of 1 by default, which is why all_ranks is the default strategy.

Returns: Optional[TelemetryHandle]

class:TelemetryHandle, or None when nemo-lens is absent or telemetry is

nemo_gym.telemetry.setup.is_telemetry_env_enabled() -> bool

True when telemetry is switched on in this process’s environment.

Cheap enough to call before doing setup work, and importantly it does not import nemo-lens — a process with telemetry off never pays for the lens import at all.

nemo_gym.telemetry.setup.server_venv_requirements() -> list

Extra requirements to install into every per-server venv.

Gym builds an isolated venv per server, and those venvs install nemo-gym[dev] — not nemo-gym[telemetry]. Without this, telemetry would be enabled in the orchestrator and simply absent in every server process, which is the one configuration that produces a trace with a hole in the middle of it.

Pinning is derived from what this process has installed rather than restated here. A [tool.uv.sources] entry only governs dependencies resolved through the local project: a bare nemo-lens[sdk] passed to uv pip install ignores it and resolves from PyPI, which would put lens 0.1.0 in the servers while the orchestrator runs the pinned commit. Reading the installed distribution’s direct_url.json makes that skew impossible by construction instead of by keeping two pins in sync.

Returns an empty list when telemetry is off or nemo-lens is not installed, so a normal run’s venvs are byte-for-byte what they are today.

nemo_gym.telemetry.setup.shutdown_telemetry(
    timeout_ms: int = 5000
) -> None

Flush and shut down this process’s telemetry providers.

Idempotent — TelemetryHandle.shutdown guards against a second call, and Gym reaches this from more than one terminal path. Never raises.

nemo_gym.telemetry.setup.telemetry_config_from_global_config(
    global_config_dict: typing.Any
) -> nemo_gym.telemetry.config.TelemetryConfig

Build a :class:TelemetryConfig from Gym’s merged global config dict.

Reads the optional top-level telemetry: block. A run that never mentions telemetry gets the all-defaults config, which is disabled.

nemo_gym.telemetry.setup.TELEMETRY_KEY_NAME = 'telemetry'

nemo_gym.telemetry.setup._ENV_FIELD_MAP = {'enabled': f'{_OTEL_PREFIX}_ENABLED', 'span_groups': f'{_OTEL_PREFIX}_SPAN_GROU...

nemo_gym.telemetry.setup._INITIALISED = False

nemo_gym.telemetry.setup._INIT_LOCK = threading.Lock()

nemo_gym.telemetry.setup._OTEL_FALLBACK_PREFIX = 'NEMO_LENS'

nemo_gym.telemetry.setup._OTEL_PREFIX = 'NEMO_GYM_OTEL'

nemo_gym.telemetry.setup._OTLP_ENV_FIELD_MAP = {'otlp_endpoint': 'OTEL_EXPORTER_OTLP_ENDPOINT', 'otlp_protocol': 'OTEL_EXPORTER...

nemo_gym.telemetry.setup._SERVER_TELEMETRY_PACKAGES = (('nemo-lens', 'nemo-lens[sdk]'), ('opentelemetry-instrumentation-fastapi', 'ope...

nemo_gym.telemetry.setup._SERVICE_NAME_ENV = 'OTEL_SERVICE_NAME'

nemo_gym.telemetry.setup._TELEMETRY_HANDLE: Optional[TelemetryHandle] = None

nemo_gym.telemetry.setup._TRUTHY = ('1', 'true', 'yes', 'on')

nemo_gym.telemetry.setup.logger = logging.getLogger(__name__)