nemo_gym.telemetry.setup
nemo_gym.telemetry.setup
Process-global nemo-lens telemetry lifecycle for NeMo Gym.
Gym’s process model is the thing this module exists to handle. Megatron-LM is one
process tree and NeMo-RL is a Ray driver plus actors, but a Gym run is N independent
FastAPI processes — a resources server, a model server, an agent server, sometimes more
— spawned by a CLI supervisor via Popen, each with its own interpreter. There is no
shared memory and no parent handle to inherit: the only thing that crosses the boundary
is the environment.
So there are two entry points:
configure_telemetry_env— called once in the supervisor (gym env start/gym env test) before any server is spawned. It translates thetelemetry:config block intoNEMO_GYM_OTEL_*env vars withsetdefault, so every server process it spawns resolves the same settings, and any env var the user set by hand still wins.init_telemetry— called once inside each server process, fromSimpleServer.run_webserver. It reads that propagated environment and builds this process’s providers.
nemo-lens allows one setup_telemetry per process. When something else in the process
already called it — NeMo-RL, when Gym runs inside an RL job — init_telemetry
reuses those providers instead of failing, and leaves their shutdown to their owner.
Importing this module never requires nemo-lens: every lens import is function-local and
guarded. With lens absent, or enabled: false, the init functions return None and
every instrumentation site stays a no-op through nemo_gym.telemetry._fallbacks.
Module Contents
Functions
Data
_MEMORY_PROFILING_ENV_FIELD_MAP
API
Process-lifetime resource attributes.
Only values constant for this process’s whole life belong here — anything that varies
per request is a span attribute instead
(kb/knowledge/conventions/telemetry-classification.md).
PEP 508 direct reference for a PEP 610 direct_url.json record.
Covers the three shapes the record can take: a VCS checkout pinned to its commit, a
local directory, and an archive such as a wheel installed from a file or a URL. Each
keeps its subdirectory. Returns None when the record has nothing reinstallable,
so the caller can fall back to a version pin.
Read a Gym-owned boolean env var, tolerating an unset or blank value.
The NEMO_LENS_* fallback for Gym’s NEMO_GYM_OTEL_* env_name.
Hand the sandbox API its extra span attributes. The sandbox package must not import rollout
correlation itself: global_config imports the sandbox registry while initialising.
Drop the process-global handle so a test can initialise again.
Test-only. Production code has exactly one init per process, which is what
_INITIALISED enforces.
Whether env_value asks for the same setting as config_value.
Whether a real OpenTelemetry SDK meter provider, not a no-op or proxy, is registered.
Whether a real OpenTelemetry SDK tracer provider, not a no-op or proxy, is registered.
A globally unique name for this process, behind a readable prefix.
The server name makes it legible and the PID separates one server’s Uvicorn workers, but neither is unique on its own: a PID repeats across hosts and across container PID namespaces. The random suffix is what actually satisfies the semantic convention. Absent parts are omitted rather than left as empty segments.
The run id is included only when this process was configured with one. A standalone
server gets <name>-<pid>-<nonce>; nemo-lens still mints a nemo.run.id for it
afterwards, inside setup_telemetry, which is too late to appear here.
Called once per process: init_telemetry sets _INITIALISED under _INIT_LOCK
and returns early afterwards, so the suffix is stable for the process’s lifetime.
Translate the telemetry: block into env vars for spawned server processes.
Call once in the supervisor, before spawning any server. os.environ is what
Popen snapshots into each child, so this is how a YAML setting reaches a server
process that shares nothing else with its parent.
Uses setdefault throughout: a raw NEMO_GYM_OTEL_* / NEMO_LENS_* /
OTEL_SERVICE_NAME / OTEL_EXPORTER_OTLP_* set by the user always wins over YAML.
Returns the run id shared by every process in this run, or None when telemetry is
disabled.
Return the event-loop sample interval and whether to count tasks, from the environment the supervisor exported.
Return this process’s telemetry handle, or None if uninitialised/disabled.
Initialise this process’s telemetry. Call once per process; idempotent.
Reads the NEMO_GYM_OTEL_* environment that configure_telemetry_env put in
place, so a server process needs no config file access to agree with its siblings.
Parameters:
This server’s Gym config name (e.g. example_single_tool_call).
Used to disambiguate service.name across the fleet.
resources_servers | responses_api_agents |
responses_api_models, or supervisor for the CLI.
Extra process-lifetime attributes to merge.
Returns: Optional[TelemetryHandle]
class:TelemetryHandle, or None when nemo-lens is absent or telemetry is
Build a requirement string pinning dist_name to the copy installed right here.
Prefers the recorded install source over the version number. A git-installed
nemo-lens reports a local version such as 0.2.0+b85578f that exists on no index,
so == would be unsatisfiable; the recorded VCS URL and commit reinstall exactly
what this process is running. The same holds for a wheel built locally and for a
plugin checked out from a repository subfolder.
Whether this process has an active metrics exporter for Gym to record into.
True after Gym’s own telemetry setup succeeds with metrics enabled, or after Gym reuses
providers that another library set up with metrics enabled. Unlike
is_telemetry_metrics_enabled, this stays false until telemetry is set up.
Whether a site that both spans group and records metrics should take its instrumented path.
Spans follow span_groups and metrics follow the metrics exporter, independently, so a
site instruments when either is on. managed_span still checks the span group itself.
True when telemetry is switched on in this process’s environment.
Cheap enough to call before doing setup work, and importantly it does not import nemo-lens — a process with telemetry off never pays for the lens import at all.
Return the effective metrics switch after Gym/Lens env precedence.
Whether this process exports spans.
With traces off, nemo-lens still enables the configured span groups, so a span site runs against a tracer that records nothing. Sites whose only output is spans, or trace context for spans downstream, check this as well to skip that work.
Resolve memory profiling after raw environment variables take precedence.
The correlation attribute Gym’s endpoint spans carry, for spans opened by code that does not know about rollouts (the sandbox API). Empty outside a rollout context.
Extra requirements to install into every per-server venv.
Gym builds an isolated venv per server, and those venvs install nemo-gym[dev] —
not nemo-gym[telemetry]. Without this, telemetry would be enabled in the
supervisor and simply absent in every server process, which is the one
configuration that produces a trace with a hole in the middle of it.
Pinning is derived from what this process has installed rather than restated here.
A [tool.uv.sources] entry only governs dependencies resolved through the local
project: a bare nemo-lens[sdk] passed to uv pip install ignores it and
resolves from PyPI, which would put lens 0.1.0 in the servers while the supervisor
runs the pinned commit. Reading the installed distribution’s direct_url.json
makes that skew impossible by construction instead of by keeping two pins in sync.
The locked component installer selects the telemetry extra and checks these parent requirements against the resulting environment. A version or source mismatch uses compatibility resolution with the inherited parent requirements. Failed syncs and stale locks remain fatal. Returns an empty list when telemetry is off or nemo-lens is not installed.
Flush and shut down this process’s telemetry providers.
Idempotent — TelemetryHandle.shutdown guards against a second call, and Gym
reaches this from more than one terminal path. A no-op when Gym reused providers that
another library set up. Never raises.
Build a TelemetryConfig from Gym’s merged global config dict.
Reads the optional top-level telemetry: block. A run that never mentions
telemetry gets the all-defaults config, which is disabled.