Skip to navigation

nemo_gym.telemetry.setup

View as MarkdownOpen in Claude

Process-global nemo-lens telemetry lifecycle for NeMo Gym.

Gym’s process model is the thing this module exists to handle. Megatron-LM is one process tree and NeMo-RL is a Ray driver plus actors, but a Gym run is N independent FastAPI processes — a resources server, a model server, an agent server, sometimes more — spawned by a CLI supervisor via Popen, each with its own interpreter. There is no shared memory and no parent handle to inherit: the only thing that crosses the boundary is the environment.

So there are two entry points:

  • configure_telemetry_env — called once in the supervisor (gym env start / gym env test) before any server is spawned. It translates the telemetry: config block into NEMO_GYM_OTEL_* env vars with setdefault, so every server process it spawns resolves the same settings, and any env var the user set by hand still wins.
  • init_telemetry — called once inside each server process, from SimpleServer.run_webserver. It reads that propagated environment and builds this process’s providers.

nemo-lens allows one setup_telemetry per process. When something else in the process already called it — NeMo-RL, when Gym runs inside an RL job — init_telemetry reuses those providers instead of failing, and leaves their shutdown to their owner.

Importing this module never requires nemo-lens: every lens import is function-local and guarded. With lens absent, or enabled: false, the init functions return None and every instrumentation site stays a no-op through nemo_gym.telemetry._fallbacks.

Module Contents

Functions

NameDescription
_build_resource_attributesProcess-lifetime resource attributes.
_direct_referencePEP 508 direct reference for a PEP 610 direct_url.json record.
_env_flagRead a Gym-owned boolean env var, tolerating an unset or blank value.
_lens_fallback_nameThe NEMO_LENS_* fallback for Gym’s NEMO_GYM_OTEL_* env_name.
_register_sandbox_span_attributesHand the sandbox API its extra span attributes. The sandbox package must not import rollout
_reset_for_testingDrop the process-global handle so a test can initialise again.
_same_settingWhether env_value asks for the same setting as config_value.
_sdk_meter_provider_installedWhether a real OpenTelemetry SDK meter provider, not a no-op or proxy, is registered.
_sdk_tracer_provider_installedWhether a real OpenTelemetry SDK tracer provider, not a no-op or proxy, is registered.
_service_instance_idA globally unique name for this process, behind a readable prefix.
configure_telemetry_envTranslate the telemetry: block into env vars for spawned server processes.
event_loop_settings_from_envReturn the event-loop sample interval and whether to count tasks, from the environment the supervisor exported.
get_telemetryReturn this process’s telemetry handle, or None if uninitialised/disabled.
init_telemetryInitialise this process’s telemetry. Call once per process; idempotent.
installed_requirementBuild a requirement string pinning dist_name to the copy installed right here.
is_metrics_exporter_activeWhether this process has an active metrics exporter for Gym to record into.
is_span_group_or_metrics_activeWhether a site that both spans group and records metrics should take its instrumented path.
is_telemetry_env_enabledTrue when telemetry is switched on in this process’s environment.
is_telemetry_metrics_enabledReturn the effective metrics switch after Gym/Lens env precedence.
is_trace_exporter_activeWhether this process exports spans.
memory_profiling_config_from_envResolve memory profiling after raw environment variables take precedence.
rollout_span_attributesThe correlation attribute Gym’s endpoint spans carry, for spans opened by code that does not
server_venv_requirementsExtra requirements to install into every per-server venv.
shutdown_telemetryFlush and shut down this process’s telemetry providers.
telemetry_config_from_global_configBuild a TelemetryConfig from Gym’s merged global config dict.

Data

TELEMETRY_KEY_NAME

_ENV_FIELD_MAP

_EVENT_LOOP_ENV_FIELD_MAP

_INITIALISED

_INIT_LOCK

_LENS_FIELDS

_MEMORY_PROFILING_ENV_FIELD_MAP

_METRICS_EXPORTING

_OTEL_FALLBACK_PREFIX

_OTEL_PREFIX

_OTLP_ENV_FIELD_MAP

_OWNS_PROVIDERS

_SERVER_TELEMETRY_PACKAGES

_SERVICE_NAME_ENV

_TELEMETRY_HANDLE

_TRACES_EXPORTING

_TRUTHY

logger

API

nemo_gym.telemetry.setup._build_resource_attributes(
server_name: typing.Optional[str],
server_type: typing.Optional[str]
) -> dict

Process-lifetime resource attributes.

Only values constant for this process’s whole life belong here — anything that varies per request is a span attribute instead (kb/knowledge/conventions/telemetry-classification.md).

nemo_gym.telemetry.setup._direct_reference(
requirement_name: str,
direct_url: dict
) -> typing.Optional[str]

PEP 508 direct reference for a PEP 610 direct_url.json record.

Covers the three shapes the record can take: a VCS checkout pinned to its commit, a local directory, and an archive such as a wheel installed from a file or a URL. Each keeps its subdirectory. Returns None when the record has nothing reinstallable, so the caller can fall back to a version pin.

nemo_gym.telemetry.setup._env_flag(
name: str,
default: bool
) -> bool

Read a Gym-owned boolean env var, tolerating an unset or blank value.

nemo_gym.telemetry.setup._lens_fallback_name(
env_name: str
) -> str

The NEMO_LENS_* fallback for Gym’s NEMO_GYM_OTEL_* env_name.

nemo_gym.telemetry.setup._register_sandbox_span_attributes() -> None

Hand the sandbox API its extra span attributes. The sandbox package must not import rollout correlation itself: global_config imports the sandbox registry while initialising.

nemo_gym.telemetry.setup._reset_for_testing() -> None

Drop the process-global handle so a test can initialise again.

Test-only. Production code has exactly one init per process, which is what _INITIALISED enforces.

nemo_gym.telemetry.setup._same_setting(
config_value: typing.Any,
env_value: str
) -> bool

Whether env_value asks for the same setting as config_value.

nemo_gym.telemetry.setup._sdk_meter_provider_installed() -> bool

Whether a real OpenTelemetry SDK meter provider, not a no-op or proxy, is registered.

nemo_gym.telemetry.setup._sdk_tracer_provider_installed() -> bool

Whether a real OpenTelemetry SDK tracer provider, not a no-op or proxy, is registered.

nemo_gym.telemetry.setup._service_instance_id(
run_id: str,
server_name: typing.Optional[str]
) -> str

A globally unique name for this process, behind a readable prefix.

The server name makes it legible and the PID separates one server’s Uvicorn workers, but neither is unique on its own: a PID repeats across hosts and across container PID namespaces. The random suffix is what actually satisfies the semantic convention. Absent parts are omitted rather than left as empty segments.

The run id is included only when this process was configured with one. A standalone server gets <name>-<pid>-<nonce>; nemo-lens still mints a nemo.run.id for it afterwards, inside setup_telemetry, which is too late to appear here.

Called once per process: init_telemetry sets _INITIALISED under _INIT_LOCK and returns early afterwards, so the suffix is stable for the process’s lifetime.

nemo_gym.telemetry.setup.configure_telemetry_env(
telemetry_config: typing.Union[nemo_gym.telemetry.config.TelemetryConfig, None]
) -> typing.Optional[str]

Translate the telemetry: block into env vars for spawned server processes.

Call once in the supervisor, before spawning any server. os.environ is what Popen snapshots into each child, so this is how a YAML setting reaches a server process that shares nothing else with its parent.

Uses setdefault throughout: a raw NEMO_GYM_OTEL_* / NEMO_LENS_* / OTEL_SERVICE_NAME / OTEL_EXPORTER_OTLP_* set by the user always wins over YAML.

Returns the run id shared by every process in this run, or None when telemetry is disabled.

nemo_gym.telemetry.setup.event_loop_settings_from_env() -> tuple[float, bool]

Return the event-loop sample interval and whether to count tasks, from the environment the supervisor exported.

nemo_gym.telemetry.setup.get_telemetry() -> typing.Optional[nemo.lens.TelemetryHandle]

Return this process’s telemetry handle, or None if uninitialised/disabled.

nemo_gym.telemetry.setup.init_telemetry(
server_name: typing.Optional[str] = None,
server_type: typing.Optional[str] = None,
resource_attributes: typing.Optional[dict] = None
) -> typing.Optional[nemo.lens.TelemetryHandle]

Initialise this process’s telemetry. Call once per process; idempotent.

Reads the NEMO_GYM_OTEL_* environment that configure_telemetry_env put in place, so a server process needs no config file access to agree with its siblings.

Parameters:

server_name
Optional[str]Defaults to None

This server’s Gym config name (e.g. example_single_tool_call). Used to disambiguate service.name across the fleet.

server_type
Optional[str]Defaults to None

resources_servers | responses_api_agents | responses_api_models, or supervisor for the CLI.

resource_attributes
Optional[dict]Defaults to None

Extra process-lifetime attributes to merge.

Returns: Optional[TelemetryHandle]

class:TelemetryHandle, or None when nemo-lens is absent or telemetry is

nemo_gym.telemetry.setup.installed_requirement(
dist_name: str,
requirement_name: str
) -> typing.Optional[str]

Build a requirement string pinning dist_name to the copy installed right here.

Prefers the recorded install source over the version number. A git-installed nemo-lens reports a local version such as 0.2.0+b85578f that exists on no index, so == would be unsatisfiable; the recorded VCS URL and commit reinstall exactly what this process is running. The same holds for a wheel built locally and for a plugin checked out from a repository subfolder.

nemo_gym.telemetry.setup.is_metrics_exporter_active() -> bool

Whether this process has an active metrics exporter for Gym to record into.

True after Gym’s own telemetry setup succeeds with metrics enabled, or after Gym reuses providers that another library set up with metrics enabled. Unlike is_telemetry_metrics_enabled, this stays false until telemetry is set up.

nemo_gym.telemetry.setup.is_span_group_or_metrics_active(
group: str
) -> bool

Whether a site that both spans group and records metrics should take its instrumented path.

Spans follow span_groups and metrics follow the metrics exporter, independently, so a site instruments when either is on. managed_span still checks the span group itself.

nemo_gym.telemetry.setup.is_telemetry_env_enabled() -> bool

True when telemetry is switched on in this process’s environment.

Cheap enough to call before doing setup work, and importantly it does not import nemo-lens — a process with telemetry off never pays for the lens import at all.

nemo_gym.telemetry.setup.is_telemetry_metrics_enabled() -> bool

Return the effective metrics switch after Gym/Lens env precedence.

nemo_gym.telemetry.setup.is_trace_exporter_active() -> bool

Whether this process exports spans.

With traces off, nemo-lens still enables the configured span groups, so a span site runs against a tracer that records nothing. Sites whose only output is spans, or trace context for spans downstream, check this as well to skip that work.

nemo_gym.telemetry.setup.memory_profiling_config_from_env(

Resolve memory profiling after raw environment variables take precedence.

nemo_gym.telemetry.setup.rollout_span_attributes() -> dict[str, str]

The correlation attribute Gym’s endpoint spans carry, for spans opened by code that does not know about rollouts (the sandbox API). Empty outside a rollout context.

nemo_gym.telemetry.setup.server_venv_requirements() -> list

Extra requirements to install into every per-server venv.

Gym builds an isolated venv per server, and those venvs install nemo-gym[dev] — not nemo-gym[telemetry]. Without this, telemetry would be enabled in the supervisor and simply absent in every server process, which is the one configuration that produces a trace with a hole in the middle of it.

Pinning is derived from what this process has installed rather than restated here. A [tool.uv.sources] entry only governs dependencies resolved through the local project: a bare nemo-lens[sdk] passed to uv pip install ignores it and resolves from PyPI, which would put lens 0.1.0 in the servers while the supervisor runs the pinned commit. Reading the installed distribution’s direct_url.json makes that skew impossible by construction instead of by keeping two pins in sync.

The locked component installer selects the telemetry extra and checks these parent requirements against the resulting environment. A version or source mismatch uses compatibility resolution with the inherited parent requirements. Failed syncs and stale locks remain fatal. Returns an empty list when telemetry is off or nemo-lens is not installed.

nemo_gym.telemetry.setup.shutdown_telemetry(
timeout_ms: int = 5000
) -> None

Flush and shut down this process’s telemetry providers.

Idempotent — TelemetryHandle.shutdown guards against a second call, and Gym reaches this from more than one terminal path. A no-op when Gym reused providers that another library set up. Never raises.

nemo_gym.telemetry.setup.telemetry_config_from_global_config(
global_config_dict: typing.Any

Build a TelemetryConfig from Gym’s merged global config dict.

Reads the optional top-level telemetry: block. A run that never mentions telemetry gets the all-defaults config, which is disabled.

nemo_gym.telemetry.setup.TELEMETRY_KEY_NAME = 'telemetry'
nemo_gym.telemetry.setup._ENV_FIELD_MAP = {'enabled': f'{_OTEL_PREFIX}_ENABLED', 'span_groups': f'{_OTEL_PREFIX}_SPAN_GROU...
nemo_gym.telemetry.setup._EVENT_LOOP_ENV_FIELD_MAP = {'event_loop_sample_interval_seconds': f'{_OTEL_PREFIX}_EVENT_LOOP_SAMPLE_INTERV...
nemo_gym.telemetry.setup._INITIALISED = False
nemo_gym.telemetry.setup._INIT_LOCK = threading.Lock()
nemo_gym.telemetry.setup._LENS_FIELDS = frozenset({'span_groups', 'traces_enabled', 'metrics_enabled', 'logs_enabled', '...
nemo_gym.telemetry.setup._MEMORY_PROFILING_ENV_FIELD_MAP = {'enabled': f'{_OTEL_PREFIX}_MEMORY_PROFILING_ENABLED', 'interval_seconds': f'{_...
nemo_gym.telemetry.setup._METRICS_EXPORTING = False
nemo_gym.telemetry.setup._OTEL_FALLBACK_PREFIX = 'NEMO_LENS'
nemo_gym.telemetry.setup._OTEL_PREFIX = 'NEMO_GYM_OTEL'
nemo_gym.telemetry.setup._OTLP_ENV_FIELD_MAP = {'otlp_endpoint': 'OTEL_EXPORTER_OTLP_ENDPOINT', 'otlp_protocol': 'OTEL_EXPORTER...
nemo_gym.telemetry.setup._OWNS_PROVIDERS = False
nemo_gym.telemetry.setup._SERVER_TELEMETRY_PACKAGES = (('nemo-lens', 'nemo-lens[sdk]'), ('opentelemetry-instrumentation-fastapi', 'ope...
nemo_gym.telemetry.setup._SERVICE_NAME_ENV = 'OTEL_SERVICE_NAME'
nemo_gym.telemetry.setup._TELEMETRY_HANDLE: Optional[TelemetryHandle] = None
nemo_gym.telemetry.setup._TRACES_EXPORTING = False
nemo_gym.telemetry.setup._TRUTHY = ('1', 'true', 'yes', 'on')
nemo_gym.telemetry.setup.logger = logging.getLogger(__name__)