API ReferenceFull Library ReferenceNemo GymNemo Gym

nemo_gym.telemetry

View as Markdown

NeMo Gym telemetry: optional OpenTelemetry instrumentation via nemo-lens.

Importing this package never requires nemo-lens. Install it with uv sync --extra telemetry and switch it on with a telemetry: block or NEMO_LENS_ENABLED=1; with either missing, every instrumentation site in nemo_gym is a ~0-cost no-op.

Public surface

  • TelemetryConfig — the telemetry: config block.
  • GymSpanGroup — Gym span groups and presets.
  • configure_telemetry_env — orchestrator side; hands the settings to spawned server processes through the environment.
  • init_telemetry / get_telemetry / shutdown_telemetry — per-process lifecycle.

Instrumentation primitives (managed_span / span_cm / trace_fn / is_span_group_enabled / safe_set_span_attributes) come from nemo_gym.telemetry._fallbacks, which resolves to the real nemo-lens implementations when it is installed and to no-op stubs when it is not.

See nemo_gym/telemetry/README.md for the design, and fern/versions/latest/pages/observability/ for user documentation.

Submodules

Package Contents

Classes

NameDescription
GymSpanGroupSpan group names for NeMo Gym instrumentation.
MemoryProfilingConfigPeriodic host and logical-server memory sampling.
TelemetryConfigOpenTelemetry / nemo-lens configuration for NeMo Gym.

Functions

NameDescription
configure_telemetry_envTranslate the telemetry: block into env vars for spawned server processes.
get_telemetryReturn this process’s telemetry handle, or None if uninitialised/disabled.
init_telemetryInitialise this process’s telemetry. Call once per process; idempotent.
is_span_group_enabledAlways returns False.
is_telemetry_env_enabledTrue when telemetry is switched on in this process’s environment.
is_telemetry_metrics_enabledReturn the effective metrics switch after Gym/Lens env precedence.
managed_spanNo-op context manager — yields None.
memory_profiling_config_from_envResolve memory profiling after raw environment variables take precedence.
safe_set_span_attributesNo-op.
shutdown_telemetryFlush and shut down this process’s telemetry providers.
span_cmNo-op context manager — yields None.
telemetry_config_from_global_configBuild a TelemetryConfig from Gym’s merged global config dict.
trace_fnNo-op decorator — returns the function unchanged.

Data

TELEMETRY_KEY_NAME

API

class nemo_gym.telemetry.GymSpanGroup()

Span group names for NeMo Gym instrumentation.

AGENT
= 'agent'

Agent-server /run and /v1/responses spans.

ALL_GROUPS
Final[frozenset]
CROSS_PROCESS_SPINE
Final[frozenset] = frozenset([SERVER, HTTP_CLIENT, ROLLOUT])
HTTP_CLIENT
= 'http_client'

Outbound spans around nemo_gym.server_utils.request — Gym’s single aiohttp egress point. The egress half of cross-process propagation: injects traceparent into the outgoing headers.

JOB
= 'job'

The whole gym eval / rollout-collection run, driver side. Shares its name with NeMo-RL’s run-level group, so one spec entry selects both.

MODEL_CALL
= 'model_call'

Model-server /v1/chat/completions, /v1/responses and /v1/messages spans.

ROLLOUT
= 'rollout'

Rollout collection spans (one per task attempt, driver side).

SANDBOX
= 'sandbox'

Sandbox provider create/exec/delete spans.

SERVER
= 'server'

Inbound FastAPI request spans on every Gym server process. The ingress half of cross-process propagation: adopts an inbound traceparent as the span’s parent.

VERIFY
= 'verify'

Resources-server /verify spans.

_PRESETS
dict
nemo_gym.telemetry.GymSpanGroup.resolve(
    spec: str
) -> frozenset
classmethod

Resolve a span_groups spec against every group registered in this process.

Entries that name nothing registered are dropped rather than raised: the library that owns them may not be imported in this process. nemo-lens logs a warning naming them when the spec is applied in setup_telemetry.

Raises:

  • RuntimeError: nemo-lens is not installed, so there is nothing to resolve.
class nemo_gym.telemetry.MemoryProfilingConfig()

Bases: BaseModel

Periodic host and logical-server memory sampling.

enabled
bool = False

Collect host memory and every server process tree.

interval_seconds
float = Field(default=1.0, gt=0)

Seconds between samples.

class nemo_gym.telemetry.TelemetryConfig()

Bases: BaseModel

OpenTelemetry / nemo-lens configuration for NeMo Gym.

Telemetry is optional twice over: it activates only when enabled is true and nemo-lens is installed (uv sync --extra telemetry). When either is missing every instrumentation site in nemo_gym degrades to a ~0-cost no-op and Gym behaves exactly as it does today.

enabled
bool = False

Master switch. When false, all instrumentation is a ~0-cost no-op.

exporter
str = 'otlp'

Exporter backend: otlp | console. console writes one JSON object per line to stdout, which is what the local validation flow greps.

instrument_aiohttp
bool = True

Use nemo-lens’s aiohttp auto-instrumentation for outbound calls, which produces a CLIENT span per request in addition to injecting traceparent.

When false, nemo_gym.server_utils.request still injects traceparent manually, so cross-process traces stay joined — you lose the client-side span, not the trace. Turn it off if the per-request patching cost matters at very high concurrency.

logs_enabled
bool = False

Bridge Python logging to OTel logs, exported with trace correlation.

memory_profiling
MemoryProfilingConfig = Field(default_factory=MemoryProfilingConfig)

Sample each Gym-managed server process tree and emit memory metrics.

metrics_enabled
bool = True

Emit metric instruments (the gym.* histograms/gauges plus the FastAPI instrumentor’s http.server.*).

otlp_endpoint
Optional[str] = None

OTEL_EXPORTER_OTLP_ENDPOINT for the otlp exporter, e.g. https://api.honeycomb.io. Optional: the standard env var still works and always wins (setdefault), so any OTLP-compatible backend or collector that expects the raw env var keeps working unchanged. Setting it here just means a run’s destination is committed with the rest of the config instead of exported by hand every time.

otlp_headers
Optional[str] = None

OTEL_EXPORTER_OTLP_HEADERS, e.g. x-honeycomb-team=.... Prefer ${oc.env:MY_API_KEY} interpolation for the secret portion so the raw credential does not live in a committed YAML file.

otlp_protocol
Optional[str] = None

OTEL_EXPORTER_OTLP_PROTOCOL: grpc (the OTel SDK default) or http/protobuf.

run_id
Optional[str] = None

Correlates every process of one gym env start / gym env test invocation. Generated in the orchestrator and inherited by the servers when unset.

service_name
str = 'nemo-gym'

service.name reported to the backend.

Passed through the standard OTEL_SERVICE_NAME env var, not through instrument_fastapi(service_name=...) — that parameter is accepted and ignored by nemo-lens at the pinned commit (contrib/fastapi.py).

Each server process appends its own Gym server name, so a run yields nemo-gym/example_single_tool_call, nemo-gym/policy_model, and so on rather than three indistinguishable nemo-gym services. Set service_name_per_server to false to opt out.

service_name_per_server
bool = True

Suffix service_name with each server’s Gym config name.

On by default because Gym runs several processes that would otherwise share one service.name, which makes a backend’s service map useless. The unsuffixed name stays available as the nemo.gym.service_group resource attribute.

span_groups
str = 'default'

Span-group spec: a preset (default | per_rollout | all) or a comma-separated list of group names (e.g. "default,sandbox"). See GymSpanGroup. An unknown name is logged as a warning and ignored rather than rejected.

traces_enabled
bool = True

Emit trace spans.

nemo_gym.telemetry.configure_telemetry_env(
    telemetry_config: typing.Union[nemo_gym.telemetry.config.TelemetryConfig, None]
) -> typing.Optional[str]

Translate the telemetry: block into env vars for spawned server processes.

Call once in the orchestrator, before spawning any server. os.environ is what Popen snapshots into each child, so this is how a YAML setting reaches a server process that shares nothing else with its parent.

Uses setdefault throughout: a raw NEMO_GYM_OTEL_* / NEMO_LENS_* / OTEL_SERVICE_NAME / OTEL_EXPORTER_OTLP_* set by the user always wins over YAML.

Returns the run id shared by every process in this run, or None when telemetry is disabled.

nemo_gym.telemetry.get_telemetry() -> typing.Optional[nemo.lens.TelemetryHandle]

Return this process’s telemetry handle, or None if uninitialised/disabled.

nemo_gym.telemetry.init_telemetry(
    server_name: typing.Optional[str] = None,
    server_type: typing.Optional[str] = None,
    resource_attributes: typing.Optional[dict] = None
) -> typing.Optional[nemo.lens.TelemetryHandle]

Initialise this process’s telemetry. Call once per process; idempotent.

Reads the NEMO_GYM_OTEL_* environment that configure_telemetry_env put in place, so a server process needs no config file access to agree with its siblings.

Parameters:

server_name
Optional[str]Defaults to None

This server’s Gym config name (e.g. example_single_tool_call). Used to disambiguate service.name across the fleet.

server_type
Optional[str]Defaults to None

resources_servers | responses_api_agents | responses_api_models, or orchestrator for the CLI.

resource_attributes
Optional[dict]Defaults to None

Extra process-lifetime attributes to merge.

Returns: Optional[TelemetryHandle]

class:TelemetryHandle, or None when nemo-lens is absent or telemetry is

nemo_gym.telemetry.is_span_group_enabled(
    group
)

Always returns False.

nemo_gym.telemetry.is_telemetry_env_enabled() -> bool

True when telemetry is switched on in this process’s environment.

Cheap enough to call before doing setup work, and importantly it does not import nemo-lens — a process with telemetry off never pays for the lens import at all.

nemo_gym.telemetry.is_telemetry_metrics_enabled() -> bool

Return the effective metrics switch after Gym/Lens env precedence.

nemo_gym.telemetry.managed_span(
    group,
    name,
    tracer = None,
    attributes = {}
)

No-op context manager — yields None.

nemo_gym.telemetry.memory_profiling_config_from_env(
    defaults: nemo_gym.telemetry.config.MemoryProfilingConfig
) -> nemo_gym.telemetry.config.MemoryProfilingConfig

Resolve memory profiling after raw environment variables take precedence.

nemo_gym.telemetry.safe_set_span_attributes(
    span,
    attributes,
    redact_keys = None
)

No-op.

nemo_gym.telemetry.shutdown_telemetry(
    timeout_ms: int = 5000
) -> None

Flush and shut down this process’s telemetry providers.

Idempotent — TelemetryHandle.shutdown guards against a second call, and Gym reaches this from more than one terminal path. A no-op when Gym reused providers that another library set up. Never raises.

nemo_gym.telemetry.span_cm(
    name,
    tracer = None,
    record_exception = True,
    attributes = {}
)

No-op context manager — yields None.

nemo_gym.telemetry.telemetry_config_from_global_config(
    global_config_dict: typing.Any
) -> nemo_gym.telemetry.config.TelemetryConfig

Build a TelemetryConfig from Gym’s merged global config dict.

Reads the optional top-level telemetry: block. A run that never mentions telemetry gets the all-defaults config, which is disabled.

nemo_gym.telemetry.trace_fn(
    group,
    name,
    tracer = None
)

No-op decorator — returns the function unchanged.

nemo_gym.telemetry.setup.TELEMETRY_KEY_NAME = 'telemetry'