Skip to navigation
API ReferenceFull Library ReferenceNemo GymNemo Gym

nemo_gym.telemetry

View as Markdown

NeMo Gym telemetry: optional OpenTelemetry instrumentation via nemo-lens.

Importing this package never requires nemo-lens. Install it with uv sync --extra telemetry and switch it on with a telemetry: block or NEMO_LENS_ENABLED=1; with either missing, every instrumentation site in nemo_gym is a ~0-cost no-op.

Public surface

  • TelemetryConfig — the telemetry: config block.
  • GymSpanGroup — Gym span groups and presets.
  • configure_telemetry_env — supervisor side; hands the settings to spawned server processes through the environment.
  • init_telemetry / get_telemetry / shutdown_telemetry — per-process lifecycle.

Instrumentation primitives (managed_span / span_cm / trace_fn / is_span_group_enabled / safe_set_span_attributes) come from nemo_gym.telemetry._fallbacks, which resolves to the real nemo-lens implementations when it is installed and to no-op stubs when it is not.

See nemo_gym/telemetry/README.md for the design, and fern/versions/latest/pages/observability/ for user documentation.

Submodules

Package Contents

Classes

NameDescription
GymSpanGroupSpan group names for NeMo Gym instrumentation.
MemoryProfilingConfigPeriodic host and logical-server memory sampling.
TelemetryConfigOpenTelemetry / nemo-lens configuration for NeMo Gym.

Functions

NameDescription
configure_telemetry_envTranslate the telemetry: block into env vars for spawned server processes.
get_telemetryReturn this process’s telemetry handle, or None if uninitialised/disabled.
init_telemetryInitialise this process’s telemetry. Call once per process; idempotent.
is_span_group_enabledAlways returns False.
is_telemetry_env_enabledTrue when telemetry is switched on in this process’s environment.
is_telemetry_metrics_enabledReturn the effective metrics switch after Gym/Lens env precedence.
managed_spanNo-op context manager — yields None.
memory_profiling_config_from_envResolve memory profiling after raw environment variables take precedence.
safe_set_span_attributesNo-op.
shutdown_telemetryFlush and shut down this process’s telemetry providers.
span_cmNo-op context manager — yields None.
telemetry_config_from_global_configBuild a TelemetryConfig from Gym’s merged global config dict.
trace_fnNo-op decorator — returns the function unchanged.

Data

TELEMETRY_KEY_NAME

API

class nemo_gym.telemetry.GymSpanGroup()

Span group names for NeMo Gym instrumentation.

AGENT
= 'agent'

Agent-server /run and /v1/responses spans.

ALL_GROUPS
Final[frozenset]
CROSS_PROCESS_SPINE
Final[frozenset] = frozenset([SERVER, HTTP_CLIENT, ROLLOUT])
HTTP_CLIENT
= 'http_client'

Outbound spans around nemo_gym.server_utils.request — Gym’s single aiohttp egress point. The egress half of cross-process propagation: injects traceparent into the outgoing headers.

JOB
= 'job'

The whole gym eval / rollout-collection run, driver side. Shares its name with NeMo-RL’s run-level group, so one spec entry selects both.

MODEL_CALL
= 'model_call'

Model-server /v1/chat/completions, /v1/responses and /v1/messages spans.

ROLLOUT
= 'rollout'

Rollout collection spans (one per task attempt, driver side).

SANDBOX
= 'sandbox'

Sandbox provider create/exec/delete spans.

SERVER
= 'server'

Inbound FastAPI request spans on every Gym server process. The ingress half of cross-process propagation: adopts an inbound traceparent as the span’s parent.

STARTUP
= 'startup'

Server startup spans, once per run.

The supervisor emits gym.startup and one gym.startup.server span per server. Each server process emits gym.server.startup or gym.server.startup.worker with one child span per stage.

VERIFY
= 'verify'

Resources-server /verify spans.

_PRESETS
dict
nemo_gym.telemetry.GymSpanGroup.resolve(
    spec: str
) -> frozenset
classmethod

Resolve a span_groups spec against every group registered in this process.

Entries that name nothing registered are dropped rather than raised: the library that owns them may not be imported in this process. nemo-lens logs a warning naming them when the spec is applied in setup_telemetry.

Raises:

  • RuntimeError: nemo-lens is not installed, so there is nothing to resolve.
class nemo_gym.telemetry.MemoryProfilingConfig()

Bases: BaseModel

Periodic host and logical-server memory sampling.

enabled
bool = False

Collect host memory and every server process tree.

interval_seconds
float = Field(default=1.0, gt=0)

Seconds between samples.

class nemo_gym.telemetry.TelemetryConfig()

Bases: BaseModel

OpenTelemetry / nemo-lens configuration for NeMo Gym.

Telemetry is optional twice over: it activates only when enabled is true and nemo-lens is installed (uv sync --extra telemetry). When either is missing every instrumentation site in nemo_gym degrades to a ~0-cost no-op and Gym behaves exactly as it does today.

enabled
bool = False

Master switch. When false, all instrumentation is a ~0-cost no-op.

event_loop_sample_interval_seconds
float = Field(default=1.0, gt=0)

Seconds between samples of each process’s event-loop lag, when metrics are on.

event_loop_task_count
bool = False

Also report how many asyncio tasks exist on each event loop, at every sample.

Off by default because counting tasks walks every task on the loop while holding it. That takes about a millisecond per sample at 16,000 tasks, which a driver at high concurrency can reach.

exporter
str = 'otlp'

Exporter backend: otlp | console. console writes one JSON object per line to stdout, which is what the local validation flow greps.

logs_enabled
bool = False

Bridge Python logging to OTel logs, exported with trace correlation.

memory_profiling
MemoryProfilingConfig = Field(default_factory=MemoryProfilingConfig)

Sample each Gym-managed server process tree and emit memory metrics.

metrics_enabled
bool = True

Emit metric instruments (the gym.* histograms/gauges plus the FastAPI instrumentor’s http.server.*).

otlp_endpoint
Optional[str] = None

OTEL_EXPORTER_OTLP_ENDPOINT for the otlp exporter, e.g. https://api.honeycomb.io. Optional: the standard env var still works and always wins (setdefault), so any OTLP-compatible backend or collector that expects the raw env var keeps working unchanged. Setting it here just means a run’s destination is committed with the rest of the config instead of exported by hand every time.

otlp_headers
Optional[str] = None

OTEL_EXPORTER_OTLP_HEADERS, e.g. x-honeycomb-team=.... Prefer ${oc.env:MY_API_KEY} interpolation for the secret portion so the raw credential does not live in a committed YAML file.

otlp_protocol
Optional[str] = None

OTEL_EXPORTER_OTLP_PROTOCOL: grpc (the OTel SDK default) or http/protobuf.

run_id
Optional[str] = None

Correlates every process of one gym env start / gym env test invocation. Generated in the supervisor and inherited by the servers when unset.

service_name
str = 'nemo-gym'

service.name reported to the backend.

Passed through the standard OTEL_SERVICE_NAME env var, not through instrument_fastapi(service_name=...) — that parameter is accepted and ignored by nemo-lens at the pinned commit (contrib/fastapi.py).

Each server process appends its own Gym server name, so a run yields nemo-gym/example_single_tool_call, nemo-gym/policy_model, and so on rather than three indistinguishable nemo-gym services. Set service_name_per_server to false to opt out.

service_name_per_server
bool = True

Suffix service_name with each server’s Gym config name.

On by default because Gym runs several processes that would otherwise share one service.name, which makes a backend’s service map useless. The unsuffixed name stays available as the nemo.gym.service_group resource attribute.

span_groups
str = 'default'

Span-group spec: a preset (default | per_rollout | all) or a comma-separated list of group names (e.g. "default,sandbox"). See GymSpanGroup. An unknown name is logged as a warning and ignored rather than rejected.

traces_enabled
bool = True

Emit trace spans.

nemo_gym.telemetry.configure_telemetry_env(
    telemetry_config: typing.Union[nemo_gym.telemetry.config.TelemetryConfig, None]
) -> typing.Optional[str]

Translate the telemetry: block into env vars for spawned server processes.

Call once in the supervisor, before spawning any server. os.environ is what Popen snapshots into each child, so this is how a YAML setting reaches a server process that shares nothing else with its parent.

Uses setdefault throughout: a raw NEMO_GYM_OTEL_* / NEMO_LENS_* / OTEL_SERVICE_NAME / OTEL_EXPORTER_OTLP_* set by the user always wins over YAML.

Returns the run id shared by every process in this run, or None when telemetry is disabled.

nemo_gym.telemetry.get_telemetry() -> typing.Optional[nemo.lens.TelemetryHandle]

Return this process’s telemetry handle, or None if uninitialised/disabled.

nemo_gym.telemetry.init_telemetry(
    server_name: typing.Optional[str] = None,
    server_type: typing.Optional[str] = None,
    resource_attributes: typing.Optional[dict] = None
) -> typing.Optional[nemo.lens.TelemetryHandle]

Initialise this process’s telemetry. Call once per process; idempotent.

Reads the NEMO_GYM_OTEL_* environment that configure_telemetry_env put in place, so a server process needs no config file access to agree with its siblings.

Parameters:

server_name
Optional[str]Defaults to None

This server’s Gym config name (e.g. example_single_tool_call). Used to disambiguate service.name across the fleet.

server_type
Optional[str]Defaults to None

resources_servers | responses_api_agents | responses_api_models, or supervisor for the CLI.

resource_attributes
Optional[dict]Defaults to None

Extra process-lifetime attributes to merge.

Returns: Optional[TelemetryHandle]

class:TelemetryHandle, or None when nemo-lens is absent or telemetry is

nemo_gym.telemetry.is_span_group_enabled(
    group
)

Always returns False.

nemo_gym.telemetry.is_telemetry_env_enabled() -> bool

True when telemetry is switched on in this process’s environment.

Cheap enough to call before doing setup work, and importantly it does not import nemo-lens — a process with telemetry off never pays for the lens import at all.

nemo_gym.telemetry.is_telemetry_metrics_enabled() -> bool

Return the effective metrics switch after Gym/Lens env precedence.

nemo_gym.telemetry.managed_span(
    group,
    name,
    tracer = None,
    attributes = {}
)

No-op context manager — yields None.

nemo_gym.telemetry.memory_profiling_config_from_env(
    defaults: nemo_gym.telemetry.config.MemoryProfilingConfig
) -> nemo_gym.telemetry.config.MemoryProfilingConfig

Resolve memory profiling after raw environment variables take precedence.

nemo_gym.telemetry.safe_set_span_attributes(
    span,
    attributes,
    redact_keys = None
)

No-op.

nemo_gym.telemetry.shutdown_telemetry(
    timeout_ms: int = 5000
) -> None

Flush and shut down this process’s telemetry providers.

Idempotent — TelemetryHandle.shutdown guards against a second call, and Gym reaches this from more than one terminal path. A no-op when Gym reused providers that another library set up. Never raises.

nemo_gym.telemetry.span_cm(
    name,
    tracer = None,
    record_exception = True,
    attributes = {}
)

No-op context manager — yields None.

nemo_gym.telemetry.telemetry_config_from_global_config(
    global_config_dict: typing.Any
) -> nemo_gym.telemetry.config.TelemetryConfig

Build a TelemetryConfig from Gym’s merged global config dict.

Reads the optional top-level telemetry: block. A run that never mentions telemetry gets the all-defaults config, which is disabled.

nemo_gym.telemetry.trace_fn(
    group,
    name,
    tracer = None
)

No-op decorator — returns the function unchanged.

nemo_gym.telemetry.setup.TELEMETRY_KEY_NAME = 'telemetry'