Configuration

Telemetry is configured through a top-level telemetry: block in your NeMo Gym config, or through environment variables. Environment variables always win.

Install

Telemetry ships as an optional extra:

uv sync --extra dev --extra telemetry

Without this extra, nemo-lens is absent, every instrumentation site is a no-op, and NeMo Gym behaves exactly as it does today.

The telemetry: Block

telemetry:
  enabled: true
  service_name: nemo-gym
  span_groups: default
  exporter: otlp

Memory Profiling

Enable memory profiling inside the telemetry block:

telemetry:
  enabled: true
  metrics_enabled: true
  memory_profiling:
    enabled: true
    interval_seconds: 1

The supervisor samples host used, available, and total memory plus every server process and its descendants, including Uvicorn workers and ordinary subprocesses. Collection starts after the servers become ready and stops before they are terminated. Server accounting reads Linux /proc; PSS is omitted when smaps_rollup is unavailable, while RSS falls back to statm.

Event Loop and Process Metrics

Whenever metrics are exported, every server process, the head server inside the process that runs gym env start or gym eval run, and the rollout driver report how busy their event loop is, how long garbage collection pauses them, their CPU, memory, threads, and open file descriptors, and how many requests they are handling. Use them to find a server whose event loop is the bottleneck. See Event Loop and Process Metrics for each metric.

Their cost stays off the request path: a request updates a few integers, and the exporter’s thread reads the totals when it exports. The event-loop task count is the exception, because counting walks every task on the loop, which takes about a millisecond per sample at 16,000 tasks. It is off by default:

telemetry:
  enabled: true
  metrics_enabled: true
  event_loop_sample_interval_seconds: 1
  event_loop_task_count: true

Why service_name_per_server Defaults to True

NeMo Gym starts several server processes. If they all report the same service.name, a backend’s service map collapses them into one node and you lose the ability to ask “how slow is the model server.” With the default, each process reports nemo-gym/<server name>, and the unsuffixed name remains available as the nemo.gym.service_group resource attribute so you can still group the fleet.

Every Server Process Exports

Each server is an independent process that reports itself as rank 0 of a world of one, so every server exports. Silencing any of them would leave a gap in every trace that crosses it. The export_strategy and export_rank fields were removed when nemo-lens dropped rank-based export filtering. Older configs that still set them keep loading, and the fields have no effect.

Environment Variables

Every field maps to an environment variable, and environment variables take precedence over the config block. This lets you enable telemetry for one run without editing a shared config file.

For the settings nemo-lens itself reads (enabled, span_groups, exporter, traces_enabled, metrics_enabled, logs_enabled, and run_id), NEMO_LENS_* works as a fallback, so NEMO_LENS_ENABLED=1 turns telemetry on. The memory profiling settings have no NEMO_LENS_* equivalent.

The order of precedence is:

1. NEMO_GYM_OTEL_*
2. NEMO_LENS_*
3. The telemetry: config block

A NEMO_LENS_* variable set by a job that runs NeMo Gym alongside other NeMo libraries, such as NeMo RL, therefore applies to NeMo Gym’s servers as well.

Boolean variables accept 1, true, yes, on and their negations.

Exporters

Console

NEMO_GYM_OTEL_EXPORTER=console

Writes one JSON object per line to stdout. Useful for local development and for confirming that instrumentation works before a collector exists. Because each record is a single line, the output pipes into jq directly.

OTLP

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
OTEL_EXPORTER_OTLP_PROTOCOL=grpc

The standard OTEL_EXPORTER_OTLP_* variables are read by the OpenTelemetry SDK, so any OTLP-compatible backend or an OpenTelemetry Collector works. gRPC is the default protocol. Set OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf to use HTTP instead.

How Settings Reach Each Server

NeMo Gym starts its servers as separate processes. They share no memory with the supervisor, so the telemetry: block is translated into NEMO_GYM_OTEL_* environment variables before any server is spawned, and each server reads that environment back on startup.

Two consequences are worth knowing:

• A variable you export in your shell reaches every server, because the supervisor copies its own environment into each child.
• A job submitted with gym eval submit needs none of this by hand: the submit sets NEMO_GYM_OTEL_ENABLED, the run id, the span groups, the logs switch and the exporter endpoint for the driver from its observability: block, pointing at the collector that runs beside the job. See Collector for Submitted Jobs.
• The supervisor generates one run_id and shares it with the fleet, so every process in a run reports the same nemo.run.id resource attribute.

Resource Attributes

Every span and metric carries these process-lifetime attributes:

Verify That It Works

Enable telemetry with the console exporter and drive one rollout. Each server process prints span records to stdout. Confirm that spans from different servers share one trace_id:

grep -h '"trace_id"' *.log | jq -r .trace_id | sort | uniq -c

One trace ID with several spans means propagation is working. Several trace IDs with one span each means it is not — see Distributed Tracing for what to check.