Configuration

Telemetry is configured through a top-level telemetry: block in your NeMo Gym config, or through environment variables. Environment variables always win.

Install

Telemetry ships as an optional extra:

uv sync --extra dev --extra telemetry

Without this extra, nemo-lens is absent, every instrumentation site is a no-op, and NeMo Gym behaves exactly as it does today.

The telemetry: Block

telemetry:
  enabled: true
  service_name: nemo-gym
  span_groups: default
  exporter: otlp

Memory Profiling

Enable memory profiling inside the telemetry block:

telemetry:
  enabled: true
  metrics_enabled: true
  memory_profiling:
    enabled: true
    interval_seconds: 1

The orchestrator samples host used, available, and total memory plus every server process and its descendants, including Uvicorn workers and ordinary subprocesses. Collection starts after the servers become ready and stops before they are terminated. Server accounting reads Linux /proc; PSS is omitted when smaps_rollup is unavailable, while RSS falls back to statm.

Why service_name_per_server Defaults to True

NeMo Gym starts several server processes. If they all report the same service.name, a backend’s service map collapses them into one node and you lose the ability to ask “how slow is the model server.” With the default, each process reports nemo-gym/<server name>, and the unsuffixed name remains available as the nemo.gym.service_group resource attribute so you can still group the fleet.

Every Server Process Exports

Each server is an independent process that reports itself as rank 0 of a world of one, so every server exports. Silencing any of them would leave a gap in every trace that crosses it. The export_strategy and export_rank fields were removed when nemo-lens dropped rank-based export filtering. Older configs that still set them keep loading, and the fields have no effect.

Environment Variables

Every field maps to an environment variable, and environment variables take precedence over the config block. This lets you enable telemetry for one run without editing a shared config file.

NEMO_LENS_* works as a fallback for any of the NEMO_GYM_OTEL_* names, so NEMO_LENS_ENABLED=1 turns telemetry on. A NEMO_GYM_OTEL_* variable beats its NEMO_LENS_* equivalent.

Boolean variables accept 1, true, yes, on and their negations.

Exporters

Console

NEMO_GYM_OTEL_EXPORTER=console

Writes one JSON object per line to stdout. Useful for local development and for confirming that instrumentation works before a collector exists. Because each record is a single line, the output pipes into jq directly.

OTLP

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
OTEL_EXPORTER_OTLP_PROTOCOL=grpc

The standard OTEL_EXPORTER_OTLP_* variables are read by the OpenTelemetry SDK, so any OTLP-compatible backend or an OpenTelemetry Collector works. gRPC is the default protocol. Set OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf to use HTTP instead.

How Settings Reach Each Server

NeMo Gym starts its servers as separate processes. They share no memory with the orchestrator, so the telemetry: block is translated into NEMO_GYM_OTEL_* environment variables before any server is spawned, and each server reads that environment back on startup.

Two consequences are worth knowing:

• A variable you export in your shell reaches every server, because the orchestrator copies its own environment into each child.
• A job submitted with gym eval submit needs none of this by hand: the submit sets NEMO_GYM_OTEL_ENABLED, the run id, the span groups, the logs switch and the exporter endpoint for the driver from its observability: block, pointing at the collector that runs beside the job. See Collector for Submitted Jobs.
• The orchestrator generates one run_id and shares it with the fleet, so every process in a run reports the same nemo.run.id resource attribute.

Resource Attributes

Every span and metric carries these process-lifetime attributes:

Verify That It Works

Enable telemetry with the console exporter and drive one rollout. Each server process prints span records to stdout. Confirm that spans from different servers share one trace_id:

grep -h '"trace_id"' *.log | jq -r .trace_id | sort | uniq -c

One trace ID with several spans means propagation is working. Several trace IDs with one span each means it is not — see Distributed Tracing for what to check.