> This page is for version 0.7.0. > For other versions, use one of these documentation indexes: > - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt > - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt > - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt > - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt > - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt > - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt > - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt > - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server. # Configuration Telemetry is configured through a top-level `telemetry:` block in your NeMo Gym config, or through environment variables. Environment variables always win. ## Install Telemetry ships as an optional extra: ```bash uv sync --extra dev --extra telemetry ``` Without this extra, `nemo-lens` is absent, every instrumentation site is a no-op, and NeMo Gym behaves exactly as it does today. ## The `telemetry:` Block ```yaml telemetry: enabled: true service_name: nemo-gym span_groups: default exporter: otlp ``` | Field | Default | Description | | ------------------------- | ---------- | ------------------------------------------------------------------------------ | | `enabled` | `false` | Master switch. When false, every site is a no-op. | | `service_name` | `nemo-gym` | The `service.name` reported to your backend. | | `service_name_per_server` | `true` | Append each server's config name, producing `nemo-gym/policy_model` and so on. | | `span_groups` | `default` | Which spans exist. See [Span Groups](/main/observability/span-groups). | | `exporter` | `otlp` | `otlp` or `console`. | | `traces_enabled` | `true` | Emit spans. | | `metrics_enabled` | `true` | Emit metrics. | | `memory_profiling` | disabled | Sample host capacity and each managed server process tree. See below. | | `logs_enabled` | `false` | Bridge Python logging to OpenTelemetry logs. | | `run_id` | generated | Correlates every process of one run. | ### Memory Profiling Enable memory profiling inside the telemetry block: ```yaml telemetry: enabled: true metrics_enabled: true memory_profiling: enabled: true interval_seconds: 1 ``` The orchestrator samples host used, available, and total memory plus every server process and its descendants, including Uvicorn workers and ordinary subprocesses. Collection starts after the servers become ready and stops before they are terminated. Server accounting reads Linux `/proc`; PSS is omitted when `smaps_rollup` is unavailable, while RSS falls back to `statm`. ### Why `service_name_per_server` Defaults to True NeMo Gym starts several server processes. If they all report the same `service.name`, a backend's service map collapses them into one node and you lose the ability to ask "how slow is the model server." With the default, each process reports `nemo-gym/`, and the unsuffixed name remains available as the `nemo.gym.service_group` resource attribute so you can still group the fleet. ### Every Server Process Exports Each server is an independent process that reports itself as rank 0 of a world of one, so every server exports. Silencing any of them would leave a gap in every trace that crosses it. The `export_strategy` and `export_rank` fields were removed when `nemo-lens` dropped rank-based export filtering. Older configs that still set them keep loading, and the fields have no effect. ## Environment Variables Every field maps to an environment variable, and environment variables take precedence over the config block. This lets you enable telemetry for one run without editing a shared config file. | Variable | Maps To | | ------------------------------------------------- | ----------------------------------- | | `NEMO_GYM_OTEL_ENABLED` | `enabled` | | `NEMO_GYM_OTEL_SPAN_GROUPS` | `span_groups` | | `NEMO_GYM_OTEL_EXPORTER` | `exporter` | | `NEMO_GYM_OTEL_TRACES_ENABLED` | `traces_enabled` | | `NEMO_GYM_OTEL_METRICS_ENABLED` | `metrics_enabled` | | `NEMO_GYM_OTEL_MEMORY_PROFILING_ENABLED` | `memory_profiling.enabled` | | `NEMO_GYM_OTEL_MEMORY_PROFILING_INTERVAL_SECONDS` | `memory_profiling.interval_seconds` | | `NEMO_GYM_OTEL_LOGS_ENABLED` | `logs_enabled` | | `NEMO_GYM_OTEL_RUN_ID` | `run_id` | | `OTEL_SERVICE_NAME` | `service_name` | `NEMO_LENS_*` works as a fallback for any of the `NEMO_GYM_OTEL_*` names, so `NEMO_LENS_ENABLED=1` turns telemetry on. A `NEMO_GYM_OTEL_*` variable beats its `NEMO_LENS_*` equivalent. Boolean variables accept `1`, `true`, `yes`, `on` and their negations. ## Exporters ### Console ```bash NEMO_GYM_OTEL_EXPORTER=console ``` Writes one JSON object per line to stdout. Useful for local development and for confirming that instrumentation works before a collector exists. Because each record is a single line, the output pipes into `jq` directly. ### OTLP ```bash OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 OTEL_EXPORTER_OTLP_PROTOCOL=grpc ``` The standard `OTEL_EXPORTER_OTLP_*` variables are read by the OpenTelemetry SDK, so any OTLP-compatible backend or an OpenTelemetry Collector works. gRPC is the default protocol. Set `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` to use HTTP instead. ## How Settings Reach Each Server NeMo Gym starts its servers as separate processes. They share no memory with the orchestrator, so the `telemetry:` block is translated into `NEMO_GYM_OTEL_*` environment variables before any server is spawned, and each server reads that environment back on startup. Two consequences are worth knowing: * A variable you export in your shell reaches every server, because the orchestrator copies its own environment into each child. * A job submitted with `gym eval submit` needs none of this by hand: the submit sets `NEMO_GYM_OTEL_ENABLED`, the run id, the span groups, the logs switch and the exporter endpoint for the driver from its `observability:` block, pointing at the collector that runs beside the job. See [Collector for Submitted Jobs](/main/observability/eval-submit-collector). * The orchestrator generates one `run_id` and shares it with the fleet, so every process in a run reports the same `nemo.run.id` resource attribute. ## Resource Attributes Every span and metric carries these process-lifetime attributes: | Attribute | Example | | ------------------------ | ----------------------- | | `service.name` | `nemo-gym/policy_model` | | `nemo.gym.service_group` | `nemo-gym` | | `nemo.gym.server.name` | `policy_model` | | `nemo.gym.server.type` | `responses_api_models` | | `nemo.gym.version` | `0.5.1` | | `nemo.run.id` | `4f2a91c0be31` | ## Verify That It Works Enable telemetry with the console exporter and drive one rollout. Each server process prints span records to stdout. Confirm that spans from different servers share one `trace_id`: ```bash grep -h '"trace_id"' *.log | jq -r .trace_id | sort | uniq -c ``` One trace ID with several spans means propagation is working. Several trace IDs with one span each means it is not — see [Distributed Tracing](/main/observability/distributed-tracing) for what to check.