> This page is for version 0.7.0.
> For other versions, use one of these documentation indexes:
> - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt
> - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt
> - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt
> - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt
> - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt
> - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt
> - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt
> - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server.

# Metrics

NeMo Gym emits a small, deliberate set of metrics. This page lists them, and explains what
is deliberately absent and why.

## Application Telemetry, Not Experiment Telemetry

These metrics answer "how is this software behaving?" — where time goes, what failed, how
a configuration performs on hardware you do not control.

They do not answer "how is my run doing?" Reward, accuracy, and benchmark scores are
experiment telemetry. They belong in Weights & Biases or MLflow, which are built for
comparing runs and keep data for the lifetime of a project. NeMo Gym already reports them
through `AggregateMetricsMixin` and continues to do so, unchanged.

Mirroring reward into OpenTelemetry would put a number in a system that aggregates across
every deployment and retains for weeks. That is the wrong home for it, and the two systems
would disagree the first time one dropped a sample.

## Metrics NeMo Gym Emits

| Metric                                       | Type               | Unit     | Meaning                                                                                                            |
| -------------------------------------------- | ------------------ | -------- | ------------------------------------------------------------------------------------------------------------------ |
| `gym.rollout.duration_ms`                    | Histogram          | ms       | Wall-clock duration of one rollout, from the agent server's `/run`.                                                |
| `gym.verify.duration_ms`                     | Histogram          | ms       | Wall-clock duration of one verification.                                                                           |
| `gym.verify.success_rate`                    | Gauge              | fraction | Cumulative fraction of verifications that completed. See below.                                                    |
| `gym.servers.active`                         | Gauge              | count    | Number of servers the orchestrator started.                                                                        |
| `gym.http.connection_pool.queue_duration_ms` | Histogram          | ms       | Time one queued connection acquisition waited for an aiohttp pool slot.                                            |
| `gym.http.connection_pool.connect_total`     | Observable counter | count    | All outbound aiohttp connection attempts (`connect()` calls), including attempts that later fail or are abandoned. |
| `gym.server.memory.usage`                    | Gauge              | bytes    | Current RSS or PSS for one managed server and its descendants.                                                     |
| `gym.server.memory.sampled_peak`             | Gauge              | bytes    | Largest value observed by the sampler for one managed server and its descendants.                                  |
| `gym.server.processes`                       | Gauge              | count    | Processes currently attributed to one managed server.                                                              |
| `gym.host.memory.usage`                      | Gauge              | bytes    | Current host memory in use, calculated as total minus available.                                                   |
| `gym.host.memory.available`                  | Gauge              | bytes    | Host memory available without swapping.                                                                            |
| `gym.host.memory.total`                      | Gauge              | bytes    | Total host memory.                                                                                                 |
| `gym.host.memory.sampled_peak`               | Gauge              | bytes    | Largest sampled host memory usage.                                                                                 |

Rollout duration is recorded even when a rollout fails. A crashed rollout still consumed
wall-clock time, and dropping those samples would bias the histogram toward the fast path.

### Memory Metrics

Memory metrics are emitted only when `telemetry.memory_profiling.enabled` is true. Server points carry `nemo.gym.server.name`, `nemo.gym.server.type`, and the root `process.pid`; server memory gauges also carry `memory.type`, which is `rss` or `pss`. Host points rely on the existing `host.name` resource attribute and carry no per-sample attributes.

RSS counts every resident page mapped by every process and can double-count pages shared by Uvicorn workers. PSS divides shared pages among the processes mapping them, so summing PSS across server trees is the better estimate of their combined host footprint. The sampled peak is not the kernel's exact lifetime peak: an allocation that begins and ends between samples can be missed.

Host usage is `total - available`, matching the memory the operating system cannot make available without swapping. Host totals describe the whole node, not only Gym processes; compare them with summed server PSS to separate Gym footprint from training, inference, Ray, and system pressure.

Descendants remain attributed while they stay in the server's Linux process tree. Detached or reparented processes and remote Ray actors are not included.

Compare these series with `http.server.active_requests` while increasing workload concurrency. Separate server processes export their HTTP metrics independently, so sum active requests by Gym server name when a server uses several Uvicorn workers.

### What `gym.verify.success_rate` Measures

It measures whether the **verification call completed**, not whether the task passed.

A verifier that correctly scores a wrong answer is a success. A verifier that raises an
exception is a failure. Task correctness is experiment telemetry, and putting it here would
turn this into a low-resolution accuracy metric living in the wrong system.

The value is cumulative over the reporting process's lifetime, not a windowed rate. A gauge
carries last-value semantics, so a rate cannot be expressed directly. A long-lived server
therefore shows a figure that moves more slowly as it ages. Read it as "this process's
success fraction so far," and use traces for anything finer.

### Why Only the Orchestrator Reports `gym.servers.active`

`gym.servers.active` is a gauge, so the exported value is whichever process wrote last.
If every server reported it, the result would be neither the fleet size nor any process's
view of it. The orchestrator is the only process that knows how many servers it started,
so it is the only writer. It sets the count after startup and zero on shutdown.

## HTTP Metrics

FastAPI auto-instrumentation emits the standard HTTP server metrics on every NeMo Gym
server:

| Metric                         | Dimensions                                                       |
| ------------------------------ | ---------------------------------------------------------------- |
| `http.server.active_requests`  | `http.server_name`, `http.method`                                |
| `http.server.request.duration` | `http.route`, `http.request.method`, `http.response.status_code` |

Use these for request latency. They are dimensioned, so you can compare `/verify` against
`/v1/responses`, or isolate 500s, which is what latency questions usually require.

The Gym-specific `gym.http.connection_pool.queue_duration_ms` histogram measures client-side
connection acquisition separately from request latency. Its bounded attributes identify the
destination server, queue outcome (`ok` or `abandoned`), and which pool constraint caused the
wait (`total`, `per_host`, or `unknown`). It records one sample for each queued connection
acquisition (`connect()`), combining repeated waiter wakeups within that acquisition.
Redirect hops, aiohttp's internal reconnects, and Gym retries record separate samples when
their acquisitions queue. Connection attempts that acquire a slot immediately do not record
a queue-duration histogram sample.
`abandoned` means the wait ended before a slot became available, including timeout and cancellation.
Divide the number of `gym.http.connection_pool.queue_duration_ms` histogram samples by
`gym.http.connection_pool.connect_total` over the same interval and destination to calculate the
fraction of connection attempts that queued.

Both instruments use `nemo.gym.http.destination.server.name` for the request destination:
the configured server name for `ServerClient` calls, `remote_agent_service` for calls to the
remote agent's external service, or `external` for unlabeled direct calls. This differs from
the resource attribute `nemo.gym.server.name`, which identifies the Gym process recording the metric.

These pool metrics require telemetry initialization before client construction. Standalone
collection does not initialize telemetry automatically, so missing metrics do not mean no
queueing. See [HTTP Connection Pool](/reference/configuration#http-connection-pool) for process coverage.

### A Note on `gym.server.request_duration_ms`

`nemo-lens` declares a `gym.server.request_duration_ms` histogram. NeMo Gym does not use
it.

At the pinned `nemo-lens` version, that instrument records without attributes. Every
endpoint of every server type would land in one undimensioned histogram: a verification,
an inference call, and a liveness probe sharing one set of buckets. The result answers no
question anyone has. The dimensioned `http.server.request.duration` above supersedes it.

## Enabling and Disabling

Metrics follow the master switch and can be turned off independently:

```yaml
telemetry:
  enabled: true
  metrics_enabled: false   # traces only
```

Metrics export on a timer. The default interval is 10 seconds and is configurable with the
standard `OTEL_METRIC_EXPORT_INTERVAL` environment variable, in milliseconds. A process
that lives for less than one interval may exit before exporting anything.