Metrics | NeMo Gym

NeMo Gym emits a small, deliberate set of metrics. This page lists them, and explains what is deliberately absent and why.

Application Telemetry, Not Experiment Telemetry

These metrics answer “how is this software behaving?” — where time goes, what failed, how a configuration performs on hardware you do not control.

They do not answer “how is my run doing?” Reward, accuracy, and benchmark scores are experiment telemetry. They belong in Weights & Biases or MLflow, which are built for comparing runs and keep data for the lifetime of a project. NeMo Gym already reports them through AggregateMetricsMixin and continues to do so, unchanged.

Mirroring reward into OpenTelemetry would put a number in a system that aggregates across every deployment and retains for weeks. That is the wrong home for it, and the two systems would disagree the first time one dropped a sample.

Metrics NeMo Gym Emits

Metric	Type	Unit	Meaning
`gym.rollout.duration_ms`	Histogram	ms	Wall-clock duration of one rollout, from the agent server’s `/run`.
`gym.verify.duration_ms`	Histogram	ms	Wall-clock duration of one verification.
`gym.verify.success_rate`	Gauge	fraction	Cumulative fraction of verifications that completed. See below.
`gym.servers.active`	Gauge	count	Number of servers the orchestrator started.
`gym.http.connection_pool.queue_duration_ms`	Histogram	ms	Time one queued connection acquisition waited for an aiohttp pool slot.
`gym.http.connection_pool.connect_total`	Observable counter	count	All outbound aiohttp connection attempts (`connect()` calls), including attempts that later fail or are abandoned.
`gym.server.memory.usage`	Gauge	bytes	Current RSS or PSS for one managed server and its descendants.
`gym.server.memory.sampled_peak`	Gauge	bytes	Largest value observed by the sampler for one managed server and its descendants.
`gym.server.processes`	Gauge	count	Processes currently attributed to one managed server.
`gym.host.memory.usage`	Gauge	bytes	Current host memory in use, calculated as total minus available.
`gym.host.memory.available`	Gauge	bytes	Host memory available without swapping.
`gym.host.memory.total`	Gauge	bytes	Total host memory.
`gym.host.memory.sampled_peak`	Gauge	bytes	Largest sampled host memory usage.

Rollout duration is recorded even when a rollout fails. A crashed rollout still consumed wall-clock time, and dropping those samples would bias the histogram toward the fast path.

Memory Metrics

Memory metrics are emitted only when telemetry.memory_profiling.enabled is true. Server points carry nemo.gym.server.name, nemo.gym.server.type, and the root process.pid; server memory gauges also carry memory.type, which is rss or pss. Host points rely on the existing host.name resource attribute and carry no per-sample attributes.

RSS counts every resident page mapped by every process and can double-count pages shared by Uvicorn workers. PSS divides shared pages among the processes mapping them, so summing PSS across server trees is the better estimate of their combined host footprint. The sampled peak is not the kernel’s exact lifetime peak: an allocation that begins and ends between samples can be missed.

Host usage is total - available, matching the memory the operating system cannot make available without swapping. Host totals describe the whole node, not only Gym processes; compare them with summed server PSS to separate Gym footprint from training, inference, Ray, and system pressure.

Descendants remain attributed while they stay in the server’s Linux process tree. Detached or reparented processes and remote Ray actors are not included.

Compare these series with http.server.active_requests while increasing workload concurrency. Separate server processes export their HTTP metrics independently, so sum active requests by Gym server name when a server uses several Uvicorn workers.

What `gym.verify.success_rate` Measures

It measures whether the verification call completed, not whether the task passed.

A verifier that correctly scores a wrong answer is a success. A verifier that raises an exception is a failure. Task correctness is experiment telemetry, and putting it here would turn this into a low-resolution accuracy metric living in the wrong system.

The value is cumulative over the reporting process’s lifetime, not a windowed rate. A gauge carries last-value semantics, so a rate cannot be expressed directly. A long-lived server therefore shows a figure that moves more slowly as it ages. Read it as “this process’s success fraction so far,” and use traces for anything finer.

Why Only the Orchestrator Reports `gym.servers.active`

gym.servers.active is a gauge, so the exported value is whichever process wrote last. If every server reported it, the result would be neither the fleet size nor any process’s view of it. The orchestrator is the only process that knows how many servers it started, so it is the only writer. It sets the count after startup and zero on shutdown.

HTTP Metrics

FastAPI auto-instrumentation emits the standard HTTP server metrics on every NeMo Gym server:

Metric	Dimensions
`http.server.active_requests`	`http.server_name`, `http.method`
`http.server.request.duration`	`http.route`, `http.request.method`, `http.response.status_code`

Use these for request latency. They are dimensioned, so you can compare /verify against /v1/responses, or isolate 500s, which is what latency questions usually require.

The Gym-specific gym.http.connection_pool.queue_duration_ms histogram measures client-side connection acquisition separately from request latency. Its bounded attributes identify the destination server, queue outcome (ok or abandoned), and which pool constraint caused the wait (total, per_host, or unknown). It records one sample for each queued connection acquisition (connect()), combining repeated waiter wakeups within that acquisition. Redirect hops, aiohttp’s internal reconnects, and Gym retries record separate samples when their acquisitions queue. Connection attempts that acquire a slot immediately do not record a queue-duration histogram sample. abandoned means the wait ended before a slot became available, including timeout and cancellation. Divide the number of gym.http.connection_pool.queue_duration_ms histogram samples by gym.http.connection_pool.connect_total over the same interval and destination to calculate the fraction of connection attempts that queued.

Both instruments use nemo.gym.http.destination.server.name for the request destination: the configured server name for ServerClient calls, remote_agent_service for calls to the remote agent’s external service, or external for unlabeled direct calls. This differs from the resource attribute nemo.gym.server.name, which identifies the Gym process recording the metric.

These pool metrics require telemetry initialization before client construction. Standalone collection does not initialize telemetry automatically, so missing metrics do not mean no queueing. See HTTP Connection Pool for process coverage.

A Note on `gym.server.request_duration_ms`

nemo-lens declares a gym.server.request_duration_ms histogram. NeMo Gym does not use it.

At the pinned nemo-lens version, that instrument records without attributes. Every endpoint of every server type would land in one undimensioned histogram: a verification, an inference call, and a liveness probe sharing one set of buckets. The result answers no question anyone has. The dimensioned http.server.request.duration above supersedes it.

Enabling and Disabling

Metrics follow the master switch and can be turned off independently:

telemetry:
  enabled: true
  metrics_enabled: false   # traces only

Metrics export on a timer. The default interval is 10 seconds and is configurable with the standard OTEL_METRIC_EXPORT_INTERVAL environment variable, in milliseconds. A process that lives for less than one interval may exit before exporting anything.