> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server.

# nemo_gym.telemetry.metrics

Gym-side wrapper over nemo-lens's `gym.*` metric instruments.

`nemo.lens.instruments.gym.record_gym_metrics` records **without attributes** at the
pinned commit — every instrument is undimensioned. That is not a detail that can be
papered over, so this module takes an explicit position on each of the five:

`gym.rollout.duration_ms` (histogram)
Used. One rollout is one comparable unit of work, so an undimensioned distribution is
still meaningful. :func:`record_rollout_duration`.

`gym.verify.duration_ms` (histogram)
Used. Same reasoning, per verification. :func:`record_verify`.

`gym.verify.success_rate` (gauge)
Used, with a stated window. A gauge is last-value, so it cannot express "rate" on its
own; this module keeps a process-local running tally and sets the gauge to the
**cumulative** success fraction since process start. That is a well-defined number,
but it is not a windowed rate, and it flattens as a process ages — see the module
docs. Counters would be the right instrument; that needs a lens change.

`gym.servers.active` (gauge)
Used from **exactly one process**. A gauge is last-value semantics, so if every
server process set it the exported value would be whichever process happened to write
last — a meaningless number that looks like a real one. :func:`record_active_servers`
is orchestrator-only and refuses to run anywhere else.

`gym.server.request_duration_ms` (histogram)
**Deliberately unused.** With no attributes it would collapse every endpoint of every
server type into one histogram: `/verify` on a resources server, `/v1/responses`
on a model server and a liveness probe would land in the same bucket set, and the
result answers no question anyone has. The FastAPI auto-instrumentation Gym enables
already emits `http.server.request.duration` dimensioned by `http.route`,
`http.request.method` and `http.response.status_code`, which is strictly better.
Use that instead; see `fern/versions/latest/pages/observability/metrics.mdx`.

Every function here is a no-op unless telemetry is initialised *and* exporting, so call
sites do not need their own guards for correctness — though they should still sit under a
span-group gate to stay free when disabled.

## Module Contents

### Functions

| Name                                                                                             | Description                                                              |
| ------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------ |
| [`_record`](#nemo_gym-telemetry-metrics-_record)                                                 | Forward to `record_gym_metrics` when a meter is available; never raise.  |
| [`_reset_verify_tally_for_testing`](#nemo_gym-telemetry-metrics-_reset_verify_tally_for_testing) | Reset the cumulative verify tally. Test-only.                            |
| [`record_active_servers`](#nemo_gym-telemetry-metrics-record_active_servers)                     | Set `gym.servers.active` — orchestrator only.                            |
| [`record_rollout_duration`](#nemo_gym-telemetry-metrics-record_rollout_duration)                 | Record one rollout's wall-clock duration into `gym.rollout.duration_ms`. |
| [`record_verify`](#nemo_gym-telemetry-metrics-record_verify)                                     | Record one verification's duration and fold it into the success rate.    |

### Data

[`_VERIFY_LOCK`](#nemo_gym-telemetry-metrics-_VERIFY_LOCK)

[`_VERIFY_SUCCEEDED`](#nemo_gym-telemetry-metrics-_VERIFY_SUCCEEDED)

[`_VERIFY_TOTAL`](#nemo_gym-telemetry-metrics-_VERIFY_TOTAL)

[`logger`](#nemo_gym-telemetry-metrics-logger)

### API

```python
nemo_gym.telemetry.metrics._record(
    kwargs = {}
) -> None
```

Forward to `record_gym_metrics` when a meter is available; never raise.

```python
nemo_gym.telemetry.metrics._reset_verify_tally_for_testing() -> None
```

Reset the cumulative verify tally. Test-only.

```python
nemo_gym.telemetry.metrics.record_active_servers(
    count: int
) -> None
```

Set `gym.servers.active` — orchestrator only.

`gym.servers.active` is a gauge, so the exported value is whatever was written last.
Calling this from more than one process produces a number that is not the fleet size,
not any process's view of it, and impossible to interpret after the fact. The
orchestrator is the only process that knows the fleet size, so it is the only caller.

```python
nemo_gym.telemetry.metrics.record_rollout_duration(
    duration_ms: float
) -> None
```

Record one rollout's wall-clock duration into `gym.rollout.duration_ms`.

```python
nemo_gym.telemetry.metrics.record_verify(
    duration_ms: float,
    succeeded: bool
) -> None
```

Record one verification's duration and fold it into the success rate.

Sets `gym.verify.success_rate` to the cumulative fraction of successful
verifications in this process since start — not a windowed rate. A long-lived server
therefore shows a figure that moves more slowly over time; read it as "this process's
success fraction so far", and use the trace data for anything finer.

```python
nemo_gym.telemetry.metrics._VERIFY_LOCK = threading.Lock()
```

```python
nemo_gym.telemetry.metrics._VERIFY_SUCCEEDED = 0
```

```python
nemo_gym.telemetry.metrics._VERIFY_TOTAL = 0
```

```python
nemo_gym.telemetry.metrics.logger = logging.getLogger(__name__)
```