> This page is for version Nightly.
> For other versions, use one of these documentation indexes:
> - 0.1.0 (default): https://docs.nvidia.com/nemo/lens/0.1.0/llms.txt
> - Nightly: https://docs.nvidia.com/nemo/lens/nightly/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/lens/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/lens/_mcp/server.

# Metrics

NeMo Lens ships opinionated metric instruments under `nemo.lens.instruments` for observability needs whose *names it owns*: GenAI inference (`gen_ai.*`) and Gym servers (`gym.*`). Metric families whose names belong to a consuming framework — RL training series in NeMo-RL, training-specific metrics such as `megatron.training.loss` — are **declared by that consumer** through the metric registry, not shipped as a module here.

`record_inference_metrics` and the registry API (`MetricSpec`, `register_metric_group`, `record_metrics`, `unregister_metric_group`, `registered_metric_groups`) are re-exported at the package level (`from nemo.lens.instruments import ...`). The Gym record function is available only through its submodule (`from nemo.lens.instruments.gym import record_gym_metrics`).

The `meter` argument is the OTel `Meter` to record on. You can use `handle.meter` from `setup_telemetry()`, or grab one directly with `get_meter(name="nemo.lens")` (`from nemo.lens import get_meter`).

## Architecture

Each module under `instruments/` follows the same pattern:

* A module-level `WeakKeyDictionary` caches instruments per `Meter`, so re-initializing the meter does not leak memory.
* A `_get_*_instruments(meter)` helper creates (and caches) all instruments for a meter on first call.
* A `record_*_metrics(meter, ...)` function takes a required `meter` plus optional per-metric arguments (best passed by keyword) and records only the ones that are not `None`.

The shipped record functions (and `record_metrics`) let callers record partial data without conditional logic — arguments left out (or passed `None`) are skipped:

```python
record_metrics(handle.meter, "rl", reward_mean=r, policy_loss=p)   # only these two
record_metrics(handle.meter, "rl", kl_divergence=kl)               # just one
```

## Inference with `instruments/inference.py`

This module emits metrics following the [OTel GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/).

```python
from nemo.lens.instruments.inference import record_inference_metrics

record_inference_metrics(
    handle.meter,
    request_duration_s=0.42,
    model="llama-3-8b",
    input_tokens=128,
    output_tokens=256,
)
```

| Instrument       | OTel name                        | Type      | Unit      |
| ---------------- | -------------------------------- | --------- | --------- |
| Request duration | `gen_ai.server.request.duration` | Histogram | s         |
| Token usage      | `gen_ai.client.token.usage`      | Histogram | `{token}` |

Token usage is labeled with `gen_ai.token.type` (`"input"` or `"output"`); filter on that label in Prometheus or Grafana.

All data points carry `gen_ai.operation.name = "text_completion"` and `gen_ai.provider.name = "nemo"` by default; override through the `operation_name=` or `provider_name=` args.

## Consumer-owned metrics with the registry

Metric families whose names belong to a consuming framework are declared by that
framework, not by lens. NeMo-RL, for example, owns the `rl.*` series: it
registers them once at startup and then records against the group, so lens never
has to carry — and NeMo-RL never has to import, extend, or override — a
per-consumer `rl.py`.

A `MetricSpec` declares one series: the `key` the caller records under, the
`name` actually emitted, the `kind` (`gauge`, `histogram`, `counter`,
`up_down_counter`, or one of the [observable kinds](#observable-series-whose-attributes-change)),
and an optional `unit` and `description`. The consumer owns
its own names; keep them in one module in the consumer's tree (as NeMo-RL does
for `rl.*`) rather than scattering literals.

```python
from nemo.lens.instruments import MetricSpec, record_metrics, register_metric_group

# Once, at startup (in the consumer, e.g. NeMo-RL):
register_metric_group(
    "rl",
    [
        MetricSpec("reward_mean", "rl.reward.mean", "gauge"),
        MetricSpec("kl_divergence", "rl.kl_divergence", "gauge"),
        MetricSpec("policy_loss", "rl.policy_loss", "gauge"),
        MetricSpec(
            "throughput_tokens_per_sec",
            "rl.throughput.tokens_per_sec",
            "gauge",
            unit="{token}/s",
        ),
        MetricSpec("generation_duration_ms", "rl.generation.duration_ms", "histogram", unit="ms"),
    ],
)

# Per step, wherever the values are available:
record_metrics(
    handle.meter,
    "rl",
    reward_mean=0.75,
    kl_divergence=0.01,
    throughput_tokens_per_sec=18500.0,
    generation_duration_ms=450.0,
)
```

Instruments are created lazily on first record against a meter and cached per
meter, exactly like the shipped modules. Register each group once;
`register_metric_group` rejects a duplicate name unless you pass
`allow_override=True`.

`record_metrics` never raises into the caller. An unregistered group, an unknown
key, a `None` value, an unusable `meter`, a malformed `values` or `attributes`
argument, and a failed instrument creation or emit are each logged once and
skipped. Warnings are deduplicated per group (and per key where it applies), so a
misconfigured call in a per-step loop does not flood the log.

`meter`, `group`, and `values` are positional-only, so you may declare metric
keys with those names and pass them as keyword arguments freely. `attributes` is
the one reserved keyword — if one of your series is genuinely called
`attributes`, record it through the `values` mapping:

```python
record_metrics(handle.meter, "mygroup", {"attributes": 7.0})
```

### Observable series whose attributes change

A synchronous instrument is pushed to and the SDK remembers what it was given.
That is wrong for a series whose *attribute set* changes during a run. A gauge
keeps its last value per attribute set and re-exports it on every cycle, and
nothing in the API can say that a set no longer applies — so a point first
reported under one label and later under another is exported under **both**,
indefinitely, and any aggregation over that label double-counts it.

The three `observable_*` kinds invert the direction: the SDK calls your callback
once per collection cycle and treats what it returns as the complete set of
points for that cycle. A label the callback stops yielding simply ends.

| Kind                         | Meter factory                       |
| ---------------------------- | ----------------------------------- |
| `observable_gauge`           | `create_observable_gauge`           |
| `observable_counter`         | `create_observable_counter`         |
| `observable_up_down_counter` | `create_observable_up_down_counter` |

The two counter kinds want a **running total**, not the change since the last
cycle. The SDK derives deltas by subtracting consecutive readings, so a callback
that returns deltas itself is wrong under either temporality — a counter whose
callback returns `7` and then `10` exports a delta of `3`, which is the intent;
one that returns `7` and then `3` exports `-4`.

```python
from opentelemetry.metrics import Observation

from nemo.lens.instruments import (
    MetricSpec,
    create_metric_instruments,
    register_metric_group,
)

def _host_memory(options):
    # Returns one point per host, with the label resolved now rather than when
    # the value was sampled, so a host that changes role is relabelled at once.
    return [
        Observation(sample.used_bytes, {"host.name": host, "role": role_of(host)})
        for host, sample in latest_samples().items()
    ]

register_metric_group(
    "rl",
    [
        MetricSpec(
            "host_memory_used",
            "rl.host.memory.used",
            "observable_gauge",
            unit="By",
            callback=_host_memory,
        )
    ],
)

# Required for observable series, and harmless on a purely synchronous group.
create_metric_instruments(handle.meter, "rl")
```

`create_metric_instruments` exists because the lazy path does not reach these.
Synchronous instruments are created on the first `record_metrics` against a
meter, but an observable one is never recorded against, so without an explicit
call it would never be created and its callback would never run. Call it once,
after `setup_telemetry` has produced the meter; calling it twice is a no-op, and
the callback is registered exactly once per meter.

Three constraints worth knowing before you reach for these:

* **The callback runs on the exporter's collection path.** Keep it cheap and
  non-blocking — reading a snapshot that some other thread maintains, not doing
  I/O. Work done here delays every metric the process exports.
* **An installed callback cannot be removed.** OTel offers no way to detach an
  observable instrument from a meter, so `unregister_metric_group` will not stop
  one. A callback that should go quiet must return no observations. For the same
  reason `register_metric_group(..., allow_override=True)` cannot replace one
  that has already been created: the old callback keeps reporting and the new
  one is dropped as a duplicate name. Lens warns when that happens; replacing an
  observable series means giving it a new metric name.
* **A callback that raises goes quiet rather than loud.** The failure is logged
  once per key and the series then reports nothing until the callback succeeds
  again, instead of writing a traceback on every collection cycle.

A `MetricSpec` is rejected at construction if an `observable_*` kind has no
`callback`, or if a synchronous kind has one. Both mistakes are otherwise
silent: the first reports nothing, and the second is never called. Passing an
observable key to `record_metrics` is logged once and dropped, since its value
comes from the callback.

## NeMo Gym with `instruments/gym.py`

This module emits NeMo Gym server metrics in the `gym.*` namespace.

```python
from nemo.lens.instruments.gym import record_gym_metrics

record_gym_metrics(
    handle.meter,
    server_request_duration_ms=42.0,
    verify_duration_ms=120.0,
    verify_success_rate=0.87,
    active_servers=4,
)
```

| Metric                           | Type           | Description                          |
| -------------------------------- | -------------- | ------------------------------------ |
| `gym.server.request_duration_ms` | Histogram (ms) | Incoming request duration            |
| `gym.rollout.duration_ms`        | Histogram (ms) | Rollout collection duration          |
| `gym.verify.duration_ms`         | Histogram (ms) | Verification endpoint duration       |
| `gym.verify.success_rate`        | Gauge          | Fraction of successful verifications |
| `gym.servers.active`             | Gauge          | Number of active Gym servers         |

## Write Custom Instruments

For project-specific metrics, prefer the registry: declare the series as `MetricSpec`s and register them under a group name in your own code. Megatron's `megatron.training.*` and NeMo-RL's `rl.*` series both live this way — in the consumer, registered against lens, never shipped here.

```python
from nemo.lens.instruments import MetricSpec, record_metrics, register_metric_group

register_metric_group(
    "myproject",
    [
        MetricSpec("latency_ms", "my.op.latency_ms", "histogram", unit="ms"),
        MetricSpec("queue_depth", "my.queue.depth", "gauge"),
    ],
)

record_metrics(handle.meter, "myproject", latency_ms=12.0, queue_depth=3)
```

Reach for a dedicated module (the `WeakKeyDictionary` pattern the shipped `inference.py` uses) only when a series needs per-point logic the registry does not express — for example inference token usage, which fans one call out into `input`/`output` points carrying `gen_ai.token.type`.

## Choose Between Gauges and Histograms

* **Gauge**: Point-in-time value. Prometheus shows the last reported value. Good for losses, rates, and counts.
* **Histogram**: Distribution of values. Prometheus computes quantiles. Good for durations, sizes, and any value where percentiles matter.
* **Counter**: Monotonic cumulative value. Prometheus shows the rate of change. Use for event counts (`skipped_iters` and `errors`).

Do not put durations on gauges; you lose the ninety-ninth percentile. Do not put event counts on histograms; the cardinality is incorrect.

## Metrics, Span Attributes, and Resource Attributes

Avoid mixing these concepts. Use the following decision table to choose the correct telemetry type:

| Value                                                                                 | Where to put it                                                              |
| ------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- |
| Changes over time, numerical (loss, throughput)                                       | **Metric**                                                                   |
| Categorical per-span context (iteration, microbatch_id, skipped)                      | **Span attribute**                                                           |
| Stable for the process lifetime (rank, parallelism configuration, model architecture) | **Resource attribute** (through `resource_attributes=` in `setup_telemetry`) |

Specifically, do not record a continuously-varying metric, such as loss, as a span attribute. Doing so wastes span storage, and Jaeger cannot aggregate across spans. Use `record_*_metrics()` to emit the value as a real metric.