Metrics

NeMo Lens ships opinionated metric instruments under nemo.lens.instruments for observability needs whose names it owns: GenAI inference (gen_ai.*) and Gym servers (gym.*). Metric families whose names belong to a consuming framework — RL training series in NeMo-RL, training-specific metrics such as megatron.training.loss — are declared by that consumer through the metric registry, not shipped as a module here.

record_inference_metrics and the registry API (MetricSpec, register_metric_group, record_metrics, unregister_metric_group, registered_metric_groups) are re-exported at the package level (from nemo.lens.instruments import ...). The Gym record function is available only through its submodule (from nemo.lens.instruments.gym import record_gym_metrics).

The meter argument is the OTel Meter to record on. You can use handle.meter from setup_telemetry(), or grab one directly with get_meter(name="nemo.lens") (from nemo.lens import get_meter).

Architecture

Each module under instruments/ follows the same pattern:

• A module-level WeakKeyDictionary caches instruments per Meter, so re-initializing the meter does not leak memory.
• A _get_*_instruments(meter) helper creates (and caches) all instruments for a meter on first call.
• A record_*_metrics(meter, ...) function takes a required meter plus optional per-metric arguments (best passed by keyword) and records only the ones that are not None.

The shipped record functions (and record_metrics) let callers record partial data without conditional logic — arguments left out (or passed None) are skipped:

record_metrics(handle.meter, "rl", reward_mean=r, policy_loss=p)   # only these two
record_metrics(handle.meter, "rl", kl_divergence=kl)               # just one

Inference with instruments/inference.py

This module emits metrics following the OTel GenAI semantic conventions.

from nemo.lens.instruments.inference import record_inference_metrics
record_inference_metrics(
    handle.meter,
    request_duration_s=0.42,
    model="llama-3-8b",
    input_tokens=128,
    output_tokens=256,
)

Token usage is labeled with gen_ai.token.type ("input" or "output"); filter on that label in Prometheus or Grafana.

All data points carry gen_ai.operation.name = "text_completion" and gen_ai.provider.name = "nemo" by default; override through the operation_name= or provider_name= args.

Consumer-owned metrics with the registry

Metric families whose names belong to a consuming framework are declared by that framework, not by lens. NeMo-RL, for example, owns the rl.* series: it registers them once at startup and then records against the group, so lens never has to carry — and NeMo-RL never has to import, extend, or override — a per-consumer rl.py.

A MetricSpec declares one series: the key the caller records under, the name actually emitted, the kind (gauge, histogram, counter, up_down_counter, or one of the observable kinds), and an optional unit and description. The consumer owns its own names; keep them in one module in the consumer’s tree (as NeMo-RL does for rl.*) rather than scattering literals.

from nemo.lens.instruments import MetricSpec, record_metrics, register_metric_group
# Once, at startup (in the consumer, e.g. NeMo-RL):
register_metric_group(
    "rl",
    [
        MetricSpec("reward_mean", "rl.reward.mean", "gauge"),
        MetricSpec("kl_divergence", "rl.kl_divergence", "gauge"),
        MetricSpec("policy_loss", "rl.policy_loss", "gauge"),
        MetricSpec(
            "throughput_tokens_per_sec",
            "rl.throughput.tokens_per_sec",
            "gauge",
            unit="{token}/s",
        ),
        MetricSpec("generation_duration_ms", "rl.generation.duration_ms", "histogram", unit="ms"),
    ],
)
# Per step, wherever the values are available:
record_metrics(
    handle.meter,
    "rl",
    reward_mean=0.75,
    kl_divergence=0.01,
    throughput_tokens_per_sec=18500.0,
    generation_duration_ms=450.0,
)

Instruments are created lazily on first record against a meter and cached per meter, exactly like the shipped modules. Register each group once; register_metric_group rejects a duplicate name unless you pass allow_override=True.

record_metrics never raises into the caller. An unregistered group, an unknown key, a None value, an unusable meter, a malformed values or attributes argument, and a failed instrument creation or emit are each logged once and skipped. Warnings are deduplicated per group (and per key where it applies), so a misconfigured call in a per-step loop does not flood the log.

meter, group, and values are positional-only, so you may declare metric keys with those names and pass them as keyword arguments freely. attributes is the one reserved keyword — if one of your series is genuinely called attributes, record it through the values mapping:

record_metrics(handle.meter, "mygroup", {"attributes": 7.0})

Observable series whose attributes change

A synchronous instrument is pushed to and the SDK remembers what it was given. That is wrong for a series whose attribute set changes during a run. A gauge keeps its last value per attribute set and re-exports it on every cycle, and nothing in the API can say that a set no longer applies — so a point first reported under one label and later under another is exported under both, indefinitely, and any aggregation over that label double-counts it.

The three observable_* kinds invert the direction: the SDK calls your callback once per collection cycle and treats what it returns as the complete set of points for that cycle. A label the callback stops yielding simply ends.

The two counter kinds want a running total, not the change since the last cycle. The SDK derives deltas by subtracting consecutive readings, so a callback that returns deltas itself is wrong under either temporality — a counter whose callback returns 7 and then 10 exports a delta of 3, which is the intent; one that returns 7 and then 3 exports -4.

from opentelemetry.metrics import Observation
from nemo.lens.instruments import (
    MetricSpec,
    create_metric_instruments,
    register_metric_group,
)
def _host_memory(options):
    # Returns one point per host, with the label resolved now rather than when
    # the value was sampled, so a host that changes role is relabelled at once.
    return [
        Observation(sample.used_bytes, {"host.name": host, "role": role_of(host)})
        for host, sample in latest_samples().items()
    ]
register_metric_group(
    "rl",
    [
        MetricSpec(
            "host_memory_used",
            "rl.host.memory.used",
            "observable_gauge",
            unit="By",
            callback=_host_memory,
        )
    ],
)
# Required for observable series, and harmless on a purely synchronous group.
create_metric_instruments(handle.meter, "rl")

create_metric_instruments exists because the lazy path does not reach these. Synchronous instruments are created on the first record_metrics against a meter, but an observable one is never recorded against, so without an explicit call it would never be created and its callback would never run. Call it once, after setup_telemetry has produced the meter; calling it twice is a no-op, and the callback is registered exactly once per meter.

Three constraints worth knowing before you reach for these:

• The callback runs on the exporter’s collection path. Keep it cheap and non-blocking — reading a snapshot that some other thread maintains, not doing I/O. Work done here delays every metric the process exports.
• An installed callback cannot be removed. OTel offers no way to detach an observable instrument from a meter, so unregister_metric_group will not stop one. A callback that should go quiet must return no observations. For the same reason register_metric_group(..., allow_override=True) cannot replace one that has already been created: the old callback keeps reporting and the new one is dropped as a duplicate name. Lens warns when that happens; replacing an observable series means giving it a new metric name.
• A callback that raises goes quiet rather than loud. The failure is logged once per key and the series then reports nothing until the callback succeeds again, instead of writing a traceback on every collection cycle.

A MetricSpec is rejected at construction if an observable_* kind has no callback, or if a synchronous kind has one. Both mistakes are otherwise silent: the first reports nothing, and the second is never called. Passing an observable key to record_metrics is logged once and dropped, since its value comes from the callback.

NeMo Gym with instruments/gym.py

This module emits NeMo Gym server metrics in the gym.* namespace.

from nemo.lens.instruments.gym import record_gym_metrics
record_gym_metrics(
    handle.meter,
    server_request_duration_ms=42.0,
    verify_duration_ms=120.0,
    verify_success_rate=0.87,
    active_servers=4,
)

Write Custom Instruments

For project-specific metrics, prefer the registry: declare the series as MetricSpecs and register them under a group name in your own code. Megatron’s megatron.training.* and NeMo-RL’s rl.* series both live this way — in the consumer, registered against lens, never shipped here.

from nemo.lens.instruments import MetricSpec, record_metrics, register_metric_group
register_metric_group(
    "myproject",
    [
        MetricSpec("latency_ms", "my.op.latency_ms", "histogram", unit="ms"),
        MetricSpec("queue_depth", "my.queue.depth", "gauge"),
    ],
)
record_metrics(handle.meter, "myproject", latency_ms=12.0, queue_depth=3)

Reach for a dedicated module (the WeakKeyDictionary pattern the shipped inference.py uses) only when a series needs per-point logic the registry does not express — for example inference token usage, which fans one call out into input/output points carrying gen_ai.token.type.

Choose Between Gauges and Histograms

• Gauge: Point-in-time value. Prometheus shows the last reported value. Good for losses, rates, and counts.
• Histogram: Distribution of values. Prometheus computes quantiles. Good for durations, sizes, and any value where percentiles matter.
• Counter: Monotonic cumulative value. Prometheus shows the rate of change. Use for event counts (skipped_iters and errors).

Do not put durations on gauges; you lose the ninety-ninth percentile. Do not put event counts on histograms; the cardinality is incorrect.

Metrics, Span Attributes, and Resource Attributes

Avoid mixing these concepts. Use the following decision table to choose the correct telemetry type:

Specifically, do not record a continuously-varying metric, such as loss, as a span attribute. Doing so wastes span storage, and Jaeger cannot aggregate across spans. Use record_*_metrics() to emit the value as a real metric.