Metrics

NeMo Lens ships opinionated metric instruments under nemo.lens.instruments for common observability needs: GenAI inference, RL training, and Gym servers. Training-specific metrics (e.g., megatron.training.loss) live in the consumer project; they are not generic.

Import each record function from its submodule, e.g., from nemo.lens.instruments.rl import record_rl_metrics. Only record_inference_metrics is re-exported at the package level (from nemo.lens.instruments import record_inference_metrics); the RL and Gym functions are available only through their submodules.

The meter argument is the OTel Meter to record on. You can use handle.meter from setup_telemetry(), or grab one directly with get_meter(name="nemo.lens") (from nemo.lens import get_meter).

Architecture

Each module under instruments/ follows the same pattern:

• A module-level WeakKeyDictionary caches instruments per Meter, so re-initializing the meter does not leak memory.
• A _get_*_instruments(meter) helper creates (and caches) all instruments for a meter on first call.
• A record_*_metrics(meter, ...) function takes a required meter plus optional per-metric arguments (best passed by keyword) and records only the ones that are not None.

Callers can record partial data without conditional logic:

1
record_rl_metrics(handle.meter, reward_mean=r, policy_loss=p)   # only these two
2
record_rl_metrics(handle.meter, kl_divergence=kl)                # just one

Inference with instruments/inference.py

This module emits metrics following the OTel GenAI semantic conventions.

1
from nemo.lens.instruments.inference import record_inference_metrics
2
3
record_inference_metrics(
4
    handle.meter,
5
    request_duration_s=0.42,
6
    model="llama-3-8b",
7
    input_tokens=128,
8
    output_tokens=256,
9
)

Token usage is labeled with gen_ai.token.type ("input" or "output"); filter on that label in Prometheus or Grafana.

All data points carry gen_ai.operation.name = "text_completion" and gen_ai.provider.name = "nemo" by default; override through the operation_name= or provider_name= args.

NeMo RL with instruments/rl.py

This module emits NeMo RL-specific gauges and histograms in the rl.* namespace.

1
from nemo.lens.instruments.rl import record_rl_metrics
2
3
record_rl_metrics(
4
    handle.meter,
5
    reward_mean=0.75,
6
    kl_divergence=0.01,
7
    policy_loss=0.3,
8
    value_loss=0.5,
9
    entropy=2.1,
10
    response_length_mean=128.0,
11
    generation_duration_ms=450.0,
12
    rollout_duration_ms=2300.0,
13
)

NeMo Gym with instruments/gym.py

This module emits NeMo Gym server metrics in the gym.* namespace.

1
from nemo.lens.instruments.gym import record_gym_metrics
2
3
record_gym_metrics(
4
    handle.meter,
5
    server_request_duration_ms=42.0,
6
    verify_duration_ms=120.0,
7
    verify_success_rate=0.87,
8
    active_servers=4,
9
)

Write Custom Instruments

The same WeakKeyDictionary pattern works for project-specific metrics. Megatron’s instruments/training.py (shipped in the Megatron repository, not NeMo Lens) emits megatron.training.* metrics the same way.

If your project has a recurring metric shape, add a module under nemo.lens.instruments.<domain> with:

1
import weakref
2
from opentelemetry import metrics
3
4
_INSTRUMENTS: weakref.WeakKeyDictionary = weakref.WeakKeyDictionary()
5
6
def _get_instruments(meter: metrics.Meter) -> dict:
7
    instruments = _INSTRUMENTS.get(meter)
8
    if instruments is None:
9
        instruments = {
10
            "latency": meter.create_histogram("my.op.latency_ms", unit="ms"),
11
            "queue_depth": meter.create_gauge("my.queue.depth"),
12
        }
13
        _INSTRUMENTS[meter] = instruments
14
    return instruments
15
16
def record_my_metrics(meter, latency_ms=None, queue_depth=None):
17
    i = _get_instruments(meter)
18
    if latency_ms is not None:
19
        i["latency"].record(latency_ms)
20
    if queue_depth is not None:
21
        i["queue_depth"].set(queue_depth)

Choose Between Gauges and Histograms

• Gauge: Point-in-time value. Prometheus shows the last reported value. Good for losses, rates, and counts.
• Histogram: Distribution of values. Prometheus computes quantiles. Good for durations, sizes, and any value where percentiles matter.
• Counter: Monotonic cumulative value. Prometheus shows the rate of change. Use for event counts (skipped_iters and errors).

Do not put durations on gauges; you lose the ninety-ninth percentile. Do not put event counts on histograms; the cardinality is incorrect.

Metrics, Span Attributes, and Resource Attributes

Avoid mixing these concepts. Use the following decision table to choose the correct telemetry type:

Specifically, do not record a continuously-varying metric, such as loss, as a span attribute. Doing so wastes span storage, and Jaeger cannot aggregate across spans. Use record_*_metrics() to emit the value as a real metric.