Metrics

NeMo-RL emits five namespaces of metrics: async efficiency metrics (rl.efficiency.*), startup phase durations (rl.setup.duration), training scalars (rl.reward.*, rl.policy.*, …), vLLM engine metrics (rl.vllm.*), and driver-side vLLM generation metrics (gen_ai.*, following the OTel GenAI semantic conventions).

Metrics are emitted only when telemetry is exporting — the driver always exports, so the rl.* series come from the driver’s metrics logger.

Every rl.* series is declared by NeMo-RL, not by lens. lens provides a consumer-driven metric registry: NeMo-RL calls register_metric_group("rl", [...]) once per process with a MetricSpec per series, then records against it with record_metrics(meter, "rl", ...). The declarations live in one table in nemo_rl/telemetry/metrics.py, so adding a series needs no lens release and no agreement on field names. record_metrics never raises into the caller and skips None.

The tee

The tee lives outside the algorithm code: nemo_rl/telemetry/metrics.py hooks nemo_rl.utils.logger.Logger.log_metrics, so after log_metrics fans a step out to the file / W&B / MLflow backends it calls tee_rl_metrics_to_otel(metrics, prefix). It is best-effort — two prefixes are read, the driver’s per-step train dicts (prefix in ("train", "")) and the one-off timing/setup dict at step 0, so other prefixes are skipped, non-scalar values are ignored, and the whole path is a no-op unless telemetry is actively exporting. The numbers you already see in W&B are therefore the same series you get in your OTLP backend, with no double bookkeeping.

Each family has its own exception handler, so a bad value in one cannot silence the others.

Training scalars

Mirrored from the keys the algorithms already log, one row per series in _TRAIN_SCALARS:

Metric	Type	Logger key
rl.reward.mean	Gauge	reward
rl.kl.divergence	Gauge	kl_penalty
rl.policy.loss	Gauge	loss
rl.value.loss	Gauge	critic/loss (PPO)
rl.entropy	Gauge	approx_entropy
rl.response.length.mean	Gauge ({token})	mean_gen_tokens_per_sample
rl.grad_norm	Gauge	grad_norm
rl.learning_rate	Gauge	lr

SFT v2 logs from its controller actor, which initializes telemetry before its first metric is logged. The controller’s fixed step measurements are also teed:

Metric	Type	Logger key
rl.sft_v2.loader.latency.max	Gauge (s)	loader_latency_max
rl.sft_v2.policy.duration	Gauge (s)	policy_time
rl.sft_v2.step.duration	Gauge (s)	total_step_time
rl.sft_v2.valid_tokens_per_second	Gauge ({token}/s)	valid_tokens_per_second

kl_penalty is named for the penalty but holds the divergence — the coefficient is divided back out in loss_functions.py — which is why the metric is named rl.kl.divergence.

Each row is declared in the module that logs its key, using the same constant the logging site uses, so a rename cannot leave a gauge silently reporting nothing.

Startup phases

Every algorithm’s setup() already timed its startup phases and logged them under the timing/setup prefix at step 0 (SetupTimingMetrics in nemo_rl/algorithms/metric_utils.py). Those are teed as one dimensioned gauge:

Metric	Type	Attributes	Description
rl.setup.duration	Gauge (s)	rl.setup.phase	Wall clock spent in one startup phase

Dimensioned rather than one series per phase because the phase set is not knowable at declaration time: alongside its declared fields SetupTimingMetrics carries a free-form extras dict, and the sparse-refit transports add vllm_<transport>_sparse_init_time_s to it at runtime. A row per phase would silently drop whatever it had not been taught about.

The phase name is the logger key with its _time_s / _s suffix stripped, so generation_init_time_s becomes phase=generation_init. Requiring one of those suffixes is also the filter that keeps non-durations out — parallel_init_enabled is a flag, not a phase.

Representative phases: generation_init (and its generation_init_reserve / generation_init_load parts), policy_init, value_init, nemo_gym_init, collective_init, weight_sync, teacher_model_init, parallel_wall, worker_setup, other_setup, total_setup.

rl.setup.duration carries no rl.bucket

Unlike rl.efficiency.seconds, these phases overlap each other by construction: total_setup contains the rest, parallel_wall covers the generation and policy builds running concurrently, and generation_init_reserve / generation_init_load are parts of generation_init. Anything summing them by bucket would count startup several times over.

Read one phase at a time. For “what did startup cost”, use phase=total_setup — that is the one value that cannot double-count. The rl.setup.* spans give the same phases their shape in a trace; see Span groups — Startup.

W&B history step axes

W&B’s internal _step is a monotonically increasing history-row number, not a NeMo-RL trainer step. Trainer-correlated metrics are buffered into one history row and carry the explicit nemo_rl/step field. Use nemo_rl/step when plotting or joining training metrics by optimizer step; relying on _step also counts independently committed telemetry rows and can misalign the series.

Independent event streams use their own custom axes and do not carry nemo_rl/step. In particular, Single-Controller rollout benchmark series under rollout/throughput/*, timing/rollout_checkpoint/*, timing/rollout_recovery/*, and rollout/checkpoint_outcome/* use telemetry/wall_time_seconds, recorded as Unix wall-clock time. This lets those series continue through a long or paused trainer step and across process restart without changing the meaning of the trainer-step axis.

Single-Controller rollout recovery metrics

Wall-clock rollout telemetry sampling is disabled by default. Enable it without changing the checkpoint cadence by setting, for example:

rollout_checkpointing:
  telemetry_interval_s: 30.0

This records one throughput and controller-pressure sample every 30 seconds. Set the value back to null to disable the sampler. Detailed per-interval metric dictionaries are printed to the controller log only when async_rl.diagnostics: true; configured metric backends receive them either way.

The rollout checkpoint benchmark focuses on the following questions:

1. Did checkpointing reduce raw generation or usable rollout throughput?

2. Which save or restore phase took the time?

3. How much live data-plane work waited behind the checkpoint barrier?

4. Did scheduled checkpoint attempts succeed at the intended cadence?

5. How much completed rollout work was reused after restart?

Prefix	Important fields	Meaning
rollout/throughput	generation_output_tokens_per_second, committed_output_tokens_per_second, committed_groups_per_second	Raw backend decoding throughput compared with output committed for training. The raw metric is absent when the backend does not expose compatible cumulative counters. With token capture, committed tokens are counted exactly from valid staged rows; without token capture, they are estimated by multiplying the reported per-sample mean by the number of completions, so compare like-for-like runs.
rollout/throughput	checkpoint_blocked_mutations, checkpoint_mutation_wait_seconds_p95, checkpoint_mutation_wait_seconds_max	Number and latency of live data-plane mutations delayed by an exclusive checkpoint.
timing/rollout_checkpoint	total_save_seconds, tq_save_seconds, barrier_wait_seconds, exclusive_hold_seconds, sidecar_save_seconds, snapshot_commit_seconds	End-to-end save latency and its storage, fencing, controller-sidecar, and atomic-publication components.
timing/rollout_checkpoint	snapshot_rows, replay_rows, staging_rows, replay_groups, ledger_groups, controller_sidecar_bytes	Logical volume captured by the snapshot. controller_sidecar_bytes excludes the native TQ payload because the current TQ checkpoint API does not report bytes written. NeMo-RL deliberately does not recursively scan the shared checkpoint directory because that scan would perturb the benchmark.
rollout/checkpoint_outcome	completed, skipped, failed, reason_*, seconds_since_previous_success, seconds_since_last_success	Result, actionable reason, and effective cadence of every scheduled checkpoint attempt.
timing/rollout_recovery	snapshot_resolution_seconds, dataloader_load_seconds, tq_load_seconds, replay_metadata_load_seconds, recovery_prepare_seconds, total_load_seconds	Rollout-state restore latency. total_load_seconds is the sum of these non-overlapping restore phases.
timing/rollout_recovery	groups_complete_restored, groups_unfinished_found, siblings_reused, siblings_rerun, redispatch_schedule_seconds	Training-ready groups restored without generation, unfinished groups found for redispatch, and sibling work preserved or rerun after restart.

Recovery counters describe work reconstructed at restore time, not a promise that every restored group will eventually train. After recovery, the configured sampler may still evict a restored group under its normal staleness rules (for example, when a windowed sampler finds that the group’s policy version has fallen outside its valid window).

barrier_wait_seconds and mutation wait latency measure opposite sides of the same fence. The former is how long the checkpoint waits for already-running mutations; the latter is how long rollout or training mutations wait for the checkpoint to release its exclusive cut.

The vLLM request, KV-cache, controller-waiter, buffer-occupancy, and per-mutation kind fields are diagnostic drill-down signals. They are useful when one of the primary throughput or barrier metrics regresses, but do not need to appear on the primary rollout checkpoint dashboard.

Async efficiency metrics (rl.efficiency.*)

Async GRPO measures where wall time goes with a Timer and logs the result as efficiency/* scalars (print_efficiency_summary in nemo_rl/algorithms/utils.py). Those same values are teed to OTel as one dimensioned gauge rather than one instrument per category, so adding a category needs no instrument change.

Metric	Type	Attributes	Description
rl.efficiency.seconds	Gauge (s)	rl.efficiency.category, rl.efficiency.measurement, rl.efficiency.window, rl.bucket	Time attributed to one efficiency category
rl.efficiency.pct	Gauge (%)	rl.efficiency.measurement, rl.efficiency.window	Productive share of driver-side wall clock over the window named by rl.efficiency.window

The single-controller path reuses two of these category names — idle/buffer_starvation for the wait when the training fleet is stalled on the buffer, and idle/refit_bubble for weight_sync — so a goodput rollup reads the same vocabulary on both paths. Both are spans only there; the single controller logs no efficiency/* scalars of its own.

Always filter on rl.efficiency.measurement

Some categories are measured on the driver against wall time; others are summed across concurrent collector threads and can exceed the wall time they happened in.

rl.efficiency.measurement	Recorded on	Categories	Safe to sum against elapsed driver time?
wall_clock	driver, sequentially	init/total, idle/buffer_starvation, idle/refit_bubble, idle/validation	yes
collector_wall_clock	collector’s collection-loop thread, sequentially	idle/refit_event_wait, idle/generation_limit_pause	no — real durations, but on a timeline that runs concurrently with the driver’s
thread_seconds	collector’s batch-worker threads, concurrently	idle/buffer_full_backoff, wasted/failed_trajectory	no — not durations at all

Eight rollout threads each backing off for 10s during the same 10s window produce a thread_seconds value of 80, not 10. Summing rl.efficiency.seconds by rl.bucket without filtering therefore overstates idle time — a wrong answer that looks like a real one. Filter to rl.efficiency.measurement="wall_clock" before comparing against elapsed time; read the other two per-phase, thread_seconds as a saturation signal.

The non-wall_clock values also carry rl.bucket so all three share one vocabulary with the spans; the measurement attribute is what keeps a bucket rollup honest.

Two deliberate metric/span disagreements to know about before comparing a metric rollup against a trace rollup:

• The two collector-loop categories carry rl.bucket="idle" as metrics and rely on you filtering by measurement, but carry no bucket at all as spans, since a trace has no equivalent filter to rely on.

• idle/validation is idle as a metric and overhead on every span covering the same seconds. Both are true of different fleets: the training GPUs are idle, which is what the driver’s timer measures, while the generation GPUs are doing necessary non-training work, which is what bucket_scope(Bucket.OVERHEAD) in validate() tags. Attributing this phase properly needs per-fleet accounting; until then, do not expect the two rollups to agree on a validation step. See span groups — why idle/validation is not a span.

rl.efficiency.window: what a value covers in time

measurement says whether values may be summed against each other; window says whether one may be summed across steps. They are independent, and the second is the easier one to get silently wrong.

rl.efficiency.window	Categories	Meaning
step	idle/buffer_starvation, idle/refit_bubble, idle/validation	per-step delta — the driver resets its Timer every step, so these sum across steps
run	init/total, all four collector-side categories, and rl.efficiency.pct on the async PPO path	cumulative since the process started — consecutive points already contain each other, so summing across steps multiplies by the step count

init/total is the driver-side exception: it is measured once, waiting for the first buffer fill before the step loop, then republished unchanged every step so it does not disappear from a dashboard after step 1. Read it as a constant. The collector’s Timer is never reset, which is why everything from it is run.

rl.efficiency.pct carries whichever window its denominator had — read the attribute, do not assume. print_efficiency_summary tags it window="step" when the caller passed a per-step wall time and window="run" when it fell back to the run’s elapsed time. Async GRPO passes the per-step value; async PPO does not, so its points are run. The per-step form is the one to trust: its numerator is the three step-window idle categories over that step’s wall time. init/total is deliberately excluded — it is a run constant, so folding it in would charge the whole startup cost to every step — and so are the collector’s categories, which are on another clock. Against the run’s elapsed time the ratio climbs toward 100% as the run lengthens no matter what the idle time does, which is why a window="run" point says little about how the run is actually doing.

vLLM engine metrics (rl.vllm.*)

Read from the vLLM engine’s own Prometheus registry, delta’d per step. snapshot_step_metrics() takes a baseline before generation and get_step_metrics() a second reading after, both fanned out to the DP-leader workers; the same snapshot pair also feeds the spec-decode metrics, so the wider coverage costs no extra RPC.

Metric	Type	Source series
rl.vllm.prompt_tokens	Counter ({token})	vllm:prompt_tokens
rl.vllm.generation_tokens	Counter ({token})	vllm:generation_tokens
rl.vllm.prompt_length.mean	Gauge ({token})	vllm:request_prompt_tokens histogram
rl.vllm.generation_length.mean	Gauge ({token})	vllm:request_generation_tokens histogram
rl.vllm.generations.ok	Counter ({generation})	vllm:request_success{finished_reason}
rl.vllm.generations.failed	Counter ({generation})	vllm:request_success{finished_reason}

The counters carry the step’s delta, so summing them across steps reconstructs the run total. The means come from each histogram’s sum/count delta, so they are exact rather than bucket-interpolated; a step in which the engine served no request omits them rather than reporting a zero-length sequence.

What counts as failed. vLLM’s finish reasons are stop, length, abort, error and repetition. Only stop and length leave a usable sample behind, and length is a normal RL outcome — a rollout routinely runs to max_tokens — so those two are ok and everything else is failed. Counting the remainder rather than an explicit deny-list keeps ok + failed equal to the engine’s own total even if a future vLLM adds a reason we have not heard of.

These are the family names vLLM registers. The reader returns family names, which prometheus_client strips of _total, so no suffixed alias is needed on this path — unlike the HTTP-scrape path in nemo_rl/models/generation/dynamo/metrics.py, where sample names do carry it. test_engine_metric_names_match_vllm checks the copies against vLLM itself. A series that matches nothing is omitted, leaving a visible gap in the dashboard rather than a plausible-looking zero, and so is one whose total went backwards, which means an engine restarted mid-step.

Driver-side generation metrics (gen_ai.*)

The driver-side vLLM generation path records token and latency metrics through lens’s record_inference_metrics with provider_name="vllm", following the OTel GenAI metrics spec.

Metric	Type	Description
gen_ai.client.token.usage	Histogram	Tokens per request, split by gen_ai.token.type (input / output)

gen_ai.server.request.duration is not recorded: it is defined per inference request, and one generate() call here is a batch across every data-parallel shard. That wall clock goes to rl.vllm.batch.duration (histogram, seconds) instead.

These overlap with rl.vllm.* on token counts but are not redundant: gen_ai.* is derived from the tensors a generate() call returns, so it measures what the driver received, while rl.vllm.* is the engine’s own accounting. When the two disagree, the gap is work the engine did that never reached the driver — the aborted requests, which appear in no returned tensor at all.

These ride the normal http/protobuf OTLP path and reach the same backend as everything else. They are distinct from vLLM’s native engine tracing (opt-in, and exported over vLLM’s own gRPC-by-default exporter) — see vLLM Tracing.

Metric vs span tag vs resource attribute

The one rule that trips people up. Classify each value before you emit it:

Kind	Use	Example
Metric	numerical value that changes over time	per-category efficiency seconds → rl.efficiency.seconds
Span tag	categorical per-span context for filtering	rl.iteration, rl.bucket, rl.num_generations_per_prompt, rl.weight_version
Resource attribute	stable for the whole run	rl.algorithm, rl.model, dl.tensor_parallel.size

Do not put a time-series number (loss, reward) on a span attribute — it produces no useful series in your backend and wastes storage. Do not put a per-step categorical (iteration number) on a metric label — that is unbounded cardinality. See lens: metrics — metric vs span attribute vs resource attribute.

Goodput (monitor-derived)

NeMo-RL does not emit rl.goodput or rl.bucket.* rollup metrics. Leaf spans carry rl.bucket ∈ {productive, overhead, idle, wasted}; umbrella spans (job / step / rollout) omit it. rl.efficiency.seconds carries the same rl.bucket tokens, but it is a per-category duration, not a rollup — and it needs the rl.efficiency.measurement filter described above. Offline monitors (e.g. wandb-monitor) SUM span / phase GPU-time by rl.bucket and compute:

rl_goodput = productive_gpu_s / (productive + overhead + idle + wasted)_gpu_s

See Span groups — goodput buckets and nemo_rl/telemetry/instrumentation.py.

Metric names use the application scope (rl.*); attribute names in the rl.* namespace are NeMo-RL’s own, defined in nemo_rl/telemetry/, not imported from lens’s semconv.py.

Filtering across runs

Every rl.* data point carries the nemo.run.id resource attribute. Use it to isolate or compare runs in your backend (Grafana/Prometheus, or any OTLP-compatible backend). See Configuration — Run identification.