nemo_rl.telemetry.metrics

Best-effort mirroring of NeMo-RL’s per-step metrics into OTel.

Kept out of nemo_rl.utils.logger (which pulls in torch/ray/wandb/etc.) so the mapping stays importable and testable without the heavy training stack. nemo_rl.utils.logger.Logger.log_metrics calls :func:tee_rl_metrics_to_otel after its normal fan-out to the file/wandb/mlflow backends.

Every series is declared by NeMo-RL rather than by lens, via lens’s consumer-driven metric registry: NeMo-RL owns the rl.* metric names, so a new series needs no lens release and no negotiation over field names. Four families are teed — the async efficiency/* phase durations, the training scalars, the vllm/* engine deltas, and the timing/setup startup phases. The first three ride the per-step train dicts; the last arrives once, at step 0.

This module names no logger key of its own beyond the efficiency/* family it also produces. The scalar rows come from the modules that log them, through

mod:

nemo_rl.telemetry.vocabulary, so adding an algorithm does not mean editing telemetry.

Logger.log_metrics fans a step out as several dicts under different prefixes, and a key is only reachable from the prefix its own dict carries — so only the prefixes those dicts actually arrive under are teed.

Module Contents

Functions

warn_once	Warn with a traceback the first time key fails, then stay quiet.
_metric_specs	Build the full MetricSpec list for the rl group.
ensure_metric_group_registered	Declare the rl metric group with lens, once per process.
_record	Record against the rl group. record_metrics never raises.
record_rl_metrics	Record declared series by registry key. No-op unless exporting.
create_rl_metric_instruments	Build the rl group’s instruments now. No-op unless exporting.
efficiency_measurements	Map each canonical efficiency category to its measurement kind.
efficiency_window	Return the period one efficiency value covers (see the constants above).
map_efficiency_seconds	Extract {category: seconds} from a raw Logger metrics dict.
_tee_setup_metrics	Emit the startup phase durations as rl.setup.duration.
map_teed_scalars	Extract {registry key: value} for every declared scalar present.
_tee_scalars	Emit the declared training and vLLM scalars, unlabelled.
_tee_efficiency_metrics	Emit the efficiency/* phase durations as rl.efficiency.*.
tee_rl_metrics_to_otel	Mirror the per-step metrics into OTel (no-op unless exporting).
_tee_rl_metrics_to_otel	Body of :func:tee_rl_metrics_to_otel, inside its exception guard.

Data

logger
_TRAIN_PREFIXES
RL_EFFICIENCY_SECONDS_METRIC
RL_EFFICIENCY_PCT_METRIC
RL_EFFICIENCY_MEASUREMENT_ATTR
WALL_CLOCK_MEASUREMENT
COLLECTOR_WALL_CLOCK_MEASUREMENT
THREAD_SECONDS_MEASUREMENT
RL_EFFICIENCY_WINDOW_ATTR
STEP_WINDOW
RUN_WINDOW
RL_SETUP_DURATION_METRIC
RL_SETUP_PHASE_ATTR
_EFFICIENCY_KEY_PREFIX
_EFFICIENCY_SECONDS_SUFFIX
RL_METRIC_GROUP
_EFFICIENCY_SECONDS_KEY
_EFFICIENCY_PCT_KEY_REGISTRY
_SETUP_DURATION_KEY
_REGISTERED
_REGISTRATION_FAILED
_WARNED
setup_phase
map_setup_seconds

API

nemo_rl.telemetry.metrics.logger

‘getLogger(…)’

nemo_rl.telemetry.metrics._TRAIN_PREFIXES: tuple[Optional[str], ...]

(‘train’, ‘’)

nemo_rl.telemetry.metrics.RL_EFFICIENCY_SECONDS_METRIC

‘rl.efficiency.seconds’

nemo_rl.telemetry.metrics.RL_EFFICIENCY_PCT_METRIC

‘rl.efficiency.pct’

nemo_rl.telemetry.metrics.RL_EFFICIENCY_MEASUREMENT_ATTR

‘rl.efficiency.measurement’

nemo_rl.telemetry.metrics.WALL_CLOCK_MEASUREMENT

‘wall_clock’

nemo_rl.telemetry.metrics.COLLECTOR_WALL_CLOCK_MEASUREMENT

‘collector_wall_clock’

nemo_rl.telemetry.metrics.THREAD_SECONDS_MEASUREMENT

‘thread_seconds’

nemo_rl.telemetry.metrics.RL_EFFICIENCY_WINDOW_ATTR

‘rl.efficiency.window’

nemo_rl.telemetry.metrics.STEP_WINDOW

‘step’

nemo_rl.telemetry.metrics.RUN_WINDOW

‘run’

nemo_rl.telemetry.metrics.RL_SETUP_DURATION_METRIC

‘rl.setup.duration’

nemo_rl.telemetry.metrics.RL_SETUP_PHASE_ATTR

‘rl.setup.phase’

nemo_rl.telemetry.metrics._EFFICIENCY_KEY_PREFIX

‘efficiency/’

nemo_rl.telemetry.metrics._EFFICIENCY_SECONDS_SUFFIX

‘_s’

nemo_rl.telemetry.metrics.RL_METRIC_GROUP

‘rl’

nemo_rl.telemetry.metrics._EFFICIENCY_SECONDS_KEY

‘registry_key(…)’

nemo_rl.telemetry.metrics._EFFICIENCY_PCT_KEY_REGISTRY

‘registry_key(…)’

nemo_rl.telemetry.metrics._SETUP_DURATION_KEY

‘registry_key(…)’

nemo_rl.telemetry.metrics._REGISTERED

False

nemo_rl.telemetry.metrics._REGISTRATION_FAILED

False

nemo_rl.telemetry.metrics._WARNED: set[str]

‘set(…)’

nemo_rl.telemetry.metrics.warn_once(key: str, message: str) → None

Warn with a traceback the first time key fails, then stay quiet.

Telemetry failures are typically per-step and deterministic – a broken instrument fails identically on every step – so warning each time would put thousands of identical tracebacks in a run’s log while telling the reader nothing the first one did not. Warning level once, so a permanently dead sink is visible at default verbosity; debug afterwards, so the repetition is still recoverable when someone is looking for it.

nemo_rl.telemetry.metrics._metric_specs() → list[nemo.lens.instruments.MetricSpec]

Build the full MetricSpec list for the rl group.

nemo_rl.telemetry.metrics.ensure_metric_group_registered() → bool

Declare the rl metric group with lens, once per process.

Returns:

Whether the group is available to record against. False means the installed lens has no metric registry, which makes every tee below a no-op rather than an error.

nemo_rl.telemetry.metrics._record(

meter: opentelemetry.metrics.Meter,

values: Mapping[str, float],

attributes: Optional[Mapping[str, str]] = None,

) → None

Record against the rl group. record_metrics never raises.

nemo_rl.telemetry.metrics.record_rl_metrics(

values: Mapping[str, float],

attributes: Optional[Mapping[str, str]] = None,

) → None

Record declared series by registry key. No-op unless exporting.

For values that never pass through Logger.log_metrics and so have no teed row. Declare them with register_recorded_metrics first.

nemo_rl.telemetry.metrics.create_rl_metric_instruments() → None

Build the rl group’s instruments now. No-op unless exporting.

Only the observable rows need this. A synchronous series gets its instrument on the first :func:record_rl_metrics that names it, but an observable one is never recorded to – the exporter pulls it through the callback – so without this nothing ever creates it and it is simply never exported. Call it once from the owner that declared the callback, after telemetry is up.

Because this registers the group, it also closes declaration (see freeze_metrics). That makes telemetry startup the deadline for declaring a row, where it used to be the first metric recorded, so a module that declares one has to be imported by then – which every entry point already does.

nemo_rl.telemetry.metrics.efficiency_measurements() → dict[str, str]

Map each canonical efficiency category to its measurement kind.

Returns:

{category: "wall_clock" | "collector_wall_clock" | "thread_seconds"}, or an empty dict when the training stack is unavailable — which makes the efficiency tee a no-op instead of an import error.

nemo_rl.telemetry.metrics.efficiency_window(category: str, measurement: str) → str

Return the period one efficiency value covers (see the constants above).

nemo_rl.telemetry.metrics.map_efficiency_seconds(

metrics: dict[str, Any],

measurement_by_category: Mapping[str, str],

) → dict[str, float]

Extract {category: seconds} from a raw Logger metrics dict.

Looks up only the categories in measurement_by_category, which keeps the aggregate efficiency/* keys (total_waste_s, productive_time_s, total_wall_time_s, thread_seconds_total_s) out of the per-category series even though they share the prefix and suffix.

Pure function (no OTel side effects) so it is trivially unit-testable.

nemo_rl.telemetry.metrics.setup_phase

None

nemo_rl.telemetry.metrics.map_setup_seconds

None

nemo_rl.telemetry.metrics._tee_setup_metrics(

meter: opentelemetry.metrics.Meter,

metrics: dict[str, Any],

) → None

Emit the startup phase durations as rl.setup.duration.

nemo_rl.telemetry.metrics.map_teed_scalars(metrics: dict[str, Any]) → dict[str, float]

Extract {registry key: value} for every declared scalar present.

Pure function (no OTel side effects) so it is trivially unit-testable.

nemo_rl.telemetry.metrics._tee_scalars(

meter: opentelemetry.metrics.Meter,

metrics: dict[str, Any],

) → None

Emit the declared training and vLLM scalars, unlabelled.

nemo_rl.telemetry.metrics._tee_efficiency_metrics(

meter: opentelemetry.metrics.Meter,

metrics: dict[str, Any],

) → None

Emit the efficiency/* phase durations as rl.efficiency.*.

nemo_rl.telemetry.metrics.tee_rl_metrics_to_otel(

metrics: dict[str, Any],

prefix: Optional[str],

) → None

Mirror the per-step metrics into OTel (no-op unless exporting).

Three families ride the driver’s per-step train dicts and are teed: the efficiency/* durations, the training scalars, and the vllm/* engine deltas. The OTel instruments are touched only when telemetry is actively exporting; everything else short-circuits to a no-op.

Never raises. Logger.log_metrics calls this unguarded on every step, so the guarantee has to live here: each emit path below has its own handler, and this one covers everything around them – reading the handle, dispatching on the prefix, declaring the group – so no shape of telemetry failure can reach a training step.

nemo_rl.telemetry.metrics._tee_rl_metrics_to_otel(

metrics: dict[str, Any],

prefix: Optional[str],

) → None

Body of :func:tee_rl_metrics_to_otel, inside its exception guard.