nemo_rl.telemetry.metrics

Best-effort mirroring of NeMo-RL’s per-step metrics into OTel.

Kept out of nemo_rl.utils.logger (which pulls in torch/ray/wandb/etc.) so the mapping stays importable and testable without the heavy training stack. nemo_rl.utils.logger.Logger.log_metrics calls :func:tee_rl_metrics_to_otel after its normal fan-out to the file/wandb/mlflow backends.

Every series is declared by NeMo-RL rather than by lens, via lens’s consumer-driven metric registry: NeMo-RL owns the rl.* metric names, so a new series needs no lens release and no negotiation over field names. Four families are teed — the async efficiency/* phase durations, the training scalars, the vllm/* engine deltas, and the timing/setup startup phases. The first three ride the per-step train dicts; the last arrives once, at step 0.

This module names no logger key of its own beyond the efficiency/* family it also produces. The scalar rows come from the modules that log them, through

mod:

nemo_rl.telemetry.vocabulary, so adding an algorithm does not mean editing telemetry.

Logger.log_metrics fans a step out as several dicts under different prefixes, and a key is only reachable from the prefix its own dict carries — so only the prefixes those dicts actually arrive under are teed.

Module Contents

Functions

warn_once	Warn with a traceback the first time key fails, then stay quiet.
_metric_specs	Build the full MetricSpec list for the rl group.
ensure_metric_group_registered	Declare the rl metric group with lens, once per process.
_record	Record against the rl group. record_metrics never raises.
record_rl_metrics	Record declared series by registry key. No-op unless exporting.
efficiency_measurements	Map each canonical efficiency category to its measurement kind.
efficiency_window	Return the period one efficiency value covers (see the constants above).
map_efficiency_seconds	Extract {category: seconds} from a raw Logger metrics dict.
_tee_setup_metrics	Emit the startup phase durations as rl.setup.duration.
map_teed_scalars	Extract {registry key: value} for every declared scalar present.
_tee_scalars	Emit the declared training and vLLM scalars, unlabelled.
_tee_efficiency_metrics	Emit the efficiency/* phase durations as rl.efficiency.*.
tee_rl_metrics_to_otel	Mirror the per-step metrics into OTel (no-op unless exporting).
_tee_rl_metrics_to_otel	Body of :func:tee_rl_metrics_to_otel, inside its exception guard.

Data

logger
_TRAIN_PREFIXES
RL_EFFICIENCY_SECONDS_METRIC
RL_EFFICIENCY_PCT_METRIC
RL_EFFICIENCY_MEASUREMENT_ATTR
WALL_CLOCK_MEASUREMENT
COLLECTOR_WALL_CLOCK_MEASUREMENT
THREAD_SECONDS_MEASUREMENT
RL_EFFICIENCY_WINDOW_ATTR
STEP_WINDOW
RUN_WINDOW
RL_SETUP_DURATION_METRIC
RL_SETUP_PHASE_ATTR
_EFFICIENCY_KEY_PREFIX
_EFFICIENCY_SECONDS_SUFFIX
_EFFICIENCY_PCT_KEY
_EFFICIENCY_PCT_PER_STEP_KEY
RL_METRIC_GROUP
_EFFICIENCY_SECONDS_KEY
_EFFICIENCY_PCT_KEY_REGISTRY
_SETUP_DURATION_KEY
_REGISTERED
_REGISTRATION_FAILED
_WARNED
setup_phase
map_setup_seconds

API

nemo_rl.telemetry.metrics.logger

‘getLogger(…)’

nemo_rl.telemetry.metrics._TRAIN_PREFIXES: tuple[Optional[str], ...]

(‘train’, ‘’)

nemo_rl.telemetry.metrics.RL_EFFICIENCY_SECONDS_METRIC

‘rl.efficiency.seconds’

nemo_rl.telemetry.metrics.RL_EFFICIENCY_PCT_METRIC

‘rl.efficiency.pct’

nemo_rl.telemetry.metrics.RL_EFFICIENCY_MEASUREMENT_ATTR

‘rl.efficiency.measurement’

nemo_rl.telemetry.metrics.WALL_CLOCK_MEASUREMENT

‘wall_clock’

nemo_rl.telemetry.metrics.COLLECTOR_WALL_CLOCK_MEASUREMENT

‘collector_wall_clock’

nemo_rl.telemetry.metrics.THREAD_SECONDS_MEASUREMENT

‘thread_seconds’

nemo_rl.telemetry.metrics.RL_EFFICIENCY_WINDOW_ATTR

‘rl.efficiency.window’

nemo_rl.telemetry.metrics.STEP_WINDOW

‘step’

nemo_rl.telemetry.metrics.RUN_WINDOW

‘run’

nemo_rl.telemetry.metrics.RL_SETUP_DURATION_METRIC

‘rl.setup.duration’

nemo_rl.telemetry.metrics.RL_SETUP_PHASE_ATTR

‘rl.setup.phase’

nemo_rl.telemetry.metrics._EFFICIENCY_KEY_PREFIX

‘efficiency/’

nemo_rl.telemetry.metrics._EFFICIENCY_SECONDS_SUFFIX

‘_s’

nemo_rl.telemetry.metrics._EFFICIENCY_PCT_KEY

‘efficiency/efficiency_pct’

nemo_rl.telemetry.metrics._EFFICIENCY_PCT_PER_STEP_KEY

‘efficiency/efficiency_pct_is_per_step’

nemo_rl.telemetry.metrics.RL_METRIC_GROUP

‘rl’

nemo_rl.telemetry.metrics._EFFICIENCY_SECONDS_KEY

‘registry_key(…)’

nemo_rl.telemetry.metrics._EFFICIENCY_PCT_KEY_REGISTRY

‘registry_key(…)’

nemo_rl.telemetry.metrics._SETUP_DURATION_KEY

‘registry_key(…)’

nemo_rl.telemetry.metrics._REGISTERED

False

nemo_rl.telemetry.metrics._REGISTRATION_FAILED

False

nemo_rl.telemetry.metrics._WARNED: set[str]

‘set(…)’

nemo_rl.telemetry.metrics.warn_once(key: str, message: str) → None

Warn with a traceback the first time key fails, then stay quiet.

Telemetry failures are typically per-step and deterministic – a broken instrument fails identically on every step – so warning each time would put thousands of identical tracebacks in a run’s log while telling the reader nothing the first one did not. Warning level once, so a permanently dead sink is visible at default verbosity; debug afterwards, so the repetition is still recoverable when someone is looking for it.

nemo_rl.telemetry.metrics._metric_specs() → list[nemo.lens.instruments.MetricSpec]

Build the full MetricSpec list for the rl group.

nemo_rl.telemetry.metrics.ensure_metric_group_registered() → bool

Declare the rl metric group with lens, once per process.

Returns:

Whether the group is available to record against. False means the installed lens has no metric registry, which makes every tee below a no-op rather than an error.

nemo_rl.telemetry.metrics._record(

meter: opentelemetry.metrics.Meter,

values: Mapping[str, float],

attributes: Optional[Mapping[str, str]] = None,

) → None

Record against the rl group. record_metrics never raises.

nemo_rl.telemetry.metrics.record_rl_metrics(

values: Mapping[str, float],

attributes: Optional[Mapping[str, str]] = None,

) → None

Record declared series by registry key. No-op unless exporting.

For values that never pass through Logger.log_metrics and so have no teed row. Declare them with register_recorded_metrics first.

nemo_rl.telemetry.metrics.efficiency_measurements() → dict[str, str]

Map each canonical efficiency category to its measurement kind.

Returns:

{category: "wall_clock" | "collector_wall_clock" | "thread_seconds"}, or an empty dict when the training stack is unavailable — which makes the efficiency tee a no-op instead of an import error.

nemo_rl.telemetry.metrics.efficiency_window(category: str, measurement: str) → str

Return the period one efficiency value covers (see the constants above).

nemo_rl.telemetry.metrics.map_efficiency_seconds(

metrics: dict[str, Any],

measurement_by_category: Mapping[str, str],

) → dict[str, float]

Extract {category: seconds} from a raw Logger metrics dict.

Looks up only the categories in measurement_by_category, which keeps the aggregate efficiency/* keys (total_waste_s, productive_time_s, total_wall_time_s, thread_seconds_total_s) out of the per-category series even though they share the prefix and suffix.

Pure function (no OTel side effects) so it is trivially unit-testable.

nemo_rl.telemetry.metrics.setup_phase

None

nemo_rl.telemetry.metrics.map_setup_seconds

None

nemo_rl.telemetry.metrics._tee_setup_metrics(

meter: opentelemetry.metrics.Meter,

metrics: dict[str, Any],

) → None

Emit the startup phase durations as rl.setup.duration.

nemo_rl.telemetry.metrics.map_teed_scalars(metrics: dict[str, Any]) → dict[str, float]

Extract {registry key: value} for every declared scalar present.

Pure function (no OTel side effects) so it is trivially unit-testable.

nemo_rl.telemetry.metrics._tee_scalars(

meter: opentelemetry.metrics.Meter,

metrics: dict[str, Any],

) → None

Emit the declared training and vLLM scalars, unlabelled.

nemo_rl.telemetry.metrics._tee_efficiency_metrics(

meter: opentelemetry.metrics.Meter,

metrics: dict[str, Any],

) → None

Emit the efficiency/* phase durations as rl.efficiency.*.

nemo_rl.telemetry.metrics.tee_rl_metrics_to_otel(

metrics: dict[str, Any],

prefix: Optional[str],

) → None

Mirror the per-step metrics into OTel (no-op unless exporting).

Three families ride the driver’s per-step train dicts and are teed: the efficiency/* durations, the training scalars, and the vllm/* engine deltas. The OTel instruments are touched only when telemetry is actively exporting; everything else short-circuits to a no-op.

Never raises. Logger.log_metrics calls this unguarded on every step, so the guarantee has to live here: each emit path below has its own handler, and this one covers everything around them – reading the handle, dispatching on the prefix, declaring the group – so no shape of telemetry failure can reach a training step.

nemo_rl.telemetry.metrics._tee_rl_metrics_to_otel(

metrics: dict[str, Any],

prefix: Optional[str],

) → None

Body of :func:tee_rl_metrics_to_otel, inside its exception guard.