`nemo_rl.telemetry.metrics`#

Best-effort mirroring of NeMo-RL’s async efficiency metrics into OTel.

Kept out of nemo_rl.utils.logger (which pulls in torch/ray/wandb/etc.) so the mapping stays importable and testable without the heavy training stack. nemo_rl.utils.logger.Logger.log_metrics calls :func:tee_rl_metrics_to_otel after its normal fan-out to the file/wandb/mlflow backends.

The async efficiency/* phase durations are emitted from instruments owned here, because they are keyed by NeMo-RL’s own efficiency-category labels and lens has no fixed field for them.

Logger.log_metrics fans a step out as several dicts under different prefixes, and a key is only reachable from the prefix its own dict carries — so only the prefixes the efficiency dict actually arrives under are teed.

Module Contents#

Functions#

`_scalar`	Coerce a logged value to float, or None when it is not a usable scalar.
`warn_once`	Warn with a traceback the first time key fails, then stay quiet.
`efficiency_measurements`	Map each canonical efficiency category to its measurement kind.
`efficiency_window`	Return the period one efficiency value covers (see the constants above).
`map_efficiency_seconds`	Extract `{category: seconds}` from a raw Logger metrics dict.
`_get_efficiency_instruments`	Create (once per Meter) the efficiency gauges.
`_tee_efficiency_metrics`	Emit the `efficiency/` phase durations as `rl.efficiency.`.
`tee_rl_metrics_to_otel`	Mirror the async efficiency durations into OTel (no-op unless exporting).
`_tee_rl_metrics_to_otel`	Body of :func:`tee_rl_metrics_to_otel`, inside its exception guard.

Data#

`logger`
`_TRAIN_PREFIXES`
`RL_EFFICIENCY_SECONDS_METRIC`
`RL_EFFICIENCY_PCT_METRIC`
`RL_EFFICIENCY_MEASUREMENT_ATTR`
`WALL_CLOCK_MEASUREMENT`
`COLLECTOR_WALL_CLOCK_MEASUREMENT`
`THREAD_SECONDS_MEASUREMENT`
`RL_EFFICIENCY_WINDOW_ATTR`
`STEP_WINDOW`
`RUN_WINDOW`
`_RUN_WINDOW_WALL_CLOCK_CATEGORIES`
`_EFFICIENCY_KEY_PREFIX`
`_EFFICIENCY_SECONDS_SUFFIX`
`_EFFICIENCY_PCT_KEY`
`_EFFICIENCY_PCT_PER_STEP_KEY`
`_EFFICIENCY_INSTRUMENTS`
`_WARNED`

API#

nemo_rl.telemetry.metrics.logger#: ‘getLogger(…)’

nemo_rl.telemetry.metrics._TRAIN_PREFIXES: tuple[Optional[str], ...]#: (‘train’, ‘’)

nemo_rl.telemetry.metrics._scalar(value: Any) → Optional[float]#: Coerce a logged value to float, or None when it is not a usable scalar.

nemo_rl.telemetry.metrics.RL_EFFICIENCY_SECONDS_METRIC#: ‘rl.efficiency.seconds’

nemo_rl.telemetry.metrics.RL_EFFICIENCY_PCT_METRIC#: ‘rl.efficiency.pct’

nemo_rl.telemetry.metrics.RL_EFFICIENCY_MEASUREMENT_ATTR#: ‘rl.efficiency.measurement’

nemo_rl.telemetry.metrics.WALL_CLOCK_MEASUREMENT#: ‘wall_clock’

nemo_rl.telemetry.metrics.COLLECTOR_WALL_CLOCK_MEASUREMENT#: ‘collector_wall_clock’

nemo_rl.telemetry.metrics.THREAD_SECONDS_MEASUREMENT#: ‘thread_seconds’

nemo_rl.telemetry.metrics.RL_EFFICIENCY_WINDOW_ATTR#: ‘rl.efficiency.window’

nemo_rl.telemetry.metrics.STEP_WINDOW#: ‘step’

nemo_rl.telemetry.metrics.RUN_WINDOW#: ‘run’

nemo_rl.telemetry.metrics._RUN_WINDOW_WALL_CLOCK_CATEGORIES: frozenset[str]#: ‘frozenset(…)’

nemo_rl.telemetry.metrics._EFFICIENCY_KEY_PREFIX#: ‘efficiency/’

nemo_rl.telemetry.metrics._EFFICIENCY_SECONDS_SUFFIX#: ‘_s’

nemo_rl.telemetry.metrics._EFFICIENCY_PCT_KEY#: ‘efficiency/efficiency_pct’

nemo_rl.telemetry.metrics._EFFICIENCY_PCT_PER_STEP_KEY#: ‘efficiency/efficiency_pct_is_per_step’

nemo_rl.telemetry.metrics._EFFICIENCY_INSTRUMENTS: weakref.WeakKeyDictionary#: ‘WeakKeyDictionary(…)’

nemo_rl.telemetry.metrics._WARNED: set[str]#: ‘set(…)’

nemo_rl.telemetry.metrics.warn_once(key: str, message: str) → None#

Warn with a traceback the first time key fails, then stay quiet.

Telemetry failures are typically per-step and deterministic – a broken instrument fails identically on every step – so warning each time would put thousands of identical tracebacks in a run’s log while telling the reader nothing the first one did not. Warning level once, so a permanently dead sink is visible at default verbosity; debug afterwards, so the repetition is still recoverable when someone is looking for it.

nemo_rl.telemetry.metrics.efficiency_measurements() → dict[str, str]#

Map each canonical efficiency category to its measurement kind.

Returns:: {category: "wall_clock" | "collector_wall_clock" | "thread_seconds"}, or an empty dict when the training stack is unavailable — which makes the efficiency tee a no-op instead of an import error.

nemo_rl.telemetry.metrics.efficiency_window(category: str, measurement: str) → str#: Return the period one efficiency value covers (see the constants above).

nemo_rl.telemetry.metrics.map_efficiency_seconds( metrics: dict[str, Any], measurement_by_category: Mapping[str, str], ) → dict[str, float]#

Extract {category: seconds} from a raw Logger metrics dict.

Looks up only the categories in measurement_by_category, which keeps the aggregate efficiency/* keys (total_waste_s, productive_time_s, total_wall_time_s, thread_seconds_total_s) out of the per-category series even though they share the prefix and suffix.

Pure function (no OTel side effects) so it is trivially unit-testable.

nemo_rl.telemetry.metrics._get_efficiency_instruments( meter: opentelemetry.metrics.Meter, ) → dict[str, Any]#: Create (once per Meter) the efficiency gauges.

nemo_rl.telemetry.metrics._tee_efficiency_metrics( meter: opentelemetry.metrics.Meter, metrics: dict[str, Any], ) → None#: Emit the efficiency/* phase durations as rl.efficiency.*.

nemo_rl.telemetry.metrics.tee_rl_metrics_to_otel( metrics: dict[str, Any], prefix: Optional[str], ) → None#

Mirror the async efficiency durations into OTel (no-op unless exporting).

Only the efficiency/* durations logged alongside the driver’s per-step train scalars are teed. The OTel instruments are touched only when telemetry is actively exporting; everything else short-circuits to a no-op.

Never raises. Logger.log_metrics calls this unguarded on every step, so the guarantee has to live here: the emit path below has its own handler, and this one covers everything around it – reading the handle, dispatching on the prefix – so no shape of telemetry failure can reach a training step.

nemo_rl.telemetry.metrics._tee_rl_metrics_to_otel( metrics: dict[str, Any], prefix: Optional[str], ) → None#: Body of :func:tee_rl_metrics_to_otel, inside its exception guard.

nemo_rl.telemetry.metrics#

Module Contents#

Functions#

Data#

API#

`nemo_rl.telemetry.metrics`#