Is this page helpful?

core.telemetry.training_metrics#

Training metric instruments (megatron.training.* namespace).

Self-contained copy of the metric recording logic from nemo.lens so that Megatron-LM can emit training metrics without a hard dependency on nemo-lens.

Module Contents#

Functions#

_get_training_instruments

record_training_metrics

Record training metrics to the OTel meter.

Data#

MEGATRON_TRAINING_STEP_DURATION_MS

MEGATRON_TRAINING_LOSS

MEGATRON_TRAINING_THROUGHPUT_TFLOPS

MEGATRON_TRAINING_GRAD_NORM

MEGATRON_TRAINING_SKIPPED_ITERS

MEGATRON_TRAINING_LEARNING_RATE

MEGATRON_TRAINING_TOKENS_PER_SEC

MEGATRON_TRAINING_MEMORY_ALLOCATED_GB

_logger

_TRAINING_INSTRUMENTS

API#

core.telemetry.training_metrics.MEGATRON_TRAINING_STEP_DURATION_MS#

‘megatron.training.step_duration_ms’

core.telemetry.training_metrics.MEGATRON_TRAINING_LOSS#

‘megatron.training.loss’

core.telemetry.training_metrics.MEGATRON_TRAINING_THROUGHPUT_TFLOPS#

‘megatron.training.throughput_tflops’

core.telemetry.training_metrics.MEGATRON_TRAINING_GRAD_NORM#

‘megatron.training.grad_norm’

core.telemetry.training_metrics.MEGATRON_TRAINING_SKIPPED_ITERS#

‘megatron.training.skipped_iters’

core.telemetry.training_metrics.MEGATRON_TRAINING_LEARNING_RATE#

‘megatron.training.learning_rate’

core.telemetry.training_metrics.MEGATRON_TRAINING_TOKENS_PER_SEC#

‘megatron.training.tokens_per_sec’

core.telemetry.training_metrics.MEGATRON_TRAINING_MEMORY_ALLOCATED_GB#

‘megatron.training.memory_allocated_gb’

core.telemetry.training_metrics._logger#

‘getLogger(…)’

core.telemetry.training_metrics._TRAINING_INSTRUMENTS: weakref.WeakKeyDictionary#

‘WeakKeyDictionary(…)’

core.telemetry.training_metrics._get_training_instruments(meter) → dict#
core.telemetry.training_metrics.record_training_metrics(
meter,
step_duration_ms: float | None = None,
loss: float | None = None,
throughput_tflops: float | None = None,
grad_norm: float | None = None,
skipped_iters: int | None = None,
learning_rate: float | None = None,
tokens_per_sec: float | None = None,
memory_allocated_gb: float | None = None,
) → None#

Record training metrics to the OTel meter.

All arguments are optional; None values are silently skipped. Safe to call when telemetry is disabled (meter is no-op).

If opentelemetry is not installed, this function is a no-op.