nemo_rl.telemetry.config

Telemetry configuration schema for NeMo-RL.

The telemetry: block of a run config. :mod:nemo_rl.telemetry.setup translates it into NEMO_RL_OTEL_* environment variables on the driver before init_ray(), so every Ray worker inherits the same settings via the Ray runtime_env. Raw NEMO_RL_OTEL_* / OTEL_EXPORTER_OTLP_* env vars always win over these YAML values (they are applied with setdefault).

This module imports only pydantic — it never requires nemo-lens, so it is safe to import unconditionally from the algorithm MasterConfig classes.

Module Contents

Classes

TelemetryConfig

OpenTelemetry / nemo-lens configuration.

API

class nemo_rl.telemetry.config.TelemetryConfig

Bases: pydantic.BaseModel

OpenTelemetry / nemo-lens configuration.

Telemetry activates only when enabled is true; otherwise every instrumentation site degrades to a ~0-cost no-op.

Fields with a fixed set of valid values are typed so that a typo is rejected when the YAML is parsed, in every process, rather than surfacing later on a GPU node – or not at all, when enabled is false and the driver returns before it validates anything.

enabled: bool

False

Master switch. When false, all instrumentation is a ~0-cost no-op.

service_name: str

‘nemo-rl’

service.name reported to the OTLP backend.

span_groups: str

‘default’

Span-group spec: a preset (default | per_step | all) or a comma-separated list of individual group names (e.g. "default,generation,reward"). See RLSpanGroup.

traces_enabled: bool

True

Emit trace spans.

metrics_enabled: bool

True

Emit metric instruments (the rl.* gauges/histograms).

logs_enabled: bool

False

Bridge Python logging to OTel logs (exported with trace correlation).

exporter: Literal[otlp, console]

‘otlp’

Exporter backend. The OTLP endpoint / headers / protocol come from the standard OTEL_EXPORTER_OTLP_* env vars, so any OTLP-compatible backend or an OpenTelemetry Collector works.

vllm_native_tracing: bool

False

Enable vLLM’s own OTLP tracing inside generation workers (opt-in). vLLM builds its exporter itself and defaults it to gRPC, reading only OTEL_EXPORTER_OTLP_TRACES_PROTOCOL – not the generic OTEL_EXPORTER_OTLP_PROTOCOL lens honours. To ride the same http/protobuf endpoint as lens, set the traces-specific var too.

vLLM traces per request, so this emits one span per rollout – thousands per step, against ~20 for the rest of the run – and vLLM offers no sampling knob. Switch it on to debug a few steps, not for a training run. For engine behaviour in aggregate the vllm/* metrics are on by default and cost two RPCs per step (a snapshot before generation and a read after).