These are set on spans for filtering — they answer “which one?” / “what kind?”, not “how much?”. Numerical values that change over time are metrics, not span tags (see Metrics).
Span group → rl.bucket
Leaf groups are tagged automatically when using
nemo_rl.telemetry.instrumentation.managed_span / trace_fn. Umbrellas are
timed but not tagged so monitors can exclude them from goodput.
Rolled-up rl.goodput is monitor-derived, not emitted by NeMo-RL.
Overriding the bucket for a region: bucket_scope
The table above classifies by what ran, but a few phases are productive or not
depending on why they ran. bucket_scope(bucket) reclassifies every leaf span
opened inside it:
with bucket_scope(Bucket.OVERHEAD):
... # generation in here is tagged overhead, not productive
The one production use is validation, in grpo.validate and ppo.validate.
Validation generates through the same generation group as a training rollout,
but its tokens are scored and discarded, so productive would count a
validation pass as goodput. The span is opened by a decorator on
VllmGeneration.generate that cannot see its caller, which is why the scope
travels with the execution context (a ContextVar) rather than an argument.
Three properties keep it from creating the double-counting problem it exists to
avoid: umbrellas stay unbucketed inside a scope, an explicit rl.bucket= passed
to managed_span still wins, and an efficiency_span keeps its category’s
bucket — that one names the phase it measures, so a caller cannot make
idle/refit_bubble productive. It propagates into coroutines started
inside the block — asyncio.run, as the rollout entrypoints use — but not into
threads or Ray actors, so a worker-side span is unaffected.
The efficiency group: idle time on async runs
Async GRPO measures its stalls with Timer under labels like
idle/buffer_starvation, on both sides of the run: the driver waiting on the
collector, and the collector waiting on the driver. efficiency_span in
nemo_rl/telemetry/instrumentation.py emits those as spans, taking the bucket
from EFFICIENCY_CATEGORY_BUCKET so idle/* lands in idle rather than
defaulting to overhead. Each span also carries
rl.efficiency.category with the raw label, so idle time can be grouped by
cause without parsing the span name.
Two driver-side phases are wired today, both children of rl.grpo.step:
With these enabled, a step’s child spans account for much more of the step
duration, so a per-step goodput breakdown leaves a smaller unattributed gap.
Why idle/validation is not a span
idle/validation is driver-side wall-clock like the two above, but it stays
Timer-only, because the window it measures is already accounted as
overhead: validate() wraps its generation in bucket_scope, so a second
span calling the same interval idle would contradict the label and, wherever
those generate spans exist, be counted twice — a rollup sums durations by
rl.bucket with no notion of nesting, so the pass would read as nearly double
its wall time.
Whether the children exist depends on the rollout path: sync validation
generates through the traced rl.vllm.generate, while async validation goes
through generate_async, which carries no span today. The overhead
attribution is the same either way, which is why this is Timer-only in both.
This is the general rule for efficiency_span: wrap a wait, not a phase that
does instrumented work. A bucketed span must be a leaf, which is the same
invariant the umbrella groups exist to preserve.
One gap remains: the phase means different things per fleet. The training GPUs
are idle while the generation GPUs do necessary non-training work, and
overhead on the generate span describes the latter only. Attributing the
former needs per-fleet accounting, not a per-phase bucket. Note also that the
val_at_start pass has no efficiency timer, so it appears in spans (as
overhead generation) but not in efficiency/*.
Trace-only: the collector’s two loop waits
idle/refit_event_wait and idle/generation_limit_pause are emitted as spans
from inside the AsyncTrajectoryCollector, but without rl.bucket:
Both are Event.wait() calls on the single collection-loop thread, so they are
honest wall-clock durations — but the collector’s wall clock runs concurrently
with the driver’s, so summing them against a driver-side denominator would
overcount. Omitting the attribute keeps them out of a bucket rollup by
construction instead of by convention. The membership list is
COLLECTOR_LOOP_CATEGORIES in nemo_rl/telemetry/instrumentation.py.
They still carry rl.efficiency.category, so they remain identifiable in a
trace and continue to be reported as efficiency/* scalars. As metrics they are
labelled rl.efficiency.measurement="collector_wall_clock" — sequential and so
real durations, unlike the batch-worker categories, but on the collector’s
timeline rather than the driver’s. See
Metrics — always filter on rl.efficiency.measurement.
Still reserved: the rest of the collector-side categories
idle/buffer_full_backoff and wasted/failed_trajectory stay Timer-only.
Both run in the batch-worker threads, so they are genuinely thread-seconds —
several workers accumulate at once and the total can exceed the wall time it
happened in. idle/buffer_full_backoff also has no clean block to wrap: it is
recorded as a precomputed duration spanning a retry loop. wasted/failed_trajectory
covers the same window as the enclosing rl.grpo.generation span, so a span
there would duplicate an existing interval. init/total is likewise still
Timer-only — it runs before the per-step loop, so it fills no step-level gap.
So goodput on async runs covers driver idle, but not collector-side idle or
wasted work — use the efficiency/* metrics for those.
Async rollout spans come from the collector actor
In an async run, no rollout is generated on the driver. Every trajectory comes
from inside AsyncTrajectoryCollector, a separate Ray actor, which calls
init_telemetry_worker(rank=0, world_size=1, always_export=True) in its
constructor. Explicit rank because it is a singleton, not a member of a ranked
group, and its runtime_env is a copy of the driver’s environment, so an
inherited RANK must not decide whether it exports. always_export goes with
that synthetic rank: an export_strategy that selects among a group’s ranks has
no meaning applied to a made-up one, and would silently mute the actor — with
export_rank: 3, every rollout span in the run would disappear. The driver uses
the same override for the same reason.
It flushes through flush_telemetry(), which the driver calls before ray.kill,
since a kill runs no atexit handler. That call stops the collection loop and
waits (bounded) for in-flight batch workers first: the shutdown is terminal, so
a still-running thread would keep opening spans against a dead processor.
Each batch worker opens one rl.grpo.generation span — the same name the sync
path uses on the driver, so the two modes read alike — carrying
rl.weight_version, rl.target_weight_version, rl.num_generations_per_prompt
and rl.num_prompt_groups. The last one is the batch width: a gap-filling batch
covers a fraction of a full one, so without it a short span looks like an
unexplained speed-up. It is in the rollout group, so it is an
umbrella and carries no rl.bucket: several batch workers run at once, so
their durations sum past wall time and cannot enter a bucket rollup.
Getting the collector into one waterfall
Ray does not propagate OTel context, so an actor’s spans start their own trace
by default. The driver captures its active span as a W3C traceparent carrier
with current_trace_carrier() — taken inside rl.grpo.job, at the point the
collector is constructed — and passes it as the actor’s trace_carrier
argument. The collector reopens it with remote_trace_context() in both the
collection-loop thread and every batch-worker thread. Per thread, not once per
process: OTel context is a ContextVar, and threading.Thread inherits none.
The result is a single trace per run:
rl.grpo.job (driver)
├── rl.grpo.step (iteration 1) (driver)
│ ├── rl.idle.buffer_starvation
│ └── rl.grpo.policy_training
├── rl.grpo.generation weight=7 (collector, thread A)
├── rl.idle.generation_limit_pause (collector, loop thread)
├── rl.grpo.generation weight=8 (collector, thread B)
└── rl.grpo.step (iteration 2) (driver)
This requires the job group to be enabled. current_trace_carrier()
returns an empty dict when no span is recording, and remote_trace_context({})
is a no-op, so the collector falls back to root spans. The default preset has
job but not rollout/efficiency, so the collector emits nothing; per_step
has rollout/efficiency but deliberately omits job so each step is its own
bounded trace. For the unified view, ask for both:
telemetry:
span_groups: per_step,job # or: all
Be deliberate about it. A run-long root span means one trace accumulating every
step and every rollout batch for the whole job, which is exactly the trace-size
problem per_step exists to avoid. Prefer it for debugging a specific run, not
as a standing default on long jobs.
Two consequences worth internalizing before reading an async trace:
One span per batch, not per sample. generate_async is dispatched one
coroutine per sample, so spanning it would emit thousands of mutually
overlapping spans per step.
There is no productive generation span in async mode, and there cannot
usefully be one. rl.vllm.generate is only reached through the synchronous
rollout path, and an async run never takes it: async_grpo_train requires an
async generation engine, so even validation goes through generate_async,
which carries no span today. Generation is a
continuously-batched pipeline overlapping training, so its productive
contribution is a utilization question — answered by fleet metrics — not a
span duration. A span-derived goodput ratio on an async run therefore has no
productive generation term; do not read it as “generation contributed
nothing.”
