nemo_gym.telemetry.spans
nemo_gym.telemetry.spans
Spans nemo-lens cannot produce: a CLIENT-kind span, a linked root for a new trace, and a child of an ended span.
client_span: nemo.lens.helpers.managed_span calls tracer.start_span(name) with no
kind argument and exposes no way to pass one, so every span it creates is
SpanKind.INTERNAL. For spans inside a service that is right. For Gym’s outbound calls it
is not: an INTERNAL span on a cross-service hop means a backend cannot tell that the agent
server called the model server, so Jaeger/Tempo/Honeycomb service maps lose the edge and
client-vs-server latency attribution stops working. Gym is pinned to nemo-lens 4d045c03,
so rather than change lens this module creates that one span directly against the OTel API.
Adding a kind parameter to managed_span is the proper fix; when it lands, delete
client_span and pass kind= instead.
linked_root_span: starts a span as the root of a new trace, linked to the span it
splits from, when the current span is that span; otherwise an ordinary child span. nemo-lens
has create_linked_span for cross-rank correlation but nothing that also decides whether
to split and attaches the result as current. The eval driver uses it so that a run is one
small job trace plus one bounded trace per rollout.
record_child_span: emits a finished span under a parent given as a SpanContext, which may
have ended already. The eval driver evaluates a rollout’s health after gym.rollout.dispatch
closed and puts the verdict on a child of it.
client_span and linked_root_span are otherwise identical to managed_span: context
attach/detach, exception recording, and an unconditional end() in finally.
Callers must gate on the span group themselves. Nothing here checks it, so that a
disabled site does not pay for entering a context manager at all
(kb/knowledge/conventions/hot-path-overhead.md).
Module Contents
Functions
API
Start a SpanKind.CLIENT span, attach it as current, and always end it.
Parameters:
Span name.
OTel tracer; defaults to the globally registered one.
Set on the span via safe_set_span_attributes, so non-scalar
values are dropped and sensitive keys are redacted rather than exported.
Start name as the root of a new trace, linked to split_from, when that span is current.
Under the run-long gym.job span, parent/child nesting would put every rollout of a run
into one trace whose size grows with the run and which stays open until the job ends;
backends cap and buffer traces as units, so long runs lose spans. Splitting gives each
rollout a trace bounded by its own work, and the span link keeps the job ↔ rollout relation.
The split happens only when the current span is split_from, the SpanContext of a
gym.job span the driver opened as a standalone root. In every other case, an embedder’s
span, no owned job, a remote non-recording context, or a child of the job being current,
the span nests as an ordinary child, so the caller’s trace keeps its shape.
Parameters:
Span name.
OTel tracer; defaults to the globally registered one.
SpanContext of the driver-owned job span, or None when no such span exists.
Set via safe_set_span_attributes.
Emit name as a finished child of parent, which may have ended. Gate on the span group first.