Skip to navigation

nemo_gym.telemetry.spans

View as MarkdownOpen in Claude

Spans nemo-lens cannot produce: a CLIENT-kind span, a linked root for a new trace, and a child of an ended span.

client_span: nemo.lens.helpers.managed_span calls tracer.start_span(name) with no kind argument and exposes no way to pass one, so every span it creates is SpanKind.INTERNAL. For spans inside a service that is right. For Gym’s outbound calls it is not: an INTERNAL span on a cross-service hop means a backend cannot tell that the agent server called the model server, so Jaeger/Tempo/Honeycomb service maps lose the edge and client-vs-server latency attribution stops working. Gym is pinned to nemo-lens 4d045c03, so rather than change lens this module creates that one span directly against the OTel API. Adding a kind parameter to managed_span is the proper fix; when it lands, delete client_span and pass kind= instead.

linked_root_span: starts a span as the root of a new trace, linked to the span it splits from, when the current span is that span; otherwise an ordinary child span. nemo-lens has create_linked_span for cross-rank correlation but nothing that also decides whether to split and attaches the result as current. The eval driver uses it so that a run is one small job trace plus one bounded trace per rollout.

record_child_span: emits a finished span under a parent given as a SpanContext, which may have ended already. The eval driver evaluates a rollout’s health after gym.rollout.dispatch closed and puts the verdict on a child of it.

client_span and linked_root_span are otherwise identical to managed_span: context attach/detach, exception recording, and an unconditional end() in finally.

Callers must gate on the span group themselves. Nothing here checks it, so that a disabled site does not pay for entering a context manager at all (kb/knowledge/conventions/hot-path-overhead.md).

Module Contents

Functions

NameDescription
client_spanStart a SpanKind.CLIENT span, attach it as current, and always end it.
linked_root_spanStart name as the root of a new trace, linked to split_from, when that span is current.
record_child_spanEmit name as a finished child of parent, which may have ended. Gate on the span group first.

API

nemo_gym.telemetry.spans.client_span(
name: str,
tracer: typing.Any = None,
attributes: typing.Any = {}
)

Start a SpanKind.CLIENT span, attach it as current, and always end it.

Parameters:

name
str

Span name.

tracer
AnyDefaults to None

OTel tracer; defaults to the globally registered one.

**attributes
AnyDefaults to {}

Set on the span via safe_set_span_attributes, so non-scalar values are dropped and sensitive keys are redacted rather than exported.

nemo_gym.telemetry.spans.linked_root_span(
name: str,
tracer: typing.Any = None,
split_from: opentelemetry.trace.SpanContext | None = None,
attributes: typing.Any = {}
) -> collections.abc.Iterator[opentelemetry.trace.Span]

Start name as the root of a new trace, linked to split_from, when that span is current.

Under the run-long gym.job span, parent/child nesting would put every rollout of a run into one trace whose size grows with the run and which stays open until the job ends; backends cap and buffer traces as units, so long runs lose spans. Splitting gives each rollout a trace bounded by its own work, and the span link keeps the job ↔ rollout relation.

The split happens only when the current span is split_from, the SpanContext of a gym.job span the driver opened as a standalone root. In every other case, an embedder’s span, no owned job, a remote non-recording context, or a child of the job being current, the span nests as an ordinary child, so the caller’s trace keeps its shape.

Parameters:

name
str

Span name.

tracer
AnyDefaults to None

OTel tracer; defaults to the globally registered one.

split_from
SpanContext | NoneDefaults to None

SpanContext of the driver-owned job span, or None when no such span exists.

**attributes
AnyDefaults to {}

Set via safe_set_span_attributes.

nemo_gym.telemetry.spans.record_child_span(
name: str,
parent: opentelemetry.trace.SpanContext,
tracer: typing.Any = None,
attributes: typing.Any = {}
) -> None

Emit name as a finished child of parent, which may have ended. Gate on the span group first.