lens.providers | NVIDIA NeMo Lens

Internal: build TracerProvider + MeterProvider + LoggerProvider.

Only imported on exporting ranks when telemetry is enabled. All heavy SDK imports live here; opentelemetry-api (no-op) is the only dependency for code paths that never reach this module.

Module Contents

Classes

Name	Description
`SeedIndependentIdGenerator`	OTel-compatible IdGenerator whose IDs survive a global `random.seed()`.
`_OpenSpanCloser`	Span processor that ends still-open spans on provider shutdown.

Functions

Name	Description
`_build_log_exporter`	-
`_build_metric_exporter`	-
`_build_span_exporter`	-
`_compact_jsonl_formatter`	Format one OTel record (span, metrics batch, or log record) as a JSON line.
`_resolve_otlp_protocol`	Resolve the OTLP wire protocol for the given signal (‘traces’ / ‘metrics’ / ‘logs’).
`_set_propagator`	Set W3C TraceContext + Baggage as the global text map propagator.
`_setup_log_provider`	Set up the OTel LoggerProvider for log bridging.
`build_noop_providers`	Register no-op providers for non-exporting ranks or disabled telemetry.
`build_providers`	Initialise TracerProvider, MeterProvider, and optionally LoggerProvider.

Data

_VALID_EXPORTERS

API

class lens.providers.SeedIndependentIdGenerator()

OTel-compatible IdGenerator whose IDs survive a global random.seed().

OTel’s default RandomIdGenerator draws from the process-global random module, which training frameworks (Megatron) seed IDENTICALLY across data-parallel ranks -> every rank would emit the SAME span/trace IDs, so a backend sees many spans sharing one span ID and parent links resolve to the wrong span. A private random.Random instance is seeded from the OS at construction and is unaffected by random.seed() on the global module.

Duck-typed rather than subclassing opentelemetry.sdk.trace.id_generator. IdGenerator (same as :class:~nemo.lens.sampling.RankAwareSampler) so this module stays importable without the SDK installed.

_rng

= random.Random()

lens.providers.SeedIndependentIdGenerator.generate_span_id() -> int

lens.providers.SeedIndependentIdGenerator.generate_trace_id() -> int

lens.providers.SeedIndependentIdGenerator.is_trace_id_random() -> bool

Declares the W3C random-trace-id trace flag (Trace Context Level 2).

class lens.providers._OpenSpanCloser()

Span processor that ends still-open spans on provider shutdown.

BatchSpanProcessor only emits a span on on_end, so a span still open when the process exits is never exported — which silently drops exactly the long-lived spans that wrap a whole run. This tracks in-flight spans and ends the leftovers on shutdown.

Must be registered BEFORE the BatchSpanProcessor: TracerProvider’s default SynchronousMultiSpanProcessor shuts its children down in registration order, so ending a straggler here still reaches a live batch processor via on_end, and its own shutdown then flushes the queue. This ordering is the whole design, and it does NOT hold for ConcurrentMultiSpanProcessor, whose shutdown runs in parallel — do not swap the provider’s active_span_processor for that one.

A force-closed span carries nemo.span.truncated and a matching event: its end time is the shutdown time, not the true end, and a consumer must be able to tell the difference.

force_flush is deliberately a no-op — it runs mid-run to push already ended spans and must not terminate live ones.

Duck-typed rather than subclassing opentelemetry.sdk.trace.SpanProcessor (same as :class:SeedIndependentIdGenerator) so this module stays importable without the SDK installed — every SDK import here is function-local.

_lock

= threading.Lock()

_open

dict = {}

lens.providers._OpenSpanCloser._on_ending(
    span
) -> None

lens.providers._OpenSpanCloser._reset_after_fork() -> None

lens.providers._OpenSpanCloser._span_id(
    span
) -> int | None

staticmethod

lens.providers._OpenSpanCloser.force_flush(
    timeout_millis: int = 30000
) -> bool

lens.providers._OpenSpanCloser.on_end(
    span
) -> None

lens.providers._OpenSpanCloser.on_start(
    span,
    parent_context = None
) -> None

lens.providers._OpenSpanCloser.shutdown() -> None

lens.providers._build_log_exporter(
    config: nemo.lens.config.NemoLensConfig
)

lens.providers._build_metric_exporter(
    config: nemo.lens.config.NemoLensConfig
)

lens.providers._build_span_exporter(
    config: nemo.lens.config.NemoLensConfig
)

lens.providers._compact_jsonl_formatter(
    record
) -> str

Format one OTel record (span, metrics batch, or log record) as a JSON line.

The SDK’s console exporters default to to_json(indent=4), which emits a multi-line block per record instead of one JSON object per line. Passing indent=None keeps each record on a single line, so a redirected console export is real JSONL that downstream tooling (e.g. perfetto conversion) can read directly.

lens.providers._resolve_otlp_protocol(
    signal: str
) -> str

Resolve the OTLP wire protocol for the given signal (‘traces’ / ‘metrics’ / ‘logs’).

Honours the OTel SDK convention: OTEL_EXPORTER_OTLP_<SIGNAL>_PROTOCOL takes precedence over OTEL_EXPORTER_OTLP_PROTOCOL. Defaults to "grpc" when neither is set, matching the OTel SDK’s default.

Recognised values: grpc, http/protobuf, http/json.

lens.providers._set_propagator() -> None

Set W3C TraceContext + Baggage as the global text map propagator.

lens.providers._setup_log_provider(
    config: nemo.lens.config.NemoLensConfig,
    resource
) -> None

Set up the OTel LoggerProvider for log bridging.

lens.providers.build_noop_providers() -> None

lens.providers.build_providers(
    config: nemo.lens.config.NemoLensConfig,
    rank: int = 0,
    world_size: int = 1,
    resource_attributes: dict | None = None,
    span_exporter = None,
    metric_reader = None
) -> None

Initialise TracerProvider, MeterProvider, and optionally LoggerProvider.

Imports the OTel SDK. Raises ImportError if not installed.

Parameters:

config

NemoLensConfig

Telemetry configuration.

rank

int" default="0

Current process rank.

world_size

int" default="1

Total number of ranks.

resource_attributes

dict | None" default="None

Extra resource attributes to merge.

span_exporter

Defaults to None

Optional custom span exporter (bypasses config-based exporter).

metric_reader

Defaults to None

Optional custom metric reader (bypasses config-based reader).

lens.providers._VALID_EXPORTERS = ('otlp', 'console')