NeMo Lens | NVIDIA NeMo Lens

NeMo Lens (nemo.lens) is a shared OpenTelemetry instrumentation library for the NVIDIA NeMo ecosystem. It provides unified tracing, metrics, and optional log bridging across distributed training, inference, and RL workloads. NeMo Lens keeps hot-path overhead low with span-group gating, where disabled groups use a fast frozenset lookup and create no span objects. Consumer projects can integrate it as an optional dependency through canonical no-op fallbacks.

Key Features

Low overhead when disabled: span group gating is a frozenset lookup; no span objects are allocated when a group is off
Distributed-training aware: caller-supplied nv.dl.rank / nv.dl.world_size resource attributes, cross-rank trace correlation
Framework-agnostic primitives: managed_span, trace_fn, span_cm work everywhere
OTel idiomatic: real TracerProvider / MeterProvider, W3C propagation, standard semconv
Optional dependency: consumers import using try/except ImportError; NeMo Lens ships canonical no-op fallbacks
Pluggable: custom exporters, custom span groups
Resource auto-detection: SLURM, Kubernetes, local GPU count out of the box

Consumers

nemo-lens is consumed by three NeMo-ecosystem projects, each registering its own domain-specific span groups:

Consumer	Domain	Span group extension
Megatron-LM	Transformer pre-training	registers `megatron`: microbatch, layer, communication, activation_offload, data_loading, inference
NeMo RL	RLHF, GRPO, DPO	registers `nemo_rl`: rollout, generation, logprob, reward, advantage, policy_update, reference_policy, data_processing
NeMo Gym	RL environments	registers `nemo_gym`: server, rollout_collection, verify, aggregate_metrics

NeMo Lens emits OTLP; choosing and running an observability stack is up to you. Quick links for sending telemetry to common destinations:

For trying things out locally, see the demo docker-compose stack (proof of concept, not for production).