> This page is for version Nightly.
> For other versions, use one of these documentation indexes:
> - 0.1.0 (default): https://docs.nvidia.com/nemo/lens/0.1.0/llms.txt
> - Nightly: https://docs.nvidia.com/nemo/lens/nightly/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/lens/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/lens/_mcp/server.

# NeMo Lens

**NeMo Lens** (`nemo.lens`) is a shared [OpenTelemetry](https://opentelemetry.io/) instrumentation library for the NVIDIA NeMo ecosystem. It provides unified tracing, metrics, and optional log bridging across distributed training, inference, and RL workloads. NeMo Lens keeps hot-path overhead low with span-group gating, where disabled groups use a fast `frozenset` lookup and create no span objects. Consumer projects can integrate it as an optional dependency through canonical no-op fallbacks.

## Key Features

* **Low overhead when disabled**: span group gating is a `frozenset` lookup; no span objects are allocated when a group is off
* **Distributed-training aware**: caller-supplied `nv.dl.rank` / `nv.dl.world_size` resource attributes, cross-rank trace correlation
* **Framework-agnostic primitives**: `managed_span`, `trace_fn`, `span_cm` work everywhere
* **OTel idiomatic**: real `TracerProvider` / `MeterProvider`, W3C propagation, standard semconv
* **Optional dependency**: consumers import using `try/except ImportError`; NeMo Lens ships canonical no-op fallbacks
* **Pluggable**: custom exporters, custom span groups
* **Resource auto-detection**: SLURM, Kubernetes, local GPU count out of the box

## Consumers

`nemo-lens` is consumed by three NeMo-ecosystem projects, each registering its own domain-specific span groups:

| Consumer                                             | Domain                   | Span group extension                                                                                                   |
| ---------------------------------------------------- | ------------------------ | ---------------------------------------------------------------------------------------------------------------------- |
| [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) | Transformer pre-training | registers `megatron`: microbatch, layer, communication, activation_offload, data_loading, inference                    |
| [NeMo RL](https://github.com/NVIDIA-NeMo/RL)         | RLHF, GRPO, DPO          | registers `nemo_rl`: rollout, generation, logprob, reward, advantage, policy_update, reference_policy, data_processing |
| [NeMo Gym](https://github.com/NVIDIA-NeMo/Gym)       | RL environments          | registers `nemo_gym`: server, rollout_collection, verify, aggregate_metrics                                            |

NeMo Lens emits OTLP; choosing and running an observability stack is up to you. Quick links for sending telemetry to common destinations:

* [Send telemetry to a file](/nemo/lens/nightly/backends-demo-stack/send-telemetry-to-a-backend#file)
* [Send telemetry to W\&B Weave](/nemo/lens/nightly/backends-demo-stack/send-telemetry-to-a-backend#wb-weave)
* [Send telemetry to Honeycomb](/nemo/lens/nightly/backends-demo-stack/send-telemetry-to-a-backend#honeycomb)
* [Send telemetry to an OTel Collector](/nemo/lens/nightly/backends-demo-stack/send-telemetry-to-a-backend#otel-collector)

For trying things out locally, see the [demo docker-compose stack](/nemo/lens/nightly/backends-demo-stack/demo-stack) (proof of concept, not for production).