> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/lens/nightly/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/lens/_mcp/server.

# Nightly

## Docs

- [Overview](https://docs.nvidia.com/nemo/lens/nightly/get-started/overview.md): 
- [Installation](https://docs.nvidia.com/nemo/lens/nightly/get-started/installation.md): 
- [Quickstart](https://docs.nvidia.com/nemo/lens/nightly/get-started/quickstart.md): 
- [Configuration](https://docs.nvidia.com/nemo/lens/nightly/user-guide/configuration.md): 
- [Instrumentation Primitives](https://docs.nvidia.com/nemo/lens/nightly/user-guide/instrumentation-primitives.md): 
- [Command Line](https://docs.nvidia.com/nemo/lens/nightly/user-guide/command-line.md): 
- [Span Groups](https://docs.nvidia.com/nemo/lens/nightly/user-guide/span-groups.md): 
- [Metrics](https://docs.nvidia.com/nemo/lens/nightly/user-guide/metrics.md): 
- [Context Propagation](https://docs.nvidia.com/nemo/lens/nightly/user-guide/context-propagation.md): 
- [Distributed Tracing](https://docs.nvidia.com/nemo/lens/nightly/user-guide/distributed-tracing.md): 
- [Custom Exporters](https://docs.nvidia.com/nemo/lens/nightly/user-guide/custom-exporters.md): 
- [Resource Detection](https://docs.nvidia.com/nemo/lens/nightly/user-guide/resource-detection.md): 
- [Contrib Helpers](https://docs.nvidia.com/nemo/lens/nightly/user-guide/contrib-helpers.md): 
- [Logging Bridge](https://docs.nvidia.com/nemo/lens/nightly/user-guide/logging-bridge.md): 
- [Troubleshooting](https://docs.nvidia.com/nemo/lens/nightly/user-guide/troubleshooting.md): 
- [Production Checklist](https://docs.nvidia.com/nemo/lens/nightly/user-guide/production-checklist.md): 
- [Architecture](https://docs.nvidia.com/nemo/lens/nightly/design/architecture.md): 
- [Optional Dependency](https://docs.nvidia.com/nemo/lens/nightly/design/optional-dependency.md): 
- [Double-Init Guard](https://docs.nvidia.com/nemo/lens/nightly/design/double-init-guard.md): 
- [Semantic Conventions](https://docs.nvidia.com/nemo/lens/nightly/design/semantic-conventions.md): 
- [Send Telemetry to a Backend](https://docs.nvidia.com/nemo/lens/nightly/backends-demo-stack/send-telemetry-to-a-backend.md): 
- [Demo Stack](https://docs.nvidia.com/nemo/lens/nightly/backends-demo-stack/demo-stack.md): 
- [Test](https://docs.nvidia.com/nemo/lens/nightly/developer-guide/test.md): 
- [Register a Span Group](https://docs.nvidia.com/nemo/lens/nightly/developer-guide/register-a-span-group.md): 
- [Contributing](https://docs.nvidia.com/nemo/lens/nightly/developer-guide/contributing.md): 
- [Build the Documentation](https://docs.nvidia.com/nemo/lens/nightly/developer-guide/build-the-documentation.md): Build, validate, and preview the NeMo Lens Fern documentation.
- [lens](https://docs.nvidia.com/nemo/lens/nightly/nemo-lens/lens.md)
- [lens.cli](https://docs.nvidia.com/nemo/lens/nightly/nemo-lens/lens/cli.md)
- [lens.config](https://docs.nvidia.com/nemo/lens/nightly/nemo-lens/lens/config.md)
- [lens.contrib](https://docs.nvidia.com/nemo/lens/nightly/nemo-lens/lens/contrib.md)
- [lens.contrib.aiohttp](https://docs.nvidia.com/nemo/lens/nightly/nemo-lens/lens/contrib/aiohttp.md)
- [lens.contrib.fastapi](https://docs.nvidia.com/nemo/lens/nightly/nemo-lens/lens/contrib/fastapi.md)
- [lens.contrib.nccl](https://docs.nvidia.com/nemo/lens/nightly/nemo-lens/lens/contrib/nccl.md)
- [lens.contrib.ray](https://docs.nvidia.com/nemo/lens/nightly/nemo-lens/lens/contrib/ray.md)
- [lens.distributed](https://docs.nvidia.com/nemo/lens/nightly/nemo-lens/lens/distributed.md)
- [lens.fallbacks](https://docs.nvidia.com/nemo/lens/nightly/nemo-lens/lens/fallbacks.md)
- [lens.groups](https://docs.nvidia.com/nemo/lens/nightly/nemo-lens/lens/groups.md)
- [lens.handle](https://docs.nvidia.com/nemo/lens/nightly/nemo-lens/lens/handle.md)
- [lens.helpers](https://docs.nvidia.com/nemo/lens/nightly/nemo-lens/lens/helpers.md)
- [lens.instruments](https://docs.nvidia.com/nemo/lens/nightly/nemo-lens/lens/instruments.md)
- [lens.instruments.gym](https://docs.nvidia.com/nemo/lens/nightly/nemo-lens/lens/instruments/gym.md)
- [lens.instruments.inference](https://docs.nvidia.com/nemo/lens/nightly/nemo-lens/lens/instruments/inference.md)
- [lens.instruments.registry](https://docs.nvidia.com/nemo/lens/nightly/nemo-lens/lens/instruments/registry.md)
- [lens.logging_bridge](https://docs.nvidia.com/nemo/lens/nightly/nemo-lens/lens/logging_bridge.md)
- [lens.package_info](https://docs.nvidia.com/nemo/lens/nightly/nemo-lens/lens/package_info.md)
- [lens.propagation](https://docs.nvidia.com/nemo/lens/nightly/nemo-lens/lens/propagation.md)
- [lens.providers](https://docs.nvidia.com/nemo/lens/nightly/nemo-lens/lens/providers.md)
- [lens.resources](https://docs.nvidia.com/nemo/lens/nightly/nemo-lens/lens/resources.md)
- [lens.resources.attributes](https://docs.nvidia.com/nemo/lens/nightly/nemo-lens/lens/resources/attributes.md)
- [lens.resources.gpu](https://docs.nvidia.com/nemo/lens/nightly/nemo-lens/lens/resources/gpu.md)
- [lens.resources.kubernetes](https://docs.nvidia.com/nemo/lens/nightly/nemo-lens/lens/resources/kubernetes.md)
- [lens.resources.local](https://docs.nvidia.com/nemo/lens/nightly/nemo-lens/lens/resources/local.md)
- [lens.resources.slurm](https://docs.nvidia.com/nemo/lens/nightly/nemo-lens/lens/resources/slurm.md)
- [lens.semconv](https://docs.nvidia.com/nemo/lens/nightly/nemo-lens/lens/semconv.md)
- [lens.semconv.attributes](https://docs.nvidia.com/nemo/lens/nightly/nemo-lens/lens/semconv/attributes.md)
- [lens.semconv.encoding](https://docs.nvidia.com/nemo/lens/nightly/nemo-lens/lens/semconv/encoding.md)
- [lens.semconv.resources](https://docs.nvidia.com/nemo/lens/nightly/nemo-lens/lens/semconv/resources.md)
- [lens.span_utilities](https://docs.nvidia.com/nemo/lens/nightly/nemo-lens/lens/span_utilities.md)
- [lens.state](https://docs.nvidia.com/nemo/lens/nightly/nemo-lens/lens/state.md)


> **Note:** This page contains both a page directory (above) and the landing page content (below). The page directory is generated for agent use and does not appear on the landing page.

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/lens/nightly/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/lens/_mcp/server.

# NeMo Lens

**NeMo Lens** (`nemo.lens`) is a shared [OpenTelemetry](https://opentelemetry.io/) instrumentation library for the NVIDIA NeMo ecosystem. It provides unified tracing, metrics, and optional log bridging across distributed training, inference, and RL workloads. NeMo Lens keeps hot-path overhead low with span-group gating, where disabled groups use a fast `frozenset` lookup and create no span objects. Consumer projects can integrate it as an optional dependency through canonical no-op fallbacks.

## Key Features

* **Low overhead when disabled**: span group gating is a `frozenset` lookup; no span objects are allocated when a group is off
* **Distributed-training aware**: caller-supplied `nv.dl.rank` / `nv.dl.world_size` resource attributes, cross-rank trace correlation
* **Framework-agnostic primitives**: `managed_span`, `trace_fn`, `span_cm` work everywhere
* **OTel idiomatic**: real `TracerProvider` / `MeterProvider`, W3C propagation, standard semconv
* **Optional dependency**: consumers import using `try/except ImportError`; NeMo Lens ships canonical no-op fallbacks
* **Pluggable**: custom exporters, custom span groups
* **Resource auto-detection**: SLURM, Kubernetes, local GPU count out of the box

## Consumers

`nemo-lens` is consumed by three NeMo-ecosystem projects, each registering its own domain-specific span groups:

| Consumer                                             | Domain                   | Span group extension                                                                                                   |
| ---------------------------------------------------- | ------------------------ | ---------------------------------------------------------------------------------------------------------------------- |
| [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) | Transformer pre-training | registers `megatron`: microbatch, layer, communication, activation_offload, data_loading, inference                    |
| [NeMo RL](https://github.com/NVIDIA-NeMo/RL)         | RLHF, GRPO, DPO          | registers `nemo_rl`: rollout, generation, logprob, reward, advantage, policy_update, reference_policy, data_processing |
| [NeMo Gym](https://github.com/NVIDIA-NeMo/Gym)       | RL environments          | registers `nemo_gym`: server, rollout_collection, verify, aggregate_metrics                                            |

NeMo Lens emits OTLP; choosing and running an observability stack is up to you. Quick links for sending telemetry to common destinations:

* [Send telemetry to a file](/nemo/lens/nightly/backends-demo-stack/send-telemetry-to-a-backend#file)
* [Send telemetry to W\&B Weave](/nemo/lens/nightly/backends-demo-stack/send-telemetry-to-a-backend#wb-weave)
* [Send telemetry to Honeycomb](/nemo/lens/nightly/backends-demo-stack/send-telemetry-to-a-backend#honeycomb)
* [Send telemetry to an OTel Collector](/nemo/lens/nightly/backends-demo-stack/send-telemetry-to-a-backend#otel-collector)

For trying things out locally, see the [demo docker-compose stack](/nemo/lens/nightly/backends-demo-stack/demo-stack) (proof of concept, not for production).