> This page is for version Nightly.
> For other versions, use one of these documentation indexes:
> - 0.1.0 (default): https://docs.nvidia.com/nemo/lens/0.1.0/llms.txt
> - Nightly: https://docs.nvidia.com/nemo/lens/nightly/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/lens/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/lens/_mcp/server.

# Overview

NeMo Lens solves a narrow but important problem: giving the NeMo ecosystem training and inference workloads a **shared, idiomatic OpenTelemetry instrumentation layer** (cheap when disabled and ergonomic when enabled) that consumers can opt into without taking on a hard dependency.

## What It Is

NeMo Lens is a thin, well-tested library built on OpenTelemetry (API required, SDK optional) that provides:

* **Unified configuration.** A configuration object (`NemoLensConfig`) with prefix and fallback environment variable support.
* **Instrumentation primitives.** Three hot-path primitives designed for performance: `managed_span`, `trace_fn`, and `span_cm`.
* **Span-group gating.** Granularity control that is coarse for production and fine-grained for debugging.
* **Rank-agnostic export.** Every process where `config.enabled` is true exports; the caller supplies its own `nv.dl.rank` / `nv.dl.world_size` and decides which ranks report.
* **Trace context propagation.** Cross-rank trace context broadcast and span linking for pipeline-parallel correlation.
* **Resource auto-detection.** Automatic environment detection for SLURM, Kubernetes, and local environments.
* **Framework integration.** Contrib modules for FastAPI, aiohttp, Ray, and NCCL.

## What It Is Not

NeMo Lens maintains clear operational boundaries and does not duplicate functionality provided by the core OpenTelemetry framework or backend storage platforms.

* **Not a tracer.** The OpenTelemetry SDK does the tracing. NeMo Lens configures it and provides ergonomic primitives.
* **Not an observability solution.** NeMo Lens emits OTLP and stops at that boundary. Choosing, running, securing, scaling, and retaining a backend is your decision, which is driven by your organization's existing observability stack and the scale of your workloads. The `docker-compose.otel.yml` file shipped with the repository is a demo and proof of concept to help you try NeMo Lens locally, not a recommended production deployment.
* **Not a backend.** Spans and metrics export through standard OTLP to any compliant backend (Jaeger, Grafana Tempo, Honeycomb, Datadog, W\&B Weave, etc.).
* **Not a requirement.** Consumers integrate by using `try/except ImportError`. NeMo Lens ships canonical no-op fallbacks so instrumented code runs unchanged when NeMo Lens is not installed.

## Architectural Principles

NeMo Lens is designed around several key architectural principles to ensure high performance and minimal overhead during training and inference.

### Cheap When Disabled

Every `managed_span` or `trace_fn` call checks `is_span_group_enabled(group)` before doing any real work. The check is a `frozenset` lookup. When the group is disabled, `managed_span` yields `None` and the body executes without creating any span objects.

### Lazy SDK Imports

`opentelemetry-api` is the only required dependency (it ships a no-op implementation). The full SDK (`opentelemetry-sdk` and OTLP exporters) is imported only when telemetry is **enabled**; a process that leaves it off never pays the import cost.

### Single Entry Point

`setup_telemetry(config)` is the only initialization call. It builds the correct providers (the real SDK or no-op) based on `config.enabled`, registers enabled span groups, and returns a `TelemetryHandle`. It takes no rank: a distributed caller passes its position as resource attributes.

### Rank-Agnostic

NeMo Lens has no notion of a rank. It exports from every process where
`config.enabled` is true, and a distributed caller passes its own position as
resource attributes (`nv.dl.rank`, `nv.dl.world_size`). Deciding which ranks report is
the caller's job — set `enabled` per rank, or let every rank export and filter on
`nv.dl.rank` in the collector. See [Production Checklist](/nemo/lens/nightly/user-guide/production-checklist).

## Next Steps

* [Install NeMo Lens](/nemo/lens/nightly/get-started/installation)
* Follow the [Quickstart](/nemo/lens/nightly/get-started/quickstart) guide to instrument a minimal script
* Read the [User Guide](/nemo/lens/nightly/user-guide/configuration) for details on each feature