Skip to navigation
Get Started

Overview

View as Markdown

NeMo Lens solves a narrow but important problem: giving the NeMo ecosystem training and inference workloads a shared, idiomatic OpenTelemetry instrumentation layer (cheap when disabled and ergonomic when enabled) that consumers can opt into without taking on a hard dependency.

What It Is

NeMo Lens is a thin, well-tested library built on OpenTelemetry (API required, SDK optional) that provides:

  • Unified configuration. A configuration object (NemoLensConfig) with prefix and fallback environment variable support.
  • Instrumentation primitives. Three hot-path primitives designed for performance: managed_span, trace_fn, and span_cm.
  • Span-group gating. Granularity control that is coarse for production and fine-grained for debugging.
  • Rank-agnostic export. Every process where config.enabled is true exports; the caller supplies its own nv.dl.rank / nv.dl.world_size and decides which ranks report.
  • Trace context propagation. Cross-rank trace context broadcast and span linking for pipeline-parallel correlation.
  • Resource auto-detection. Automatic environment detection for SLURM, Kubernetes, and local environments.
  • Framework integration. Contrib modules for FastAPI, aiohttp, Ray, and NCCL.

What It Is Not

NeMo Lens maintains clear operational boundaries and does not duplicate functionality provided by the core OpenTelemetry framework or backend storage platforms.

  • Not a tracer. The OpenTelemetry SDK does the tracing. NeMo Lens configures it and provides ergonomic primitives.
  • Not an observability solution. NeMo Lens emits OTLP and stops at that boundary. Choosing, running, securing, scaling, and retaining a backend is your decision, which is driven by your organization’s existing observability stack and the scale of your workloads. The docker-compose.otel.yml file shipped with the repository is a demo and proof of concept to help you try NeMo Lens locally, not a recommended production deployment.
  • Not a backend. Spans and metrics export through standard OTLP to any compliant backend (Jaeger, Grafana Tempo, Honeycomb, Datadog, W&B Weave, etc.).
  • Not a requirement. Consumers integrate by using try/except ImportError. NeMo Lens ships canonical no-op fallbacks so instrumented code runs unchanged when NeMo Lens is not installed.

Architectural Principles

NeMo Lens is designed around several key architectural principles to ensure high performance and minimal overhead during training and inference.

Cheap When Disabled

Every managed_span or trace_fn call checks is_span_group_enabled(group) before doing any real work. The check is a frozenset lookup. When the group is disabled, managed_span yields None and the body executes without creating any span objects.

Lazy SDK Imports

opentelemetry-api is the only required dependency (it ships a no-op implementation). The full SDK (opentelemetry-sdk and OTLP exporters) is imported only when telemetry is enabled; a process that leaves it off never pays the import cost.

Single Entry Point

setup_telemetry(config) is the only initialization call. It builds the correct providers (the real SDK or no-op) based on config.enabled, registers enabled span groups, and returns a TelemetryHandle. It takes no rank: a distributed caller passes its position as resource attributes.

Rank-Agnostic

NeMo Lens has no notion of a rank. It exports from every process where config.enabled is true, and a distributed caller passes its own position as resource attributes (nv.dl.rank, nv.dl.world_size). Deciding which ranks report is the caller’s job — set enabled per rank, or let every rank export and filter on nv.dl.rank in the collector. See Production Checklist.

Next Steps