> This page is for version Nightly.
> For other versions, use one of these documentation indexes:
> - 0.1.0 (default): https://docs.nvidia.com/nemo/lens/0.1.0/llms.txt
> - Nightly: https://docs.nvidia.com/nemo/lens/nightly/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/lens/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/lens/_mcp/server.

# Troubleshooting

You enabled NeMo Lens, and something is not showing up where you expect. This page walks through the common failure modes in the order you should check them.

If you're here because of a `RuntimeError` from `setup_telemetry`, skip ahead to [the double-init section](#double-init).

## Nothing Is Being Exported

NeMo Lens never crashes your training job over a broken pipe; export failures are logged and swallowed. That is the right default, but it means you have to go looking when nothing shows up.

### Is `NEMO_LENS_ENABLED` Actually Set for the Python Process?

The env var is read inside the Python process, not by your shell. Check what Python sees:

```python
import os
print(os.environ.get("NEMO_LENS_ENABLED"))
```

Common causes of a mismatch:

* `sudo`, `srun`, or `torchrun` wrappers that strip env vars.
* Docker `ENV` vs. `docker run -e`, where one gets baked in and the other is per-container.
* A virtualenv activation script that overrides the parent shell's env.

If `os.environ` disagrees with your shell, fix the launcher, not NeMo Lens.

### Is the OTLP Endpoint Reachable?

`OTEL_EXPORTER_OTLP_ENDPOINT` defaults to `localhost:4317` when unset. If your collector is elsewhere, set it explicitly:

```bash
export OTEL_EXPORTER_OTLP_ENDPOINT=http://collector.internal:4317
```

From the training host, verify reachability with `curl -v $OTEL_EXPORTER_OTLP_ENDPOINT`. A firewall between training and collector is a common and silent failure mode.

### Is Telemetry Enabled on This Rank?

NeMo Lens exports from every process where `config.enabled` is true; it does not
elect an exporting rank. If your launcher sets `enabled` per rank, a quiet rank
is simply one where it came out false.

```python
handle = setup_telemetry(config)
print(f"rank {rank}: is_exporting={handle.is_exporting}")
```

### Wrong OTLP Protocol

NeMo Lens's exporter builder falls back between gRPC and HTTP automatically (it tries the protocol you ask for, then the other), and `nemo-lens[sdk]` installs both exporter packages, so a missing exporter package is rarely the issue. The real failure mode is a protocol/endpoint mismatch: NeMo Lens defaults to gRPC on port 4317, but your collector might only listen for HTTP on 4318 (or vice versa). Match NeMo Lens to the collector's listener:

```bash
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_ENDPOINT=http://collector.internal:4318
```

### Verify Locally First

Before debugging collector plumbing, confirm NeMo Lens itself is producing spans by routing to a file. See [Send telemetry to a file](/nemo/lens/nightly/backends-demo-stack/send-telemetry-to-a-backend#file). If you see spans in the file, the problem is downstream; if you don't, it's in NeMo Lens or your config.

## Spans Show Up in Jaeger but They Are Fragmented

You see isolated per-rank traces instead of one cross-rank waterfall. There are two common causes:

### PP Ranks Are Not Linked

For pipeline-parallel Megatron jobs, the `step` / `microbatch` span groups must be enabled, and the PP broadcast must be running. Look for `megatron.pp.recv_forward.linked` spans on non-first-stage ranks; their absence means the broadcast step didn't happen.

If every rank has telemetry enabled but the appropriate span groups are off, each process creates its own root trace, and nothing links them together.

### Incoming Context Is Not Extracted

For HTTP or gRPC services receiving traced requests (e.g., NeMo Gym servers), the request headers carry a `traceparent` that must be extracted, or the server-side span starts a fresh trace.

Use the FastAPI helper:

```python
from nemo.lens.contrib.fastapi import instrument_fastapi
instrument_fastapi(app)
```

Alternatively, you can extract the context manually. See [Context Propagation](/nemo/lens/nightly/user-guide/context-propagation).

## `setup_telemetry` Raises Already Been Initialized

```
RuntimeError: setup_telemetry() has already been initialized for this process.
```

You're calling `setup_telemetry` twice with `config.enabled=True`. This is almost always a bug, as double-init silently breaks telemetry on the OTel SDK, and NeMo Lens fails loudly instead. See [Double-Init Guard](/nemo/lens/nightly/design/double-init-guard) for the rationale.

If you have a legitimate need (multi-rank simulation in one process, tests), pass `_allow_reinit=True`:

```python
setup_telemetry(config, _allow_reinit=True)
```

The leading underscore is deliberate. If you're reaching for it in production code, step back.

## Span Group Is Not Active

You instrumented a site with `managed_span('my_group', ...)`, and nothing exports. There are two usual suspects:

### `my_group` Is Not in the Enabled Set

NeMo Lens defines no groups; the authoritative set is whatever the libraries in your process registered, unioned. Presets merge across libraries, so `default` means every registered library's idea of default. Groups like `step` and `microbatch` typically live in `per_step` or `all`. Check what actually resolved:

```python
from nemo.lens import enabled_span_groups, is_span_group_enabled
from nemo.lens.groups import SpanRegistry

print(is_span_group_enabled("my_group"))
print(sorted(enabled_span_groups()))
print(SpanRegistry.namespaces(), sorted(SpanRegistry.groups()))
```

Set `NEMO_LENS_SPAN_GROUPS` to include it:

```bash
NEMO_LENS_SPAN_GROUPS=per_step
# or
NEMO_LENS_SPAN_GROUPS=default,my_group
```

### The Name in Your Spec Was Never Registered

Resolution never raises. An entry that names nothing registered in this process
is warned about, listing the registered groups, presets, and namespaces, and the
rest of the spec still applies. Ask what did not resolve:

```python
from nemo.lens import pending_span_groups
print(sorted(pending_span_groups()))    # ['my_gropu']
```

A name in there is either a typo, or it belongs to a library this process never
imported. The second case is normal in a multi-process job: a launcher agent or a
spawned worker inherits one `NEMO_LENS_SPAN_GROUPS` from the trainer but loads a
different set of libraries. Give such a process its own env prefix
(`from_env(prefix="NVRX_OTEL", fallback_prefix="NEMO_LENS")`) rather than looking
for one spec that fits every process.

If you instead see a `WARNING` about groups "registered after
`setup_telemetry()`", that is an import-order problem caught from the other side:
the groups do work, but they were not selectable when your spec was resolved.

### Your Spec Resolved, But to Nothing

`pending_span_groups()` is empty and you still get no spans. Every name in your
spec matched a preset, but that preset is currently empty. The usual cause is
that the library owning the groups the preset borrowed is not registered, so
those members are pruned away. NeMo Lens warns that the spec "resolved to no span
groups at all" and lists what *is* registered. Either import the missing library,
or name a group directly instead of going through the preset.

A `WARNING` that no library registered any span groups means nothing
group-gated will be emitted at all — the library you expected to instrument was
never imported.

### Cannot Filter By Rank

If you cannot filter or group spans by rank in your backend, or
`service.instance.id` carries no rank, you did not pass `nv.dl.rank`. NeMo Lens
logs this at startup:

```
No nv.dl.rank resource attribute was supplied, so telemetry from this process
cannot be filtered by rank in the collector, and service.instance.id is not
rank-derived ...
```

Fix it at the `setup_telemetry` call:

```python
from nemo.lens.semconv import NV_DL_RANK, NV_DL_WORLD_SIZE

setup_telemetry(
    cfg,
    resource_attributes={NV_DL_RANK: rank, NV_DL_WORLD_SIZE: world_size},
)
```

A single-process caller can pass rank `0` to silence the warning.

### Telemetry Is Disabled in This Process

When `config.enabled` is false, `setup_telemetry` installs no-op providers and
sets the enabled group set to empty regardless of the rest of the config. Nothing
in this process will export, by design.

## Metrics Not Showing Up in Prometheus

### Give It One Export Cycle

Metrics go through `PeriodicExportingMetricReader`. The default interval is 10 seconds; wait at least one cycle after your first metric record before expecting to see anything.

### Prometheus Scrapes the Collector, Not You

The application exports OTLP. The collector converts this data into the Prometheus-format endpoint scraped by Prometheus. Confirm the collector is exposing its metrics endpoint (typically `:8889/metrics`) and that Prometheus has it in its scrape config.

### SDK-Appended Unit Suffixes

The OTel SDK appends units to metric names when it has them. A metric like `rl.reward.mean` becomes `rl_reward_mean` (dots to underscores); duration metrics ending in `_ms` (e.g., `gym.server.request_duration_ms`) might get `_milliseconds` appended. Use the Prometheus metric browser to find the exact name rather than guessing.

## Tracing Is Slow or Blocking My Training Loop

Telemetry collection is designed to be lightweight and non-blocking. If you experience performance degradation or backpressure in your training loop, check these potential bottlenecks.

### Confirm It's Telemetry, Not Something Else

Toggle NeMo Lens off (`NEMO_LENS_ENABLED=0`) and retime the workload. If speed comes back, the overhead is real and configurable; if it doesn't, look elsewhere.

### Narrow the Scope

Switch from `per_step` or `all` to `default`. Spans in the hot loop (microbatch, communication) are the likely cost. See [Span Groups](/nemo/lens/nightly/user-guide/span-groups) for how to dial this in.

### BatchSpanProcessor Should Never Block

`BatchSpanProcessor` is asynchronous and batches on a background thread. If it's blocking, the collector is probably unreachable and the internal queue is full, so span creation starts backpressuring. Fix the collector or route to a file while you debug.

## My Custom Exporter Is Not Receiving Spans

If you are using a custom exporter but it is not receiving any telemetry data, ensure it is wired correctly. Check these common setup mistakes.

### Wire It at `setup_telemetry` Time

Custom exporters are attached through the `span_exporter=` argument to `setup_telemetry`. Constructing one after `setup_telemetry` returned doesn't do anything, as the provider is already built.

```python
handle = setup_telemetry(config, span_exporter=my_exporter)
```

See [Custom Exporters](/nemo/lens/nightly/user-guide/custom-exporters).

### Check `traces_enabled`

If `config.traces_enabled=False` (or `NEMO_LENS_TRACES_ENABLED=0`), the TracerProvider is a no-op regardless of what exporter you passed in; metrics toggle independently.