> This page is for version Nightly.
> For other versions, use one of these documentation indexes:
> - 0.1.0 (default): https://docs.nvidia.com/nemo/lens/0.1.0/llms.txt
> - Nightly: https://docs.nvidia.com/nemo/lens/nightly/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/lens/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/lens/_mcp/server.

# Configuration

`NemoLensConfig` is the single configuration object consumed by `setup_telemetry`. It holds every knob exposed by the library.

## Construction

Construct a `NemoLensConfig` object using either environment variables or direct instantiation in Python.

### From Environment

```python
from nemo.lens import NemoLensConfig

cfg = NemoLensConfig.from_env()
```

Reads `NEMO_LENS_*` env vars. For library-specific prefixes, pass `prefix` and `fallback_prefix`:

```python
cfg = NemoLensConfig.from_env(
    prefix='MEGATRON_OTEL',
    fallback_prefix='NEMO_LENS',
)
```

The **prefix and fallback** pattern lets each consumer have library-scoped env vars while sharing common defaults. The primary prefix wins; the fallback applies only if the primary is unset.

### Direct Construction

```python
cfg = NemoLensConfig(
    enabled=True,
    service_name='my-training-run',
    span_groups='per_step',
)
```

## Fields

A `NemoLensConfig` object contains fields divided into several functional categories.

### Core

| Field          | Default  | Description                                              |
| -------------- | -------- | -------------------------------------------------------- |
| `enabled`      | `False`  | Master toggle. Must be `True` to activate any telemetry. |
| `service_name` | `"nemo"` | OTLP service name. Overridden by `OTEL_SERVICE_NAME`.    |

### Restricting Which Processes Export

NeMo Lens has no rank-based export filtering and no notion of a rank. Every
process that calls `setup_telemetry` with `enabled=True` exports.

To keep most ranks quiet in a large job, decide it in the caller:

```python
from nemo.lens.semconv import NV_DL_RANK, NV_DL_WORLD_SIZE

cfg = NemoLensConfig.from_env()
cfg.enabled = cfg.enabled and rank == world_size - 1   # last rank only
setup_telemetry(
    cfg,
    resource_attributes={NV_DL_RANK: rank, NV_DL_WORLD_SIZE: world_size},
)
```

Alternatively, let every rank export and filter in the collector on the `nv.dl.rank`
resource attribute, or reduce volume with the OTel SDK's own
`OTEL_TRACES_SAMPLER`. Enabling telemetry on every rank of a large job without
one of these in place multiplies your span volume by the world size.

### Signal Toggles

| Field             | Default | Description                                                                                     |
| ----------------- | ------- | ----------------------------------------------------------------------------------------------- |
| `traces_enabled`  | `True`  | Enable trace spans.                                                                             |
| `metrics_enabled` | `True`  | Enable metric instruments.                                                                      |
| `logs_enabled`    | `False` | Enable the OTel log bridge. See [Logging Bridge](/nemo/lens/nightly/user-guide/logging-bridge). |

### Granularity

| Field         | Default     | Description                                                                                                                                            |
| ------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `span_groups` | `"default"` | Comma-separated spec of preset keywords or group names, resolved against `SpanRegistry`. See [Span Groups](/nemo/lens/nightly/user-guide/span-groups). |

### Backend

| Field      | Default  | Description                                                                       |
| ---------- | -------- | --------------------------------------------------------------------------------- |
| `exporter` | `"otlp"` | `"otlp"` (gRPC, falls back to HTTP) or `"console"` (stdout, for local debugging). |

With `exporter="console"` (the env var `NEMO_LENS_EXPORTER=console`), NeMo Lens uses the `ConsoleSpanExporter`, `ConsoleMetricExporter`, and `ConsoleLogExporter` from the SDK, which print spans, metrics, and log records to stdout. NeMo Lens overrides the SDK's default formatter on all three: the SDK pretty-prints with `indent=4`, whereas NeMo Lens emits one compact JSON object per line. Redirecting stdout therefore yields real JSONL that line-oriented tooling can consume without reassembling records. Any value other than `"otlp"` or `"console"` raises a `ValueError("Unknown exporter type: ...")` when the providers are built.

### Identification

| Field    | Default | Description                                                                                    |
| -------- | ------- | ---------------------------------------------------------------------------------------------- |
| `run_id` | auto    | Unique run ID. Auto-generated from `SLURM_JOB_ID` or a UUID if empty. Shared across all ranks. |
| `user`   | `""`    | Optional user/team label. Emitted as `nemo.user.id`.                                           |

### W\&B Weave

| Field           | Default | Description                                                     |
| --------------- | ------- | --------------------------------------------------------------- |
| `wandb_entity`  | `""`    | W\&B team/user name — set as `wandb.entity` resource attribute. |
| `wandb_project` | `""`    | W\&B project name — set as `wandb.project` resource attribute.  |

## Environment Variables

Most configuration fields have a corresponding `<PREFIX>_<KEY>` env var. Three are exceptions that bypass the prefix and fallback model entirely: `service_name` reads the bare `OTEL_SERVICE_NAME`, and `wandb_entity` and `wandb_project` read the bare `WANDB_ENTITY` and `WANDB_PROJECT` (without a prefix or fallback). Note also that the `user` field's env var is `<PREFIX>_USER_ID` (e.g., `NEMO_LENS_USER_ID`), not `_USER`. Using `NEMO_LENS` as the prefix:

| Variable                    | Field             |
| --------------------------- | ----------------- |
| `NEMO_LENS_ENABLED`         | `enabled`         |
| `NEMO_LENS_TRACES_ENABLED`  | `traces_enabled`  |
| `NEMO_LENS_METRICS_ENABLED` | `metrics_enabled` |
| `NEMO_LENS_LOGS_ENABLED`    | `logs_enabled`    |
| `NEMO_LENS_SPAN_GROUPS`     | `span_groups`     |
| `NEMO_LENS_EXPORTER`        | `exporter`        |
| `NEMO_LENS_RUN_ID`          | `run_id`          |
| `NEMO_LENS_USER_ID`         | `user`            |
| `WANDB_ENTITY`              | `wandb_entity`    |
| `WANDB_PROJECT`             | `wandb_project`   |

Boolean parsing accepts: `1`/`0`, `true`/`false`, `yes`/`no`, `on`/`off` (case-insensitive). Any other value raises `ValueError`.

## Standard OTel SDK Env Vars

Standard OTel SDK env vars are honored; most are read by the SDK directly, but a few are consumed by NeMo Lens's provider-building code: `OTEL_EXPORTER_OTLP_PROTOCOL` (and the signal-specific `OTEL_EXPORTER_OTLP_{TRACES,METRICS,LOGS}_PROTOCOL`, which wins; default `grpc`) selects the OTLP transport, and `OTEL_METRIC_EXPORT_INTERVAL` (default `10000` ms) sets the metric reader's export interval. `OTEL_RESOURCE_ATTRIBUTES` is read by the SDK but consulted by NeMo Lens too, because it is the only channel that survives a spawn — see [Resource Detection](/nemo/lens/nightly/user-guide/resource-detection):

| Variable                      | Example                              |
| ----------------------------- | ------------------------------------ |
| `OTEL_SERVICE_NAME`           | `my-training-run`                    |
| `OTEL_RESOURCE_ATTRIBUTES`    | `nv.dl.rank=5,nv.dl.world_size=8`    |
| `OTEL_EXPORTER_OTLP_ENDPOINT` | `http://localhost:4317`              |
| `OTEL_EXPORTER_OTLP_PROTOCOL` | `grpc` or `http/protobuf`            |
| `OTEL_EXPORTER_OTLP_HEADERS`  | `Authorization=Bearer <token>` |
| `OTEL_TRACES_SAMPLER`         | `parentbased_traceidratio`           |
| `OTEL_TRACES_SAMPLER_ARG`     | `0.1`                                |
| `OTEL_METRIC_EXPORT_INTERVAL` | `10000` (ms)                         |
| `OTEL_SDK_DISABLED`           | `true`                               |

In addition, `build_providers` reads the non-prefixed `DEPLOYMENT_ENV` (falling back to `ENVIRONMENT`) and, when set, emits it as the `deployment.environment` resource attribute.

## `setup_telemetry` Signature

```python
setup_telemetry(
    config: NemoLensConfig,
    *,
    resource_attributes: dict | None = None,
    span_exporter=None,
    metric_reader=None,
    _allow_reinit: bool = False,
) -> TelemetryHandle
```

| Parameter             | Description                                                                                                                                                                                                                                                               |
| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `config`              | The `NemoLensConfig` object, typically from `from_env()`. The only positional parameter — everything after it is keyword-only.                                                                                                                                            |
| `resource_attributes` | Extra attributes to merge into the OTel `Resource` (become Jaeger "Process" tags). A distributed caller supplies its own position here, as `{NV_DL_RANK: rank, NV_DL_WORLD_SIZE: world_size}` — lens does not derive it, and warns at startup if `nv.dl.rank` is missing. |
| `span_exporter`       | Optional custom `SpanExporter`, bypasses config-based construction. See [Custom Exporters](/nemo/lens/nightly/user-guide/custom-exporters).                                                                                                                               |
| `metric_reader`       | Optional custom `MetricReader`, bypasses config-based construction.                                                                                                                                                                                                       |
| `_allow_reinit`       | Escape hatch for testing only, which bypasses the [double-initialization guard](/nemo/lens/nightly/design/double-init-guard).                                                                                                                                             |

Returns a `TelemetryHandle` exposing:

* `.tracer` and `.meter`: read-only properties holding the OTel tracer and meter (no-op objects when telemetry is disabled).
* `.is_exporting`: a `bool` indicating whether real exporting providers were built. Now equivalent to `config.enabled`.
* `.shutdown(timeout_ms=5000)`: force-flushes and shuts down both the tracer and meter providers.

> **Info**
>
> Call `setup_telemetry` once per process. A second call with `config.enabled=True` raises `RuntimeError`. See [Double-Init Guard](/nemo/lens/nightly/design/double-init-guard) for rationale.