> This page is for version Nightly.
> For other versions, use one of these documentation indexes:
> - 0.1.0 (default): https://docs.nvidia.com/nemo/lens/0.1.0/llms.txt
> - Nightly: https://docs.nvidia.com/nemo/lens/nightly/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/lens/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/lens/_mcp/server.

# Production Checklist

Transitioning from a local telemetry configuration to a deployment that spans an entire training campaign requires careful consideration. This page provides recommendations on which configuration options matter in production and how to select them deliberately rather than by accident.

If any recommendations here conflict with your site's operational conventions, follow those guidelines.

## Decide Which Processes Export

NeMo Lens exports from every process that calls `setup_telemetry` with
`enabled=True`. It has no rank-based export filtering, so on a large job this
decision is yours to make and worth making deliberately: enabling telemetry
everywhere multiplies span volume by the world size.

Set `enabled` per rank in the caller:

* **A single rank's view is representative.** The simplest and usual choice. Enable on one rank (commonly the last) and leave the rest off.
* **Attribute hangs per node.** Enable where `LOCAL_RANK == 0` to get one exporting rank per machine. Launchers such as torchrun set `LOCAL_RANK` automatically.
* **Scale across fleets.** Enable on a deterministic subset, for example `rank % 64 == 0`, when you want several vantage points without full export.
* **Investigate per-rank issues.** Enable everywhere for the duration of a hang, `NaN`, or faulty-node investigation, then revert.

```python
from nemo.lens.semconv import NV_DL_RANK, NV_DL_WORLD_SIZE

cfg = NemoLensConfig.from_env()
cfg.enabled = cfg.enabled and rank == world_size - 1
handle = setup_telemetry(
    cfg,
    resource_attributes={NV_DL_RANK: rank, NV_DL_WORLD_SIZE: world_size},
)
```

If you would rather let every rank export and decide later, filter on the
`nv.dl.rank` resource attribute in the collector instead.

## Choose a Span Group Preset

* **Quiet production telemetry.** Use the `default` preset to emit only `job`, `checkpoint`, and `evaluate` spans without per-step overhead.
* **Production profiling window.** Use the `per_step` preset combined with aggressive SDK trace sampling. For example, setting `OTEL_TRACES_SAMPLER=parentbased_traceidratio` with a low ratio keeps telemetry volume manageable while you gather a performance profile.
* **Active debugging.** Use the `all` preset, but only for short-duration runs. Do not leave this preset enabled in production.

See [Span Groups](/nemo/lens/nightly/user-guide/span-groups) for complete details.

## Layer Your Sampling

You can configure sampling across three composable layers:

1. **Which processes are enabled.** `config.enabled`, decided per rank by the caller, determines which ranks emit telemetry at all. Disabled ranks are fully set to no-op.
2. **OTel SDK trace sampler.** Configured through `OTEL_TRACES_SAMPLER`, this sampler makes a per-trace decision on whether to record the trace.
3. **Collector-side tail sampling.** This optional layer allows you to make routing decisions after traces are generated, such as keeping all error traces while sampling successful ones.

Combine these layers. Layer one decides which ranks emit telemetry; layer two decides the volume those ranks produce; and layer three decides which traces are retained. Attempting to perform all sampling at a single layer either over-samples routine cases or under-samples and hides critical data.

## Use `nemo.run.id` as the Partition Key

Every backend requires a partition key to scope queries to a single run. NeMo Lens sets the `nemo.run.id` attribute on every span and metric, automatically deriving the value from `NEMO_LENS_RUN_ID`, `SLURM_JOB_ID`, or a generated UUID.

Use this key in all query interfaces:

* **Jaeger.** Filter by tag using `nemo.run.id=<value>`.
* **Grafana.** Populate a dashboard variable `nemo_run_id` using `label_values(<metric>, nemo_run_id)`.
* **Honeycomb.** Filter by the `nemo.run.id` facet.
* **Datadog.** Filter by the `@nemo.run.id` facet.
* **Kibana.** Filter by the `nemo.run.id` field.

Set `NEMO_LENS_RUN_ID` explicitly when you want a human-readable identifier (such as `llama3-pretrain-2026-04-22`) rather than a UUID. Slurm job identifiers are also compatible, but a specific job name is easier to search for than a system-generated job identifier.

## Use Resource Attributes for Run Comparison

Parallelism configuration, model architecture, precision, and cluster name belong in `resource_attributes` because they are stable for the process lifetime. Do not record these properties on individual spans. In Jaeger, these attributes appear as **Process** tags, allowing you to filter across every span and metric in the run without cluttering the individual span views.

```python
handle = setup_telemetry(
    config,
    resource_attributes={
        "nv.dl.rank": rank,
        "nv.dl.world_size": world_size,
        "dl.tensor_parallel.size": 4,
        "dl.pipeline_parallel.size": 2,
        "dl.data_parallel.size": 8,
        "megatron.num_layers": 80,
        "megatron.precision": "bf16",
    },
)
```

See [Resource Detection](/nemo/lens/nightly/user-guide/resource-detection) for the full picture and the `dl.*` versus project-scoped conventions.

## Size Your Collector

The collector must keep pace with your peak trace rate. If its receive queue backs up, the SDK's `BatchSpanProcessor` eventually backs up, resulting in dropped spans. Monitor the collector's own telemetry (Prometheus metrics on `:8888/metrics`) and alert on processor queue saturation, such as `otelcol_processor_batch_send_size`, exporter send failures, and queue depth.

If a collector cannot keep pace, the honest solution is to add collector capacity or apply more aggressive sampling instead of muting alerts.

## Export Only What You Need

Avoid enabling the `all` span group as a default precaution. Span groups allow you to enable and disable specific telemetry sites without redeploying your code; enable only the `default` preset in production and escalate to other groups during troubleshooting. Every additional group increases collector volume, backend cost, and noise in the trace view.

If you require the `all` preset permanently, this indicates a need to move some of those groups into `default` at the library level, rather than shipping the `all` preset to every production run.

## Configure Logging Scope

If you use the log bridge (`NEMO_LENS_LOGS_ENABLED=1`), configure log level and scope deliberately. Enabling `DEBUG` logging across a large cluster can overwhelm any storage system. Bridge logs from specific subsystems instead of the root logger:

```python
setup_logging_bridge(logger_name="megatron.training")
```

See [Logging Bridge](/nemo/lens/nightly/user-guide/logging-bridge).

## Configure a Backup Sink

The collector fans out with minimal overhead. A common pattern is to route one OTLP stream from the application to the collector, and then configure the collector to export to two backends (such as W\&B Weave for trace visualization and Prometheus for alerting). The application exports once, while the plumbing remains in the collector configuration.

## Account for Spans Truncated at Shutdown

A span that is still open when `handle.shutdown()` runs is force-closed so that it is exported instead of dropped. This is what makes a run-scoped span that wraps the entire job appear in your backend at all.

The end timestamp of such a span is the shutdown time, not the time the work finished. For a span that the application abandoned early in a long run, the recorded duration is wrong by however long the run continued afterward. Every force-closed span carries the `nemo.span.truncated` attribute set to `true` and an event named `nemo.span.truncated`.

Exclude `nemo.span.truncated` spans from duration percentiles and latency alerts, or chart them separately. Treat a steady stream of them as a bug in the instrumented code: a span that the application never ends is a leak, and truncation only makes the leak visible.

Spans are recovered on a graceful exit and on an uncaught exception. A process terminated by `SIGKILL`, or by a `SIGTERM` with no handler, still loses them. Install a `SIGTERM` handler that calls `handle.shutdown()` to close that gap.

## Complete Pre-Live Checks

* [ ] Call `setup_telemetry` exactly once at process startup.
* [ ] Ensure `handle.shutdown()` runs in a `finally` block so that traces flush during process exit.
* [ ] Exclude spans carrying `nemo.span.truncated` from duration percentiles and latency alerts.
* [ ] Verify that `nemo.run.id` is configured or automatically derived and visible in your dashboards.
* [ ] Confirm that Jaeger, Grafana, or your specific backend can filter query results by `nemo.run.id`.
* [ ] Test connection reachability to the OTLP endpoint using `curl` from a training host.
* [ ] Decide per rank which processes set `enabled`, and select an `OTEL_TRACES_SAMPLER` deliberately, instead of relying on default values.
* [ ] Verify that dashboards contain panels for step duration, loss, throughput, and gradient norm.
* [ ] Confirm a rollback plan exists (such as setting `NEMO_LENS_ENABLED=0` and restarting your application without code changes).