Skip to navigation
User Guide

Production Checklist

View as Markdown

Transitioning from a local telemetry configuration to a deployment that spans an entire training campaign requires careful consideration. This page provides recommendations on which configuration options matter in production and how to select them deliberately rather than by accident.

If any recommendations here conflict with your site’s operational conventions, follow those guidelines.

Decide Which Processes Export

NeMo Lens exports from every process that calls setup_telemetry with enabled=True. It has no rank-based export filtering, so on a large job this decision is yours to make and worth making deliberately: enabling telemetry everywhere multiplies span volume by the world size.

Set enabled per rank in the caller:

  • A single rank’s view is representative. The simplest and usual choice. Enable on one rank (commonly the last) and leave the rest off.
  • Attribute hangs per node. Enable where LOCAL_RANK == 0 to get one exporting rank per machine. Launchers such as torchrun set LOCAL_RANK automatically.
  • Scale across fleets. Enable on a deterministic subset, for example rank % 64 == 0, when you want several vantage points without full export.
  • Investigate per-rank issues. Enable everywhere for the duration of a hang, NaN, or faulty-node investigation, then revert.
from nemo.lens.semconv import NV_DL_RANK, NV_DL_WORLD_SIZE
cfg = NemoLensConfig.from_env()
cfg.enabled = cfg.enabled and rank == world_size - 1
handle = setup_telemetry(
    cfg,
    resource_attributes={NV_DL_RANK: rank, NV_DL_WORLD_SIZE: world_size},
)

If you would rather let every rank export and decide later, filter on the nv.dl.rank resource attribute in the collector instead.

Choose a Span Group Preset

  • Quiet production telemetry. Use the default preset to emit only job, checkpoint, and evaluate spans without per-step overhead.
  • Production profiling window. Use the per_step preset combined with aggressive SDK trace sampling. For example, setting OTEL_TRACES_SAMPLER=parentbased_traceidratio with a low ratio keeps telemetry volume manageable while you gather a performance profile.
  • Active debugging. Use the all preset, but only for short-duration runs. Do not leave this preset enabled in production.

See Span Groups for complete details.

Layer Your Sampling

You can configure sampling across three composable layers:

  1. Which processes are enabled. config.enabled, decided per rank by the caller, determines which ranks emit telemetry at all. Disabled ranks are fully set to no-op.
  2. OTel SDK trace sampler. Configured through OTEL_TRACES_SAMPLER, this sampler makes a per-trace decision on whether to record the trace.
  3. Collector-side tail sampling. This optional layer allows you to make routing decisions after traces are generated, such as keeping all error traces while sampling successful ones.

Combine these layers. Layer one decides which ranks emit telemetry; layer two decides the volume those ranks produce; and layer three decides which traces are retained. Attempting to perform all sampling at a single layer either over-samples routine cases or under-samples and hides critical data.

Use nemo.run.id as the Partition Key

Every backend requires a partition key to scope queries to a single run. NeMo Lens sets the nemo.run.id attribute on every span and metric, automatically deriving the value from NEMO_LENS_RUN_ID, SLURM_JOB_ID, or a generated UUID.

Use this key in all query interfaces:

  • Jaeger. Filter by tag using nemo.run.id=<value>.
  • Grafana. Populate a dashboard variable nemo_run_id using label_values(<metric>, nemo_run_id).
  • Honeycomb. Filter by the nemo.run.id facet.
  • Datadog. Filter by the @nemo.run.id facet.
  • Kibana. Filter by the nemo.run.id field.

Set NEMO_LENS_RUN_ID explicitly when you want a human-readable identifier (such as llama3-pretrain-2026-04-22) rather than a UUID. Slurm job identifiers are also compatible, but a specific job name is easier to search for than a system-generated job identifier.

Use Resource Attributes for Run Comparison

Parallelism configuration, model architecture, precision, and cluster name belong in resource_attributes because they are stable for the process lifetime. Do not record these properties on individual spans. In Jaeger, these attributes appear as Process tags, allowing you to filter across every span and metric in the run without cluttering the individual span views.

handle = setup_telemetry(
    config,
    resource_attributes={
        "nv.dl.rank": rank,
        "nv.dl.world_size": world_size,
        "dl.tensor_parallel.size": 4,
        "dl.pipeline_parallel.size": 2,
        "dl.data_parallel.size": 8,
        "megatron.num_layers": 80,
        "megatron.precision": "bf16",
    },
)

See Resource Detection for the full picture and the dl.* versus project-scoped conventions.

Size Your Collector

The collector must keep pace with your peak trace rate. If its receive queue backs up, the SDK’s BatchSpanProcessor eventually backs up, resulting in dropped spans. Monitor the collector’s own telemetry (Prometheus metrics on :8888/metrics) and alert on processor queue saturation, such as otelcol_processor_batch_send_size, exporter send failures, and queue depth.

If a collector cannot keep pace, the honest solution is to add collector capacity or apply more aggressive sampling instead of muting alerts.

Export Only What You Need

Avoid enabling the all span group as a default precaution. Span groups allow you to enable and disable specific telemetry sites without redeploying your code; enable only the default preset in production and escalate to other groups during troubleshooting. Every additional group increases collector volume, backend cost, and noise in the trace view.

If you require the all preset permanently, this indicates a need to move some of those groups into default at the library level, rather than shipping the all preset to every production run.

Configure Logging Scope

If you use the log bridge (NEMO_LENS_LOGS_ENABLED=1), configure log level and scope deliberately. Enabling DEBUG logging across a large cluster can overwhelm any storage system. Bridge logs from specific subsystems instead of the root logger:

setup_logging_bridge(logger_name="megatron.training")

See Logging Bridge.

Configure a Backup Sink

The collector fans out with minimal overhead. A common pattern is to route one OTLP stream from the application to the collector, and then configure the collector to export to two backends (such as W&B Weave for trace visualization and Prometheus for alerting). The application exports once, while the plumbing remains in the collector configuration.

Account for Spans Truncated at Shutdown

A span that is still open when handle.shutdown() runs is force-closed so that it is exported instead of dropped. This is what makes a run-scoped span that wraps the entire job appear in your backend at all.

The end timestamp of such a span is the shutdown time, not the time the work finished. For a span that the application abandoned early in a long run, the recorded duration is wrong by however long the run continued afterward. Every force-closed span carries the nemo.span.truncated attribute set to true and an event named nemo.span.truncated.

Exclude nemo.span.truncated spans from duration percentiles and latency alerts, or chart them separately. Treat a steady stream of them as a bug in the instrumented code: a span that the application never ends is a leak, and truncation only makes the leak visible.

Spans are recovered on a graceful exit and on an uncaught exception. A process terminated by SIGKILL, or by a SIGTERM with no handler, still loses them. Install a SIGTERM handler that calls handle.shutdown() to close that gap.

Complete Pre-Live Checks

  • Call setup_telemetry exactly once at process startup.
  • Ensure handle.shutdown() runs in a finally block so that traces flush during process exit.
  • Exclude spans carrying nemo.span.truncated from duration percentiles and latency alerts.
  • Verify that nemo.run.id is configured or automatically derived and visible in your dashboards.
  • Confirm that Jaeger, Grafana, or your specific backend can filter query results by nemo.run.id.
  • Test connection reachability to the OTLP endpoint using curl from a training host.
  • Decide per rank which processes set enabled, and select an OTEL_TRACES_SAMPLER deliberately, instead of relying on default values.
  • Verify that dashboards contain panels for step duration, loss, throughput, and gradient norm.
  • Confirm a rollback plan exists (such as setting NEMO_LENS_ENABLED=0 and restarting your application without code changes).