> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/lens/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/lens/_mcp/server. # Send Telemetry to a Backend **NeMo Lens does not provide or recommend an observability solution.** It emits OTLP. Where that OTLP goes, how it is stored, how it is queried, and how it is visualized are your decisions, which are shaped by your organization's existing observability investments and the scale of your workloads. This page shows how to configure NeMo Lens for common destinations. NeMo Lens exports through standard OTLP, so any OTLP-compatible backend works without code changes. Four destinations are covered in depth: * [File](#export-telemetry-to-a-file): local trace and metric capture for offline analysis or archival * [W\&B Weave](#integrate-with-wb-weave): Weights & Biases' trace UI, co-located with training run metadata * [Honeycomb](#send-telemetry-to-honeycomb): hosted APM that accepts all three signals on one OTLP endpoint * [OTel Collector](#configure-otel-collector): a routing and aggregation layer in front of other backends This guide also provides a quick reference for other hosted backends. --- ## Export Telemetry to a File Writing traces to a local file is useful for offline analysis, CI captures, and archival. Choose one of the following four approaches depending on your requirements. ### Use Console Exporter with Shell Redirect (Simplest) ```bash export NEMO_LENS_ENABLED=1 export NEMO_LENS_EXPORTER=console python train.py > traces.jsonl 2>&1 ``` Setting `NEMO_LENS_EXPORTER=console` installs the `ConsoleSpanExporter`, which writes one JSON line per span to stdout. Redirect stdout to a file, and you have a span log. **Drawbacks:** * Mixes application stdout with span data; separate them with selective logging to stderr. * Does not capture metrics (the metric exporter writes a different format). ### Point Custom ConsoleSpanExporter to a File Handle ```python from pathlib import Path from opentelemetry.sdk.trace.export import ConsoleSpanExporter from nemo.lens import NemoLensConfig, setup_telemetry trace_file = Path("traces.jsonl").open("a", buffering=1) # line-buffered exporter = ConsoleSpanExporter(out=trace_file) config = NemoLensConfig(enabled=True, exporter="console") # falls back if custom skipped handle = setup_telemetry( config, rank=0, world_size=1, span_exporter=exporter, ) try: # ... your workload ... finally: handle.shutdown() trace_file.close() ``` The `ConsoleSpanExporter` accepts any file-like object using `out=`. This separates trace data from application stdout without a shell redirect. **Caveats:** * Line-buffer the output using `buffering=1` so that lines are not lost if the process crashes. * Close the file after calling `handle.shutdown()`. * Each line is a Python `repr` representation of the span, not strict JSON. For strict JSON, write a custom exporter as described in the next approach. ### Implement Custom SpanExporter for Full Control For structured JSON, compression, rotation, or any custom format: ```python import json from opentelemetry.sdk.trace.export import SpanExporter, SpanExportResult class JSONLFileSpanExporter(SpanExporter): def __init__(self, path: str): self._fh = open(path, "a", buffering=1) def export(self, spans) -> SpanExportResult: try: for span in spans: record = { "name": span.name, "trace_id": f"{span.context.trace_id:032x}", "span_id": f"{span.context.span_id:016x}", "parent_id": ( f"{span.parent.span_id:016x}" if span.parent else None ), "start_time_unix_nano": span.start_time, "end_time_unix_nano": span.end_time, "attributes": dict(span.attributes or {}), "status": str(span.status.status_code), } self._fh.write(json.dumps(record) + "\n") return SpanExportResult.SUCCESS except Exception: return SpanExportResult.FAILURE def shutdown(self) -> None: self._fh.close() def force_flush(self, timeout_millis: int = 30000) -> bool: self._fh.flush() return True handle = setup_telemetry( config, span_exporter=JSONLFileSpanExporter("traces.jsonl"), ) ``` This approach provides strict JSONL that is trivial to query using `jq`. Extend this implementation with gzip compression, rotation, or remote write as needed. ### Use OTel Collector File Exporter If you are already running an OTel Collector, add a `file` exporter to its pipeline: ```yaml # otel-collector.yaml exporters: file/traces: path: /var/log/otel/traces.jsonl rotation: max_megabytes: 100 max_days: 7 service: pipelines: traces: receivers: [otlp] processors: [batch] exporters: [file/traces, jaeger] # fan out to both ``` The application still exports OTLP as normal, and the Collector handles file writes, rotation, and retention. Use this approach when you want a single file with spans from multiple ranks or multiple services. ### Export Metrics to a File For metrics, use a `PeriodicExportingMetricReader` with a `ConsoleMetricExporter`: ```python from opentelemetry.sdk.metrics.export import ConsoleMetricExporter, PeriodicExportingMetricReader metric_file = open("metrics.jsonl", "a", buffering=1) reader = PeriodicExportingMetricReader( ConsoleMetricExporter(out=metric_file), export_interval_millis=10000, ) handle = setup_telemetry(config, metric_reader=reader) ``` Alternatively, use the file exporter of the Collector in the metrics pipeline using the same pattern as traces. --- ## Integrate with W\&B Weave [Weave](https://wandb.ai/site/weave) is the Weights & Biases trace visualization tool. It ingests OTLP spans and renders them in the same UI as your training runs, so that traces and training metrics live together. ### Configure the Integration Configure the integration by choosing one of the two patterns below depending on whether you are using a collector. #### Configure Pattern A for Direct Export from the Application Use this pattern when you want to export telemetry directly to Weights & Biases without running a local OTel Collector. ```bash # Required: W&B identification (set as resource attributes) export WANDB_ENTITY=my-team # or your personal entity export WANDB_PROJECT=megatron-training # Required: traces-signal-specific endpoint + auth header export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://trace.wandb.ai/otel/v1/traces export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf export OTEL_EXPORTER_OTLP_TRACES_HEADERS="wandb-api-key=$WANDB_API_KEY" # Weave ingests traces only; disable the other two signals. export MEGATRON_OTEL_METRICS_ENABLED=0 export NEMO_LENS_LOGS_ENABLED=0 # Required: lens activation export NEMO_LENS_ENABLED=1 ``` A ready-to-run compose file for this pattern is at `docker-compose.weave.yml`, which brings up only the Megatron container and points traces straight to Weave. Use this file when you do not want to run the local stack. #### Configure Pattern B for Export Through an OTel Collector Use this pattern when you want batching, filtering, or multi-backend fan-out. See `observability/otel-collector-weave.yaml` in the repository for a ready-to-run example, toggled through the `--config=/etc/otel/collector-weave.yaml` mode of the `docker-compose.otel.yml` file. ### Notes on the Direct Path * W\&B Weave currently ingests **traces only** (as of early 2026). Metrics still require a separate sink, such as Prometheus, an OTel Collector, or native `wandb.log()`. * Use the `OTEL_EXPORTER_OTLP_TRACES_*` variants instead of the signal-agnostic `OTEL_EXPORTER_OTLP_*`. The Weave URL is a full path ending in `/v1/traces`. Signal-specific environment variables are treated as full URLs, whereas the generic variant appends `/v1/traces` automatically; setting both would produce `/v1/traces/v1/traces` and a 404 error. * NeMo Lens honors `OTEL_EXPORTER_OTLP_PROTOCOL` and signal-specific variants, so `http/protobuf` routes to the HTTP exporter class because Weave is HTTP-only. * `NemoLensConfig.from_env()` reads `WANDB_ENTITY` and `WANDB_PROJECT` directly and sets them as `wandb.entity` and `wandb.project` resource attributes on every span, which is required for Weave to route correctly. ### Run the Application ```bash python pretrain_gpt.py ... ``` Traces appear in the Weave tab of your W\&B run within a few seconds. The trace tree mirrors Jaeger's structure: a `megatron.train_step` root span with child spans for `forward_backward`, `optimizer`, and other tasks. ### Link Traces to Runs Because `WANDB_ENTITY` and `WANDB_PROJECT` are set as span attributes, Weave automatically associates traces with the right W\&B run. The `nemo.run.id` resource attribute (auto-generated or from `SLURM_JOB_ID`) serves as a unique run identifier you can filter on in the Weave UI. ### Configure Sampling for Cost Management W\&B bills by ingested trace volume. For long `per_step` runs, sample aggressively: ```bash export OTEL_TRACES_SAMPLER=parentbased_traceidratio export OTEL_TRACES_SAMPLER_ARG=0.01 # keep 1% of traces ``` See the [sampling](/nemo/lens/user-guide/sampling) documentation for how this composes with NeMo Lens's export strategies. ### Understand Exported Data Everything the SDK exports, including span names, attributes, events, links, and status. Weave renders: * Attribute key-value pairs * Error events through `span.record_exception` * OTel links as clickable references, which are useful for pipeline-parallel correlation --- ## Send Telemetry to Honeycomb [Honeycomb](https://honeycomb.io) is a hosted APM that ingests OpenTelemetry data natively. Unlike W\&B Weave, it accepts all three signals (traces, metrics, and logs) on a single OTLP endpoint. This is a good fit if you want one hosted destination for everything and already have (or are happy to adopt) Honeycomb's query model. ### Configure Honeycomb Configure the integration with Honeycomb by choosing one of the following two patterns. #### Configure Pattern A for Direct Export from the Application ```bash # One endpoint covers all three signals. export OTEL_EXPORTER_OTLP_ENDPOINT=https://api.honeycomb.io:443 export OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=${HONEYCOMB_API_KEY},x-honeycomb-dataset=${HONEYCOMB_DATASET}" export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf export NEMO_LENS_ENABLED=1 ``` * **`x-honeycomb-team`**: Your ingest API key. In the Honeycomb UI, select **Environment Settings**, and then select **API Keys** to find this key. * **`x-honeycomb-dataset`**: The dataset name. This name is required for metrics, and it is optional but recommended for traces and logs. Choose any meaningful name; Honeycomb automatically creates the dataset on the first write. * **`OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf`**: Honeycomb supports both gRPC and HTTP, but HTTP is more forgiving behind load balancers. Default to HTTP unless you have a reason otherwise. NeMo Lens honors `OTEL_EXPORTER_OTLP_PROTOCOL` (and the signal-specific variants `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL` and `OTEL_EXPORTER_OTLP_METRICS_PROTOCOL`) when picking between gRPC and HTTP exporters, so this works without code changes. For EU instance, substitute `https://api.eu1.honeycomb.io:443`. A ready-to-run compose file for this pattern is at `docker-compose.honeycomb.yml`, which brings up only the Megatron container and points it straight to Honeycomb. Use this file when you do not want to run the local stack. #### Configure Pattern B for Export Through an OTel Collector Use this pattern when you want batching, filtering, or multi-backend fan-out between the application and Honeycomb. See [collector-honeycomb.yaml](https://github.com/NVIDIA-NeMo/Lens/blob/main/observability/otel-collector-honeycomb.yaml) in the repository for a ready-to-run example: ```yaml exporters: otlphttp/honeycomb: endpoint: https://api.honeycomb.io:443 headers: x-honeycomb-team: ${env:HONEYCOMB_API_KEY} x-honeycomb-dataset: ${env:HONEYCOMB_DATASET} service: pipelines: traces: { receivers: [otlp], processors: [batch], exporters: [otlphttp/honeycomb] } metrics: { receivers: [otlp], processors: [batch], exporters: [otlphttp/honeycomb] } logs: { receivers: [otlp], processors: [batch], exporters: [otlphttp/honeycomb] } ``` The application then points at your collector (`OTEL_EXPORTER_OTLP_ENDPOINT=http://collector:4317`), and the collector handles Honeycomb auth and routing. The repo's `docker-compose.otel.yml` has a one-line toggle for this: uncomment `--config=/etc/otel/collector-honeycomb.yaml` and set `HONEYCOMB_API_KEY` and `HONEYCOMB_DATASET` in `.env`. ### Compare Classic and Current Honeycomb Accounts Honeycomb migrated from dataset-per-service (Classic) to environment-based organization. If you are on a Classic account, the `x-honeycomb-dataset` header is required for every signal, and the dataset field has specific semantics. For current Honeycomb it is still required for metrics and optional but recommended for traces and logs. If you are unsure which account type you have, your account page displays this information. ### Manage Data Volume through Sampling Honeycomb bills on event volume. A `per_step` Megatron run on many ranks will ship a lot of events. Layer your sampling: 1. NeMo Lens `export_strategy` at the rank level (start with `single_rank`). 2. OTel SDK `OTEL_TRACES_SAMPLER=parentbased_traceidratio` at the trace level. 3. Honeycomb **Refinery** tail sampling, which provides access to the full trace before making a decision. This is recommended for production; see [Honeycomb's Refinery docs](https://docs.honeycomb.io/manage-data-volume/refinery/). ### Understand Exported Honeycomb Data Every attribute, event, and link the SDK exports. Honeycomb's UI is especially good at high-cardinality attribute queries (`BubbleUp`, `HEATMAP`, etc.), so set span attributes liberally; attribute cardinality is what Honeycomb is best at. --- ## Configure OTel Collector The OpenTelemetry Collector is a common intermediary between your application and your observability backends. Running a Collector instead of exporting directly from the SDK can provide several benefits: * **Fan-out capabilities.** Send the same telemetry to multiple backends, such as Jaeger, Prometheus, and an S3 archive. * **Sampling and filtering.** Drop spans at the Collector instead of within each SDK instance. * **Batching and resilience.** Buffer during network outages without losing data. * **Transforms.** Rename attributes, redact PII, or enrich with external metadata. * **Centralized configuration.** Change backends without restarting training jobs. ### Set Minimum Configuration ```yaml # otel-collector.yaml receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318 processors: batch: timeout: 5s send_batch_size: 1024 exporters: # Traces to Jaeger otlp/jaeger: endpoint: jaeger:4317 tls: insecure: true # Metrics to Prometheus (pull model) prometheus: endpoint: 0.0.0.0:8889 # Optional: file archival file: path: /var/log/otel/telemetry.jsonl service: pipelines: traces: receivers: [otlp] processors: [batch] exporters: [otlp/jaeger, file] metrics: receivers: [otlp] processors: [batch] exporters: [prometheus] ``` ### Run the Collector Docker (simplest): ```bash docker run --rm \ -p 4317:4317 -p 4318:4318 -p 8889:8889 \ -v $(pwd)/otel-collector.yaml:/etc/otel-collector.yaml \ otel/opentelemetry-collector-contrib:latest \ --config=/etc/otel-collector.yaml ``` This is a generic standalone example. The repository's bundled configurations live under `observability/` and are mounted at `/etc/otel/collector*.yaml` by `docker-compose.otel.yml` (selected using the `--config` line); adjust the path if you copy from there. On a cluster, deploy as a sidecar, DaemonSet, or shared service. Typical patterns: * **Sidecar deployment.** Deploy one Collector per application pod. This provides low latency and an isolated failure domain. * **DaemonSet deployment.** Deploy one Collector per host, where every local application exports to it. This is a good fit for Kubernetes. * **Shared service deployment.** Deploy one fleet of Collectors behind a load balancer. This is the most cost-effective option but adds a network hop. ### Configure the Application ```bash export NEMO_LENS_ENABLED=1 export OTEL_EXPORTER_OTLP_ENDPOINT=http://collector.internal:4317 ``` That's it. Lens discovers the endpoint from the standard env var; no code changes. ### Apply Useful Processors Beyond `batch`, consider: ```yaml processors: # Drop noisy spans at the Collector filter/drop_health_checks: traces: span: - 'name == "GET /healthz"' # Sample smart: keep errors, sample 10% of successes tail_sampling: decision_wait: 10s policies: - name: keep-errors type: status_code status_code: { status_codes: [ERROR] } - name: sample-successes type: probabilistic probabilistic: { sampling_percentage: 10 } # Enrich with cluster metadata resource: attributes: - key: cluster.name value: prod-us-west action: upsert # Redact sensitive attributes attributes/redact: actions: - key: user.email action: delete ``` Attach to a pipeline: ```yaml service: pipelines: traces: receivers: [otlp] processors: [filter/drop_health_checks, tail_sampling, resource, batch] exporters: [otlp/jaeger, file] ``` ### Configure Multi-Backend Routing Send traces to two places simultaneously, such as Jaeger for interactive debugging and W\&B Weave for run history: ```yaml exporters: otlp/jaeger: endpoint: jaeger:4317 tls: { insecure: true } otlphttp/wandb: endpoint: https://trace.wandb.ai/otel/v1/traces headers: wandb-api-key: ${env:WANDB_API_KEY} service: pipelines: traces: receivers: [otlp] processors: [batch] exporters: [otlp/jaeger, otlphttp/wandb] ``` The application exports to one endpoint (the Collector), and the Collector fans out the telemetry. ### Configure Collector-Side Sampling Instead of sampling at the SDK through `OTEL_TRACES_SAMPLER`, sample at the Collector. The advantage of this approach is that you can make the decision based on the complete trace (for example, keep all traces containing an error), which the SDK cannot do because it has not seen the whole trace yet. The `tail_sampling` processor is the standard tool. See the full [tail sampling documentation](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor). ### Production Considerations * **Backpressure**. If a backend is slow, the Collector buffers. Configure `sending_queue` limits to cap memory. * **TLS**. Enable TLS between the SDK and the Collector, and between the Collector and backends, in any multi-tenant setup. * **Health checks**. Enable the `health_check` extension (as the bundled configurations do) to expose `:13133/` and monitor it. * **Version pinning**. The `opentelemetry-collector-contrib` image changes, so pin to a version and upgrade deliberately. ### Debug the Collector ```bash # Increase logging --set service.telemetry.logs.level=debug # Enable the debug exporter to see spans on stdout exporters: debug: verbosity: detailed # Add to a pipeline for testing service: pipelines: traces: exporters: [debug] ``` The Collector's own telemetry (`:8888/metrics`) shows incoming span rates, processor queue depth, and exporter success counts; scrape it with Prometheus to monitor the monitoring. --- ## Send Telemetry to Other Hosted Backends Configure other popular hosted backends by using standard OpenTelemetry environment variables. ### Configure Grafana Cloud ```bash export OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-gateway-.grafana.net/otlp export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic " ``` This routes traces to Tempo, metrics to Mimir, and logs to Loki, all queryable from a unified Grafana UI. ### Configure Datadog ```bash export OTEL_EXPORTER_OTLP_ENDPOINT=https://trace.agent.datadoghq.com export OTEL_EXPORTER_OTLP_HEADERS="DD-API-KEY=" ``` Datadog also ships their own Collector preset; see their documentation for advanced configuration. ### Configure New Relic ```bash export OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.nr-data.net export OTEL_EXPORTER_OTLP_HEADERS="api-key=" ``` ### Configure Self-Hosted Jaeger or Tempo Both accept OTLP natively: ```bash export OTEL_EXPORTER_OTLP_ENDPOINT=http://jaeger.internal:4317 # or tempo.internal ``` --- ## Select a Backend | Factor | Prefer | | -------------------------------------------- | --------------------------------------------------------------------------------------------------------- | | Quick local iteration | [Console or File](#export-telemetry-to-a-file) | | Small team, no infrastructure team | Hosted ([W\&B Weave](#integrate-with-wb-weave), [Honeycomb](#send-telemetry-to-honeycomb), Grafana Cloud) | | Training runs tied to W\&B | [W\&B Weave](#integrate-with-wb-weave) | | One hosted destination for all three signals | [Honeycomb](#send-telemetry-to-honeycomb) | | High-cardinality attribute queries matter | [Honeycomb](#send-telemetry-to-honeycomb) | | Production with multi-backend routing | [OTel Collector](#configure-otel-collector) and your chosen backends | | Data residency and compliance | Self-hosted Collector and self-hosted backends | | Already have one APM vendor | Their OTLP endpoint | All destinations work the same from NeMo Lens's perspective; the choice is about cost, operational burden, and integration with your existing stack. ## Partition Telemetry by Run Regardless of the backend, filter by `nemo.run.id` (auto-set by NeMo Lens) to isolate a specific training run's data: * **Jaeger**: Use the tag filter `nemo.run.id=<value>`. * **Grafana**: Use the dashboard variable `nemo_run_id`. * **Honeycomb**: Use the filter `nemo.run.id`. * **Datadog**: Use the facet `@nemo.run.id`. * **Weave**: Use the run-level association through `WANDB_ENTITY` and `WANDB_PROJECT`. Multiple runs land in the same index or project; the attribute is the partition key. ## gRPC vs HTTP OTLP has two transport variants: ```bash # gRPC (default, faster, persistent connections) OTEL_EXPORTER_OTLP_PROTOCOL=grpc OTEL_EXPORTER_OTLP_ENDPOINT=http://collector:4317 # HTTP/Protobuf (firewall-friendly, HTTPS works easily) OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf OTEL_EXPORTER_OTLP_ENDPOINT=http://collector:4318 ``` The NeMo Lens `providers.py` tries gRPC first, and falls back to HTTP if the gRPC exporter is not installed. If you only installed `opentelemetry-exporter-otlp-proto-http`, set the protocol explicitly. ## Limit Ingested Spans A `per_step` run on 1,000 ranks can produce over 100,000 spans per second. Most backends cannot (or will not affordably) ingest that volume. Layer your sampling: 1. NeMo Lens `export_strategy` at the rank level, where `single_rank` sends one rank's data. This is usually the correct starting point. 2. The OTel SDK sampler at the trace level, where setting `OTEL_TRACES_SAMPLER=parentbased_traceidratio` with `OTEL_TRACES_SAMPLER_ARG=0.1` keeps 10% of traces. 3. Collector tail sampling for intelligent decisions, which keeps all errors and samples 1% of successes. Combine aggressively. It is easier to re-enable telemetry when you are debugging than to pay for ingestion nobody looks at.