Send Telemetry to a Backend

NeMo Lens does not provide or recommend an observability solution. It emits OTLP. Where that OTLP goes, how it is stored, how it is queried, and how it is visualized are your decisions, which are shaped by your organization’s existing observability investments and the scale of your workloads.

This page shows how to configure NeMo Lens for common destinations. NeMo Lens exports through standard OTLP, so any OTLP-compatible backend works without code changes. Four destinations are covered in depth:

• File: local trace and metric capture for offline analysis or archival
• W&B Weave: Weights & Biases’ trace UI, co-located with training run metadata
• Honeycomb: hosted APM that accepts all three signals on one OTLP endpoint
• OTel Collector: a routing and aggregation layer in front of other backends

This guide also provides a quick reference for other hosted backends.

Export Telemetry to a File

Writing traces to a local file is useful for offline analysis, CI captures, and archival. Choose one of the following four approaches depending on your requirements.

Use Console Exporter with Shell Redirect (Simplest)

$
export NEMO_LENS_ENABLED=1
$
export NEMO_LENS_EXPORTER=console
$
$
python train.py > traces.jsonl 2>&1

Setting NEMO_LENS_EXPORTER=console installs the ConsoleSpanExporter, which writes one JSON line per span to stdout. Redirect stdout to a file, and you have a span log.

Drawbacks:

• Mixes application stdout with span data; separate them with selective logging to stderr.
• Does not capture metrics (the metric exporter writes a different format).

Point Custom ConsoleSpanExporter to a File Handle

1
from pathlib import Path
2
from opentelemetry.sdk.trace.export import ConsoleSpanExporter
3
from nemo.lens import NemoLensConfig, setup_telemetry
4
5
trace_file = Path("traces.jsonl").open("a", buffering=1)   # line-buffered
6
exporter = ConsoleSpanExporter(out=trace_file)
7
8
config = NemoLensConfig(enabled=True, exporter="console")  # falls back if custom skipped
9
handle = setup_telemetry(
10
    config,
11
    rank=0,
12
    world_size=1,
13
    span_exporter=exporter,
14
)
15
16
try:
17
    # ... your workload ...
18
finally:
19
    handle.shutdown()
20
    trace_file.close()

The ConsoleSpanExporter accepts any file-like object using out=. This separates trace data from application stdout without a shell redirect.

Caveats:

• Line-buffer the output using buffering=1 so that lines are not lost if the process crashes.
• Close the file after calling handle.shutdown().
• Each line is a Python repr representation of the span, not strict JSON. For strict JSON, write a custom exporter as described in the next approach.

Implement Custom SpanExporter for Full Control

For structured JSON, compression, rotation, or any custom format:

1
import json
2
from opentelemetry.sdk.trace.export import SpanExporter, SpanExportResult
3
4
class JSONLFileSpanExporter(SpanExporter):
5
    def __init__(self, path: str):
6
        self._fh = open(path, "a", buffering=1)
7
8
    def export(self, spans) -> SpanExportResult:
9
        try:
10
            for span in spans:
11
                record = {
12
                    "name": span.name,
13
                    "trace_id": f"{span.context.trace_id:032x}",
14
                    "span_id": f"{span.context.span_id:016x}",
15
                    "parent_id": (
16
                        f"{span.parent.span_id:016x}" if span.parent else None
17
                    ),
18
                    "start_time_unix_nano": span.start_time,
19
                    "end_time_unix_nano": span.end_time,
20
                    "attributes": dict(span.attributes or {}),
21
                    "status": str(span.status.status_code),
22
                }
23
                self._fh.write(json.dumps(record) + "\n")
24
            return SpanExportResult.SUCCESS
25
        except Exception:
26
            return SpanExportResult.FAILURE
27
28
    def shutdown(self) -> None:
29
        self._fh.close()
30
31
    def force_flush(self, timeout_millis: int = 30000) -> bool:
32
        self._fh.flush()
33
        return True
34
35
handle = setup_telemetry(
36
    config,
37
    span_exporter=JSONLFileSpanExporter("traces.jsonl"),
38
)

This approach provides strict JSONL that is trivial to query using jq. Extend this implementation with gzip compression, rotation, or remote write as needed.

Use OTel Collector File Exporter

If you are already running an OTel Collector, add a file exporter to its pipeline:

1
# otel-collector.yaml
2
exporters:
3
  file/traces:
4
    path: /var/log/otel/traces.jsonl
5
    rotation:
6
      max_megabytes: 100
7
      max_days: 7
8
9
service:
10
  pipelines:
11
    traces:
12
      receivers: [otlp]
13
      processors: [batch]
14
      exporters: [file/traces, jaeger]   # fan out to both

The application still exports OTLP as normal, and the Collector handles file writes, rotation, and retention.

Use this approach when you want a single file with spans from multiple ranks or multiple services.

Export Metrics to a File

For metrics, use a PeriodicExportingMetricReader with a ConsoleMetricExporter:

1
from opentelemetry.sdk.metrics.export import ConsoleMetricExporter, PeriodicExportingMetricReader
2
3
metric_file = open("metrics.jsonl", "a", buffering=1)
4
reader = PeriodicExportingMetricReader(
5
    ConsoleMetricExporter(out=metric_file),
6
    export_interval_millis=10000,
7
)
8
9
handle = setup_telemetry(config, metric_reader=reader)

Alternatively, use the file exporter of the Collector in the metrics pipeline using the same pattern as traces.

Integrate with W&B Weave

Weave is the Weights & Biases trace visualization tool. It ingests OTLP spans and renders them in the same UI as your training runs, so that traces and training metrics live together.

Configure the Integration

Configure the integration by choosing one of the two patterns below depending on whether you are using a collector.

Configure Pattern A for Direct Export from the Application

Use this pattern when you want to export telemetry directly to Weights & Biases without running a local OTel Collector.

$
# Required: W&B identification (set as resource attributes)
$
export WANDB_ENTITY=my-team                    # or your personal entity
$
export WANDB_PROJECT=megatron-training
$
$
# Required: traces-signal-specific endpoint + auth header
$
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://trace.wandb.ai/otel/v1/traces
$
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf
$
export OTEL_EXPORTER_OTLP_TRACES_HEADERS="wandb-api-key=$WANDB_API_KEY"
$
$
# Weave ingests traces only; disable the other two signals.
$
export MEGATRON_OTEL_METRICS_ENABLED=0
$
export NEMO_LENS_LOGS_ENABLED=0
$
$
# Required: lens activation
$
export NEMO_LENS_ENABLED=1

A ready-to-run compose file for this pattern is at docker-compose.weave.yml, which brings up only the Megatron container and points traces straight to Weave. Use this file when you do not want to run the local stack.

Configure Pattern B for Export Through an OTel Collector

Use this pattern when you want batching, filtering, or multi-backend fan-out. See observability/otel-collector-weave.yaml in the repository for a ready-to-run example, toggled through the --config=/etc/otel/collector-weave.yaml mode of the docker-compose.otel.yml file.

Notes on the Direct Path

• W&B Weave currently ingests traces only (as of early 2026). Metrics still require a separate sink, such as Prometheus, an OTel Collector, or native wandb.log().
• Use the OTEL_EXPORTER_OTLP_TRACES_* variants instead of the signal-agnostic OTEL_EXPORTER_OTLP_*. The Weave URL is a full path ending in /v1/traces. Signal-specific environment variables are treated as full URLs, whereas the generic variant appends /v1/traces automatically; setting both would produce /v1/traces/v1/traces and a 404 error.
• NeMo Lens honors OTEL_EXPORTER_OTLP_PROTOCOL and signal-specific variants, so http/protobuf routes to the HTTP exporter class because Weave is HTTP-only.
• NemoLensConfig.from_env() reads WANDB_ENTITY and WANDB_PROJECT directly and sets them as wandb.entity and wandb.project resource attributes on every span, which is required for Weave to route correctly.

Run the Application

$
python pretrain_gpt.py ...

Traces appear in the Weave tab of your W&B run within a few seconds. The trace tree mirrors Jaeger’s structure: a megatron.train_step root span with child spans for forward_backward, optimizer, and other tasks.

Link Traces to Runs

Because WANDB_ENTITY and WANDB_PROJECT are set as span attributes, Weave automatically associates traces with the right W&B run. The nemo.run.id resource attribute (auto-generated or from SLURM_JOB_ID) serves as a unique run identifier you can filter on in the Weave UI.

Configure Sampling for Cost Management

W&B bills by ingested trace volume. For long per_step runs, sample aggressively:

$
export OTEL_TRACES_SAMPLER=parentbased_traceidratio
$
export OTEL_TRACES_SAMPLER_ARG=0.01    # keep 1% of traces

See the sampling documentation for how this composes with NeMo Lens’s export strategies.

Understand Exported Data

Everything the SDK exports, including span names, attributes, events, links, and status. Weave renders:

• Attribute key-value pairs
• Error events through span.record_exception
• OTel links as clickable references, which are useful for pipeline-parallel correlation

Send Telemetry to Honeycomb

Honeycomb is a hosted APM that ingests OpenTelemetry data natively. Unlike W&B Weave, it accepts all three signals (traces, metrics, and logs) on a single OTLP endpoint. This is a good fit if you want one hosted destination for everything and already have (or are happy to adopt) Honeycomb’s query model.

Configure Honeycomb

Configure the integration with Honeycomb by choosing one of the following two patterns.

Configure Pattern A for Direct Export from the Application

$
# One endpoint covers all three signals.
$
export OTEL_EXPORTER_OTLP_ENDPOINT=https://api.honeycomb.io:443
$
export OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=${HONEYCOMB_API_KEY},x-honeycomb-dataset=${HONEYCOMB_DATASET}"
$
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
$
$
export NEMO_LENS_ENABLED=1

• x-honeycomb-team: Your ingest API key. In the Honeycomb UI, select Environment Settings, and then select API Keys to find this key.
• x-honeycomb-dataset: The dataset name. This name is required for metrics, and it is optional but recommended for traces and logs. Choose any meaningful name; Honeycomb automatically creates the dataset on the first write.
• OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf: Honeycomb supports both gRPC and HTTP, but HTTP is more forgiving behind load balancers. Default to HTTP unless you have a reason otherwise.

NeMo Lens honors OTEL_EXPORTER_OTLP_PROTOCOL (and the signal-specific variants OTEL_EXPORTER_OTLP_TRACES_PROTOCOL and OTEL_EXPORTER_OTLP_METRICS_PROTOCOL) when picking between gRPC and HTTP exporters, so this works without code changes.

For EU instance, substitute https://api.eu1.honeycomb.io:443.

A ready-to-run compose file for this pattern is at docker-compose.honeycomb.yml, which brings up only the Megatron container and points it straight to Honeycomb. Use this file when you do not want to run the local stack.

Configure Pattern B for Export Through an OTel Collector

Use this pattern when you want batching, filtering, or multi-backend fan-out between the application and Honeycomb. See collector-honeycomb.yaml in the repository for a ready-to-run example:

1
exporters:
2
  otlphttp/honeycomb:
3
    endpoint: https://api.honeycomb.io:443
4
    headers:
5
      x-honeycomb-team: ${env:HONEYCOMB_API_KEY}
6
      x-honeycomb-dataset: ${env:HONEYCOMB_DATASET}
7
8
service:
9
  pipelines:
10
    traces:   { receivers: [otlp], processors: [batch], exporters: [otlphttp/honeycomb] }
11
    metrics:  { receivers: [otlp], processors: [batch], exporters: [otlphttp/honeycomb] }
12
    logs:     { receivers: [otlp], processors: [batch], exporters: [otlphttp/honeycomb] }

The application then points at your collector (OTEL_EXPORTER_OTLP_ENDPOINT=http://collector:4317), and the collector handles Honeycomb auth and routing.

The repo’s docker-compose.otel.yml has a one-line toggle for this: uncomment --config=/etc/otel/collector-honeycomb.yaml and set HONEYCOMB_API_KEY and HONEYCOMB_DATASET in .env.

Compare Classic and Current Honeycomb Accounts

Honeycomb migrated from dataset-per-service (Classic) to environment-based organization. If you are on a Classic account, the x-honeycomb-dataset header is required for every signal, and the dataset field has specific semantics. For current Honeycomb it is still required for metrics and optional but recommended for traces and logs. If you are unsure which account type you have, your account page displays this information.

Manage Data Volume through Sampling

Honeycomb bills on event volume. A per_step Megatron run on many ranks will ship a lot of events. Layer your sampling:

1. NeMo Lens export_strategy at the rank level (start with single_rank).
2. OTel SDK OTEL_TRACES_SAMPLER=parentbased_traceidratio at the trace level.
3. Honeycomb Refinery tail sampling, which provides access to the full trace before making a decision. This is recommended for production; see Honeycomb’s Refinery docs.

Understand Exported Honeycomb Data

Every attribute, event, and link the SDK exports. Honeycomb’s UI is especially good at high-cardinality attribute queries (BubbleUp, HEATMAP, etc.), so set span attributes liberally; attribute cardinality is what Honeycomb is best at.

Configure OTel Collector

The OpenTelemetry Collector is a common intermediary between your application and your observability backends. Running a Collector instead of exporting directly from the SDK can provide several benefits:

• Fan-out capabilities. Send the same telemetry to multiple backends, such as Jaeger, Prometheus, and an S3 archive.
• Sampling and filtering. Drop spans at the Collector instead of within each SDK instance.
• Batching and resilience. Buffer during network outages without losing data.
• Transforms. Rename attributes, redact PII, or enrich with external metadata.
• Centralized configuration. Change backends without restarting training jobs.

Set Minimum Configuration

1
# otel-collector.yaml
2
receivers:
3
  otlp:
4
    protocols:
5
      grpc:
6
        endpoint: 0.0.0.0:4317
7
      http:
8
        endpoint: 0.0.0.0:4318
9
10
processors:
11
  batch:
12
    timeout: 5s
13
    send_batch_size: 1024
14
15
exporters:
16
  # Traces to Jaeger
17
  otlp/jaeger:
18
    endpoint: jaeger:4317
19
    tls:
20
      insecure: true
21
22
  # Metrics to Prometheus (pull model)
23
  prometheus:
24
    endpoint: 0.0.0.0:8889
25
26
  # Optional: file archival
27
  file:
28
    path: /var/log/otel/telemetry.jsonl
29
30
service:
31
  pipelines:
32
    traces:
33
      receivers: [otlp]
34
      processors: [batch]
35
      exporters: [otlp/jaeger, file]
36
37
    metrics:
38
      receivers: [otlp]
39
      processors: [batch]
40
      exporters: [prometheus]

Run the Collector

Docker (simplest):

$
docker run --rm \
>
  -p 4317:4317 -p 4318:4318 -p 8889:8889 \
>
  -v $(pwd)/otel-collector.yaml:/etc/otel-collector.yaml \
>
  otel/opentelemetry-collector-contrib:latest \
>
  --config=/etc/otel-collector.yaml

This is a generic standalone example. The repository’s bundled configurations live under observability/ and are mounted at /etc/otel/collector*.yaml by docker-compose.otel.yml (selected using the --config line); adjust the path if you copy from there.

On a cluster, deploy as a sidecar, DaemonSet, or shared service. Typical patterns:

• Sidecar deployment. Deploy one Collector per application pod. This provides low latency and an isolated failure domain.
• DaemonSet deployment. Deploy one Collector per host, where every local application exports to it. This is a good fit for Kubernetes.
• Shared service deployment. Deploy one fleet of Collectors behind a load balancer. This is the most cost-effective option but adds a network hop.

Configure the Application

$
export NEMO_LENS_ENABLED=1
$
export OTEL_EXPORTER_OTLP_ENDPOINT=http://collector.internal:4317

That’s it. Lens discovers the endpoint from the standard env var; no code changes.

Apply Useful Processors

Beyond batch, consider:

1
processors:
2
  # Drop noisy spans at the Collector
3
  filter/drop_health_checks:
4
    traces:
5
      span:
6
        - 'name == "GET /healthz"'
7
8
  # Sample smart: keep errors, sample 10% of successes
9
  tail_sampling:
10
    decision_wait: 10s
11
    policies:
12
      - name: keep-errors
13
        type: status_code
14
        status_code: { status_codes: [ERROR] }
15
      - name: sample-successes
16
        type: probabilistic
17
        probabilistic: { sampling_percentage: 10 }
18
19
  # Enrich with cluster metadata
20
  resource:
21
    attributes:
22
      - key: cluster.name
23
        value: prod-us-west
24
        action: upsert
25
26
  # Redact sensitive attributes
27
  attributes/redact:
28
    actions:
29
      - key: user.email
30
        action: delete

Attach to a pipeline:

1
service:
2
  pipelines:
3
    traces:
4
      receivers: [otlp]
5
      processors: [filter/drop_health_checks, tail_sampling, resource, batch]
6
      exporters: [otlp/jaeger, file]

Configure Multi-Backend Routing

Send traces to two places simultaneously, such as Jaeger for interactive debugging and W&B Weave for run history:

1
exporters:
2
  otlp/jaeger:
3
    endpoint: jaeger:4317
4
    tls: { insecure: true }
5
6
  otlphttp/wandb:
7
    endpoint: https://trace.wandb.ai/otel/v1/traces
8
    headers:
9
      wandb-api-key: ${env:WANDB_API_KEY}
10
11
service:
12
  pipelines:
13
    traces:
14
      receivers: [otlp]
15
      processors: [batch]
16
      exporters: [otlp/jaeger, otlphttp/wandb]

The application exports to one endpoint (the Collector), and the Collector fans out the telemetry.

Configure Collector-Side Sampling

Instead of sampling at the SDK through OTEL_TRACES_SAMPLER, sample at the Collector. The advantage of this approach is that you can make the decision based on the complete trace (for example, keep all traces containing an error), which the SDK cannot do because it has not seen the whole trace yet.

The tail_sampling processor is the standard tool. See the full tail sampling documentation.

Production Considerations

• Backpressure. If a backend is slow, the Collector buffers. Configure sending_queue limits to cap memory.
• TLS. Enable TLS between the SDK and the Collector, and between the Collector and backends, in any multi-tenant setup.
• Health checks. Enable the health_check extension (as the bundled configurations do) to expose :13133/ and monitor it.
• Version pinning. The opentelemetry-collector-contrib image changes, so pin to a version and upgrade deliberately.

Debug the Collector

$
# Increase logging
$
--set service.telemetry.logs.level=debug
$
$
# Enable the debug exporter to see spans on stdout
$
exporters:
$
  debug:
$
    verbosity: detailed
$
$
# Add to a pipeline for testing
$
service:
$
  pipelines:
$
    traces:
$
      exporters: [debug]

The Collector’s own telemetry (:8888/metrics) shows incoming span rates, processor queue depth, and exporter success counts; scrape it with Prometheus to monitor the monitoring.

Send Telemetry to Other Hosted Backends

Configure other popular hosted backends by using standard OpenTelemetry environment variables.

Configure Grafana Cloud

$
export OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-gateway-<region>.grafana.net/otlp
$
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic <base64-encoded-instance-id-and-token>"

This routes traces to Tempo, metrics to Mimir, and logs to Loki, all queryable from a unified Grafana UI.

Configure Datadog

$
export OTEL_EXPORTER_OTLP_ENDPOINT=https://trace.agent.datadoghq.com
$
export OTEL_EXPORTER_OTLP_HEADERS="DD-API-KEY=<your-api-key>"

Datadog also ships their own Collector preset; see their documentation for advanced configuration.

Configure New Relic

$
export OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.nr-data.net
$
export OTEL_EXPORTER_OTLP_HEADERS="api-key=<your-ingest-key>"

Configure Self-Hosted Jaeger or Tempo

Both accept OTLP natively:

$
export OTEL_EXPORTER_OTLP_ENDPOINT=http://jaeger.internal:4317    # or tempo.internal

Select a Backend

All destinations work the same from NeMo Lens’s perspective; the choice is about cost, operational burden, and integration with your existing stack.

Partition Telemetry by Run

Regardless of the backend, filter by nemo.run.id (auto-set by NeMo Lens) to isolate a specific training run’s data:

• Jaeger: Use the tag filter nemo.run.id=<value>.
• Grafana: Use the dashboard variable nemo_run_id.
• Honeycomb: Use the filter nemo.run.id.
• Datadog: Use the facet @nemo.run.id.
• Weave: Use the run-level association through WANDB_ENTITY and WANDB_PROJECT.

Multiple runs land in the same index or project; the attribute is the partition key.

gRPC vs HTTP

OTLP has two transport variants:

$
# gRPC (default, faster, persistent connections)
$
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
$
OTEL_EXPORTER_OTLP_ENDPOINT=http://collector:4317
$
$
# HTTP/Protobuf (firewall-friendly, HTTPS works easily)
$
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
$
OTEL_EXPORTER_OTLP_ENDPOINT=http://collector:4318

The NeMo Lens providers.py tries gRPC first, and falls back to HTTP if the gRPC exporter is not installed. If you only installed opentelemetry-exporter-otlp-proto-http, set the protocol explicitly.

Limit Ingested Spans

A per_step run on 1,000 ranks can produce over 100,000 spans per second. Most backends cannot (or will not affordably) ingest that volume. Layer your sampling:

1. NeMo Lens export_strategy at the rank level, where single_rank sends one rank’s data. This is usually the correct starting point.
2. The OTel SDK sampler at the trace level, where setting OTEL_TRACES_SAMPLER=parentbased_traceidratio with OTEL_TRACES_SAMPLER_ARG=0.1 keeps 10% of traces.
3. Collector tail sampling for intelligent decisions, which keeps all errors and samples 1% of successes.

Combine aggressively. It is easier to re-enable telemetry when you are debugging than to pay for ingestion nobody looks at.