Skip to navigation
Backends & Demo Stack

Send Telemetry to a Backend

View as Markdown

NeMo Lens does not provide or recommend an observability solution. It emits OTLP. Where that OTLP goes, how it is stored, how it is queried, and how it is visualized are your decisions, which are shaped by your organization’s existing observability investments and the scale of your workloads.

This page shows how to configure NeMo Lens for common destinations. NeMo Lens exports through standard OTLP, so any OTLP-compatible backend works without code changes. Five destinations are covered in depth:

  • File: local trace and metric capture for offline analysis or archival
  • W&B Weave: Weights & Biases’ trace UI, co-located with training run metadata
  • Honeycomb: hosted APM that accepts all three signals on one OTLP endpoint
  • RabbitMQ: a message broker that queues every signal for a consumer you write
  • OTel Collector: a routing and aggregation layer in front of other backends

This guide also provides a quick reference for other hosted backends.

Export Telemetry to a File

Writing traces to a local file is useful for offline analysis, CI captures, and archival. Choose one of the following four approaches depending on your requirements.

Use Console Exporter with Shell Redirect (Simplest)

export NEMO_LENS_ENABLED=1
export NEMO_LENS_EXPORTER=console
python train.py > traces.jsonl 2>&1

Setting NEMO_LENS_EXPORTER=console installs the ConsoleSpanExporter with a compact formatter, so it writes exactly one JSON object per line to stdout. Redirect stdout to a file, and you have a JSONL span log that line-oriented tooling can read directly.

Drawbacks:

  • Mixes application stdout with span data; separate them with selective logging to stderr.
  • Interleaves metric batches into the same stream when metrics_enabled is on. Those lines are also JSONL, but they carry a different schema than spans, so a reader must tell the two apart.

Point Custom ConsoleSpanExporter to a File Handle

from pathlib import Path
from opentelemetry.sdk.trace.export import ConsoleSpanExporter
from nemo.lens import NemoLensConfig, setup_telemetry
trace_file = Path("traces.jsonl").open("a", buffering=1)   # line-buffered
exporter = ConsoleSpanExporter(
    out=trace_file,
    # The SDK default is indent=4, which spreads one span over many lines.
    formatter=lambda span: span.to_json(indent=None) + "\n",
)
config = NemoLensConfig(enabled=True, exporter="console")  # falls back if custom skipped
handle = setup_telemetry(
    config,
    span_exporter=exporter,
)
try:
    # ... your workload ...
finally:
    handle.shutdown()
    trace_file.close()

The ConsoleSpanExporter accepts any file-like object using out=. This separates trace data from application stdout without a shell redirect.

Caveats:

  • Line-buffer the output using buffering=1 so that lines are not lost if the process crashes.
  • Close the file after calling handle.shutdown().
  • Pass the compact formatter shown above. Constructing ConsoleSpanExporter yourself does not pick up the one NeMo Lens installs for exporter="console", and the SDK default pretty-prints with indent=4, which is not valid JSONL.
  • The line format is the OTel SDK’s span JSON schema. To control the field set yourself, write a custom exporter as described in the next approach.

Implement Custom SpanExporter for Full Control

For structured JSON, compression, rotation, or any custom format:

import json
from opentelemetry.sdk.trace.export import SpanExporter, SpanExportResult
class JSONLFileSpanExporter(SpanExporter):
    def __init__(self, path: str):
        self._fh = open(path, "a", buffering=1)
    def export(self, spans) -> SpanExportResult:
        try:
            for span in spans:
                record = {
                    "name": span.name,
                    "trace_id": f"{span.context.trace_id:032x}",
                    "span_id": f"{span.context.span_id:016x}",
                    "parent_id": (
                        f"{span.parent.span_id:016x}" if span.parent else None
                    ),
                    "start_time_unix_nano": span.start_time,
                    "end_time_unix_nano": span.end_time,
                    "attributes": dict(span.attributes or {}),
                    "status": str(span.status.status_code),
                }
                self._fh.write(json.dumps(record) + "\n")
            return SpanExportResult.SUCCESS
        except Exception:
            return SpanExportResult.FAILURE
    def shutdown(self) -> None:
        self._fh.close()
    def force_flush(self, timeout_millis: int = 30000) -> bool:
        self._fh.flush()
        return True
handle = setup_telemetry(
    config,
    span_exporter=JSONLFileSpanExporter("traces.jsonl"),
)

This approach provides strict JSONL that is trivial to query using jq. Extend this implementation with gzip compression, rotation, or remote write as needed.

Use OTel Collector File Exporter

If you are already running an OTel Collector, add a file exporter to its pipeline:

# otel-collector.yaml
exporters:
  file/traces:
    path: /var/log/otel/traces.jsonl
    rotation:
      max_megabytes: 100
      max_days: 7
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [file/traces, jaeger]   # fan out to both

The application still exports OTLP as normal, and the Collector handles file writes, rotation, and retention.

Use this approach when you want a single file with spans from multiple ranks or multiple services.

Export Metrics to a File

For metrics, use a PeriodicExportingMetricReader with a ConsoleMetricExporter:

from opentelemetry.sdk.metrics.export import ConsoleMetricExporter, PeriodicExportingMetricReader
metric_file = open("metrics.jsonl", "a", buffering=1)
reader = PeriodicExportingMetricReader(
    ConsoleMetricExporter(
        out=metric_file,
        # As with ConsoleSpanExporter, the SDK default is indent=4.
        formatter=lambda metrics_data: metrics_data.to_json(indent=None) + "\n",
    ),
    export_interval_millis=10000,
)
handle = setup_telemetry(config, metric_reader=reader)

One line is written per export interval, and each line holds the whole metrics batch rather than a single data point.

Alternatively, use the file exporter of the Collector in the metrics pipeline using the same pattern as traces.

Integrate with W&B Weave

Weave is the Weights & Biases trace visualization tool. It ingests OTLP spans and renders them in the same UI as your training runs, so that traces and training metrics live together.

Configure the Integration

Configure the integration by choosing one of the two patterns below depending on whether you are using a collector.

Configure Pattern A for Direct Export from the Application

Use this pattern when you want to export telemetry directly to Weights & Biases without running a local OTel Collector.

# Required: W&B identification (set as resource attributes)
export WANDB_ENTITY=my-team                    # or your personal entity
export WANDB_PROJECT=megatron-training
# Required: traces-signal-specific endpoint + auth header
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://trace.wandb.ai/otel/v1/traces
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_TRACES_HEADERS="wandb-api-key=$WANDB_API_KEY"
# Weave ingests traces only; disable the other two signals.
export MEGATRON_OTEL_METRICS_ENABLED=0
export NEMO_LENS_LOGS_ENABLED=0
# Required: lens activation
export NEMO_LENS_ENABLED=1

A ready-to-run compose file for this pattern is at docker-compose.weave.yml, which brings up only the Megatron container and points traces straight to Weave. Use this file when you do not want to run the local stack.

Configure Pattern B for Export Through an OTel Collector

Use this pattern when you want batching, filtering, or multi-backend fan-out. See observability/otel-collector-weave.yaml in the repository for a ready-to-run example, toggled through the --config=/etc/otel/collector-weave.yaml mode of the docker-compose.otel.yml file.

Notes on the Direct Path

  • W&B Weave currently ingests traces only (as of early 2026). Metrics still require a separate sink, such as Prometheus, an OTel Collector, or native wandb.log().
  • Use the OTEL_EXPORTER_OTLP_TRACES_* variants instead of the signal-agnostic OTEL_EXPORTER_OTLP_*. The Weave URL is a full path ending in /v1/traces. Signal-specific environment variables are treated as full URLs, whereas the generic variant appends /v1/traces automatically; setting both would produce /v1/traces/v1/traces and a 404 error.
  • NeMo Lens honors OTEL_EXPORTER_OTLP_PROTOCOL and signal-specific variants, so http/protobuf routes to the HTTP exporter class because Weave is HTTP-only.
  • NemoLensConfig.from_env() reads WANDB_ENTITY and WANDB_PROJECT directly and sets them as wandb.entity and wandb.project resource attributes on every span, which is required for Weave to route correctly.

Run the Application

python pretrain_gpt.py ...

Traces appear in the Weave tab of your W&B run within a few seconds. The trace tree mirrors Jaeger’s structure: a megatron.train_step root span with child spans for forward_backward, optimizer, and other tasks.

Because WANDB_ENTITY and WANDB_PROJECT are set as span attributes, Weave automatically associates traces with the right W&B run. The nemo.run.id resource attribute (auto-generated or from SLURM_JOB_ID) serves as a unique run identifier you can filter on in the Weave UI.

Configure Sampling for Cost Management

W&B bills by ingested trace volume. For long per_step runs, sample aggressively:

export OTEL_TRACES_SAMPLER=parentbased_traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.01    # keep 1% of traces

NeMo Lens does no sampling of its own, so this SDK-level setting is the only trace sampling in play unless you add tail sampling in the collector.

Understand Exported Data

Everything the SDK exports, including span names, attributes, events, links, and status. Weave renders:

  • Attribute key-value pairs
  • Error events through span.record_exception
  • OTel links as clickable references, which are useful for pipeline-parallel correlation

Send Telemetry to Honeycomb

Honeycomb is a hosted APM that ingests OpenTelemetry data natively. Unlike W&B Weave, it accepts all three signals (traces, metrics, and logs) on a single OTLP endpoint. This is a good fit if you want one hosted destination for everything and already have (or are happy to adopt) Honeycomb’s query model.

Configure Honeycomb

Configure the integration with Honeycomb by choosing one of the following two patterns.

Configure Pattern A for Direct Export from the Application

# One endpoint covers all three signals.
export OTEL_EXPORTER_OTLP_ENDPOINT=https://api.honeycomb.io:443
export OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=${HONEYCOMB_API_KEY},x-honeycomb-dataset=${HONEYCOMB_DATASET}"
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export NEMO_LENS_ENABLED=1
  • x-honeycomb-team: Your ingest API key. In the Honeycomb UI, select Environment Settings, and then select API Keys to find this key.
  • x-honeycomb-dataset: The dataset name. This name is required for metrics, and it is optional but recommended for traces and logs. Choose any meaningful name; Honeycomb automatically creates the dataset on the first write.
  • OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf: Honeycomb supports both gRPC and HTTP, but HTTP is more forgiving behind load balancers. Default to HTTP unless you have a reason otherwise.

NeMo Lens honors OTEL_EXPORTER_OTLP_PROTOCOL (and the signal-specific variants OTEL_EXPORTER_OTLP_TRACES_PROTOCOL and OTEL_EXPORTER_OTLP_METRICS_PROTOCOL) when picking between gRPC and HTTP exporters, so this works without code changes.

For EU instance, substitute https://api.eu1.honeycomb.io:443.

A ready-to-run compose file for this pattern is at docker-compose.honeycomb.yml, which brings up only the Megatron container and points it straight to Honeycomb. Use this file when you do not want to run the local stack.

Configure Pattern B for Export Through an OTel Collector

Use this pattern when you want batching, filtering, or multi-backend fan-out between the application and Honeycomb. See collector-honeycomb.yaml in the repository for a ready-to-run example:

exporters:
  otlphttp/honeycomb:
    endpoint: https://api.honeycomb.io:443
    headers:
      x-honeycomb-team: ${env:HONEYCOMB_API_KEY}
      x-honeycomb-dataset: ${env:HONEYCOMB_DATASET}
service:
  pipelines:
    traces:   { receivers: [otlp], processors: [batch], exporters: [otlphttp/honeycomb] }
    metrics:  { receivers: [otlp], processors: [batch], exporters: [otlphttp/honeycomb] }
    logs:     { receivers: [otlp], processors: [batch], exporters: [otlphttp/honeycomb] }

The application then points at your collector (OTEL_EXPORTER_OTLP_ENDPOINT=http://collector:4317), and the collector handles Honeycomb auth and routing.

The repo’s docker-compose.otel.yml has a one-line toggle for this: uncomment --config=/etc/otel/collector-honeycomb.yaml and set HONEYCOMB_API_KEY and HONEYCOMB_DATASET in .env.

Compare Classic and Current Honeycomb Accounts

Honeycomb migrated from dataset-per-service (Classic) to environment-based organization. If you are on a Classic account, the x-honeycomb-dataset header is required for every signal, and the dataset field has specific semantics. For current Honeycomb it is still required for metrics and optional but recommended for traces and logs. If you are unsure which account type you have, your account page displays this information.

Manage Data Volume through Sampling

Honeycomb bills on event volume. A per_step Megatron run on many ranks will ship a lot of events. Layer your sampling:

  1. Leaving enabled false on ranks that need not report (start with one rank).
  2. OTel SDK OTEL_TRACES_SAMPLER=parentbased_traceidratio at the trace level.
  3. Honeycomb Refinery tail sampling, which provides access to the full trace before making a decision. This is recommended for production; see Honeycomb’s Refinery docs.

Understand Exported Honeycomb Data

Every attribute, event, and link the SDK exports. Honeycomb’s UI is especially good at high-cardinality attribute queries (BubbleUp, HEATMAP, etc.), so set span attributes liberally; attribute cardinality is what Honeycomb is best at.

Stream Telemetry to RabbitMQ

RabbitMQ is a message broker. Instead of shipping telemetry to a query UI, it places every signal on a durable queue that a service you write consumes. This is a good fit when you want to post-process, re-route, or archive telemetry with custom logic rather than read it in a vendor dashboard.

Unlike W&B Weave and Honeycomb, there is no direct-export pattern. The OpenTelemetry SDK has no AMQP exporter, so telemetry always flows through an OTel Collector, which republishes it to RabbitMQ over AMQP:

Megatron ──OTLP──► OTel Collector ──AMQP──► exchange nemo_lens.otel.raw ──► queue nemo_lens-processing ──► your consumer

Configure RabbitMQ

The application exports OTLP to the collector as usual:

export NEMO_LENS_ENABLED=1
export OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317

The collector uses the contrib rabbitmq exporter to publish each signal to the exchange. See otel-collector-rabbitmq.yaml in the repository for a ready-to-run example:

exporters:
  rabbitmq/default:
    connection:
      endpoint: amqp://rabbitmq:5672
      auth:
        plain:
          username: nemo_lens
          password: example
    routing:
      exchange: nemo_lens.otel.raw
    encoding_extension: otlp_encoding/rabbitmq   # emit OTLP-JSON
    durable: true
extensions:
  otlp_encoding/rabbitmq:
    protocol: otlp_json
service:
  # The encoding extension must be enabled here for the exporter to find it.
  extensions: [health_check, otlp_encoding/rabbitmq]
  pipelines:
    traces:   { receivers: [otlp], processors: [batch/traces],  exporters: [rabbitmq/default] }
    metrics:  { receivers: [otlp], processors: [batch/metrics], exporters: [rabbitmq/default] }
    logs:     { receivers: [otlp], processors: [batch/logs],    exporters: [rabbitmq/default] }

A ready-to-run stack is at docker-compose.rabbitmq.yml, which brings up the Megatron container, an OTel Collector using the config above, and a RabbitMQ broker. The broker topology (exchange, queues, bindings, and the nemo_lens user) is seeded at boot from observability/rabbitmq/definitions.json. Start it with:

docker compose -f docker-compose.rabbitmq.yml up -d

The RabbitMQ management UI is at http://localhost:15672 (login nemo_lens / example).

Minimal vs. Full Infra Coverage

Two ready-to-run stacks are provided, so you can start minimal and scale up:

  • docker-compose.rabbitmq.yml (above) — the minimal integration. The collector forwards only the training process’s own OTLP (traces, metrics, logs).
  • docker-compose.rabbitmq+infra.yml — full infrastructure coverage. It adds a DCGM Exporter (per-GPU utilization, memory, temperature, power, NVLink, ECC) and a Node Exporter (InfiniBand, RDMA, NIC bandwidth, TCP retransmits) as sidecars, and points the collector at otel-collector-rabbitmq-infra.yaml, which scrapes both plus host CPU/memory/disk metrics and publishes them all to the same queue. This variant requires a GPU host with the NVIDIA container runtime.

Adding the sidecars alone is not enough — the collector config must scrape them, which is why the two variants ship with separate collector configs.

Understand the Message Contract

  • Exchange: nemo_lens.otel.raw (topic). Each signal is published with its default routing key: otlp_spans, otlp_metrics, or otlp_logs.
  • Queues: the bundled topology binds nemo_lens-processing with #, so it receives all three signals. A second queue, nemo_lens-poison, is provided for dead-lettering messages your consumer cannot process.
  • Payload: OTLP-JSON. Each message is a JSON ExportTraceServiceRequest, ExportMetricsServiceRequest, or ExportLogsServiceRequest. Set protocol: otlp_proto on the encoding extension for the compact protobuf form instead.
  • Durability: durable: true marks published messages persistent and the queues are declared durable, so telemetry survives a broker restart.

Replace the bundled definitions.json with your own exchange, queues, bindings, and policies to fit your deployment; the exporter only needs the target exchange to exist.

Configure OTel Collector

The OpenTelemetry Collector is a common intermediary between your application and your observability backends. Running a Collector instead of exporting directly from the SDK can provide several benefits:

  • Fan-out capabilities. Send the same telemetry to multiple backends, such as Jaeger, Prometheus, and an S3 archive.
  • Sampling and filtering. Drop spans at the Collector instead of within each SDK instance.
  • Batching and resilience. Buffer during network outages without losing data.
  • Transforms. Rename attributes, redact PII, or enrich with external metadata.
  • Centralized configuration. Change backends without restarting training jobs.

Set Minimum Configuration

# otel-collector.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
processors:
  batch:
    timeout: 5s
    send_batch_size: 1024
exporters:
  # Traces to Jaeger
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true
  # Metrics to Prometheus (pull model)
  prometheus:
    endpoint: 0.0.0.0:8889
  # Optional: file archival
  file:
    path: /var/log/otel/telemetry.jsonl
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/jaeger, file]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus]

Run the Collector

Docker (simplest):

docker run --rm \
  -p 4317:4317 -p 4318:4318 -p 8889:8889 \
  -v $(pwd)/otel-collector.yaml:/etc/otel-collector.yaml \
  otel/opentelemetry-collector-contrib:latest \
  --config=/etc/otel-collector.yaml

This is a generic standalone example. The repository’s bundled configurations live under observability/ and are mounted at /etc/otel/collector*.yaml by docker-compose.otel.yml (selected using the --config line); adjust the path if you copy from there.

On a cluster, deploy as a sidecar, DaemonSet, or shared service. Typical patterns:

  • Sidecar deployment. Deploy one Collector per application pod. This provides low latency and an isolated failure domain.
  • DaemonSet deployment. Deploy one Collector per host, where every local application exports to it. This is a good fit for Kubernetes.
  • Shared service deployment. Deploy one fleet of Collectors behind a load balancer. This is the most cost-effective option but adds a network hop.

Configure the Application

export NEMO_LENS_ENABLED=1
export OTEL_EXPORTER_OTLP_ENDPOINT=http://collector.internal:4317

That’s it. Lens discovers the endpoint from the standard env var; no code changes.

Apply Useful Processors

Beyond batch, consider:

processors:
  # Drop noisy spans at the Collector
  filter/drop_health_checks:
    traces:
      span:
        - 'name == "GET /healthz"'
  # Sample smart: keep errors, sample 10% of successes
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: keep-errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: sample-successes
        type: probabilistic
        probabilistic: { sampling_percentage: 10 }
  # Enrich with cluster metadata
  resource:
    attributes:
      - key: cluster.name
        value: prod-us-west
        action: upsert
  # Redact sensitive attributes
  attributes/redact:
    actions:
      - key: user.email
        action: delete

Attach to a pipeline:

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [filter/drop_health_checks, tail_sampling, resource, batch]
      exporters: [otlp/jaeger, file]

Configure Multi-Backend Routing

Send traces to two places simultaneously, such as Jaeger for interactive debugging and W&B Weave for run history:

exporters:
  otlp/jaeger:
    endpoint: jaeger:4317
    tls: { insecure: true }
  otlphttp/wandb:
    endpoint: https://trace.wandb.ai/otel/v1/traces
    headers:
      wandb-api-key: ${env:WANDB_API_KEY}
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/jaeger, otlphttp/wandb]

The application exports to one endpoint (the Collector), and the Collector fans out the telemetry.

Configure Collector-Side Sampling

Instead of sampling at the SDK through OTEL_TRACES_SAMPLER, sample at the Collector. The advantage of this approach is that you can make the decision based on the complete trace (for example, keep all traces containing an error), which the SDK cannot do because it has not seen the whole trace yet.

The tail_sampling processor is the standard tool. See the full tail sampling documentation.

Production Considerations

  • Backpressure. If a backend is slow, the Collector buffers. Configure sending_queue limits to cap memory.
  • TLS. Enable TLS between the SDK and the Collector, and between the Collector and backends, in any multi-tenant setup.
  • Health checks. Enable the health_check extension (as the bundled configurations do) to expose :13133/ and monitor it.
  • Version pinning. The opentelemetry-collector-contrib image changes, so pin to a version and upgrade deliberately.

Debug the Collector

# Increase logging
--set service.telemetry.logs.level=debug
# Enable the debug exporter to see spans on stdout
exporters:
  debug:
    verbosity: detailed
# Add to a pipeline for testing
service:
  pipelines:
    traces:
      exporters: [debug]

The Collector’s own telemetry (:8888/metrics) shows incoming span rates, processor queue depth, and exporter success counts; scrape it with Prometheus to monitor the monitoring.

Send Telemetry to Other Hosted Backends

Configure other popular hosted backends by using standard OpenTelemetry environment variables.

Configure Grafana Cloud

export OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-gateway-<region>.grafana.net/otlp
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic <base64-encoded-instance-id-and-token>"

This routes traces to Tempo, metrics to Mimir, and logs to Loki, all queryable from a unified Grafana UI.

Configure Datadog

export OTEL_EXPORTER_OTLP_ENDPOINT=https://trace.agent.datadoghq.com
export OTEL_EXPORTER_OTLP_HEADERS="DD-API-KEY=<your-api-key>"

Datadog also ships their own Collector preset; see their documentation for advanced configuration.

Configure New Relic

export OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.nr-data.net
export OTEL_EXPORTER_OTLP_HEADERS="api-key=<your-ingest-key>"

Configure Self-Hosted Jaeger or Tempo

Both accept OTLP natively:

export OTEL_EXPORTER_OTLP_ENDPOINT=http://jaeger.internal:4317    # or tempo.internal

Select a Backend

FactorPrefer
Quick local iterationConsole or File
Small team, no infrastructure teamHosted (W&B Weave, Honeycomb, Grafana Cloud)
Training runs tied to W&BW&B Weave
One hosted destination for all three signalsHoneycomb
High-cardinality attribute queries matterHoneycomb
Custom post-processing or async consumptionRabbitMQ
Production with multi-backend routingOTel Collector and your chosen backends
Data residency and complianceSelf-hosted Collector and self-hosted backends
Already have one APM vendorTheir OTLP endpoint

All destinations work the same from NeMo Lens’s perspective; the choice is about cost, operational burden, and integration with your existing stack.

Partition Telemetry by Run

Regardless of the backend, filter by nemo.run.id (auto-set by NeMo Lens) to isolate a specific training run’s data:

  • Jaeger: Use the tag filter nemo.run.id=&lt;value&gt;.
  • Grafana: Use the dashboard variable nemo_run_id.
  • Honeycomb: Use the filter nemo.run.id.
  • Datadog: Use the facet @nemo.run.id.
  • Weave: Use the run-level association through WANDB_ENTITY and WANDB_PROJECT.

Multiple runs land in the same index or project; the attribute is the partition key.

gRPC vs HTTP

OTLP has two transport variants:

# gRPC (default, faster, persistent connections)
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
OTEL_EXPORTER_OTLP_ENDPOINT=http://collector:4317
# HTTP/Protobuf (firewall-friendly, HTTPS works easily)
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_ENDPOINT=http://collector:4318

The NeMo Lens providers.py tries gRPC first, and falls back to HTTP if the gRPC exporter is not installed. If you only installed opentelemetry-exporter-otlp-proto-http, set the protocol explicitly.

Limit Ingested Spans

A per_step run on 1,000 ranks can produce over 100,000 spans per second. Most backends cannot (or will not affordably) ingest that volume. Layer your sampling:

  1. Leaving enabled false on ranks that need not report, so a single rank sends data. This is usually the correct starting point.
  2. The OTel SDK sampler at the trace level, where setting OTEL_TRACES_SAMPLER=parentbased_traceidratio with OTEL_TRACES_SAMPLER_ARG=0.1 keeps 10% of traces.
  3. Collector tail sampling for intelligent decisions, which keeps all errors and samples 1% of successes.

Combine aggressively. It is easier to re-enable telemetry when you are debugging than to pay for ingestion nobody looks at.