Custom Exporters

setup_telemetry supports injecting custom SpanExporter and MetricReader instances, which bypasses the configuration-based construction. This is the extension point for the following use cases:

• In-memory exporters for testing
• Custom exporters for proprietary backends
• Exporters that require fine-grained configuration that NeMo Lens does not expose

Signature

1
setup_telemetry(
2
    config,
3
    rank=0,
4
    world_size=1,
5
    resource_attributes=None,
6
    span_exporter=None,        # optional custom SpanExporter
7
    metric_reader=None,         # optional custom MetricReader
8
)

When provided, these override the config’s exporter field for that signal. You can mix these options by passing a custom span_exporter and letting metrics use the config’s exporter.

Custom Span Exporter

1
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
2
3
exporter = InMemorySpanExporter()
4
5
handle = setup_telemetry(
6
    config,
7
    rank=0,
8
    world_size=1,
9
    span_exporter=exporter,
10
)
11
12
# ... do work ...
13
14
spans = exporter.get_finished_spans()

This is how the NeMo Lens test suite captures spans for assertions.

Custom Metric Reader

1
from opentelemetry.sdk.metrics.export import InMemoryMetricReader
2
3
reader = InMemoryMetricReader()
4
5
handle = setup_telemetry(
6
    config,
7
    rank=0,
8
    world_size=1,
9
    metric_reader=reader,
10
)
11
12
# ... record metrics ...
13
14
data = reader.get_metrics_data()

Write a Custom Exporter

To route telemetry to a backend not supported by OTLP, subclass SpanExporter:

1
from opentelemetry.sdk.trace.export import SpanExporter, SpanExportResult
2
3
class MyBackendExporter(SpanExporter):
4
    def __init__(self, client):
5
        self._client = client
6
7
    def export(self, spans) -> SpanExportResult:
8
        try:
9
            for span in spans:
10
                self._client.send(
11
                    name=span.name,
12
                    trace_id=span.context.trace_id,
13
                    attributes=dict(span.attributes),
14
                )
15
            return SpanExportResult.SUCCESS
16
        except Exception:
17
            return SpanExportResult.FAILURE
18
19
    def shutdown(self) -> None:
20
        self._client.close()
21
22
    def force_flush(self, timeout_millis=30000) -> bool:
23
        return True
24
25
handle = setup_telemetry(config, span_exporter=MyBackendExporter(my_client))

Use the same pattern for MetricReader (refer to the OpenTelemetry documentation).

Custom Exporters and Sampling

Custom exporters plug into the BatchSpanProcessor that NeMo Lens installs. By default (sampler_enabled=False), no span-level sampling occurs, and a custom exporter receives every span produced on an exporting rank. When sampler_enabled=True, NeMo Lens’s RankAwareSampler makes a single per-rank decision at construction time (sampling.py), where all spans on a rank are either kept or dropped together, meaning it is not a per-span sample. If you want a parallel processor regardless of sampling, install your own SpanProcessor as shown below:

1
from opentelemetry import trace
2
from opentelemetry.sdk.trace.export import BatchSpanProcessor
3
4
handle = setup_telemetry(config)
5
provider = trace.get_tracer_provider()
6
provider.add_span_processor(BatchSpanProcessor(MyArchiveExporter()))

NeMo Lens does not expose an extension point for custom processors, so if you need one, add it to the provider directly after setup_telemetry returns. This is a lower-level interface, but it is stable within the OpenTelemetry SDK.

Why This API

Before this extension point, connecting NeMo Lens to a non-OTLP backend required subclassing build_providers or monkey-patching internals. Now it is a supported extension point: pass your exporter, and NeMo Lens does the rest.

The design goal is to keep NeMo Lens’s core surface small (OTLP and console are plenty for most users) while letting power users plug in whatever they need without forking NeMo Lens.