Configuration

NemoLensConfig is the single configuration object consumed by setup_telemetry. It holds every knob exposed by the library.

Construction

Construct a NemoLensConfig object using either environment variables or direct instantiation in Python.

From Environment

from nemo.lens import NemoLensConfig
cfg = NemoLensConfig.from_env()

Reads NEMO_LENS_* env vars. For library-specific prefixes, pass prefix and fallback_prefix:

cfg = NemoLensConfig.from_env(
    prefix='MEGATRON_OTEL',
    fallback_prefix='NEMO_LENS',
)

The prefix and fallback pattern lets each consumer have library-scoped env vars while sharing common defaults. The primary prefix wins; the fallback applies only if the primary is unset.

Direct Construction

cfg = NemoLensConfig(
    enabled=True,
    service_name='my-training-run',
    span_groups='per_step',
)

Fields

A NemoLensConfig object contains fields divided into several functional categories.

Core

Restricting Which Processes Export

NeMo Lens has no rank-based export filtering and no notion of a rank. Every process that calls setup_telemetry with enabled=True exports.

To keep most ranks quiet in a large job, decide it in the caller:

from nemo.lens.semconv import NV_DL_RANK, NV_DL_WORLD_SIZE
cfg = NemoLensConfig.from_env()
cfg.enabled = cfg.enabled and rank == world_size - 1   # last rank only
setup_telemetry(
    cfg,
    resource_attributes={NV_DL_RANK: rank, NV_DL_WORLD_SIZE: world_size},
)

Alternatively, let every rank export and filter in the collector on the nv.dl.rank resource attribute, or reduce volume with the OTel SDK’s own OTEL_TRACES_SAMPLER. Enabling telemetry on every rank of a large job without one of these in place multiplies your span volume by the world size.

Signal Toggles

Granularity

Backend

With exporter="console" (the env var NEMO_LENS_EXPORTER=console), NeMo Lens uses the ConsoleSpanExporter, ConsoleMetricExporter, and ConsoleLogExporter from the SDK, which print spans, metrics, and log records to stdout. NeMo Lens overrides the SDK’s default formatter on all three: the SDK pretty-prints with indent=4, whereas NeMo Lens emits one compact JSON object per line. Redirecting stdout therefore yields real JSONL that line-oriented tooling can consume without reassembling records. Any value other than "otlp" or "console" raises a ValueError("Unknown exporter type: ...") when the providers are built.

Identification

W&B Weave

Environment Variables

Most configuration fields have a corresponding <PREFIX>_<KEY> env var. Three are exceptions that bypass the prefix and fallback model entirely: service_name reads the bare OTEL_SERVICE_NAME, and wandb_entity and wandb_project read the bare WANDB_ENTITY and WANDB_PROJECT (without a prefix or fallback). Note also that the user field’s env var is <PREFIX>_USER_ID (e.g., NEMO_LENS_USER_ID), not _USER. Using NEMO_LENS as the prefix:

Boolean parsing accepts: 1/0, true/false, yes/no, on/off (case-insensitive). Any other value raises ValueError.

Standard OTel SDK Env Vars

Standard OTel SDK env vars are honored; most are read by the SDK directly, but a few are consumed by NeMo Lens’s provider-building code: OTEL_EXPORTER_OTLP_PROTOCOL (and the signal-specific OTEL_EXPORTER_OTLP_{TRACES,METRICS,LOGS}_PROTOCOL, which wins; default grpc) selects the OTLP transport, and OTEL_METRIC_EXPORT_INTERVAL (default 10000 ms) sets the metric reader’s export interval. OTEL_RESOURCE_ATTRIBUTES is read by the SDK but consulted by NeMo Lens too, because it is the only channel that survives a spawn — see Resource Detection:

In addition, build_providers reads the non-prefixed DEPLOYMENT_ENV (falling back to ENVIRONMENT) and, when set, emits it as the deployment.environment resource attribute.

setup_telemetry Signature

setup_telemetry(
    config: NemoLensConfig,
    *,
    resource_attributes: dict | None = None,
    span_exporter=None,
    metric_reader=None,
    _allow_reinit: bool = False,
) -> TelemetryHandle

Returns a TelemetryHandle exposing:

• .tracer and .meter: read-only properties holding the OTel tracer and meter (no-op objects when telemetry is disabled).
• .is_exporting: a bool indicating whether real exporting providers were built. Now equivalent to config.enabled.
• .shutdown(timeout_ms=5000): force-flushes and shuts down both the tracer and meter providers.