> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/lens/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/lens/_mcp/server. # Configuration `NemoLensConfig` is the single configuration object consumed by `setup_telemetry`. It holds every knob exposed by the library. ## Construction Construct a `NemoLensConfig` object using either environment variables or direct instantiation in Python. ### From Environment ```python from nemo.lens import NemoLensConfig cfg = NemoLensConfig.from_env() ``` Reads `NEMO_LENS_*` env vars. For library-specific prefixes, pass `prefix` and `fallback_prefix`: ```python cfg = NemoLensConfig.from_env( prefix='MEGATRON_OTEL', fallback_prefix='NEMO_LENS', span_group_cls=MegatronSpanGroup, ) ``` The **prefix and fallback** pattern lets each consumer have library-scoped env vars while sharing common defaults. The primary prefix wins; the fallback applies only if the primary is unset. ### Direct Construction ```python cfg = NemoLensConfig( enabled=True, service_name='my-training-run', export_strategy='all_ranks', span_groups='per_step', ) ``` Field validation runs in `__post_init__`: `export_sample_rate` must be in `[0.0, 1.0]`, otherwise `ValueError`. ## Fields A `NemoLensConfig` object contains fields divided into several functional categories. ### Core | Field | Default | Description | | -------------- | -------- | -------------------------------------------------------- | | `enabled` | `False` | Master toggle. Must be `True` to activate any telemetry. | | `service_name` | `"nemo"` | OTLP service name. Overridden by `OTEL_SERVICE_NAME`. | ### Export Strategy Controls which ranks send telemetry to the collector. Three strategies are available: `single_rank` (default), `all_ranks`, and `sampled`. See [Sampling](/nemo/lens/user-guide/sampling) for detailed semantics, when to use each, and how they compose with OTel SDK samplers. Unknown strategy names raise a `ValueError` at `setup_telemetry` time, not at configuration construction; register custom strategies before initializing telemetry. See [Custom Export Strategies](/nemo/lens/user-guide/custom-export-strategies). | Field | Default | Description | | -------------------- | --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `export_strategy` | `"single_rank"` | `"single_rank"`, `"all_ranks"`, `"sampled"`, `"first_rank_per_node"`, or any name registered through [`register_export_strategy`](/nemo/lens/user-guide/custom-export-strategies). | | `export_rank` | `-1` | For `single_rank`: which rank exports. `-1` means the last rank. | | `export_sample_rate` | `1.0` | For `sampled`: fraction of ranks in `[0.0, 1.0]`. Validated at config time. | | `sampler_enabled` | `False` | Install `RankAwareSampler` on the TracerProvider for SDK-level per-rank filtering. See [Sampling](/nemo/lens/user-guide/sampling). | ### Signal Toggles | Field | Default | Description | | ----------------- | ------- | --------------------------------------------------------------------------------------- | | `traces_enabled` | `True` | Enable trace spans. | | `metrics_enabled` | `True` | Enable metric instruments. | | `logs_enabled` | `False` | Enable the OTel log bridge. See [Logging Bridge](/nemo/lens/user-guide/logging-bridge). | ### Granularity | Field | Default | Description | | ------------- | ----------- | ------------------------------------------------------------------------------------------------------------- | | `span_groups` | `"default"` | Comma-separated spec of preset keywords or group names. See [Span Groups](/nemo/lens/user-guide/span-groups). | ### Backend | Field | Default | Description | | ---------- | -------- | --------------------------------------------------------------------------------- | | `exporter` | `"otlp"` | `"otlp"` (gRPC, falls back to HTTP) or `"console"` (stdout, for local debugging). | With `exporter="console"` (the env var `NEMO_LENS_EXPORTER=console`), NeMo Lens uses the `ConsoleSpanExporter` and `ConsoleMetricExporter` from the SDK, which print spans and metrics to stdout. Any value other than `"otlp"` or `"console"` raises a `ValueError("Unknown exporter type: ...")` when the providers are built. ### Identification | Field | Default | Description | | -------- | ------- | ---------------------------------------------------------------------------------------------- | | `run_id` | auto | Unique run ID. Auto-generated from `SLURM_JOB_ID` or a UUID if empty. Shared across all ranks. | | `user` | `""` | Optional user/team label. Emitted as `nemo.user.id`. | ### W\&B Weave | Field | Default | Description | | --------------- | ------- | --------------------------------------------------------------- | | `wandb_entity` | `""` | W\&B team/user name — set as `wandb.entity` resource attribute. | | `wandb_project` | `""` | W\&B project name — set as `wandb.project` resource attribute. | ## Environment Variables Most configuration fields have a corresponding `<PREFIX>_<KEY>` env var. Three are exceptions that bypass the prefix and fallback model entirely: `service_name` reads the bare `OTEL_SERVICE_NAME`, and `wandb_entity` and `wandb_project` read the bare `WANDB_ENTITY` and `WANDB_PROJECT` (without a prefix or fallback). Note also that the `user` field's env var is `<PREFIX>_USER_ID` (e.g., `NEMO_LENS_USER_ID`), not `_USER`. Using `NEMO_LENS` as the prefix: | Variable | Field | | ------------------------------ | -------------------- | | `NEMO_LENS_ENABLED` | `enabled` | | `NEMO_LENS_EXPORT_STRATEGY` | `export_strategy` | | `NEMO_LENS_EXPORT_RANK` | `export_rank` | | `NEMO_LENS_EXPORT_SAMPLE_RATE` | `export_sample_rate` | | `NEMO_LENS_SAMPLER_ENABLED` | `sampler_enabled` | | `NEMO_LENS_TRACES_ENABLED` | `traces_enabled` | | `NEMO_LENS_METRICS_ENABLED` | `metrics_enabled` | | `NEMO_LENS_LOGS_ENABLED` | `logs_enabled` | | `NEMO_LENS_SPAN_GROUPS` | `span_groups` | | `NEMO_LENS_EXPORTER` | `exporter` | | `NEMO_LENS_RUN_ID` | `run_id` | | `NEMO_LENS_USER_ID` | `user` | | `WANDB_ENTITY` | `wandb_entity` | | `WANDB_PROJECT` | `wandb_project` | Boolean parsing accepts: `1`/`0`, `true`/`false`, `yes`/`no`, `on`/`off` (case-insensitive). Any other value raises `ValueError`. ## Standard OTel SDK Env Vars Standard OTel SDK env vars are honored; most are read by the SDK directly, but a few are consumed by NeMo Lens's provider-building code: `OTEL_EXPORTER_OTLP_PROTOCOL` (and the signal-specific `OTEL_EXPORTER_OTLP_{TRACES,METRICS,LOGS}_PROTOCOL`, which wins; default `grpc`) selects the OTLP transport, and `OTEL_METRIC_EXPORT_INTERVAL` (default `10000` ms) sets the metric reader's export interval: | Variable | Example | | ----------------------------- | ------------------------------------ | | `OTEL_SERVICE_NAME` | `my-training-run` | | `OTEL_EXPORTER_OTLP_ENDPOINT` | `http://localhost:4317` | | `OTEL_EXPORTER_OTLP_PROTOCOL` | `grpc` or `http/protobuf` | | `OTEL_EXPORTER_OTLP_HEADERS` | `Authorization=Bearer <token>` | | `OTEL_TRACES_SAMPLER` | `parentbased_traceidratio` | | `OTEL_TRACES_SAMPLER_ARG` | `0.1` | | `OTEL_METRIC_EXPORT_INTERVAL` | `10000` (ms) | | `OTEL_SDK_DISABLED` | `true` | In addition, `build_providers` reads the non-prefixed `DEPLOYMENT_ENV` (falling back to `ENVIRONMENT`) and, when set, emits it as the `deployment.environment` resource attribute. ## `setup_telemetry` Signature ```python setup_telemetry( config: NemoLensConfig, rank: int = 0, world_size: int = 1, resource_attributes: dict | None = None, span_exporter=None, metric_reader=None, export_strategy: ExportStrategy | None = None, _allow_reinit: bool = False, ) -> TelemetryHandle ``` | Parameter | Description | | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `config` | The `NemoLensConfig` object, typically from `from_env()`. | | `rank` and `world_size` | Distributed position, used for export strategy and resource attributes. | | `resource_attributes` | Extra attributes to merge into the OTel `Resource` (become Jaeger "Process" tags). | | `span_exporter` | Optional custom `SpanExporter`, bypasses config-based construction. See [Custom Exporters](/nemo/lens/user-guide/custom-exporters). | | `metric_reader` | Optional custom `MetricReader`, bypasses config-based construction. | | `export_strategy` | Optional callable `(config, rank, world_size) -> bool` that bypasses the registry-based strategy dispatch (per-call override, no global registration needed). See [Custom Export Strategies](/nemo/lens/user-guide/custom-export-strategies). | | `_allow_reinit` | Escape hatch for testing only, which bypasses the [double-initialization guard](/nemo/lens/design/double-init-guard). | Returns a `TelemetryHandle` exposing: * `.tracer` and `.meter`: read-only properties holding the OTel tracer and meter (no-op objects on non-exporting ranks). * `.is_exporting`: a `bool` indicating whether this rank built real exporting providers. * `.shutdown(timeout_ms=5000)`: force-flushes and shuts down both the tracer and meter providers. Call `setup_telemetry` once per process. A second call with `config.enabled=True` raises `RuntimeError`. See [Double-Init Guard](/nemo/lens/design/double-init-guard) for rationale.