> This page is for version Nightly. > For other versions, use one of these documentation indexes: > - 0.1.0 (default): https://docs.nvidia.com/nemo/lens/0.1.0/llms.txt > - Nightly: https://docs.nvidia.com/nemo/lens/nightly/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/lens/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/lens/_mcp/server. # Resource Detection OpenTelemetry [Resources](https://opentelemetry.io/docs/specs/otel/resource/sdk/) describe the entity producing telemetry: service name, version, host, cloud provider, and other properties. NeMo Lens auto-detects a handful of environment-specific attributes to make runs filterable without manual configuration. ## Default Resource Attributes Every exporter-rank process emits these attributes (set in `providers.py:build_providers`): | Attribute | Source | Example | | ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------- | | `service.name` | `config.service_name` (populated from `OTEL_SERVICE_NAME` through `NemoLensConfig.from_env()`, default `"nemo"`) | `"megatron-lm"` | | `service.version` | `nemo.lens.__version__` | `"0.1.0"`, `"0.1.0.post3+gabc1234"` | | `service.instance.id` | `"{run_id}-rank{rank}"` when `nv.dl.rank` is available through either channel and `service.instance.id` is not explicit; otherwise left to the SDK | `"abc123-rank0"` | | `nv.dl.rank` | caller-supplied `resource_attributes` or `OTEL_RESOURCE_ATTRIBUTES` | `0` | | `nv.dl.world_size` | caller-supplied `resource_attributes` or `OTEL_RESOURCE_ATTRIBUTES` | `8` | | `nemo.run.id` | `config.run_id` (auto-generated if empty) | `"abc123"` | | `nemo.user.id` | `config.user` (if set) | `"my-team"` | | `deployment.environment` | `DEPLOYMENT_ENV` or `ENVIRONMENT` env var | `"production"` | ## Auto-Detected Resource Attributes The `nemo.lens.resources.detect_resource()` function merges attributes from three sources: ### Local Process Attributes with `detect_local()` | Attribute | Description | | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `host.name` | Hostname from `socket.gethostname()` | | `process.pid` | Python's `os.getpid()` | | `host.gpu.count` | GPU count from `CUDA_VISIBLE_DEVICES` or `nvidia-smi`. Best-effort; omitted entirely when undetectable (no `nvidia-smi` and `CUDA_VISIBLE_DEVICES` unset). An empty `CUDA_VISIBLE_DEVICES` reports `0`. | ### Slurm Environment Attributes with `detect_slurm()` This helper returns attributes when `SLURM_JOB_ID` is set or when launch-provided `OTEL_RESOURCE_ATTRIBUTES` contains a selected Slurm or run identity value. With a local job ID, it derives Slurm resource attributes from the local environment. Inherited values can activate detection independently and take precedence over caller-provided `resource_attributes` defaults and locally derived fallback values. Empty inherited values are ignored. | Attribute | Source env var | | ----------------------------- | ------------------------------------------------------------------------------------ | | `slurm.job.id` | `SLURM_JOB_ID` | | `slurm.job.id.raw` | Raw `SLURM_JOB_ID` | | `slurm.array.job_id` | `SLURM_ARRAY_JOB_ID`, or `SLURM_JOB_ID` for non-array jobs | | `slurm.array.task_id` | `SLURM_ARRAY_TASK_ID`, or `"0"` for non-array jobs | | `slurm.array.count` | `SLURM_ARRAY_TASK_COUNT`, or `1` when unset | | `slurm.sluid` | `LENS_SLURM_SLUID` or `NV_LAUNCH_SLUID` | | `slurm.array.sluid` | `LENS_SLURM_ARRAY_SLUID` or `NV_LAUNCH_ARRAY_SLUID` | | `slurm.job.name` | `SLURM_JOB_NAME` | | `slurm.cluster.name` | `SLURM_CLUSTER_NAME` | | `slurm.partition` | `SLURM_JOB_PARTITION` or `SLURM_PARTITION` | | `slurm.head_node.name` | `SLURMD_NODENAME` outside a Slurm step, or inherited from launch attributes | | `slurm.nnodes` | `SLURM_JOB_NUM_NODES` or `SLURM_NNODES` | | `slurm.ntasks` | `SLURM_NTASKS` | | `slurm.restart_count` | `SLURM_RESTART_COUNT`, or `0` when unset | | `slurm.job.user` | `SLURM_JOB_USER` | | `slurm.job.account` | `SLURM_JOB_ACCOUNT` | | `slurm.job.qos` | `SLURM_JOB_QOS` | | `slurm.job.reservation` | `SLURM_JOB_RESERVATION` | | `slurm.segment` | `SLURM_SEGMENT` | | `slurm.topology.addr` | `SLURM_TOPOLOGY_ADDR` | | `slurm.topology.addr_pattern` | `SLURM_TOPOLOGY_ADDR_PATTERN` | | `nv.dl.job.uuid` | Derived from Slurm cluster and submitted job ID, or inherited from launch attributes | | `nv.dl.run.uuid` | Inherited from launch attributes | `detect_slurm()` returns `slurm.array.count`, `slurm.nnodes`, `slurm.ntasks`, and `slurm.restart_count` as integers whether they are derived locally or inherited through `OTEL_RESOURCE_ATTRIBUTES`. Other inherited values remain strings. Complete signed ASCII base-10 strings are accepted. An empty inherited value, or any other form for one of those four attributes, is discarded so that a valid locally derived value can be used as the fallback. Locally detected integer values use the same conversion rule. If no valid local value exists, the attribute is omitted. The table above is the complete allowlist of attributes that `detect_slurm()` reads from `OTEL_RESOURCE_ATTRIBUTES`; every accepted key has an explicit normalizer. Other resource attributes are ignored by this helper, but they remain in `OTEL_RESOURCE_ATTRIBUTES` and are still read independently by the OpenTelemetry SDK. Retired keys such as `slurm.nodelist`, `slurm.cluster`, and `slurm.torchelastic.restart_count` are not emitted for new telemetry. ### Kubernetes Environment Attributes with `detect_kubernetes()` These attributes are active when `KUBERNETES_SERVICE_HOST` is set or the `/var/run/secrets/kubernetes.io` directory exists. This helper maps the following variables: | Attribute | Source env var | | -------------------- | ----------------------------------------- | | `k8s.namespace.name` | `K8S_NAMESPACE` | | `k8s.pod.name` | `K8S_POD_NAME` (falls back to `HOSTNAME`) | | `k8s.pod.uid` | `K8S_POD_UID` | | `k8s.node.name` | `K8S_NODE_NAME` | | `k8s.container.name` | `K8S_CONTAINER_NAME` | | `k8s.job.name` | `K8S_JOB_NAME` | `HOSTNAME` is used as a fallback for `k8s.pod.name` only when `K8S_POD_NAME` is unset. ## Add Custom Resource Attributes Pass the `resource_attributes` argument to `setup_telemetry`: ```python handle = setup_telemetry( config, resource_attributes={ 'nv.dl.rank': rank, 'nv.dl.world_size': world_size, 'dl.tensor_parallel.size': 4, 'dl.pipeline_parallel.size': 2, 'dl.data_parallel.size': 8, 'megatron.num_layers': 32, 'megatron.precision': 'bf16', }, ) ``` These attributes merge with the auto-detected set. In Jaeger, they appear as **Process** tags and are filterable across every span in the run. > **Warning** > > NeMo Lens does not derive `nv.dl.rank` — it has no notion of a rank. If you omit > it, a warning is logged at startup: telemetry from this process cannot be > filtered by rank in the collector, and `service.instance.id` is not rank-derived. > > The process itself stays identifiable either way — `host.name` and `process.pid` > are always attached, and `service.instance.id` is left to the SDK, which supplies > a per-process UUID from version 1.43.0 onward. What you lose is rank-based > filtering and correlation, not the ability to tell two processes apart. > > NeMo Lens deliberately does **not** read `RANK` or `WORLD_SIZE` to fill the gap — > that would restore the rank-awareness it was designed to shed, and would be wrong > for any caller whose rank is not the process's global rank. > > The warning is suppressed (dropped to `DEBUG`) when you supply your own > `service.instance.id`, through either channel. A launcher agent or sidecar that > names itself and has no training rank to claim is a correct configuration, not > one to nag about once per node per job. A genuinely single-process caller can > also pass rank `0`. Pass the `resource_attributes` argument to `setup_telemetry`: ```python handle = setup_telemetry( config, resource_attributes={ 'nv.dl.rank': rank, 'nv.dl.world_size': world_size, 'dl.tensor_parallel.size': 4, 'dl.pipeline_parallel.size': 2, 'dl.data_parallel.size': 8, 'megatron.num_layers': 32, 'megatron.precision': 'bf16', }, ) ``` These attributes merge with the auto-detected set. In Jaeger, they appear as **Process** tags and are filterable across every span in the run. ## Two Channels for Supplying Attributes Resource attributes reach NeMo Lens two ways, and both are first-class: | Channel | Use when | | --------------------------------------------- | ------------------------------------------------------------------------------ | | `resource_attributes=` on `setup_telemetry` | You control the call site. | | `OTEL_RESOURCE_ATTRIBUTES` in the process env | You do not — a spawned worker, an `exec`'d relaunch, a launcher-started agent. | The env var is the standard OTel channel (`key=value,key=value`, percent-encoded) and is the **only one that survives a `spawn` or `exec`**, so it is how a child process supplies its own `nv.dl.rank` when it has no `setup_telemetry` call site to reach. NeMo Lens resolves both before deriving anything from them, so an env-supplied rank produces the same `service.instance.id` a caller-supplied one would. On conflict the caller's dict wins, matching what the OTel SDK does with the same inputs. > **Note** > > The Resource carrier parser returns strings. During Resource construction, > Lens restores integers for these fields: > > * `nv.dl.rank`, `nv.dl.world_size`, and `nv.dl.local_rank` > * `nv.dl.topology.size.tp`, `nv.dl.topology.size.pp`, and `nv.dl.topology.size.dp` > * `nv.dl.training.config.global_batch_size`, > `nv.dl.training.config.micro_batch_size`, and `nv.dl.training.config.sequence_length` > * `nv.dl.training.target.train_iters`, `nv.dl.training.target.train_samples`, > and `nv.dl.training.target.train_tokens` > * `nv.gpu.index` and `nv.gpu.memory_total` > > Complete signed base-10 strings are accepted. Invalid values remain visible and > produce a warning. Unknown fields, UUIDs, IDs, software versions, and GPU compute > capability remain strings; Lens does not infer types from a numeric-looking value > such as `"007"`. Slurm retains the selection and invalid-value fallback described > above while using the same strict integer conversion. ```bash # parent, before spawning the child export OTEL_RESOURCE_ATTRIBUTES="nv.dl.rank=5,nv.dl.world_size=8,nemo.run.id=abc123" ``` > **Note** > > Hand-rolling that string is easy to get subtly wrong — values need > percent-encoding, keys must **not** be encoded, and an inherited value has to be > merged without piling up duplicate keys. Use the helpers described in > [Propagate Attributes to a Child Process](#propagate-attributes-to-a-child-process) > instead of building it yourself. ## Propagate Attributes to a Child Process A process that cannot reach a `setup_telemetry` call site — a spawned checkpoint worker, an `exec`'d relaunch, a launcher-started agent — receives its resource attributes through `OTEL_RESOURCE_ATTRIBUTES`. That is the only channel that survives the boundary: `multiprocessing.Process` has no `env` parameter at all, so mutating `os.environ` before `start()` is not one option among several. When both channels provide the same key, the caller's `resource_attributes` mapping wins, matching the OpenTelemetry SDK. Use the Resource carrier helpers rather than formatting the value by hand: ```python from nemo.lens.resources.attributes import ( get_otel_resource_attributes, publish_otel_resource_attributes, ) from nemo.lens.semconv.encoding import compose_attributes current = get_otel_resource_attributes() defaults = {'service.name': 'checkpoint-worker'} overrides = {'nv.dl.role': 'ckpt_worker'} resolved = compose_attributes(current, defaults=defaults, overrides=overrides) # Publish the exact resolved map while the child starts, then restore the environment. with publish_otel_resource_attributes(resolved): ctx.Process(target=worker).start() ``` When a consumer needs a child carrier string without changing the environment, use the pure string transformation directly: ```python from nemo.lens.resources.attributes import ( extend_otel_resource_attributes, ) child_carrier = extend_otel_resource_attributes( selected_carrier, defaults={'nv.dl.rank': rank}, overrides={'nv.dl.role': 'worker'}, ) ``` Composition follows ordinary dictionary precedence: defaults apply only when a key is absent, current values replace defaults, and overrides replace both. Empty strings and `None` are ordinary overriding values. Encoding retains an empty string as `key=` and omits a `None` entry; it does not reapply a default. The function returns a new dictionary and does not mutate any input mapping. Type reconstruction remains a separate Resource operation. Publication installs only the encoded resolved map. It never retains current attributes implicitly, so include them in composition when they should survive. Passing an already resolved map requires no second decode or merge. `extend_otel_resource_attributes()` parses, composes, and formats a string without changing the environment. `set_otel_resource_attributes()` formats a resolved map and permanently replaces the selected environment value; it does not merge with the old value. `publish_otel_resource_attributes()` temporarily replaces the value and restores the exact incoming state. A resolved map needs no second parse or composition before it is set or published. Publisher contexts may nest within one thread, but they must not overlap across threads because environment variables are process-global. Serialize publication when multiple threads can launch children. ### What It Handles | Concern | Behavior | | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Separators in values | Percent-encoded. Unencoded, a comma truncates its value *and* invents a key — a run id of `exp/2026-01,seed=7` arrives as `nemo.run.id="exp/2026-01"` plus a spurious `seed="7"`, with no error. | | Value types | `str`, `bool`, `int` and `float` are stringified. An empty string is retained as `key=` and `None` is omitted. Other values — `bytes`, a list, a dict — are dropped with a warning, because this channel is string-typed and a repr is not a round trip. | | An inherited value | Extend applies defaults → parsed current values → overrides. Later layers win even when the value is empty or `None`. Parsing and formatting may canonicalize escaping and collapse duplicate keys to the last decoded value. | | Repeated calls | Parse/compose/format produces one entry per decoded key, so repeated transformations do not accumulate stale duplicates. | | Non-string keys | Dropped with a warning. `str(key)` would ship `None` as the literal attribute name `"None"`. | | Separators in *keys* | Dropped with a warning. The SDK unquotes only the value half, so such a key has no representation; encoding it would ship the escape sequence as the attribute name. No semconv name contains `,` or `=`. | > **Note** > > Values arrive from the environment carrier as **strings** because the format has > no type metadata. During Resource construction, NeMo Lens restores the documented > integer Resource fields. Unknown fields remain strings rather than being inferred > from numeric-looking text. For selected Slurm and run identity keys, launch-provided `OTEL_RESOURCE_ATTRIBUTES` values are authoritative. Treat `resource_attributes` as consumer defaults for those keys. ## Use Cases ### Filter by Global Rank In Jaeger, use: `nv.dl.rank=0` ### Compare Distinct Runs In Grafana, configure `nemo.run.id` as a dashboard variable, list all values, and then select the specific runs to compare. ### Filter by Parallelism Configuration In Jaeger, use: `dl.tensor_parallel.size=4 AND dl.pipeline_parallel.size=2` Because these are resource attributes instead of span attributes, they apply to every span without cluttering the span view. ## Attribute Conventions * **Use standard names.** Apply standard OTel attribute names where they exist, such as `service.*`, `k8s.*`, and `host.*`. * **Use distributed learning prefixes.** Apply the `nv.dl.*` prefix for participant resource attributes and the `dl.*` prefix for other training-specific attributes that are shared across consumers. * **Use project-specific prefixes.** Apply the `<project>.*` prefix for project-specific attributes, such as `megatron.*`, `rl.*`, and `gym.*`. See [semconv](/nemo/lens/nightly/design/semantic-conventions) for the full attribute namespace conventions. ## Resource Detection Order The `detect_resource()` function merges resource attributes in the following order: local, Slurm, and then Kubernetes. If a key collision occurs, the attribute from the later source takes precedence. In practice, collisions are rare because each layer uses its own namespace. ## Run Telemetry Locally On a development machine with no Slurm or Kubernetes environment, only the `detect_local()` helper executes. Run identifiers are automatically generated as UUID values, so you can still filter by `nemo.run.id` to isolate a single local run. ## Deriving an attempt UUID Use `from nemo.lens import derive_nv_dl_run_uuid` to derive an attempt UUID from scheduler identity or a supplied `run_id`. Call it when the application establishes an attempt. This standalone helper does not configure telemetry or publish attributes; callers choose where to record the result.