> This page is for version Nightly.
> For other versions, use one of these documentation indexes:
> - 0.1.0 (default): https://docs.nvidia.com/nemo/lens/0.1.0/llms.txt
> - Nightly: https://docs.nvidia.com/nemo/lens/nightly/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/lens/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/lens/_mcp/server.

# Semantic Conventions

`nemo.lens.semconv` centralizes the attribute names NeMo Lens emits. The
`attributes` module owns names and version metadata, while `resources` owns
Resource field sets and type mappings. Existing imports from `nemo.lens.semconv`
remain supported. Two categories of attributes coexist:

1. **Standard OTel semconv**: `gen_ai.*` and `k8s.*`. These track upstream OTel specs. (Resource attributes like `service.*` and `host.*` are also standard OTel names but are set as literals in `providers.py` and `resources/local.py` rather than mirrored as constants.)
2. **NeMo custom namespaces**: `nv.dl.*` / `dl.*` (distributed learning), `rl.*`, `gym.*`, `slurm.*`, `nemo.*`, `wandb.*`. These are NeMo-specific extensions that do not exist upstream.

## Why Constants Instead of Strings

Using constants instead of raw strings provides three key benefits:

1. **Grep-ability**: renaming an attribute across the codebase means changing one constant, not every call site.
2. **Type safety** (weak but real): `NV_DL_RANK` is an exported name; typos become `ImportError`s. `"nv.dl.rank"` typos become silent data loss.
3. **Central definitions**: one package lists the attribute names and Resource
   type mappings. This keeps them reviewable and documented.

Callers who want the string can use `NV_DL_RANK` directly; Python strings-as-constants have no boxing cost.

## Version Tracking

```python
SEMCONV_VERSION = "1.29.0"
```

This constant documents which upstream OTel semconv version the standard namespaces (`gen_ai.*`, `k8s.*`) are aligned with. Upstream bumps namespace conventions periodically; `gen_ai.*` graduated from experimental to stable around 1.30.

When upgrading NeMo Lens to a new semconv version, complete these tasks:

1. Review the upstream changelog for renamed or removed attributes.
2. Update `SEMCONV_VERSION`.
3. Update any changed constants.
4. Add the version bump to the NeMo Lens changelog so consumers know to update.

## Resource Types and Attribute Carriers

`semconv/resources.py` defines explicit type mappings for Resource fields.
`normalize_resource_attributes()` applies those mappings after decoding an
environment carrier. Unknown values are unchanged, so numeric-looking identifiers
such as `"007"` stay strings. Every mapped Resource integer field, including
Slurm fields, uses one converter that accepts native integers or complete signed
ASCII base-10 strings. Invalid DL/GPU values remain visible and produce a
warning. Invalid Slurm values are omitted so locally detected values can be
used; the conversion rule is shared while callers retain their failure policy.

`semconv/encoding.py` contains generic dictionary composition and explicit
converter lookup. The `resources/attributes.py` module owns formatting and
parsing for the comma-separated, percent-encoded `OTEL_RESOURCE_ATTRIBUTES`
carrier. Parsing returns strings. Type reconstruction is a separate step and
always uses a caller-selected mapping; the Resource codec does not infer types.

## Stability Markers

From `semconv/attributes.py`:

```
# gen_ai.*  — Experimental (upstream, stabilising in semconv 1.30+)
# k8s.*     — Stable (upstream)
# nv.dl.*   — NeMo custom (stable within NeMo ecosystem)
# dl.*      — NeMo custom (stable within NeMo ecosystem)
# rl.*      — NeMo custom (stable within NeMo ecosystem)
# gym.*     — NeMo custom (stable within NeMo ecosystem)
# slurm.*   — NeMo custom (stable within NeMo ecosystem)
# nemo.*    — NeMo custom (stable within NeMo ecosystem)
# wandb.*   — NeMo custom (stable within NeMo ecosystem)
```

"Stable" for custom namespaces means these names will not change in minor releases. Breaking changes bump the major version.

"Experimental" for `gen_ai.*` matches upstream; OTel considers them stable-in-practice but reserves the right to tweak until the full semconv 1.30 stabilization.

## Namespace Conventions

NeMo Lens organizes attributes into standard upstream namespaces and NeMo-specific custom namespaces.

### Standard Upstream Namespaces

Follow upstream OTel spec exactly. Do not redefine, and do not rename. If upstream says `gen_ai.request.model`, NeMo Lens uses `gen_ai.request.model`.

### `nv.dl.*` / `dl.*` Distributed Learning

Shared across Megatron-LM, NeMo RL, NeMo Gym. Anything a distributed training job needs:

```
nv.dl.job.uuid               — stable submitted-job identifier
nv.dl.run.uuid               — stable run-attempt identifier
nv.dl.rank                   — global rank resource attribute
nv.dl.world_size             — total ranks resource attribute
nv.dl.local_rank             — rank on this node resource attribute
nv.dl.role                   — trainer or worker process role
nv.dl.provider.name          — training framework or provider name
nv.dl.topology.size.{tp,pp,dp}
                             — tensor, pipeline, and data parallel sizes (integers)
nv.dl.training.config.global_batch_size
nv.dl.training.config.micro_batch_size
nv.dl.training.config.sequence_length
                             — stable integer training configuration
nv.dl.training.config.optimizer
nv.dl.training.config.recompute_granularity
                             — stable text training configuration
nv.dl.training.target.{train_iters,train_samples,train_tokens}
                             — configured integer training targets
nv.dl.software.{torch,cuda,nccl,transformer_engine}
                             — application-observed software versions
dl.tensor_parallel.{rank,size}
dl.pipeline_parallel.{rank,size}
dl.data_parallel.{rank,size}
dl.iteration                 — training iteration
dl.microbatch_id             — microbatch index
dl.batch_size, dl.sequence_length
dl.loss, dl.grad_norm, dl.learning_rate
dl.throughput_tflops, dl.throughput_tokens_per_sec
```

### `<project>.*` Project-Specific Attributes

Use project-specific namespaces for attributes that apply to one consumer rather than the full NeMo ecosystem:

* `megatron.*`: Megatron-LM consumer-specific span attributes (e.g., model architecture `megatron.num_layers`/`megatron.hidden_size`, iteration-level `megatron.skipped`/`megatron.update_successful`). These are set as inline string attributes in the Megatron-LM fork and are **not** defined as constants in `nemo.lens.semconv`. The semconv package holds the shared and standard namespaces (`dl.*`, `gen_ai.*`, `rl.*`, `gym.*`, `slurm.*`, `nemo.*`, `wandb.*`, `k8s.*`); for `rl.*` it holds the span-attribute names only (see below).
* `rl.*` — RL-specific. The **span-attribute** names (`rl.algorithm`, `rl.reward`, `rl.generation.backend`, `rl.num_rollouts`) are defined here as constants. The **metric** series are **owned by the NeMo-RL consumer**.

  NeMo-RL declares its `rl.*` metric series through the metric registry (`register_metric_group`) in
  its own tree, so those names are **not** defined here. Among them, `rl.grad_norm`,
  `rl.learning_rate` and `rl.throughput.tokens_per_sec` are RL-scoped counterparts to quantities the
  `dl.*` namespace registers, describing the *policy update* specifically, so an RL job can report
  them without claiming the shared `dl.*` series that a co-resident trainer owns. The first two are
  deliberate spelling matches for `dl.grad_norm`/`dl.learning_rate`, so the two namespaces differ
  only by prefix. Throughput is the exception: `rl.*` uses the dotted `rl.throughput.tokens_per_sec`
  to match the `rl.response_length.mean`/`rl.generation.duration_ms` subgroup convention, whereas
  `dl.*` uses the flat `dl.throughput_tokens_per_sec`. Query the two by their own names rather than
  by swapping the prefix.
* `gym.*` — Gym server-specific (`verify.success_rate`, `rollout.batch_size`)

### `nemo.*` NeMo-Wide Identification

* `nemo.run.id` — unique run identifier, shared across all ranks
* `nemo.user.id` — optional team/user label
* `nemo.span.truncated` — optional; set to `true` only on a span that was still open at telemetry shutdown and was force-closed. Its end timestamp is the shutdown time, not the time the work finished, so exclude these spans from duration statistics. Absent on every normally ended span. See [Production](/nemo/lens/nightly/user-guide/production-checklist).

### Environment Namespaces

* `slurm.*` — SLURM job attributes
* `k8s.*` — Kubernetes pod/node attributes (standard OTel)
* `wandb.*` — W\&B Weave integration metadata

## Adding New Attributes

Checklist before adding a constant:

1. **Is there an upstream OTel semconv name?** Use it.
2. **Is this shared across consumers?** Use `dl.*` or another shared namespace.
3. **Is this project-specific?** Use `<project>.*`. Such namespaces are set inline in the consumer fork (e.g. Megatron-LM), not as constants in `semconv/attributes.py`.
4. **Is this a metric or a span attribute?** Metric instruments go in `instruments/`; attribute names go in `semconv/attributes.py`.
5. **Is the attribute always available?** If it is not, document it as "optional" so query authors know to handle missing values.

Do not add attributes speculatively. A constant with no call site is dead code that becomes stale.

## Attribute vs. Metric Decision

| Value                                                         | Goes where                            |
| ------------------------------------------------------------- | ------------------------------------- |
| Categorical, per-span (which iteration, what kind of request) | Span attribute                        |
| Stable for the process lifetime (rank, parallelism config)    | Resource attribute                    |
| Numerical, varies over time (loss, duration)                  | Metric                                |
| Text payload (prompt, completion)                             | Span event, or do not emit (PII risk) |

Putting a continuously-varying value such as loss on a span attribute produces thousands of span-attribute time series in Jaeger that cannot be aggregated. Use a metric.

## Attribute Cardinality

OTel metric attributes have a cardinality budget — each distinct combination of attribute values creates a new time series. `gen_ai.operation.name="text_completion"` is fine (one value). `gen_ai.request.model="llama-3-8b"` is fine (a handful of values). `user.session.id="abc-123"` is dangerous (unbounded cardinality — one time series per user session).

Rule of thumb: metric attributes should have **tens to hundreds** of distinct values, not millions. Use span attributes for high-cardinality values.