Semantic Conventions

nemo.lens.semconv centralizes the attribute names NeMo Lens emits. The attributes module owns names and version metadata, while resources owns Resource field sets and type mappings. Existing imports from nemo.lens.semconv remain supported. Two categories of attributes coexist:

1. Standard OTel semconv: gen_ai.* and k8s.*. These track upstream OTel specs. (Resource attributes like service.* and host.* are also standard OTel names but are set as literals in providers.py and resources/local.py rather than mirrored as constants.)
2. NeMo custom namespaces: nv.dl.* / dl.* (distributed learning), rl.*, gym.*, slurm.*, nemo.*, wandb.*. These are NeMo-specific extensions that do not exist upstream.

Why Constants Instead of Strings

Using constants instead of raw strings provides three key benefits:

1. Grep-ability: renaming an attribute across the codebase means changing one constant, not every call site.
2. Type safety (weak but real): NV_DL_RANK is an exported name; typos become ImportErrors. "nv.dl.rank" typos become silent data loss.
3. Central definitions: one package lists the attribute names and Resource type mappings. This keeps them reviewable and documented.

Callers who want the string can use NV_DL_RANK directly; Python strings-as-constants have no boxing cost.

Version Tracking

SEMCONV_VERSION = "1.29.0"

This constant documents which upstream OTel semconv version the standard namespaces (gen_ai.*, k8s.*) are aligned with. Upstream bumps namespace conventions periodically; gen_ai.* graduated from experimental to stable around 1.30.

When upgrading NeMo Lens to a new semconv version, complete these tasks:

1. Review the upstream changelog for renamed or removed attributes.
2. Update SEMCONV_VERSION.
3. Update any changed constants.
4. Add the version bump to the NeMo Lens changelog so consumers know to update.

Resource Types and Attribute Carriers

semconv/resources.py defines explicit type mappings for Resource fields. normalize_resource_attributes() applies those mappings after decoding an environment carrier. Unknown values are unchanged, so numeric-looking identifiers such as "007" stay strings. Every mapped Resource integer field, including Slurm fields, uses one converter that accepts native integers or complete signed ASCII base-10 strings. Invalid DL/GPU values remain visible and produce a warning. Invalid Slurm values are omitted so locally detected values can be used; the conversion rule is shared while callers retain their failure policy.

semconv/encoding.py contains generic dictionary composition and explicit converter lookup. The resources/attributes.py module owns formatting and parsing for the comma-separated, percent-encoded OTEL_RESOURCE_ATTRIBUTES carrier. Parsing returns strings. Type reconstruction is a separate step and always uses a caller-selected mapping; the Resource codec does not infer types.

Stability Markers

From semconv/attributes.py:

# gen_ai.*  — Experimental (upstream, stabilising in semconv 1.30+)
# k8s.*     — Stable (upstream)
# nv.dl.*   — NeMo custom (stable within NeMo ecosystem)
# dl.*      — NeMo custom (stable within NeMo ecosystem)
# rl.*      — NeMo custom (stable within NeMo ecosystem)
# gym.*     — NeMo custom (stable within NeMo ecosystem)
# slurm.*   — NeMo custom (stable within NeMo ecosystem)
# nemo.*    — NeMo custom (stable within NeMo ecosystem)
# wandb.*   — NeMo custom (stable within NeMo ecosystem)

“Stable” for custom namespaces means these names will not change in minor releases. Breaking changes bump the major version.

“Experimental” for gen_ai.* matches upstream; OTel considers them stable-in-practice but reserves the right to tweak until the full semconv 1.30 stabilization.

Namespace Conventions

NeMo Lens organizes attributes into standard upstream namespaces and NeMo-specific custom namespaces.

Standard Upstream Namespaces

Follow upstream OTel spec exactly. Do not redefine, and do not rename. If upstream says gen_ai.request.model, NeMo Lens uses gen_ai.request.model.

nv.dl.* / dl.* Distributed Learning

Shared across Megatron-LM, NeMo RL, NeMo Gym. Anything a distributed training job needs:

nv.dl.job.uuid               — stable submitted-job identifier
nv.dl.run.uuid               — stable run-attempt identifier
nv.dl.rank                   — global rank resource attribute
nv.dl.world_size             — total ranks resource attribute
nv.dl.local_rank             — rank on this node resource attribute
nv.dl.role                   — trainer or worker process role
nv.dl.provider.name          — training framework or provider name
nv.dl.topology.size.{tp,pp,dp}
                             — tensor, pipeline, and data parallel sizes (integers)
nv.dl.training.config.global_batch_size
nv.dl.training.config.micro_batch_size
nv.dl.training.config.sequence_length
                             — stable integer training configuration
nv.dl.training.config.optimizer
nv.dl.training.config.recompute_granularity
                             — stable text training configuration
nv.dl.training.target.{train_iters,train_samples,train_tokens}
                             — configured integer training targets
nv.dl.software.{torch,cuda,nccl,transformer_engine}
                             — application-observed software versions
dl.tensor_parallel.{rank,size}
dl.pipeline_parallel.{rank,size}
dl.data_parallel.{rank,size}
dl.iteration                 — training iteration
dl.microbatch_id             — microbatch index
dl.batch_size, dl.sequence_length
dl.loss, dl.grad_norm, dl.learning_rate
dl.throughput_tflops, dl.throughput_tokens_per_sec

<project>.* Project-Specific Attributes

Use project-specific namespaces for attributes that apply to one consumer rather than the full NeMo ecosystem:

• megatron.*: Megatron-LM consumer-specific span attributes (e.g., model architecture megatron.num_layers/megatron.hidden_size, iteration-level megatron.skipped/megatron.update_successful). These are set as inline string attributes in the Megatron-LM fork and are not defined as constants in nemo.lens.semconv. The semconv package holds the shared and standard namespaces (dl.*, gen_ai.*, rl.*, gym.*, slurm.*, nemo.*, wandb.*, k8s.*); for rl.* it holds the span-attribute names only (see below).

• rl.* — RL-specific. The span-attribute names (rl.algorithm, rl.reward, rl.generation.backend, rl.num_rollouts) are defined here as constants. The metric series are owned by the NeMo-RL consumer.

  NeMo-RL declares its rl.* metric series through the metric registry (register_metric_group) in its own tree, so those names are not defined here. Among them, rl.grad_norm, rl.learning_rate and rl.throughput.tokens_per_sec are RL-scoped counterparts to quantities the dl.* namespace registers, describing the policy update specifically, so an RL job can report them without claiming the shared dl.* series that a co-resident trainer owns. The first two are deliberate spelling matches for dl.grad_norm/dl.learning_rate, so the two namespaces differ only by prefix. Throughput is the exception: rl.* uses the dotted rl.throughput.tokens_per_sec to match the rl.response_length.mean/rl.generation.duration_ms subgroup convention, whereas dl.* uses the flat dl.throughput_tokens_per_sec. Query the two by their own names rather than by swapping the prefix.

• gym.* — Gym server-specific (verify.success_rate, rollout.batch_size)

nemo.* NeMo-Wide Identification

• nemo.run.id — unique run identifier, shared across all ranks
• nemo.user.id — optional team/user label
• nemo.span.truncated — optional; set to true only on a span that was still open at telemetry shutdown and was force-closed. Its end timestamp is the shutdown time, not the time the work finished, so exclude these spans from duration statistics. Absent on every normally ended span. See Production.

Environment Namespaces

• slurm.* — SLURM job attributes
• k8s.* — Kubernetes pod/node attributes (standard OTel)
• wandb.* — W&B Weave integration metadata

Adding New Attributes

Checklist before adding a constant:

1. Is there an upstream OTel semconv name? Use it.
2. Is this shared across consumers? Use dl.* or another shared namespace.
3. Is this project-specific? Use <project>.*. Such namespaces are set inline in the consumer fork (e.g. Megatron-LM), not as constants in semconv/attributes.py.
4. Is this a metric or a span attribute? Metric instruments go in instruments/; attribute names go in semconv/attributes.py.
5. Is the attribute always available? If it is not, document it as “optional” so query authors know to handle missing values.

Do not add attributes speculatively. A constant with no call site is dead code that becomes stale.

Attribute vs. Metric Decision

Putting a continuously-varying value such as loss on a span attribute produces thousands of span-attribute time series in Jaeger that cannot be aggregated. Use a metric.

Attribute Cardinality

OTel metric attributes have a cardinality budget — each distinct combination of attribute values creates a new time series. gen_ai.operation.name="text_completion" is fine (one value). gen_ai.request.model="llama-3-8b" is fine (a handful of values). user.session.id="abc-123" is dangerous (unbounded cardinality — one time series per user session).

Rule of thumb: metric attributes should have tens to hundreds of distinct values, not millions. Use span attributes for high-cardinality values.