Resource Detection

OpenTelemetry Resources describe the entity producing telemetry: service name, version, host, cloud provider, and other properties. NeMo Lens auto-detects a handful of environment-specific attributes to make runs filterable without manual configuration.

Default Resource Attributes

Every exporter-rank process emits these attributes (set in providers.py:build_providers):

Auto-Detected Resource Attributes

The nemo.lens.resources.detect_resource() function merges attributes from three sources:

Local Process Attributes with detect_local()

Slurm Environment Attributes with detect_slurm()

This helper returns attributes when SLURM_JOB_ID is set or when launch-provided OTEL_RESOURCE_ATTRIBUTES contains a selected Slurm or run identity value. With a local job ID, it derives Slurm resource attributes from the local environment. Inherited values can activate detection independently and take precedence over caller-provided resource_attributes defaults and locally derived fallback values. Empty inherited values are ignored.

detect_slurm() returns slurm.array.count, slurm.nnodes, slurm.ntasks, and slurm.restart_count as integers whether they are derived locally or inherited through OTEL_RESOURCE_ATTRIBUTES. Other inherited values remain strings. Complete signed ASCII base-10 strings are accepted. An empty inherited value, or any other form for one of those four attributes, is discarded so that a valid locally derived value can be used as the fallback. Locally detected integer values use the same conversion rule. If no valid local value exists, the attribute is omitted.

The table above is the complete allowlist of attributes that detect_slurm() reads from OTEL_RESOURCE_ATTRIBUTES; every accepted key has an explicit normalizer. Other resource attributes are ignored by this helper, but they remain in OTEL_RESOURCE_ATTRIBUTES and are still read independently by the OpenTelemetry SDK.

Retired keys such as slurm.nodelist, slurm.cluster, and slurm.torchelastic.restart_count are not emitted for new telemetry.

Kubernetes Environment Attributes with detect_kubernetes()

These attributes are active when KUBERNETES_SERVICE_HOST is set or the /var/run/secrets/kubernetes.io directory exists. This helper maps the following variables:

HOSTNAME is used as a fallback for k8s.pod.name only when K8S_POD_NAME is unset.

Add Custom Resource Attributes

Pass the resource_attributes argument to setup_telemetry:

handle = setup_telemetry(
    config,
    resource_attributes={
        'nv.dl.rank': rank,
        'nv.dl.world_size': world_size,
        'dl.tensor_parallel.size': 4,
        'dl.pipeline_parallel.size': 2,
        'dl.data_parallel.size': 8,
        'megatron.num_layers': 32,
        'megatron.precision': 'bf16',
    },
)

These attributes merge with the auto-detected set. In Jaeger, they appear as Process tags and are filterable across every span in the run.

Pass the resource_attributes argument to setup_telemetry:

handle = setup_telemetry(
    config,
    resource_attributes={
        'nv.dl.rank': rank,
        'nv.dl.world_size': world_size,
        'dl.tensor_parallel.size': 4,
        'dl.pipeline_parallel.size': 2,
        'dl.data_parallel.size': 8,
        'megatron.num_layers': 32,
        'megatron.precision': 'bf16',
    },
)

These attributes merge with the auto-detected set. In Jaeger, they appear as Process tags and are filterable across every span in the run.

Two Channels for Supplying Attributes

Resource attributes reach NeMo Lens two ways, and both are first-class:

The env var is the standard OTel channel (key=value,key=value, percent-encoded) and is the only one that survives a spawn or exec, so it is how a child process supplies its own nv.dl.rank when it has no setup_telemetry call site to reach. NeMo Lens resolves both before deriving anything from them, so an env-supplied rank produces the same service.instance.id a caller-supplied one would.

On conflict the caller’s dict wins, matching what the OTel SDK does with the same inputs.

# parent, before spawning the child
export OTEL_RESOURCE_ATTRIBUTES="nv.dl.rank=5,nv.dl.world_size=8,nemo.run.id=abc123"

Propagate Attributes to a Child Process

A process that cannot reach a setup_telemetry call site — a spawned checkpoint worker, an exec’d relaunch, a launcher-started agent — receives its resource attributes through OTEL_RESOURCE_ATTRIBUTES. That is the only channel that survives the boundary: multiprocessing.Process has no env parameter at all, so mutating os.environ before start() is not one option among several. When both channels provide the same key, the caller’s resource_attributes mapping wins, matching the OpenTelemetry SDK.

Use the Resource carrier helpers rather than formatting the value by hand:

from nemo.lens.resources.attributes import (
    get_otel_resource_attributes,
    publish_otel_resource_attributes,
)
from nemo.lens.semconv.encoding import compose_attributes
current = get_otel_resource_attributes()
defaults = {'service.name': 'checkpoint-worker'}
overrides = {'nv.dl.role': 'ckpt_worker'}
resolved = compose_attributes(current, defaults=defaults, overrides=overrides)
# Publish the exact resolved map while the child starts, then restore the environment.
with publish_otel_resource_attributes(resolved):
    ctx.Process(target=worker).start()

When a consumer needs a child carrier string without changing the environment, use the pure string transformation directly:

from nemo.lens.resources.attributes import (
    extend_otel_resource_attributes,
)
child_carrier = extend_otel_resource_attributes(
    selected_carrier,
    defaults={'nv.dl.rank': rank},
    overrides={'nv.dl.role': 'worker'},
)

Composition follows ordinary dictionary precedence: defaults apply only when a key is absent, current values replace defaults, and overrides replace both. Empty strings and None are ordinary overriding values. Encoding retains an empty string as key= and omits a None entry; it does not reapply a default. The function returns a new dictionary and does not mutate any input mapping. Type reconstruction remains a separate Resource operation.

Publication installs only the encoded resolved map. It never retains current attributes implicitly, so include them in composition when they should survive. Passing an already resolved map requires no second decode or merge. extend_otel_resource_attributes() parses, composes, and formats a string without changing the environment. set_otel_resource_attributes() formats a resolved map and permanently replaces the selected environment value; it does not merge with the old value. publish_otel_resource_attributes() temporarily replaces the value and restores the exact incoming state. A resolved map needs no second parse or composition before it is set or published. Publisher contexts may nest within one thread, but they must not overlap across threads because environment variables are process-global. Serialize publication when multiple threads can launch children.

What It Handles

For selected Slurm and run identity keys, launch-provided OTEL_RESOURCE_ATTRIBUTES values are authoritative. Treat resource_attributes as consumer defaults for those keys.

Use Cases

Filter by Global Rank

In Jaeger, use: nv.dl.rank=0

Compare Distinct Runs

In Grafana, configure nemo.run.id as a dashboard variable, list all values, and then select the specific runs to compare.

Filter by Parallelism Configuration

In Jaeger, use: dl.tensor_parallel.size=4 AND dl.pipeline_parallel.size=2

Because these are resource attributes instead of span attributes, they apply to every span without cluttering the span view.

Attribute Conventions

• Use standard names. Apply standard OTel attribute names where they exist, such as service.*, k8s.*, and host.*.
• Use distributed learning prefixes. Apply the nv.dl.* prefix for participant resource attributes and the dl.* prefix for other training-specific attributes that are shared across consumers.
• Use project-specific prefixes. Apply the <project>.* prefix for project-specific attributes, such as megatron.*, rl.*, and gym.*.

See semconv for the full attribute namespace conventions.

Resource Detection Order

The detect_resource() function merges resource attributes in the following order: local, Slurm, and then Kubernetes. If a key collision occurs, the attribute from the later source takes precedence. In practice, collisions are rare because each layer uses its own namespace.

Run Telemetry Locally

On a development machine with no Slurm or Kubernetes environment, only the detect_local() helper executes. Run identifiers are automatically generated as UUID values, so you can still filter by nemo.run.id to isolate a single local run.

Deriving an attempt UUID

Use from nemo.lens import derive_nv_dl_run_uuid to derive an attempt UUID from scheduler identity or a supplied run_id. Call it when the application establishes an attempt. This standalone helper does not configure telemetry or publish attributes; callers choose where to record the result.