Resource Detection

OpenTelemetry Resources describe the entity producing telemetry: service name, version, host, cloud provider, and other properties. NeMo Lens auto-detects a handful of environment-specific attributes to make runs filterable without manual configuration.

Default Resource Attributes

Every exporter-rank process emits these attributes (set in providers.py:build_providers):

Auto-Detected Resource Attributes

The nemo.lens.resources.detect_resource() function merges attributes from three sources:

Local Process Attributes with detect_local()

Slurm Environment Attributes with detect_slurm()

These attributes are active when SLURM_JOB_ID is set. This helper maps the following variables:

Kubernetes Environment Attributes with detect_kubernetes()

These attributes are active when KUBERNETES_SERVICE_HOST is set or the /var/run/secrets/kubernetes.io directory exists. This helper maps the following variables:

HOSTNAME is used as a fallback for k8s.pod.name only when K8S_POD_NAME is unset.

Add Custom Resource Attributes

Pass the resource_attributes argument to setup_telemetry:

1
handle = setup_telemetry(
2
    config,
3
    rank=rank,
4
    world_size=world_size,
5
    resource_attributes={
6
        'dl.tensor_parallel.size': 4,
7
        'dl.pipeline_parallel.size': 2,
8
        'dl.data_parallel.size': 8,
9
        'megatron.num_layers': 32,
10
        'megatron.precision': 'bf16',
11
    },
12
)

These attributes merge with the auto-detected set. In Jaeger, they appear as Process tags and are filterable across every span in the run.

Use Cases

Filter by Global Rank

In Jaeger, use: dl.rank=0

Compare Distinct Runs

In Grafana, configure nemo.run.id as a dashboard variable, list all values, and then select the specific runs to compare.

Filter by Parallelism Configuration

In Jaeger, use: dl.tensor_parallel.size=4 AND dl.pipeline_parallel.size=2

Because these are resource attributes instead of span attributes, they apply to every span without cluttering the span view.

Attribute Conventions

• Use standard names. Apply standard OTel attribute names where they exist, such as service.*, k8s.*, and host.*.
• Use distributed learning prefix. Apply the dl.* (distributed learning) prefix for training-specific attributes that are shared across consumers.
• Use project-specific prefixes. Apply the <project>.* prefix for project-specific attributes, such as megatron.*, rl.*, and gym.*.

See semconv for the full attribute namespace conventions.

Resource Detection Order

The detect_resource() function merges resource attributes in the following order: local, Slurm, and then Kubernetes. If a key collision occurs, the attribute from the later source takes precedence. In practice, collisions are rare because each layer uses its own namespace.

Run Telemetry Locally

On a development machine with no Slurm or Kubernetes environment, only the detect_local() helper executes. Run identifiers are automatically generated as UUID values, so you can still filter by nemo.run.id to isolate a single local run.