Command Line

NeMo Lens installs a nemo-lens command for launch scripts and other process boundaries that need to emit telemetry outside a normal Python application lifecycle.

set-slurm-resource-attrs

set-slurm-resource-attrs prints a shell export that appends Lens-derived Slurm resource attributes to OTEL_RESOURCE_ATTRIBUTES.

Use it with eval at Slurm launch boundaries:

eval "$(nemo-lens set-slurm-resource-attrs --stage sbatch)"

Run the task-stage form inside each srun task:

eval "$(nemo-lens set-slurm-resource-attrs \
  --stage task \
  --container-image "$CONTAINER_IMAGE")"

--stage selects one launch boundary per invocation. A Slurm launch can run the command once in the sbatch script context and again inside each task. The sbatch stage captures job/scheduler attributes and slurm.head_node.name when Slurm exposes it. The task stage captures task-local attributes such as host.name, nv.dl.launch.container.image, and slurm.topology.*.

The command prints exactly one shell assignment:

export OTEL_RESOURCE_ATTRIBUTES=...

Each invocation appends only missing derived attributes. Existing OTEL_RESOURCE_ATTRIBUTES values are preserved and take precedence over locally derived fallbacks.

--container-image records the launcher/container image identity as nv.dl.launch.container.image. It is valid only with --stage task.

emit-spans

emit-spans emits completed spans from explicit Unix epoch-second intervals. Use it when a shell launcher or wrapper has already measured a start and end time and needs to export that interval as a trace span.

Install the SDK dependencies before using the command:

pip install 'nemo-lens[sdk]'

Pass one or more --span values:

NEMO_LENS_EXPORTER=console nemo-lens emit-spans \
  --service nv.dl.launch \
  --span launch,1700000000.0,1700000003.0 \
  --span python_startup,1700000000.2,1700000001.1,launch

Each span is encoded as:

NAME,START_EPOCH_SECONDS,END_EPOCH_SECONDS[,PARENT_NAME]

Parent spans can appear before or after their children on the command line. The command orders them before emission so the exported spans have parent context. Parent/child intervals are not required to be nested; launch flows sometimes use short anchor spans as parents for longer measured work.

If an end timestamp precedes its start, the shared emit_span() helper clamps the end to the start and emits a zero-duration span. It logs a warning with the original timestamps and inversion amount, and attaches these attributes to the corrected span:

Strings preserve the supplied timestamp precision. The flag identifies an observed inversion, which can result from clock skew or a clock adjustment; the difference is not a measurement of the true offset between clocks. Clamping a parent preserves its children and their parent context. Each interval is corrected independently: valid children retain their measured timestamps, even when they extend outside the corrected parent’s zero-duration interval. Ordinary zero-duration spans and correctly ordered spans receive no inversion attributes. Consumers should exclude flagged spans from duration analysis. Successful emission of corrected spans exits zero.

If any span input is invalid, nothing is emitted. Invalid input includes malformed fields, duplicate span names, missing parents, parent cycles, and non-finite timestamps. Inverted finite intervals are corrected as described above rather than treated as invalid input. The CLI preserves decimal timestamp precision and delegates correction to the same helper used by Python callers.

--service sets the exported service.name. Other attributes from OTEL_RESOURCE_ATTRIBUTES are inherited by the SDK resource detector, but --service takes precedence over service.name from the environment.

The command uses the same exporter configuration as NeMo Lens tracing. It defaults to OTLP; set NEMO_LENS_EXPORTER=console to write spans to stdout. Standard OTel endpoint and protocol variables configure OTLP, including the trace-specific variables, which take precedence over their generic equivalents.

The command exits nonzero if provider setup, flushing, or span export fails.