> This page is for version Nightly.
> For other versions, use one of these documentation indexes:
> - 0.1.0 (default): https://docs.nvidia.com/nemo/lens/0.1.0/llms.txt
> - Nightly: https://docs.nvidia.com/nemo/lens/nightly/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/lens/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/lens/_mcp/server.

# Command Line

NeMo Lens installs a `nemo-lens` command for launch scripts and other process
boundaries that need to emit telemetry outside a normal Python application
lifecycle.

## `set-slurm-resource-attrs`

`set-slurm-resource-attrs` prints a shell export that appends Lens-derived Slurm
resource attributes to `OTEL_RESOURCE_ATTRIBUTES`.

Use it with `eval` at Slurm launch boundaries:

```bash
eval "$(nemo-lens set-slurm-resource-attrs --stage sbatch)"
```

Run the task-stage form inside each `srun` task:

```bash
eval "$(nemo-lens set-slurm-resource-attrs \
  --stage task \
  --container-image "$CONTAINER_IMAGE")"
```

`--stage` selects one launch boundary per invocation. A Slurm launch can run the
command once in the sbatch script context and again inside each task. The sbatch
stage captures job/scheduler attributes and `slurm.head_node.name` when Slurm
exposes it. The task stage captures task-local attributes such as `host.name`,
`nv.dl.launch.container.image`, and `slurm.topology.*`.

The command prints exactly one shell assignment:

```bash
export OTEL_RESOURCE_ATTRIBUTES=...
```

Each invocation appends only missing derived attributes. Existing
`OTEL_RESOURCE_ATTRIBUTES` values are preserved and take precedence over locally
derived fallbacks.

`--container-image` records the launcher/container image identity as
`nv.dl.launch.container.image`. It is valid only with `--stage task`.

## `emit-spans`

`emit-spans` emits completed spans from explicit Unix epoch-second intervals.
Use it when a shell launcher or wrapper has already measured a start and end
time and needs to export that interval as a trace span.

Install the SDK dependencies before using the command:

```bash
pip install 'nemo-lens[sdk]'
```

Pass one or more `--span` values:

```bash
NEMO_LENS_EXPORTER=console nemo-lens emit-spans \
  --service nv.dl.launch \
  --span launch,1700000000.0,1700000003.0 \
  --span python_startup,1700000000.2,1700000001.1,launch
```

Each span is encoded as:

```text
NAME,START_EPOCH_SECONDS,END_EPOCH_SECONDS[,PARENT_NAME]
```

Parent spans can appear before or after their children on the command line. The
command orders them before emission so the exported spans have parent context.
Parent/child intervals are not required to be nested; launch flows sometimes use
short anchor spans as parents for longer measured work.

If an end timestamp precedes its start, the shared `emit_span()` helper clamps
the end to the start and emits a zero-duration span. It logs a warning with the original timestamps
and inversion amount, and attaches these attributes to the corrected span:

| Attribute                      | Value                                                               |
| ------------------------------ | ------------------------------------------------------------------- |
| `inverted.skew`                | Boolean `true`                                                      |
| `inverted.start_epoch_seconds` | Original start timestamp as a string                                |
| `inverted.end_epoch_seconds`   | Original end timestamp as a string                                  |
| `inverted.delta_seconds`       | Positive start-minus-end difference in seconds, as a decimal string |

Strings preserve the supplied timestamp precision. The flag identifies an
observed inversion, which can result from clock skew or a clock adjustment;
the difference is not a measurement of the true offset between clocks.
Clamping a parent preserves its children and their parent context. Each interval
is corrected independently: valid children retain their measured timestamps,
even when they extend outside the corrected parent's zero-duration interval.
Ordinary zero-duration spans and correctly ordered spans receive no inversion
attributes. Consumers should exclude flagged spans from duration analysis.
Successful emission of corrected spans exits zero.

If any span input is invalid, nothing is emitted. Invalid input includes
malformed fields, duplicate span names, missing parents, parent cycles,
and non-finite timestamps. Inverted finite intervals are corrected as described
above rather than treated as invalid input. The CLI preserves decimal timestamp
precision and delegates correction to the same helper used by Python callers.

`--service` sets the exported `service.name`. Other attributes from
`OTEL_RESOURCE_ATTRIBUTES` are inherited by the SDK resource detector, but
`--service` takes precedence over `service.name` from the environment.

The command uses the same exporter configuration as NeMo Lens tracing. It
defaults to OTLP; set `NEMO_LENS_EXPORTER=console` to write spans to stdout.
Standard OTel endpoint and protocol variables configure OTLP, including the
trace-specific variables, which take precedence over their generic equivalents.

The command exits nonzero if provider setup, flushing, or span export fails.