> This page is for version Nightly.
> For other versions, use one of these documentation indexes:
> - 0.1.0 (default): https://docs.nvidia.com/nemo/lens/0.1.0/llms.txt
> - Nightly: https://docs.nvidia.com/nemo/lens/nightly/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/lens/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/lens/_mcp/server.

# Instrumentation Primitives

NeMo Lens provides live-span primitives and explicitly timed emission helpers.
Select a primitive based on the call site.

## Explicit Intervals and Instant Markers

Use the existing `nemo.lens.span_utilities.emit_span` for both. It returns the
completed **Span**; use `.get_span_context()` when only its identity is needed.

```python
from nemo.lens.span_utilities import emit_span
from opentelemetry import trace

marker = emit_span(None, "app.phase_start", 1700000000.25, 1700000000.25,
                   group="startup", attributes={"phase": "startup"})
if marker is not None:
    emit_span(None, "app.phase", 1700000000.25, 1700000001.5,
              group="startup", context=trace.set_span_in_context(marker))
```

`emit_span()` uses the context active when it is called unless you pass
`context=`. Backdated timestamps do not recover an earlier context. Pass
`Context()` to start a root span. The emitted span is never made current.

Times are Unix-epoch seconds. Equal timestamps produce a zero-duration span.
If the end precedes the start, `emit_span()` always clamps the end to the start,
logs a warning about possible clock skew, and records `inverted.skew=true` on
the span. The original values are retained as string attributes
`inverted.start_epoch_seconds` and `inverted.end_epoch_seconds`, together with
the positive difference as the decimal string `inverted.delta_seconds`.
The difference records the observed inversion, not a measured clock offset.
Pass decimal strings or `Decimal` values to retain precision below that of a
floating-point epoch timestamp.

Corrected parents remain available as parents for subsequent spans; children
retain their own timestamps and are corrected independently if needed.
Non-inverted spans, including ordinary zero-duration markers, receive no skew
attributes. Exclude corrected spans from duration analysis using the skew flag.
Caller attributes are preserved without mutating the supplied dictionary;
the correction attributes take precedence if the same keys were supplied.

Timestamps must be finite. There is no tolerance or rejection threshold for an
inversion. The former `eps_ms` parameter has been removed; callers that supplied
it should remove that keyword argument.
The [CLI](/nemo/lens/nightly/user-guide/command-line) uses this same helper and correction policy.

Pass `None` for the default tracer. A disabled `group` returns `None` before
validation; otherwise the function returns a completed span. Attributes are
filtered and redacted, and the span ends even if attribute processing raises.

## `managed_span` as a Group-Gated Context Manager

```python
from nemo.lens import managed_span

with managed_span('step', 'train.step', iteration=42) as span:
    do_work()
    if span is not None:
        span.set_attribute('loss', loss_value)
```

### How It Behaves

* When the `step` group is **disabled**, the function yields `None`, the body runs unchanged, and no span object is created. The gating check is a fast `frozenset` lookup.
* When the `step` group is **enabled**, the function starts a span, sets attributes, attaches its context, and yields the span. On exit, the function detaches the context and ends the span. If the body raises an exception, that exception is recorded on the span with `StatusCode.ERROR` before re-raising.

### When to Use It

* **Hot paths with configurable granularity.** Use this for training steps, microbatches, or communication operations where configurable granularity is required.
* **Blocks where telemetry is frequently disabled.** Use this for any block of code where performance overhead must be minimized when telemetry is disabled.

### Tracer Resolution

If you do not pass a `tracer` argument, `managed_span` obtains a named tracer from the global `TracerProvider` using the `nemo.lens.helpers` instrumentation scope. This defaults to the same scope used by `span_cm`, whereas `trace_fn` defaults to the `nemo.lens` scope instead. Passing `tracer=handle.tracer` (which uses the `nemo.lens` scope) skips this `get_tracer` call, allowing spans to share the handle's instrumentation scope.

Lens reuses default tracers for each scope and provider. This avoids repeated SDK tracer lookups, which in SDK 1.44 modify Python's warning filters and allow unrelated warnings to print again. The cache resets when the global provider changes or the process forks, so initialization and child processes still resolve the appropriate tracer. Disabled span groups do not resolve a tracer, and explicitly supplied tracers are used directly. Lens does not install a warning-suppression policy; rank-aware warning handling belongs to the consuming application.

## `trace_fn` as a Group-Gated Decorator

```python
from nemo.lens import trace_fn

@trace_fn('microbatch', 'train.microbatch.forward')
def forward_step(batch):
    ...
```

### How It Behaves

This decorator has identical gating semantics to `managed_span`, but is applied as a function decorator, requiring no re-indentation of the function body. The span group is checked at call time rather than at decoration time, which enables dynamic toggling of groups.

### When to Use It

* **Instrument existing functions.** Use this to instrument existing functions without restructuring their bodies.
* **Span name matches function name.** Use this when the desired span name matches the function name, which is typical.

### Limitations

`trace_fn` cannot set span attributes from function arguments without a custom wrapper. For attribute-rich spans, use `managed_span` inside the function body instead.

## `span_cm` as a Simple Ungated Context Manager

```python
from nemo.lens import span_cm

with span_cm('evaluate', tracer=handle.tracer, dataset='mmlu'):
    ...
```

### How It Behaves

This context manager always creates a span with no group gating. Attributes are set through `safe_set_span_attributes`, which processes only scalars and scalar sequences; non-scalar values are silently dropped.

### When to Use It

* **Required code paths.** Use this in code paths where you always want a span when telemetry is active, such as evaluation, setup, or shutdown procedures.
* **Utility modules.** Use this in utility modules that do not contain information about span groups.
* **Cold paths.** Use this for spans outside hot paths where the per-call cost of always creating a span is negligible.

### No-Op Interaction

When the global `TracerProvider` is a no-op instance (that is, when telemetry is disabled), `start_as_current_span` is still called but behaves as a cheap no-op operation within the OTel API, resulting in no exporter round-trip and no attribute processing cost. This call does not bypass span object creation entirely the way `managed_span` does when its group is disabled, but the performance cost is negligible.

## `safe_set_span_attributes`

Use this utility to bulk-set span attributes with sensible filtering and redaction:

```python
from nemo.lens import safe_set_span_attributes

safe_set_span_attributes(span, {
    'iteration': 42,
    'skipped': False,
    'loss': 1.23,
    'prompt': 'user input here',     # redacted to '[REDACTED]'
    'complex_obj': {...},             # silently dropped (not scalar)
})
```

### Behavior and Evaluation Rules

1. **Verify recording status.** If `span.is_recording()` is `False`, the call is a no-op.
2. **Skip null values.** `None` values are silently skipped.
3. **Drop non-scalars.** Non-scalar values (such as dictionaries or complex objects) are silently skipped, as OTel attributes must be scalars or sequences of scalars.
4. **Format sequences.** Sequences of scalars are converted to a `list`.
5. **Redact sensitive strings.** String values whose keys match a redact key are replaced with the string `'[REDACTED]'`.

### Default Redact Keys

```python
from nemo.lens import DEFAULT_REDACT_KEYS
# frozenset({'prompt', 'input_text', 'output_text', 'text',
#            'password', 'token', 'secret', 'key'})
```

Pass a custom `redact_keys` set to override.

### `redact_value`

`redact_value(key, value, redact_keys=DEFAULT_REDACT_KEYS)` is the single-value primitive that `safe_set_span_attributes` uses internally:

```python
from nemo.lens import redact_value, DEFAULT_REDACT_KEYS

redact_value('prompt', 'user input here')   # '[REDACTED]'  (key is in DEFAULT_REDACT_KEYS)
redact_value('iteration', 'user input here') # 'user input here' (key not redacted)
```

The function returns `'[REDACTED]'` if and only if the `key` is present in `redact_keys`; otherwise, it returns `value` unchanged. Redaction is decided by the attribute key name rather than by inspecting the value itself.

## `get_tracer` and `get_meter`

Both are top-level exports that return the globally registered tracer/meter from the active provider:

```python
from nemo.lens import get_tracer, get_meter

tracer = get_tracer()   # default instrumentation scope 'nemo.lens'
meter = get_meter()     # default instrumentation scope 'nemo.lens'
```

Each function accepts an optional `name` argument to set the instrumentation scope, which defaults to `'nemo.lens'`. Use these functions when you require a tracer or meter outside the span primitives; for example, you can use them to create custom metric instruments.

## Choose the Correct Primitive

| Call frequency           | Attribute-heavy? | Need group gating? | Use            |
| ------------------------ | ---------------- | ------------------ | -------------- |
| Hot (per-microbatch)     | Yes              | Yes                | `managed_span` |
| Hot (per-microbatch)     | No               | Yes                | `trace_fn`     |
| Cold (per-job, per-eval) | Yes              | No                 | `span_cm`      |
| Cold (per-job, per-eval) | No               | No                 | `span_cm`      |

## Check Group Status Before Expensive Preparation

If building the attributes dict is itself expensive, gate it:

```python
from nemo.lens import is_span_group_enabled

if is_span_group_enabled('step'):
    attrs = build_expensive_attributes()
    with managed_span('step', 'train.step', **attrs) as span:
        ...
```

This saves the `build_expensive_attributes()` cost when `step` is disabled.

## Scoped span attributes

Use `span_attributes` for values shared by operations within a block:

```python
from nemo import lens

with (
    lens.span_attributes({"nv.dl.run.uuid": run_uuid}),
    lens.managed_span("checkpoint", "checkpoint.save", **{"checkpoint.size": size}),
):
    save_checkpoint()
```

The first context manager sets attributes; the second creates a span. The UUID
applies to spans created within the scope. `checkpoint.size` belongs only to the
save span; child spans do not inherit it. Inner attribute scopes override outer
values, and explicit span attributes override scoped values. Exit restores the
previous scope, including on exceptions.

Lens installs a span processor during provider setup. It adds scoped values when
spans start, including spans created directly through OTel using that provider.
A tracer from a separate provider does not use this processor. Span-group
settings still control whether a span is created.

This API does not update existing spans or provide attributes to the initial
sampler. Backdated spans receive the scope active when they are created. Scopes
follow Python context-variable rules: tasks inherit their creation context;
new threads require explicit context copying. This API does not propagate values
to worker processes or change Resource, metric, or log attributes. It is separate
from OTel's instrumentation scope, which identifies the instrumentation library.