> This page is for version 0.5.0 · 26.06.
> For other versions, use one of these documentation indexes:
> - Nightly (default): https://docs.nvidia.com/nemo/automodel/nightly/llms.txt
> - Latest: https://docs.nvidia.com/nemo/automodel/latest/llms.txt
> - 0.7.0 · 26.10: https://docs.nvidia.com/nemo/automodel/v0.7/llms.txt
> - 0.5.0 · 26.06: https://docs.nvidia.com/nemo/automodel/v0.5/llms.txt
> - 0.4.0 · 26.04: https://docs.nvidia.com/nemo/automodel/v0.4/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt.

# nemo_automodel.components.models.nemotron_nas.packing

Thread public FlashAttention boundaries through the released DeciLM API.

The publisher's classes and checkpoint layout are retained. Reference interface:
nvidia/Llama-3_3-Nemotron-Super-49B-v1_5 at
420ba7d28211abf116b8b103ab700d92619daf98 (Apache-2.0 modeling_decilm.py).
No publisher implementation is copied here.

## Module Contents

### Classes

| Name                                                                                          | Description                                               |
| --------------------------------------------------------------------------------------------- | --------------------------------------------------------- |
| [`_FlashBoundaries`](#nemo_automodel-components-models-nemotron_nas-packing-_FlashBoundaries) | Generic FlashAttention varlen boundaries for packed rows. |

### Functions

| Name                                                                                                      | Description                                                                     |
| --------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- |
| [`_attention_forward`](#nemo_automodel-components-models-nemotron_nas-packing-_attention_forward)         | Dispatch packed calls to varlen attention and others to the released forward.   |
| [`_causal_mask`](#nemo_automodel-components-models-nemotron_nas-packing-_causal_mask)                     | Keep packed boundaries instead of building the release's causal mask.           |
| [`_packed_attention`](#nemo_automodel-components-models-nemotron_nas-packing-_packed_attention)           | Use the released projections/RoPE with explicit, padding-aware segments.        |
| [`_root_forward`](#nemo_automodel-components-models-nemotron_nas-packing-_root_forward)                   | Route generic FlashAttention packing metadata into the released DeciLM forward. |
| [`_validate_backend`](#nemo_automodel-components-models-nemotron_nas-packing-_validate_backend)           | -                                                                               |
| [`apply_packing_adapters`](#nemo_automodel-components-models-nemotron_nas-packing-apply_packing_adapters) | Add explicit packed boundaries without changing unpacked/generation calls.      |

### API

```python
class nemo_automodel.components.models.nemotron_nas.packing._FlashBoundaries(
    cu_q: torch.Tensor,
    cu_k: torch.Tensor,
    max_q: int,
    max_k: int
)
```

Dataclass

Generic FlashAttention varlen boundaries for packed rows.

**`cu_k`** `Tensor`

int32 Tensor of shape \[segments + 1] holding the matching key
boundaries.

---

**`cu_q`** `Tensor`

int32 Tensor of shape \[segments + 1] holding cumulative query
lengths over the row-major flattened \[batch \* sequence] stream.
Physical padding runs are segments of their own.

---

**`max_k`** `int`

Longest key segment, in tokens.

---

**`max_q`** `int`

Longest query segment, in tokens.

---

```python
nemo_automodel.components.models.nemotron_nas.packing._attention_forward(
    self: torch.nn.Module,
    args: typing.Any = (),
    cu_seq_lens_q: torch.Tensor | None = None,
    cu_seq_lens_k: torch.Tensor | None = None,
    max_length_q: int | None = None,
    max_length_k: int | None = None,
    kwargs: typing.Any = {}
) -> typing.Any
```

Dispatch packed calls to varlen attention and others to the released forward.

**Parameters:**

**`*args`** `Any` — default: ()

Positional release arguments; the first is `hidden_states`.

---

**`cu_seq_lens_q`** `torch.Tensor | None` — default: None

Query boundaries forwarded by the shared pipeline
forward; see `_FlashBoundaries.cu_q`.

---

**`cu_seq_lens_k`** `torch.Tensor | None` — default: None

Key boundaries; see `_FlashBoundaries.cu_k`.

---

**`max_length_q`** `int | None` — default: None

Longest query segment, in tokens.

---

**`max_length_k`** `int | None` — default: None

Longest key segment, in tokens.

---

**`**kwargs`** `Any` — default: \{}

Release arguments, including `hidden_states`: Tensor of
shape \[batch, sequence, hidden] and `attention_mask`: Tensor of
shape \[batch, sequence] or packed segment boundaries.

---

**Returns:** `Any`

The release's attention tuple, whose first element is a Tensor of shape

**Raises:**

* `ValueError`: If pipeline metadata is incomplete or a cache is requested
  with packing.

```python
nemo_automodel.components.models.nemotron_nas.packing._causal_mask(
    self: torch.nn.Module,
    attention_mask: torch.Tensor | nemo_automodel.components.models.nemotron_nas.packing._FlashBoundaries | None,
    args: typing.Any = (),
    kwargs: typing.Any = {}
) -> typing.Any
```

Keep packed boundaries instead of building the release's causal mask.

**Parameters:**

**`attention_mask`** `torch.Tensor | _FlashBoundaries | None`

Tensor of shape \[batch, sequence] holding the release's
0/1 padding mask, or packed segment boundaries.

---

**`*args`** `Any` — default: ()

Remaining release mask arguments.

---

**`**kwargs`** `Any` — default: \{}

Remaining release mask arguments.

---

**Returns:** `Any`

The boundaries unchanged on packed calls; otherwise the release's mask.

```python
nemo_automodel.components.models.nemotron_nas.packing._packed_attention(
    self: torch.nn.Module,
    hidden: torch.Tensor,
    metadata: nemo_automodel.components.models.nemotron_nas.packing._FlashBoundaries,
    kwargs: typing.Any = {}
) -> typing.Any
```

Use the released projections/RoPE with explicit, padding-aware segments.

As in the release, FP32 query, key and value return to the autocast dtype
before FlashAttention; without autocast, to the pre-quantization or
query-projection weight dtype.

**Parameters:**

**`hidden`** `torch.Tensor`

Tensor of shape \[batch, sequence, hidden] holding normalized
packed rows.

---

**`metadata`** `_FlashBoundaries`

Segment boundaries over the flattened rows.

---

**`**kwargs`** `Any` — default: \{}

Release attention arguments. `position_embeddings` is a
tuple of cos and sin Tensors of shape \[batch, sequence, head_dim];
without it, `position_ids` (Tensor of shape \[batch, sequence])
rebuilds them.

---

**Returns:** `Any`

A tuple of a Tensor of shape \[batch, sequence, hidden] holding the

**Raises:**

* `ValueError`: If a KV cache or sink attention is requested.

```python
nemo_automodel.components.models.nemotron_nas.packing._root_forward(
    self: torch.nn.Module,
    input_ids: torch.Tensor | None = None,
    attention_mask: torch.Tensor | None = None,
    position_ids: torch.Tensor | None = None,
    past_key_values: transformers.cache_utils.Cache | None = None,
    inputs_embeds: torch.Tensor | None = None,
    labels: torch.Tensor | None = None,
    use_cache: bool | None = None,
    output_attentions: bool | None = None,
    output_hidden_states: bool | None = None,
    return_dict: bool | None = None,
    cache_position: torch.Tensor | None = None,
    cu_seq_lens_q: torch.Tensor | None = None,
    cu_seq_lens_k: torch.Tensor | None = None,
    max_length_q: int | None = None,
    max_length_k: int | None = None,
    padding_mask: torch.Tensor | None = None,
    _packed_seq_ids: torch.Tensor | None = None
) -> typing.Any
```

Route generic FlashAttention packing metadata into the released DeciLM forward.

**Parameters:**

**`input_ids`** `torch.Tensor | None` — default: None

Tensor of shape \[batch, sequence] holding token IDs. Packed
rows hold several documents followed by padding.

---

**`attention_mask`** `torch.Tensor | None` — default: None

Tensor of shape \[batch, sequence] holding the release's
0/1 padding mask. Packed calls replace it with segment boundaries.

---

**`position_ids`** `torch.Tensor | None` — default: None

Tensor of shape \[batch, sequence] holding positions that
restart at each packed document.

---

**`past_key_values`** `Cache | None` — default: None

Release KV cache; packed calls reject it.

---

**`inputs_embeds`** `torch.Tensor | None` — default: None

Tensor of shape \[batch, sequence, hidden], used instead
of `input_ids`.

---

**`labels`** `torch.Tensor | None` — default: None

Tensor of shape \[batch, sequence] for the release's own loss.

---

**`use_cache`** `bool | None` — default: None

Whether to return a cache; packed calls reject True.

---

**`output_attentions`** `bool | None` — default: None

Release flag.

---

**`output_hidden_states`** `bool | None` — default: None

Release flag.

---

**`return_dict`** `bool | None` — default: None

Whether the release returns a `ModelOutput`.

---

**`cache_position`** `torch.Tensor | None` — default: None

Tensor of shape \[sequence] holding cache positions.

---

**`cu_seq_lens_q`** `torch.Tensor | None` — default: None

Query boundaries; see `_FlashBoundaries.cu_q`.

---

**`cu_seq_lens_k`** `torch.Tensor | None` — default: None

Key boundaries; see `_FlashBoundaries.cu_k`.

---

**`max_length_q`** `int | None` — default: None

Longest query segment, in tokens.

---

**`max_length_k`** `int | None` — default: None

Longest key segment, in tokens.

---

**`padding_mask`** `torch.Tensor | None` — default: None

bool Tensor of shape \[batch, sequence] from the collater;
ignored.

---

**`_packed_seq_ids`** `torch.Tensor | None` — default: None

Tensor of shape \[batch, sequence] holding document IDs
from the collater; ignored.

---

**Returns:** `Any`

The release's causal-LM output, whose logits have shape \[batch,

**Raises:**

* `ValueError`: If packing metadata is incomplete or a cache is requested
  with packing.

```python
nemo_automodel.components.models.nemotron_nas.packing._validate_backend(
    self: torch.nn.Module,
    attn_implementation: str
) -> None
```

```python
nemo_automodel.components.models.nemotron_nas.packing.apply_packing_adapters(
    model: torch.nn.Module,
    mesh: nemo_automodel.components.distributed.mesh.MeshContext | None = None
) -> torch.nn.Module
```

Add explicit packed boundaries without changing unpacked/generation calls.

**Parameters:**

**`model`** `nn.Module`

Released remote DeciLM causal LM, before FSDP/PP wrapping.

---

**`mesh`** `MeshContext | None` — default: None

Construction context supplied by the runtime-hook dispatcher.

---

**Returns:** `nn.Module`

The same module, with its parameter names and ordinary forwards intact.