> This page is for version 0.5.0 · 26.06.
> For other versions, use one of these documentation indexes:
> - Nightly (default): https://docs.nvidia.com/nemo/automodel/nightly/llms.txt
> - Latest: https://docs.nvidia.com/nemo/automodel/latest/llms.txt
> - 0.7.0 · 26.10: https://docs.nvidia.com/nemo/automodel/v0.7/llms.txt
> - 0.5.0 · 26.06: https://docs.nvidia.com/nemo/automodel/v0.5/llms.txt
> - 0.4.0 · 26.04: https://docs.nvidia.com/nemo/automodel/v0.4/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt.

# nemo_automodel.components.models.nemotron_flash.packing

Packed execution for the released Flash-1B modules, including memory tokens.

Uses the unmodified modules from nvidia/Nemotron-Flash-1B revision
da87f707f678b5936e67fdcfe598f76ebcc5fdbc. Each document receives its own
learned prefix; attention, short convolutions, and recurrent states share the
same expanded document boundaries. Checkpoint parameter names remain unchanged.

## Module Contents

### Classes

| Name                                                                                                | Description                                                                                |
| --------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ |
| [`_PackedInput`](#nemo_automodel-components-models-nemotron_flash-packing-_PackedInput)             | Loader metadata for the documents packed into padded rows.                                 |
| [`_PhysicalDocuments`](#nemo_automodel-components-models-nemotron_flash-packing-_PhysicalDocuments) | Boundaries of the expanded stream, where every document starts with its own memory prefix. |

### Functions

| Name                                                                                                                | Description                                                                         |
| ------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- |
| [`_attention_forward`](#nemo_automodel-components-models-nemotron_flash-packing-_attention_forward)                 | Run released FlashAttention-2 inputs with variable-length document boundaries.      |
| [`_backbone_forward`](#nemo_automodel-components-models-nemotron_flash-packing-_backbone_forward)                   | Run the released backbone over prefixed documents when packing metadata is present. |
| [`_delta_forward`](#nemo_automodel-components-models-nemotron_flash-packing-_delta_forward)                         | Pass packed document boundaries to the released DeltaNet kernels.                   |
| [`_expand_documents`](#nemo_automodel-components-models-nemotron_flash-packing-_expand_documents)                   | Insert one learned prefix per document, retaining differentiable gathers.           |
| [`_mamba_forward`](#nemo_automodel-components-models-nemotron_flash-packing-_mamba_forward)                         | Reset the Mamba2 convolution and scan state at packed document boundaries.          |
| [`_root_forward`](#nemo_automodel-components-models-nemotron_flash-packing-_root_forward)                           | Route loader packing metadata into the released causal-LM forward.                  |
| [`_short_convolution_forward`](#nemo_automodel-components-models-nemotron_flash-packing-_short_convolution_forward) | Translate packed boundaries into the installed FLA short-convolution API.           |
| [`_validate_backend`](#nemo_automodel-components-models-nemotron_flash-packing-_validate_backend)                   | -                                                                                   |
| [`apply_packing_adapters`](#nemo_automodel-components-models-nemotron_flash-packing-apply_packing_adapters)         | Add document-aware training while preserving released weights and generation.       |

### API

```python
class nemo_automodel.components.models.nemotron_flash.packing._PackedInput(
    indices: torch.Tensor,
    cu: torch.Tensor,
    maximum: int
)
```

Dataclass

Loader metadata for the documents packed into padded rows.

**`cu`** `Tensor`

int32 Tensor of shape \[batch, max_documents + 1] holding row-local
cumulative document lengths with `-1` after each row's last
boundary, or a pre-flattened Tensor of shape \[documents + 1] over
the unpadded token stream.

---

**`indices`** `Tensor`

int64 Tensor of shape \[batch, sequence] holding row-local token
positions with `-1` padding, or a pre-flattened Tensor of shape
\[tokens] indexing the row-major \[batch, sequence] layout.

---

**`maximum`** `int`

Longest document in tokens, excluding the memory prefix.

---

```python
class nemo_automodel.components.models.nemotron_flash.packing._PhysicalDocuments(
    cu: torch.Tensor,
    seq_idx: torch.Tensor,
    maximum: int
)
```

Dataclass

Boundaries of the expanded stream, where every document starts with its own memory prefix.

**`cu`** `Tensor`

int32 Tensor of shape \[documents + 1] holding cumulative lengths of
the prefixed documents over the \[physical_tokens] stream, where
physical_tokens = tokens + documents \* prefix.

---

**`maximum`** `int`

Longest prefixed document, in tokens.

---

**`seq_idx`** `Tensor`

int32 Tensor of shape \[1, physical_tokens] holding each
physical token's document index.

---

```python
nemo_automodel.components.models.nemotron_flash.packing._attention_forward(
    self: torch.nn.Module,
    query: torch.Tensor,
    key: torch.Tensor,
    value: torch.Tensor,
    attention_mask: torch.Tensor | nemo_automodel.components.models.nemotron_flash.packing._PhysicalDocuments | None,
    query_length: int,
    kwargs: typing.Any = {}
) -> torch.Tensor
```

Run released FlashAttention-2 inputs with variable-length document boundaries.

**Parameters:**

**`query`** `torch.Tensor`

Tensor of shape \[batch, sequence, heads, head_dim]. Packed calls
pass \[1, physical_tokens, heads, head_dim].

---

**`key`** `torch.Tensor`

Tensor of shape \[batch, sequence, heads, head_dim], already
repeated to the query heads by the release.

---

**`value`** `torch.Tensor`

Tensor of shape \[batch, sequence, heads, value_head_dim].

---

**`attention_mask`** `torch.Tensor | _PhysicalDocuments | None`

Tensor of shape \[batch, sequence] holding the release's
0/1 padding mask, or the prefixed documents' boundaries.

---

**`query_length`** `int`

Query sequence length.

---

**`**kwargs`** `Any` — default: \{}

Release options: `dropout`, `softmax_scale` and
`use_sliding_windows`.

---

**Returns:** `torch.Tensor`

Tensor of shape \[batch, sequence, heads, value_head_dim].

```python
nemo_automodel.components.models.nemotron_flash.packing._backbone_forward(
    self: torch.nn.Module,
    input_ids: torch.Tensor | None = None,
    attention_mask: torch.Tensor | nemo_automodel.components.models.nemotron_flash.packing._PackedInput | None = None,
    position_ids: torch.Tensor | None = None,
    inputs_embeds: torch.Tensor | None = None,
    kwargs: typing.Any = {}
) -> typing.Any
```

Run the released backbone over prefixed documents when packing metadata is present.

**Parameters:**

**`input_ids`** `torch.Tensor | None` — default: None

Tensor of shape \[batch, sequence] holding token IDs.

---

**`attention_mask`** `torch.Tensor | _PackedInput | None` — default: None

Tensor of shape \[batch, sequence] holding the release's
0/1 padding mask, or the packing metadata on packed calls.

---

**`position_ids`** `torch.Tensor | None` — default: None

Tensor of shape \[batch, sequence]; unused on packed calls.

---

**`inputs_embeds`** `torch.Tensor | None` — default: None

Tensor of shape \[batch, sequence, hidden], used instead
of `input_ids`.

---

**`**kwargs`** `Any` — default: \{}

Remaining release arguments, forwarded unchanged on unpacked
calls.

---

**Returns:** `Any`

On packed calls, a `MoeModelOutputWithPast` whose last_hidden_state

**Raises:**

* `ValueError`: If a packed call requests attention or hidden-state outputs.

```python
nemo_automodel.components.models.nemotron_flash.packing._delta_forward(
    self: torch.nn.Module,
    hidden_states: torch.Tensor,
    attention_mask: torch.Tensor | nemo_automodel.components.models.nemotron_flash.packing._PhysicalDocuments | None = None,
    kwargs: typing.Any = {}
) -> typing.Any
```

Pass packed document boundaries to the released DeltaNet kernels.

**Parameters:**

**`hidden_states`** `torch.Tensor`

Tensor of shape \[batch, sequence, hidden]. Packed calls
pass the prefixed stream as \[1, physical_tokens, hidden].

---

**`attention_mask`** `torch.Tensor | _PhysicalDocuments | None` — default: None

Tensor of shape \[batch, sequence] holding the release's
0/1 padding mask, or the prefixed documents' boundaries. On packed
calls the boundaries reach the delta rule as `cu_seqlens` and the
short convolutions through the release's `position_ids` argument.

---

**`**kwargs`** `Any` — default: \{}

Remaining release arguments.

---

**Returns:** `Any`

The release's DeltaNet output tuple, whose first element has the shape

```python
nemo_automodel.components.models.nemotron_flash.packing._expand_documents(
    embeddings: torch.Tensor,
    memory: torch.Tensor,
    packed: nemo_automodel.components.models.nemotron_flash.packing._PackedInput
) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor, nemo_automodel.components.models.nemotron_flash.packing._PhysicalDocuments]
```

Insert one learned prefix per document, retaining differentiable gathers.

**Parameters:**

**`embeddings`** `torch.Tensor`

Tensor of shape \[batch, sequence, hidden] holding the padded
packed rows' token embeddings.

---

**`memory`** `torch.Tensor`

Tensor of shape \[prefix, hidden] holding the learned memory
tokens prepended to every document.

---

**`packed`** `_PackedInput`

Locations of the real tokens and document boundaries in
`embeddings`.

---

**Returns:** `torch.Tensor`

A tuple of:

```python
nemo_automodel.components.models.nemotron_flash.packing._mamba_forward(
    self: torch.nn.Module,
    hidden_states: torch.Tensor,
    attention_mask: torch.Tensor | nemo_automodel.components.models.nemotron_flash.packing._PhysicalDocuments | None = None,
    kwargs: typing.Any = {}
) -> typing.Any
```

Reset the Mamba2 convolution and scan state at packed document boundaries.

**Parameters:**

**`hidden_states`** `torch.Tensor`

Tensor of shape \[batch, sequence, hidden]. Packed calls
pass the prefixed stream as \[1, physical_tokens, hidden].

---

**`attention_mask`** `torch.Tensor | _PhysicalDocuments | None` — default: None

The release's mask, or the prefixed documents'
boundaries on packed calls.

---

**`**kwargs`** `Any` — default: \{}

Remaining release arguments, forwarded unchanged on unpacked
calls.

---

**Returns:** `Any`

A tuple of a Tensor with the shape of `hidden_states` and the

**Raises:**

* `ValueError`: If a packed call uses Mamba tensor parallelism.

```python
nemo_automodel.components.models.nemotron_flash.packing._root_forward(
    self: torch.nn.Module,
    input_ids: torch.Tensor | None = None,
    attention_mask: torch.Tensor | None = None,
    position_ids: torch.Tensor | None = None,
    past_key_values: transformers.cache_utils.Cache | list[torch.Tensor] | None = None,
    inputs_embeds: torch.Tensor | None = None,
    labels: torch.Tensor | None = None,
    use_cache: bool | None = None,
    output_attentions: bool | None = None,
    output_hidden_states: bool | None = None,
    return_dict: bool | None = None,
    calc_logits_for_entire_prompt: bool = True,
    fla_past_key_values: transformers.cache_utils.Cache | None = None,
    mamba_inference_params: mamba_ssm.utils.generation.InferenceParams | None = None,
    packed_token_indices: torch.Tensor | None = None,
    cu_seqlens: torch.Tensor | None = None,
    max_seqlen: int | None = None,
    cu_seq_lens_q: torch.Tensor | None = None,
    cu_seq_lens_k: torch.Tensor | None = None,
    max_length_q: int | None = None,
    max_length_k: int | None = None,
    padding_mask: torch.Tensor | None = None,
    _packed_seq_ids: torch.Tensor | None = None
) -> typing.Any
```

Route loader packing metadata into the released causal-LM forward.

**Parameters:**

**`input_ids`** `torch.Tensor | None` — default: None

Tensor of shape \[batch, sequence] holding token IDs. Packed
rows hold several documents followed by padding.

---

**`attention_mask`** `torch.Tensor | None` — default: None

Tensor of shape \[batch, sequence] holding the release's
0/1 padding mask. Packed calls replace it with the packing metadata.

---

**`position_ids`** `torch.Tensor | None` — default: None

Tensor of shape \[batch, sequence]. Packed calls ignore it
and restart positions at each document's memory prefix.

---

**`past_key_values`** `Cache | list[torch.Tensor] | None` — default: None

Release attention cache; packed calls reject it.

---

**`inputs_embeds`** `torch.Tensor | None` — default: None

Tensor of shape \[batch, sequence, hidden], used instead
of `input_ids`.

---

**`labels`** `torch.Tensor | None` — default: None

Tensor of shape \[batch, sequence] for the release's own loss.

---

**`use_cache`** `bool | None` — default: None

Whether to return caches; packed calls reject True.

---

**`output_attentions`** `bool | None` — default: None

Release flag; packed calls reject True.

---

**`output_hidden_states`** `bool | None` — default: None

Release flag; packed calls reject True.

---

**`return_dict`** `bool | None` — default: None

Whether the release returns a `ModelOutput`.

---

**`calc_logits_for_entire_prompt`** `bool` — default: True

Release flag selecting full-prompt logits.

---

**`fla_past_key_values`** `Cache | None` — default: None

Release DeltaNet cache; packed calls reject it.

---

**`mamba_inference_params`** `InferenceParams | None` — default: None

Release Mamba cache; packed calls reject it.

---

**`packed_token_indices`** `torch.Tensor | None` — default: None

Real-token positions; see `_PackedInput.indices`.

---

**`cu_seqlens`** `torch.Tensor | None` — default: None

Document boundaries; see `_PackedInput.cu`.

---

**`max_seqlen`** `int | None` — default: None

Longest document in tokens.

---

**`cu_seq_lens_q`** `torch.Tensor | None` — default: None

int32 Tensor of shape \[segments + 1] from the generic
FlashAttention packer; rejected without `packed_token_indices`.

---

**`cu_seq_lens_k`** `torch.Tensor | None` — default: None

int32 Tensor of shape \[segments + 1]; rejected likewise.

---

**`max_length_q`** `int | None` — default: None

Generic FlashAttention maximum; rejected likewise.

---

**`max_length_k`** `int | None` — default: None

Generic FlashAttention maximum; rejected likewise.

---

**`padding_mask`** `torch.Tensor | None` — default: None

bool Tensor of shape \[batch, sequence] from the collater;
ignored.

---

**`_packed_seq_ids`** `torch.Tensor | None` — default: None

Tensor of shape \[batch, sequence] holding document IDs
from the collater; ignored.

---

**Returns:** `Any`

The release's causal-LM output. Its logits have shape \[batch, sequence,

**Raises:**

* `ValueError`: If packing metadata is incomplete, a cache is requested
  with packing, or only generic FlashAttention metadata is supplied.

```python
nemo_automodel.components.models.nemotron_flash.packing._short_convolution_forward(
    self: torch.nn.Module,
    args: typing.Any = (),
    seq_idx: torch.Tensor | nemo_automodel.components.models.nemotron_flash.packing._PhysicalDocuments | None = None,
    kwargs: typing.Any = {}
) -> typing.Any
```

Translate packed boundaries into the installed FLA short-convolution API.

**Parameters:**

**`*args`** `Any` — default: ()

Positional release arguments.

---

**`seq_idx`** `torch.Tensor | _PhysicalDocuments | None` — default: None

int32 Tensor of shape \[batch, sequence] holding document
indices, or the prefixed documents' boundaries on packed calls.
Those become `cu_seqlens` when the installed convolution accepts
it and otherwise its `seq_idx` tensor.

---

**`**kwargs`** `Any` — default: \{}

Remaining release arguments, including `x`: Tensor of shape
\[batch, sequence, channels].

---

**Returns:** `Any`

The convolution's output tuple, whose first element has the shape of

```python
nemo_automodel.components.models.nemotron_flash.packing._validate_backend(
    self: torch.nn.Module,
    attn_implementation: str
) -> None
```

```python
nemo_automodel.components.models.nemotron_flash.packing.apply_packing_adapters(
    model: torch.nn.Module,
    mesh: nemo_automodel.components.distributed.mesh.MeshContext | None = None
) -> torch.nn.Module
```

Add document-aware training while preserving released weights and generation.

**Parameters:**

**`model`** `nn.Module`

Released remote Flash-1B causal LM, before distributed wrapping.

---

**`mesh`** `MeshContext | None` — default: None

Construction context supplied by the runtime-hook dispatcher.

---

**Returns:** `nn.Module`

The original model with an opt-in packed path and unchanged parameter names.