Skip to navigation

nemo_automodel.components.models.nemotron_flash.packing

View as MarkdownOpen in Claude

Packed execution for the released Flash-1B modules, including memory tokens.

Uses the unmodified modules from nvidia/Nemotron-Flash-1B revision da87f707f678b5936e67fdcfe598f76ebcc5fdbc. Each document receives its own learned prefix; attention, short convolutions, and recurrent states share the same expanded document boundaries. Checkpoint parameter names remain unchanged.

Module Contents

Classes

NameDescription
_PackedInputLoader metadata for the documents packed into padded rows.
_PhysicalDocumentsBoundaries of the expanded stream, where every document starts with its own memory prefix.

Functions

NameDescription
_attention_forwardRun released FlashAttention-2 inputs with variable-length document boundaries.
_backbone_forwardRun the released backbone over prefixed documents when packing metadata is present.
_delta_forwardPass packed document boundaries to the released DeltaNet kernels.
_expand_documentsInsert one learned prefix per document, retaining differentiable gathers.
_mamba_forwardReset the Mamba2 convolution and scan state at packed document boundaries.
_root_forwardRoute loader packing metadata into the released causal-LM forward.
_short_convolution_forwardTranslate packed boundaries into the installed FLA short-convolution API.
_validate_backend-
apply_packing_adaptersAdd document-aware training while preserving released weights and generation.

API

class nemo_automodel.components.models.nemotron_flash.packing._PackedInput(
indices: torch.Tensor,
cu: torch.Tensor,
maximum: int
)
Dataclass

Loader metadata for the documents packed into padded rows.

cu
Tensor

int32 Tensor of shape [batch, max_documents + 1] holding row-local cumulative document lengths with -1 after each row’s last boundary, or a pre-flattened Tensor of shape [documents + 1] over the unpadded token stream.

indices
Tensor

int64 Tensor of shape [batch, sequence] holding row-local token positions with -1 padding, or a pre-flattened Tensor of shape [tokens] indexing the row-major [batch, sequence] layout.

maximum
int

Longest document in tokens, excluding the memory prefix.

class nemo_automodel.components.models.nemotron_flash.packing._PhysicalDocuments(
cu: torch.Tensor,
seq_idx: torch.Tensor,
maximum: int
)
Dataclass

Boundaries of the expanded stream, where every document starts with its own memory prefix.

cu
Tensor

int32 Tensor of shape [documents + 1] holding cumulative lengths of the prefixed documents over the [physical_tokens] stream, where physical_tokens = tokens + documents * prefix.

maximum
int

Longest prefixed document, in tokens.

seq_idx
Tensor

int32 Tensor of shape [1, physical_tokens] holding each physical token’s document index.

nemo_automodel.components.models.nemotron_flash.packing._attention_forward(
self: torch.nn.Module,
query: torch.Tensor,
key: torch.Tensor,
value: torch.Tensor,
query_length: int,
kwargs: typing.Any = {}
) -> torch.Tensor

Run released FlashAttention-2 inputs with variable-length document boundaries.

Parameters:

query
torch.Tensor

Tensor of shape [batch, sequence, heads, head_dim]. Packed calls pass [1, physical_tokens, heads, head_dim].

key
torch.Tensor

Tensor of shape [batch, sequence, heads, head_dim], already repeated to the query heads by the release.

value
torch.Tensor

Tensor of shape [batch, sequence, heads, value_head_dim].

attention_mask
torch.Tensor | _PhysicalDocuments | None

Tensor of shape [batch, sequence] holding the release’s 0/1 padding mask, or the prefixed documents’ boundaries.

query_length
int

Query sequence length.

**kwargs
AnyDefaults to {}

Release options: dropout, softmax_scale and use_sliding_windows.

Returns: torch.Tensor

Tensor of shape [batch, sequence, heads, value_head_dim].

nemo_automodel.components.models.nemotron_flash.packing._backbone_forward(
self: torch.nn.Module,
input_ids: torch.Tensor | None = None,
attention_mask: torch.Tensor | nemo_automodel.components.models.nemotron_flash.packing._PackedInput | None = None,
position_ids: torch.Tensor | None = None,
inputs_embeds: torch.Tensor | None = None,
kwargs: typing.Any = {}
) -> typing.Any

Run the released backbone over prefixed documents when packing metadata is present.

Parameters:

input_ids
torch.Tensor | NoneDefaults to None

Tensor of shape [batch, sequence] holding token IDs.

attention_mask
torch.Tensor | _PackedInput | NoneDefaults to None

Tensor of shape [batch, sequence] holding the release’s 0/1 padding mask, or the packing metadata on packed calls.

position_ids
torch.Tensor | NoneDefaults to None

Tensor of shape [batch, sequence]; unused on packed calls.

inputs_embeds
torch.Tensor | NoneDefaults to None

Tensor of shape [batch, sequence, hidden], used instead of input_ids.

**kwargs
AnyDefaults to {}

Remaining release arguments, forwarded unchanged on unpacked calls.

Returns: Any

On packed calls, a MoeModelOutputWithPast whose last_hidden_state

Raises:

  • ValueError: If a packed call requests attention or hidden-state outputs.
nemo_automodel.components.models.nemotron_flash.packing._delta_forward(
self: torch.nn.Module,
hidden_states: torch.Tensor,
kwargs: typing.Any = {}
) -> typing.Any

Pass packed document boundaries to the released DeltaNet kernels.

Parameters:

hidden_states
torch.Tensor

Tensor of shape [batch, sequence, hidden]. Packed calls pass the prefixed stream as [1, physical_tokens, hidden].

attention_mask
torch.Tensor | _PhysicalDocuments | NoneDefaults to None

Tensor of shape [batch, sequence] holding the release’s 0/1 padding mask, or the prefixed documents’ boundaries. On packed calls the boundaries reach the delta rule as cu_seqlens and the short convolutions through the release’s position_ids argument.

**kwargs
AnyDefaults to {}

Remaining release arguments.

Returns: Any

The release’s DeltaNet output tuple, whose first element has the shape

nemo_automodel.components.models.nemotron_flash.packing._expand_documents(
embeddings: torch.Tensor,
memory: torch.Tensor,
) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor, nemo_automodel.components.models.nemotron_flash.packing._PhysicalDocuments]

Insert one learned prefix per document, retaining differentiable gathers.

Parameters:

embeddings
torch.Tensor

Tensor of shape [batch, sequence, hidden] holding the padded packed rows’ token embeddings.

memory
torch.Tensor

Tensor of shape [prefix, hidden] holding the learned memory tokens prepended to every document.

packed
_PackedInput

Locations of the real tokens and document boundaries in embeddings.

Returns: torch.Tensor

A tuple of:

nemo_automodel.components.models.nemotron_flash.packing._mamba_forward(
self: torch.nn.Module,
hidden_states: torch.Tensor,
kwargs: typing.Any = {}
) -> typing.Any

Reset the Mamba2 convolution and scan state at packed document boundaries.

Parameters:

hidden_states
torch.Tensor

Tensor of shape [batch, sequence, hidden]. Packed calls pass the prefixed stream as [1, physical_tokens, hidden].

attention_mask
torch.Tensor | _PhysicalDocuments | NoneDefaults to None

The release’s mask, or the prefixed documents’ boundaries on packed calls.

**kwargs
AnyDefaults to {}

Remaining release arguments, forwarded unchanged on unpacked calls.

Returns: Any

A tuple of a Tensor with the shape of hidden_states and the

Raises:

  • ValueError: If a packed call uses Mamba tensor parallelism.
nemo_automodel.components.models.nemotron_flash.packing._root_forward(
self: torch.nn.Module,
input_ids: torch.Tensor | None = None,
attention_mask: torch.Tensor | None = None,
position_ids: torch.Tensor | None = None,
past_key_values: transformers.cache_utils.Cache | list[torch.Tensor] | None = None,
inputs_embeds: torch.Tensor | None = None,
labels: torch.Tensor | None = None,
use_cache: bool | None = None,
output_attentions: bool | None = None,
output_hidden_states: bool | None = None,
return_dict: bool | None = None,
calc_logits_for_entire_prompt: bool = True,
fla_past_key_values: transformers.cache_utils.Cache | None = None,
mamba_inference_params: mamba_ssm.utils.generation.InferenceParams | None = None,
packed_token_indices: torch.Tensor | None = None,
cu_seqlens: torch.Tensor | None = None,
max_seqlen: int | None = None,
cu_seq_lens_q: torch.Tensor | None = None,
cu_seq_lens_k: torch.Tensor | None = None,
max_length_q: int | None = None,
max_length_k: int | None = None,
padding_mask: torch.Tensor | None = None,
_packed_seq_ids: torch.Tensor | None = None
) -> typing.Any

Route loader packing metadata into the released causal-LM forward.

Parameters:

input_ids
torch.Tensor | NoneDefaults to None

Tensor of shape [batch, sequence] holding token IDs. Packed rows hold several documents followed by padding.

attention_mask
torch.Tensor | NoneDefaults to None

Tensor of shape [batch, sequence] holding the release’s 0/1 padding mask. Packed calls replace it with the packing metadata.

position_ids
torch.Tensor | NoneDefaults to None

Tensor of shape [batch, sequence]. Packed calls ignore it and restart positions at each document’s memory prefix.

past_key_values
Cache | list[torch.Tensor] | NoneDefaults to None

Release attention cache; packed calls reject it.

inputs_embeds
torch.Tensor | NoneDefaults to None

Tensor of shape [batch, sequence, hidden], used instead of input_ids.

labels
torch.Tensor | NoneDefaults to None

Tensor of shape [batch, sequence] for the release’s own loss.

use_cache
bool | NoneDefaults to None

Whether to return caches; packed calls reject True.

output_attentions
bool | NoneDefaults to None

Release flag; packed calls reject True.

output_hidden_states
bool | NoneDefaults to None

Release flag; packed calls reject True.

return_dict
bool | NoneDefaults to None

Whether the release returns a ModelOutput.

calc_logits_for_entire_prompt
boolDefaults to True

Release flag selecting full-prompt logits.

fla_past_key_values
Cache | NoneDefaults to None

Release DeltaNet cache; packed calls reject it.

mamba_inference_params
InferenceParams | NoneDefaults to None

Release Mamba cache; packed calls reject it.

packed_token_indices
torch.Tensor | NoneDefaults to None

Real-token positions; see _PackedInput.indices.

cu_seqlens
torch.Tensor | NoneDefaults to None

Document boundaries; see _PackedInput.cu.

max_seqlen
int | NoneDefaults to None

Longest document in tokens.

cu_seq_lens_q
torch.Tensor | NoneDefaults to None

int32 Tensor of shape [segments + 1] from the generic FlashAttention packer; rejected without packed_token_indices.

cu_seq_lens_k
torch.Tensor | NoneDefaults to None

int32 Tensor of shape [segments + 1]; rejected likewise.

max_length_q
int | NoneDefaults to None

Generic FlashAttention maximum; rejected likewise.

max_length_k
int | NoneDefaults to None

Generic FlashAttention maximum; rejected likewise.

padding_mask
torch.Tensor | NoneDefaults to None

bool Tensor of shape [batch, sequence] from the collater; ignored.

_packed_seq_ids
torch.Tensor | NoneDefaults to None

Tensor of shape [batch, sequence] holding document IDs from the collater; ignored.

Returns: Any

The release’s causal-LM output. Its logits have shape [batch, sequence,

Raises:

  • ValueError: If packing metadata is incomplete, a cache is requested with packing, or only generic FlashAttention metadata is supplied.
nemo_automodel.components.models.nemotron_flash.packing._short_convolution_forward(
self: torch.nn.Module,
args: typing.Any = (),
kwargs: typing.Any = {}
) -> typing.Any

Translate packed boundaries into the installed FLA short-convolution API.

Parameters:

*args
AnyDefaults to ()

Positional release arguments.

seq_idx
torch.Tensor | _PhysicalDocuments | NoneDefaults to None

int32 Tensor of shape [batch, sequence] holding document indices, or the prefixed documents’ boundaries on packed calls. Those become cu_seqlens when the installed convolution accepts it and otherwise its seq_idx tensor.

**kwargs
AnyDefaults to {}

Remaining release arguments, including x: Tensor of shape [batch, sequence, channels].

Returns: Any

The convolution’s output tuple, whose first element has the shape of

nemo_automodel.components.models.nemotron_flash.packing._validate_backend(
self: torch.nn.Module,
attn_implementation: str
) -> None
nemo_automodel.components.models.nemotron_flash.packing.apply_packing_adapters(
model: torch.nn.Module,
) -> torch.nn.Module

Add document-aware training while preserving released weights and generation.

Parameters:

model
nn.Module

Released remote Flash-1B causal LM, before distributed wrapping.

mesh
MeshContext | NoneDefaults to None

Construction context supplied by the runtime-hook dispatcher.

Returns: nn.Module

The original model with an opt-in packed path and unchanged parameter names.