nemo_automodel.components.models.nemotron_flash.packing
nemo_automodel.components.models.nemotron_flash.packing
Packed execution for the released Flash-1B modules, including memory tokens.
Uses the unmodified modules from nvidia/Nemotron-Flash-1B revision da87f707f678b5936e67fdcfe598f76ebcc5fdbc. Each document receives its own learned prefix; attention, short convolutions, and recurrent states share the same expanded document boundaries. Checkpoint parameter names remain unchanged.
Module Contents
Classes
Functions
API
Loader metadata for the documents packed into padded rows.
int32 Tensor of shape [batch, max_documents + 1] holding row-local
cumulative document lengths with -1 after each row’s last
boundary, or a pre-flattened Tensor of shape [documents + 1] over
the unpadded token stream.
int64 Tensor of shape [batch, sequence] holding row-local token
positions with -1 padding, or a pre-flattened Tensor of shape
[tokens] indexing the row-major [batch, sequence] layout.
Longest document in tokens, excluding the memory prefix.
Boundaries of the expanded stream, where every document starts with its own memory prefix.
int32 Tensor of shape [documents + 1] holding cumulative lengths of the prefixed documents over the [physical_tokens] stream, where physical_tokens = tokens + documents * prefix.
Longest prefixed document, in tokens.
int32 Tensor of shape [1, physical_tokens] holding each physical token’s document index.
Run released FlashAttention-2 inputs with variable-length document boundaries.
Parameters:
Tensor of shape [batch, sequence, heads, head_dim]. Packed calls pass [1, physical_tokens, heads, head_dim].
Tensor of shape [batch, sequence, heads, head_dim], already repeated to the query heads by the release.
Tensor of shape [batch, sequence, heads, value_head_dim].
Tensor of shape [batch, sequence] holding the release’s 0/1 padding mask, or the prefixed documents’ boundaries.
Query sequence length.
Release options: dropout, softmax_scale and
use_sliding_windows.
Returns: torch.Tensor
Tensor of shape [batch, sequence, heads, value_head_dim].
Run the released backbone over prefixed documents when packing metadata is present.
Parameters:
Tensor of shape [batch, sequence] holding token IDs.
Tensor of shape [batch, sequence] holding the release’s 0/1 padding mask, or the packing metadata on packed calls.
Tensor of shape [batch, sequence]; unused on packed calls.
Tensor of shape [batch, sequence, hidden], used instead
of input_ids.
Remaining release arguments, forwarded unchanged on unpacked calls.
Returns: Any
On packed calls, a MoeModelOutputWithPast whose last_hidden_state
Raises:
ValueError: If a packed call requests attention or hidden-state outputs.
Pass packed document boundaries to the released DeltaNet kernels.
Parameters:
Tensor of shape [batch, sequence, hidden]. Packed calls pass the prefixed stream as [1, physical_tokens, hidden].
Tensor of shape [batch, sequence] holding the release’s
0/1 padding mask, or the prefixed documents’ boundaries. On packed
calls the boundaries reach the delta rule as cu_seqlens and the
short convolutions through the release’s position_ids argument.
Remaining release arguments.
Returns: Any
The release’s DeltaNet output tuple, whose first element has the shape
Insert one learned prefix per document, retaining differentiable gathers.
Parameters:
Tensor of shape [batch, sequence, hidden] holding the padded packed rows’ token embeddings.
Tensor of shape [prefix, hidden] holding the learned memory tokens prepended to every document.
Locations of the real tokens and document boundaries in
embeddings.
Returns: torch.Tensor
A tuple of:
Reset the Mamba2 convolution and scan state at packed document boundaries.
Parameters:
Tensor of shape [batch, sequence, hidden]. Packed calls pass the prefixed stream as [1, physical_tokens, hidden].
The release’s mask, or the prefixed documents’ boundaries on packed calls.
Remaining release arguments, forwarded unchanged on unpacked calls.
Returns: Any
A tuple of a Tensor with the shape of hidden_states and the
Raises:
ValueError: If a packed call uses Mamba tensor parallelism.
Route loader packing metadata into the released causal-LM forward.
Parameters:
Tensor of shape [batch, sequence] holding token IDs. Packed rows hold several documents followed by padding.
Tensor of shape [batch, sequence] holding the release’s 0/1 padding mask. Packed calls replace it with the packing metadata.
Tensor of shape [batch, sequence]. Packed calls ignore it and restart positions at each document’s memory prefix.
Release attention cache; packed calls reject it.
Tensor of shape [batch, sequence, hidden], used instead
of input_ids.
Tensor of shape [batch, sequence] for the release’s own loss.
Whether to return caches; packed calls reject True.
Release flag; packed calls reject True.
Release flag; packed calls reject True.
Whether the release returns a ModelOutput.
Release flag selecting full-prompt logits.
Release DeltaNet cache; packed calls reject it.
Release Mamba cache; packed calls reject it.
Real-token positions; see _PackedInput.indices.
Document boundaries; see _PackedInput.cu.
Longest document in tokens.
int32 Tensor of shape [segments + 1] from the generic
FlashAttention packer; rejected without packed_token_indices.
int32 Tensor of shape [segments + 1]; rejected likewise.
Generic FlashAttention maximum; rejected likewise.
Generic FlashAttention maximum; rejected likewise.
bool Tensor of shape [batch, sequence] from the collater; ignored.
Tensor of shape [batch, sequence] holding document IDs from the collater; ignored.
Returns: Any
The release’s causal-LM output. Its logits have shape [batch, sequence,
Raises:
ValueError: If packing metadata is incomplete, a cache is requested with packing, or only generic FlashAttention metadata is supplied.
Translate packed boundaries into the installed FLA short-convolution API.
Parameters:
Positional release arguments.
int32 Tensor of shape [batch, sequence] holding document
indices, or the prefixed documents’ boundaries on packed calls.
Those become cu_seqlens when the installed convolution accepts
it and otherwise its seq_idx tensor.
Remaining release arguments, including x: Tensor of shape
[batch, sequence, channels].
Returns: Any
The convolution’s output tuple, whose first element has the shape of
Add document-aware training while preserving released weights and generation.
Parameters:
Released remote Flash-1B causal LM, before distributed wrapping.
Construction context supplied by the runtime-hook dispatcher.
Returns: nn.Module
The original model with an opt-in packed path and unchanged parameter names.