nemo_automodel.components.models.nemotron_nas.packing
nemo_automodel.components.models.nemotron_nas.packing
Thread public FlashAttention boundaries through the released DeciLM API.
The publisher’s classes and checkpoint layout are retained. Reference interface: nvidia/Llama-3_3-Nemotron-Super-49B-v1_5 at 420ba7d28211abf116b8b103ab700d92619daf98 (Apache-2.0 modeling_decilm.py). No publisher implementation is copied here.
Module Contents
Classes
Functions
API
Generic FlashAttention varlen boundaries for packed rows.
int32 Tensor of shape [segments + 1] holding the matching key boundaries.
int32 Tensor of shape [segments + 1] holding cumulative query lengths over the row-major flattened [batch * sequence] stream. Physical padding runs are segments of their own.
Longest key segment, in tokens.
Longest query segment, in tokens.
Dispatch packed calls to varlen attention and others to the released forward.
Parameters:
Positional release arguments; the first is hidden_states.
Query boundaries forwarded by the shared pipeline
forward; see _FlashBoundaries.cu_q.
Key boundaries; see _FlashBoundaries.cu_k.
Longest query segment, in tokens.
Longest key segment, in tokens.
Release arguments, including hidden_states: Tensor of
shape [batch, sequence, hidden] and attention_mask: Tensor of
shape [batch, sequence] or packed segment boundaries.
Returns: Any
The release’s attention tuple, whose first element is a Tensor of shape
Raises:
ValueError: If pipeline metadata is incomplete or a cache is requested with packing.
Keep packed boundaries instead of building the release’s causal mask.
Parameters:
Tensor of shape [batch, sequence] holding the release’s 0/1 padding mask, or packed segment boundaries.
Remaining release mask arguments.
Remaining release mask arguments.
Returns: Any
The boundaries unchanged on packed calls; otherwise the release’s mask.
Use the released projections/RoPE with explicit, padding-aware segments.
As in the release, FP32 query, key and value return to the autocast dtype before FlashAttention; without autocast, to the pre-quantization or query-projection weight dtype.
Parameters:
Tensor of shape [batch, sequence, hidden] holding normalized packed rows.
Segment boundaries over the flattened rows.
Release attention arguments. position_embeddings is a
tuple of cos and sin Tensors of shape [batch, sequence, head_dim];
without it, position_ids (Tensor of shape [batch, sequence])
rebuilds them.
Returns: Any
A tuple of a Tensor of shape [batch, sequence, hidden] holding the
Raises:
ValueError: If a KV cache or sink attention is requested.
Route generic FlashAttention packing metadata into the released DeciLM forward.
Parameters:
Tensor of shape [batch, sequence] holding token IDs. Packed rows hold several documents followed by padding.
Tensor of shape [batch, sequence] holding the release’s 0/1 padding mask. Packed calls replace it with segment boundaries.
Tensor of shape [batch, sequence] holding positions that restart at each packed document.
Release KV cache; packed calls reject it.
Tensor of shape [batch, sequence, hidden], used instead
of input_ids.
Tensor of shape [batch, sequence] for the release’s own loss.
Whether to return a cache; packed calls reject True.
Release flag.
Release flag.
Whether the release returns a ModelOutput.
Tensor of shape [sequence] holding cache positions.
Query boundaries; see _FlashBoundaries.cu_q.
Key boundaries; see _FlashBoundaries.cu_k.
Longest query segment, in tokens.
Longest key segment, in tokens.
bool Tensor of shape [batch, sequence] from the collater; ignored.
Tensor of shape [batch, sequence] holding document IDs from the collater; ignored.
Returns: Any
The release’s causal-LM output, whose logits have shape [batch,
Raises:
ValueError: If packing metadata is incomplete or a cache is requested with packing.
Add explicit packed boundaries without changing unpacked/generation calls.
Parameters:
Released remote DeciLM causal LM, before FSDP/PP wrapping.
Construction context supplied by the runtime-hook dispatcher.
Returns: nn.Module
The same module, with its parameter names and ordinary forwards intact.