Skip to navigation

nemo_automodel.components.models.nemotron_nas.packing

View as MarkdownOpen in Claude

Thread public FlashAttention boundaries through the released DeciLM API.

The publisher’s classes and checkpoint layout are retained. Reference interface: nvidia/Llama-3_3-Nemotron-Super-49B-v1_5 at 420ba7d28211abf116b8b103ab700d92619daf98 (Apache-2.0 modeling_decilm.py). No publisher implementation is copied here.

Module Contents

Classes

NameDescription
_FlashBoundariesGeneric FlashAttention varlen boundaries for packed rows.

Functions

NameDescription
_attention_forwardDispatch packed calls to varlen attention and others to the released forward.
_causal_maskKeep packed boundaries instead of building the release’s causal mask.
_packed_attentionUse the released projections/RoPE with explicit, padding-aware segments.
_root_forwardRoute generic FlashAttention packing metadata into the released DeciLM forward.
_validate_backend-
apply_packing_adaptersAdd explicit packed boundaries without changing unpacked/generation calls.

API

class nemo_automodel.components.models.nemotron_nas.packing._FlashBoundaries(
cu_q: torch.Tensor,
cu_k: torch.Tensor,
max_q: int,
max_k: int
)
Dataclass

Generic FlashAttention varlen boundaries for packed rows.

cu_k
Tensor

int32 Tensor of shape [segments + 1] holding the matching key boundaries.

cu_q
Tensor

int32 Tensor of shape [segments + 1] holding cumulative query lengths over the row-major flattened [batch * sequence] stream. Physical padding runs are segments of their own.

max_k
int

Longest key segment, in tokens.

max_q
int

Longest query segment, in tokens.

nemo_automodel.components.models.nemotron_nas.packing._attention_forward(
self: torch.nn.Module,
args: typing.Any = (),
cu_seq_lens_q: torch.Tensor | None = None,
cu_seq_lens_k: torch.Tensor | None = None,
max_length_q: int | None = None,
max_length_k: int | None = None,
kwargs: typing.Any = {}
) -> typing.Any

Dispatch packed calls to varlen attention and others to the released forward.

Parameters:

*args
AnyDefaults to ()

Positional release arguments; the first is hidden_states.

cu_seq_lens_q
torch.Tensor | NoneDefaults to None

Query boundaries forwarded by the shared pipeline forward; see _FlashBoundaries.cu_q.

cu_seq_lens_k
torch.Tensor | NoneDefaults to None

Key boundaries; see _FlashBoundaries.cu_k.

max_length_q
int | NoneDefaults to None

Longest query segment, in tokens.

max_length_k
int | NoneDefaults to None

Longest key segment, in tokens.

**kwargs
AnyDefaults to {}

Release arguments, including hidden_states: Tensor of shape [batch, sequence, hidden] and attention_mask: Tensor of shape [batch, sequence] or packed segment boundaries.

Returns: Any

The release’s attention tuple, whose first element is a Tensor of shape

Raises:

  • ValueError: If pipeline metadata is incomplete or a cache is requested with packing.
nemo_automodel.components.models.nemotron_nas.packing._causal_mask(
self: torch.nn.Module,
args: typing.Any = (),
kwargs: typing.Any = {}
) -> typing.Any

Keep packed boundaries instead of building the release’s causal mask.

Parameters:

attention_mask
torch.Tensor | _FlashBoundaries | None

Tensor of shape [batch, sequence] holding the release’s 0/1 padding mask, or packed segment boundaries.

*args
AnyDefaults to ()

Remaining release mask arguments.

**kwargs
AnyDefaults to {}

Remaining release mask arguments.

Returns: Any

The boundaries unchanged on packed calls; otherwise the release’s mask.

nemo_automodel.components.models.nemotron_nas.packing._packed_attention(
self: torch.nn.Module,
hidden: torch.Tensor,
kwargs: typing.Any = {}
) -> typing.Any

Use the released projections/RoPE with explicit, padding-aware segments.

As in the release, FP32 query, key and value return to the autocast dtype before FlashAttention; without autocast, to the pre-quantization or query-projection weight dtype.

Parameters:

hidden
torch.Tensor

Tensor of shape [batch, sequence, hidden] holding normalized packed rows.

metadata
_FlashBoundaries

Segment boundaries over the flattened rows.

**kwargs
AnyDefaults to {}

Release attention arguments. position_embeddings is a tuple of cos and sin Tensors of shape [batch, sequence, head_dim]; without it, position_ids (Tensor of shape [batch, sequence]) rebuilds them.

Returns: Any

A tuple of a Tensor of shape [batch, sequence, hidden] holding the

Raises:

  • ValueError: If a KV cache or sink attention is requested.
nemo_automodel.components.models.nemotron_nas.packing._root_forward(
self: torch.nn.Module,
input_ids: torch.Tensor | None = None,
attention_mask: torch.Tensor | None = None,
position_ids: torch.Tensor | None = None,
past_key_values: transformers.cache_utils.Cache | None = None,
inputs_embeds: torch.Tensor | None = None,
labels: torch.Tensor | None = None,
use_cache: bool | None = None,
output_attentions: bool | None = None,
output_hidden_states: bool | None = None,
return_dict: bool | None = None,
cache_position: torch.Tensor | None = None,
cu_seq_lens_q: torch.Tensor | None = None,
cu_seq_lens_k: torch.Tensor | None = None,
max_length_q: int | None = None,
max_length_k: int | None = None,
padding_mask: torch.Tensor | None = None,
_packed_seq_ids: torch.Tensor | None = None
) -> typing.Any

Route generic FlashAttention packing metadata into the released DeciLM forward.

Parameters:

input_ids
torch.Tensor | NoneDefaults to None

Tensor of shape [batch, sequence] holding token IDs. Packed rows hold several documents followed by padding.

attention_mask
torch.Tensor | NoneDefaults to None

Tensor of shape [batch, sequence] holding the release’s 0/1 padding mask. Packed calls replace it with segment boundaries.

position_ids
torch.Tensor | NoneDefaults to None

Tensor of shape [batch, sequence] holding positions that restart at each packed document.

past_key_values
Cache | NoneDefaults to None

Release KV cache; packed calls reject it.

inputs_embeds
torch.Tensor | NoneDefaults to None

Tensor of shape [batch, sequence, hidden], used instead of input_ids.

labels
torch.Tensor | NoneDefaults to None

Tensor of shape [batch, sequence] for the release’s own loss.

use_cache
bool | NoneDefaults to None

Whether to return a cache; packed calls reject True.

output_attentions
bool | NoneDefaults to None

Release flag.

output_hidden_states
bool | NoneDefaults to None

Release flag.

return_dict
bool | NoneDefaults to None

Whether the release returns a ModelOutput.

cache_position
torch.Tensor | NoneDefaults to None

Tensor of shape [sequence] holding cache positions.

cu_seq_lens_q
torch.Tensor | NoneDefaults to None

Query boundaries; see _FlashBoundaries.cu_q.

cu_seq_lens_k
torch.Tensor | NoneDefaults to None

Key boundaries; see _FlashBoundaries.cu_k.

max_length_q
int | NoneDefaults to None

Longest query segment, in tokens.

max_length_k
int | NoneDefaults to None

Longest key segment, in tokens.

padding_mask
torch.Tensor | NoneDefaults to None

bool Tensor of shape [batch, sequence] from the collater; ignored.

_packed_seq_ids
torch.Tensor | NoneDefaults to None

Tensor of shape [batch, sequence] holding document IDs from the collater; ignored.

Returns: Any

The release’s causal-LM output, whose logits have shape [batch,

Raises:

  • ValueError: If packing metadata is incomplete or a cache is requested with packing.
nemo_automodel.components.models.nemotron_nas.packing._validate_backend(
self: torch.nn.Module,
attn_implementation: str
) -> None
nemo_automodel.components.models.nemotron_nas.packing.apply_packing_adapters(
model: torch.nn.Module,
) -> torch.nn.Module

Add explicit packed boundaries without changing unpacked/generation calls.

Parameters:

model
nn.Module

Released remote DeciLM causal LM, before FSDP/PP wrapping.

mesh
MeshContext | NoneDefaults to None

Construction context supplied by the runtime-hook dispatcher.

Returns: nn.Module

The same module, with its parameter names and ordinary forwards intact.