Skip to navigation

nemo_automodel.components.datasets.packing

View as MarkdownOpen in Claude

Dataset-owned packed-sequence construction contracts and helpers.

Module Contents

Classes

NameDescription
PackedSequenceContractModel-derived requests passed to every loader, independently of format.
PackedSequenceMetadataBatch-major metadata that remains valid after microbatch splitting.
_DefaultPackedSequenceContractBlock-causal packing defaults for callers without model requirements.

Functions

NameDescription
build_packed_sequence_metadataBuild batch-major metadata for a padded indexed packing mask.
get_seqlens_in_batchExtract document lengths from an indexed packed-sequence mask.
get_unpad_dataBuild varlen metadata for an indexed or binary attention mask.
resolve_packing_contractTranslate the deprecated attention keyword to a packing contract.

Data

BatchOutputFormat

DEFAULT_PACKED_SEQUENCE_CONTRACT

PackedMaskType

_LEGACY_FLASH_ATTENTION_IMPLEMENTATIONS

API

class nemo_automodel.components.datasets.packing.PackedSequenceContract()
Protocol

Model-derived requests passed to every loader, independently of format.

Batched packed inputs interpret packed_mask_type and return_packed_sequence_metadata. Native THD inputs derive their kernel metadata from lengths instead. return_document_ids requests optional original membership in either format; effective segmentation always remains available separately. Unpacked native loaders ignore packing-only requests. max_sequence_length is the model’s input-token limit per packed document, before batch flattening, and applies to both batched and THD output.

max_sequence_length
int | None

Decoder input-token limit, or None when the model does not declare one.

packed_mask_type
PackedMaskType

Attention representation for batched packed inputs; ignored by THD.

prepare_attention_inputs
Callable[[torch.Tensor], dict[str, Tensor | int]] | None

Model-owned conversion supplied by the recipe for native collation.

The callback accepts integer document IDs [batch, sequence] and returns an attention mask [batch, 1, sequence, sequence], compact IDs of shape [batch, sequence], or HF Flash offsets [segments + 1] and integer maxima. It preserves the input device and does not mutate the IDs. It must be serializable without capturing a model or GPU state. Deferred PP and THD inputs do not invoke this conversion.

return_document_ids
bool

Return original membership as document_ids in either output format.

Effective segmentation always uses _packed_seq_ids. Original IDs are opt-in metadata for model policies such as independent image groups; they do not select attention or MTP loss boundaries.

return_packed_sequence_metadata
bool

Return the gathered-token bundle for batched packed output only.

This requests packed_token_indices [batch, sequence] (-1 at padding), cu_seqlens [batch, boundaries] for the non-padding token stream (-1 at unused entries), and max_seqlen. AutoModel’s FA4 and recurrent consumers use these to gather tokens and delimit kernel/state sequences. Their model-entry code flattens the bundle after PP microbatch splitting. build_packed_sequence_metadata constructs this layout from IDs; a loader’s physical padded offsets cannot simply be renamed to match it. THD ignores this request and derives its own metadata from seq_lens and seq_lens_padded. The flag does not select an output format.

class nemo_automodel.components.datasets.packing.PackedSequenceMetadata

Bases: typing.TypedDict

Batch-major metadata that remains valid after microbatch splitting.

cu_seqlens
Tensor
max_seqlen
int
packed_token_indices
Tensor
class nemo_automodel.components.datasets.packing._DefaultPackedSequenceContract(
return_packed_sequence_metadata: bool = False,
return_document_ids: bool = False,
max_sequence_length: int | None = None,
prepare_attention_inputs: collections.abc.Callable[[torch.Tensor], dict[str, torch.Tensor | int]] | None = None
)
Dataclass

Block-causal packing defaults for callers without model requirements.

max_sequence_length
int | None = None
packed_mask_type
PackedMaskType = 'block_causal'
prepare_attention_inputs
Callable[[torch.Tensor], dict[str, Tensor | int]] | None = None
return_document_ids
bool = False
return_packed_sequence_metadata
bool = False
nemo_automodel.components.datasets.packing.build_packed_sequence_metadata(
attention_mask: torch.Tensor

Build batch-major metadata for a padded indexed packing mask.

Parameters:

attention_mask
torch.Tensor

Integer tensor of shape [batch, sequence] containing 1-based document IDs and zero-valued padding.

Returns: PackedSequenceMetadata

Metadata containing row-local packed_token_indices of shape

nemo_automodel.components.datasets.packing.get_seqlens_in_batch(
attention_mask: torch.Tensor
) -> torch.Tensor

Extract document lengths from an indexed packed-sequence mask.

Parameters:

attention_mask
torch.Tensor

Integer tensor of shape [batch, sequence]. Each nonzero value is a 1-based document index local to its batch row; zero marks padding.

Returns: torch.Tensor

Tensor of shape [documents] containing nonzero document lengths in

nemo_automodel.components.datasets.packing.get_unpad_data(
attention_mask: torch.Tensor
) -> tuple[torch.Tensor, torch.Tensor, int]

Build varlen metadata for an indexed or binary attention mask.

Indexed masks treat every distinct positive document index in each batch row as a separate sequence. Binary masks treat every nonempty batch row as one sequence. Padding tokens are omitted from the flattened token stream.

Parameters:

attention_mask
torch.Tensor

Integer or boolean tensor of shape [batch, sequence]. Positive values identify valid tokens and zero marks padding.

Returns: torch.Tensor

A tuple containing indices of shape [tokens] into the flattened

Raises:

  • ValueError: If the mask is not rank two or contains no valid tokens.
nemo_automodel.components.datasets.packing.resolve_packing_contract(
attn_implementation: str | None

Translate the deprecated attention keyword to a packing contract.

Parameters:

packing
PackedSequenceContract

Explicit structural packing contract. It takes precedence when both migration surfaces are supplied.

attn_implementation
str | None

Deprecated attention-backend name, or None.

Returns: PackedSequenceContract

The explicit contract, or a compatibility contract matching the legacy

nemo_automodel.components.datasets.packing.BatchOutputFormat = Literal['batched', 'thd']
nemo_automodel.components.datasets.packing.DEFAULT_PACKED_SEQUENCE_CONTRACT: Final[PackedSequenceContract] = _DefaultPackedSequenceContract()
nemo_automodel.components.datasets.packing.PackedMaskType = Literal['block_causal', 'document_ids', 'flash_varlen']
nemo_automodel.components.datasets.packing._LEGACY_FLASH_ATTENTION_IMPLEMENTATIONS = frozenset({'flash_attention_2', 'flash_attention_3', 'flash_attention_4'})