nemo_automodel.components.datasets.packing
nemo_automodel.components.datasets.packing
Dataset-owned packed-sequence construction contracts and helpers.
Module Contents
Classes
Functions
Data
DEFAULT_PACKED_SEQUENCE_CONTRACT
_LEGACY_FLASH_ATTENTION_IMPLEMENTATIONS
API
Model-derived requests passed to every loader, independently of format.
Batched packed inputs interpret packed_mask_type and
return_packed_sequence_metadata. Native THD inputs derive their kernel
metadata from lengths instead. return_document_ids requests optional
original membership in either format; effective segmentation always remains
available separately. Unpacked native loaders ignore packing-only requests.
max_sequence_length is the model’s input-token limit per packed document,
before batch flattening, and applies to both batched and THD output.
Decoder input-token limit, or None when the model does not declare one.
Attention representation for batched packed inputs; ignored by THD.
Model-owned conversion supplied by the recipe for native collation.
The callback accepts integer document IDs [batch, sequence] and returns an attention mask [batch, 1, sequence, sequence], compact IDs of shape [batch, sequence], or HF Flash offsets [segments + 1] and integer maxima. It preserves the input device and does not mutate the IDs. It must be serializable without capturing a model or GPU state. Deferred PP and THD inputs do not invoke this conversion.
Return original membership as document_ids in either output format.
Effective segmentation always uses _packed_seq_ids. Original IDs
are opt-in metadata for model policies such as independent image groups;
they do not select attention or MTP loss boundaries.
Return the gathered-token bundle for batched packed output only.
This requests packed_token_indices [batch, sequence] (-1 at padding),
cu_seqlens [batch, boundaries] for the non-padding token stream (-1
at unused entries), and max_seqlen. AutoModel’s FA4 and recurrent
consumers use these to gather tokens and delimit kernel/state sequences.
Their model-entry code flattens the bundle after PP microbatch splitting.
build_packed_sequence_metadata constructs this layout from IDs; a
loader’s physical padded offsets cannot simply be renamed to match it.
THD ignores this request and derives its own metadata from seq_lens
and seq_lens_padded. The flag does not select an output format.
Bases: typing.TypedDict
Batch-major metadata that remains valid after microbatch splitting.
Block-causal packing defaults for callers without model requirements.
Build batch-major metadata for a padded indexed packing mask.
Parameters:
Integer tensor of shape [batch, sequence] containing 1-based document IDs and zero-valued padding.
Returns: PackedSequenceMetadata
Metadata containing row-local packed_token_indices of shape
Extract document lengths from an indexed packed-sequence mask.
Parameters:
Integer tensor of shape [batch, sequence]. Each nonzero value is a 1-based document index local to its batch row; zero marks padding.
Returns: torch.Tensor
Tensor of shape [documents] containing nonzero document lengths in
Build varlen metadata for an indexed or binary attention mask.
Indexed masks treat every distinct positive document index in each batch row as a separate sequence. Binary masks treat every nonempty batch row as one sequence. Padding tokens are omitted from the flattened token stream.
Parameters:
Integer or boolean tensor of shape [batch, sequence]. Positive values identify valid tokens and zero marks padding.
Returns: torch.Tensor
A tuple containing indices of shape [tokens] into the flattened
Raises:
ValueError: If the mask is not rank two or contains no valid tokens.
Translate the deprecated attention keyword to a packing contract.
Parameters:
Explicit structural packing contract. It takes precedence when both migration surfaces are supplied.
Deprecated attention-backend name, or None.
Returns: PackedSequenceContract
The explicit contract, or a compatibility contract matching the legacy