nemo_automodel.components.models.lfm3.context_parallel
nemo_automodel.components.models.lfm3.context_parallel
Packed-document boundaries and context parallelism for LFM3.
Each CP rank owns one contiguous shard of the sequence. A convolution or attention layer that needs tokens before the shard asks the earlier ranks for them: only the part of the document that the shard starts in, and no more than the layer can reach (the kernel width minus one for a convolution, the window for sliding attention, the whole document for full attention).
Module Contents
Classes
Functions
Data
API
For every CP rank, the number of tokens before its shard that belong to the document the shard starts in.
Document boundaries of the local tokens, rows flattened in order.
cu_seq_lens is an int32 Tensor of shape [documents + 1] and max_length the longest
document, as FlashAttention’s varlen interface takes them. halo is set under context
parallelism.
Return the document of each token as a long Tensor of shape [tokens].
Bases: Function
Send shard suffixes to later ranks and receive this rank’s halo; backward returns the gradients.
Return the halo’s gradient to the ranks that own the tokens.
Parameters:
Autograd context holding the exchange plan of the forward.
Gradient of the halo, a Tensor of shape [batch, channels, halo_tokens, …].
Returns: torch.Tensor
Gradient of the local shard, a Tensor of shape [batch, channels, sequence, …] that
Exchange shard suffixes.
Parameters:
Autograd context that records the exchange plan.
This rank’s shard, a Tensor of shape [batch, channels, sequence, …]; axis 2 is this rank’s contiguous interval of the global sequence.
Context-parallel process group.
Pairs of source rank and token count, in sequence order.
Pairs of destination rank and the number of trailing tokens it reads.
Returns: torch.Tensor
The received pieces in sequence order, a Tensor of shape
Return document boundaries over the flattened [batch * sequence] tokens and drop the entries they came from.
Parameters:
Token Tensors of shape [batch, sequence]. The boundaries come from the first of:
seq_lens_padded [batch, documents] of a THD batch (qkv_format and seq_lens
are removed with it); cu_seq_lens_q [documents + 1] of a batch packed for
FlashAttention; _packed_seq_ids [batch, sequence], 1-based document indices with
0 at padding (removed); a 2D attention_mask [batch, sequence], nonzero at real
tokens. Padding runs of the last two become segments of their own, so real tokens
never attend to or convolve over padding. Any other batch holds one document per row.
Returns: list[int]
Boundaries as ints from 0 to batch * sequence; every row starts a new document.
Return the (rank, token count) pairs that hold a rank’s halo, in sequence order.
Run one collective on the CP group, once per process.
A batch of point-to-point operations may be the first operation on a process group only if every rank takes part in it, and a rank whose shard starts at a document boundary can have nothing to send or receive.
Send and receive Tensors between CP ranks in one batch of point-to-point operations.
Parameters:
Context-parallel process group; the ranks below are ranks within it.
Pairs of destination rank and the Tensor to send, of shape [batch, channels, piece_tokens, …]. Each piece is copied before it is sent.
Pairs of source rank and the buffer that receives its piece, of shape [batch, channels, piece_tokens, …] with the sender’s dtype, device and trailing dimensions. The buffers are filled in place.
Run the causal convolution without reading across documents.
Documents are laid out kernel_size - 1 zeros apart, so each one gets the left padding
it has when it runs alone.
Parameters:
Depthwise convolution padded by kernel_size - 1 on both ends.
Tensor of shape [batch, channels, sequence].
Boundaries of the flattened [batch * sequence] tokens.
Returns: torch.Tensor
Tensor of shape [batch, channels, sequence].
Add this rank’s halo to the keys and values and describe the documents to the attention backend.
Parameters:
Boundaries of the flattened [batch * sequence] tokens.
Tensor of shape [batch, heads, sequence, head_dim].
Tensor of shape [batch, heads, sequence, head_dim].
Number of keys a query sees, itself included; None for full attention.
Hugging Face attention backend of the layer.
Returns: torch.Tensor
Keys and values of shape [batch, heads, halo + sequence, head_dim], the attention mask
Fetch the tokens before this rank’s shard that belong to its first document.
Parameters:
This rank’s shard, a Tensor of shape [batch, channels, sequence, …].
Halo length of every CP rank.
Largest number of tokens a layer can use; None fetches the whole halo.
Returns: torch.Tensor
Tensor of shape [batch, channels, halo, …] in sequence order. Gradients flow back
Return the sharder for packed THD batches at any CP size and for unpacked batches under CP.
Parameters:
Unused here; the sharder reads the batch, with token Tensors of shape [batch, sequence], when it shards.
Unused; LFM3 shards whole rows.
Returns: dict[str, ContextParallelSharder]
Mapping with the cp_sharder whose shard_batch is shard_documents and
Keep this CP rank’s contiguous shard and describe its documents to the model.
Parameters:
CP mesh, or None; every rank of it receives the same batch.
Unused TP mesh forwarded to the shared sharder.
Token Tensors of shape [batch, sequence]. A THD batch carries the document
spans of each row in seq_lens_padded, a batch packed for FlashAttention
carries cu_seq_lens_q or the document indices _packed_seq_ids, and an
unpacked batch holds one document per row, split where a 2D attention_mask
marks padding (see _document_boundaries).
Optional Tensor of shape [batch, sequence] sharded with the batch.
Token used when the sequence is padded to a multiple of the CP size.
Returns: Callable[[], contextlib.AbstractContextManager]
A null context, the batch with Tensors of shape [batch, sequence / CP] plus