Skip to navigation
ReferenceFull Library ReferenceNemo AutomodelNemo AutomodelComponentsModelsLfm3nemo_automodel.components.models.lfm3.context_parallel

nemo_automodel.components.models.lfm3.context_parallel

View as MarkdownOpen in Claude

Packed-document boundaries and context parallelism for LFM3.

Each CP rank owns one contiguous shard of the sequence. A convolution or attention layer that needs tokens before the shard asks the earlier ranks for them: only the part of the document that the shard starts in, and no more than the layer can reach (the kernel width minus one for a convolution, the window for sliding attention, the whole document for full attention).

Module Contents

Classes

NameDescription
DocumentHaloFor every CP rank, the number of tokens before its shard that belong to the document the shard starts in.
PackedDocumentsDocument boundaries of the local tokens, rows flattened in order.
_HaloExchangeSend shard suffixes to later ranks and receive this rank’s halo; backward returns the gradients.

Functions

NameDescription
_document_boundariesReturn document boundaries over the flattened [batch * sequence] tokens and drop the entries they came from.
_halo_sourcesReturn the (rank, token count) pairs that hold a rank’s halo, in sequence order.
_join_groupRun one collective on the CP group, once per process.
_transferSend and receive Tensors between CP ranks in one batch of point-to-point operations.
convolve_documentsRun the causal convolution without reading across documents.
document_attention_inputsAdd this rank’s halo to the keys and values and describe the documents to the attention backend.
exchange_haloFetch the tokens before this rank’s shard that belong to its first document.
prepare_cp_inputsReturn the sharder for packed THD batches at any CP size and for unpacked batches under CP.
shard_documentsKeep this CP rank’s contiguous shard and describe its documents to the model.

Data

CPBatch

API

class nemo_automodel.components.models.lfm3.context_parallel.DocumentHalo(
group: torch.distributed.ProcessGroup,
lengths: tuple[int, ...]
)
Dataclass

For every CP rank, the number of tokens before its shard that belong to the document the shard starts in.

group
ProcessGroup
lengths
tuple[int, ...]
class nemo_automodel.components.models.lfm3.context_parallel.PackedDocuments(
cu_seq_lens: torch.Tensor,
max_length: int,
)
Dataclass

Document boundaries of the local tokens, rows flattened in order.

cu_seq_lens is an int32 Tensor of shape [documents + 1] and max_length the longest document, as FlashAttention’s varlen interface takes them. halo is set under context parallelism.

cu_seq_lens
Tensor
halo
DocumentHalo | None = None
max_length
int
nemo_automodel.components.models.lfm3.context_parallel.PackedDocuments.document_index(
tokens: int
) -> torch.Tensor

Return the document of each token as a long Tensor of shape [tokens].

class nemo_automodel.components.models.lfm3.context_parallel._HaloExchange()

Bases: Function

Send shard suffixes to later ranks and receive this rank’s halo; backward returns the gradients.

nemo_automodel.components.models.lfm3.context_parallel._HaloExchange.backward(
ctx: torch.autograd.function.FunctionCtx,
grad: torch.Tensor
) -> tuple[torch.Tensor, None, None, None]
staticmethod

Return the halo’s gradient to the ranks that own the tokens.

Parameters:

ctx
torch.autograd.function.FunctionCtx

Autograd context holding the exchange plan of the forward.

grad
torch.Tensor

Gradient of the halo, a Tensor of shape [batch, channels, halo_tokens, …].

Returns: torch.Tensor

Gradient of the local shard, a Tensor of shape [batch, channels, sequence, …] that

nemo_automodel.components.models.lfm3.context_parallel._HaloExchange.forward(
ctx: torch.autograd.function.FunctionCtx,
tensor: torch.Tensor,
group: torch.distributed.ProcessGroup,
receives: list[tuple[int, int]],
sends: list[tuple[int, int]]
) -> torch.Tensor
staticmethod

Exchange shard suffixes.

Parameters:

ctx
torch.autograd.function.FunctionCtx

Autograd context that records the exchange plan.

tensor
torch.Tensor

This rank’s shard, a Tensor of shape [batch, channels, sequence, …]; axis 2 is this rank’s contiguous interval of the global sequence.

group
dist.ProcessGroup

Context-parallel process group.

receives
list[tuple[int, int]]

Pairs of source rank and token count, in sequence order.

sends
list[tuple[int, int]]

Pairs of destination rank and the number of trailing tokens it reads.

Returns: torch.Tensor

The received pieces in sequence order, a Tensor of shape

nemo_automodel.components.models.lfm3.context_parallel._document_boundaries(
) -> list[int]

Return document boundaries over the flattened [batch * sequence] tokens and drop the entries they came from.

Parameters:

batch
CPBatch

Token Tensors of shape [batch, sequence]. The boundaries come from the first of: seq_lens_padded [batch, documents] of a THD batch (qkv_format and seq_lens are removed with it); cu_seq_lens_q [documents + 1] of a batch packed for FlashAttention; _packed_seq_ids [batch, sequence], 1-based document indices with 0 at padding (removed); a 2D attention_mask [batch, sequence], nonzero at real tokens. Padding runs of the last two become segments of their own, so real tokens never attend to or convolve over padding. Any other batch holds one document per row.

Returns: list[int]

Boundaries as ints from 0 to batch * sequence; every row starts a new document.

nemo_automodel.components.models.lfm3.context_parallel._halo_sources(
lengths: tuple[int, ...],
shard_length: int,
rank: int,
limit: int | None
) -> list[tuple[int, int]]

Return the (rank, token count) pairs that hold a rank’s halo, in sequence order.

nemo_automodel.components.models.lfm3.context_parallel._join_group(
group: torch.distributed.ProcessGroup,
device: torch.device
) -> None

Run one collective on the CP group, once per process.

A batch of point-to-point operations may be the first operation on a process group only if every rank takes part in it, and a rank whose shard starts at a document boundary can have nothing to send or receive.

nemo_automodel.components.models.lfm3.context_parallel._transfer(
group: torch.distributed.ProcessGroup,
sends: list[tuple[int, torch.Tensor]],
receives: list[tuple[int, torch.Tensor]]
) -> None

Send and receive Tensors between CP ranks in one batch of point-to-point operations.

Parameters:

group
dist.ProcessGroup

Context-parallel process group; the ranks below are ranks within it.

sends
list[tuple[int, torch.Tensor]]

Pairs of destination rank and the Tensor to send, of shape [batch, channels, piece_tokens, …]. Each piece is copied before it is sent.

receives
list[tuple[int, torch.Tensor]]

Pairs of source rank and the buffer that receives its piece, of shape [batch, channels, piece_tokens, …] with the sender’s dtype, device and trailing dimensions. The buffers are filled in place.

nemo_automodel.components.models.lfm3.context_parallel.convolve_documents(
conv: torch.nn.Conv1d,
gated: torch.Tensor,
) -> torch.Tensor

Run the causal convolution without reading across documents.

Documents are laid out kernel_size - 1 zeros apart, so each one gets the left padding it has when it runs alone.

Parameters:

conv
nn.Conv1d

Depthwise convolution padded by kernel_size - 1 on both ends.

gated
torch.Tensor

Tensor of shape [batch, channels, sequence].

documents
PackedDocuments

Boundaries of the flattened [batch * sequence] tokens.

Returns: torch.Tensor

Tensor of shape [batch, channels, sequence].

nemo_automodel.components.models.lfm3.context_parallel.document_attention_inputs(
key: torch.Tensor,
value: torch.Tensor,
sliding_window: int | None,
attn_implementation: str
) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor | None, dict[str, torch.Tensor | int]]

Add this rank’s halo to the keys and values and describe the documents to the attention backend.

Parameters:

documents
PackedDocuments

Boundaries of the flattened [batch * sequence] tokens.

key
torch.Tensor

Tensor of shape [batch, heads, sequence, head_dim].

value
torch.Tensor

Tensor of shape [batch, heads, sequence, head_dim].

sliding_window
int | None

Number of keys a query sees, itself included; None for full attention.

attn_implementation
str

Hugging Face attention backend of the layer.

Returns: torch.Tensor

Keys and values of shape [batch, heads, halo + sequence, head_dim], the attention mask

nemo_automodel.components.models.lfm3.context_parallel.exchange_halo(
tensor: torch.Tensor,
limit: int | None = None
) -> torch.Tensor

Fetch the tokens before this rank’s shard that belong to its first document.

Parameters:

tensor
torch.Tensor

This rank’s shard, a Tensor of shape [batch, channels, sequence, …].

halo
DocumentHalo

Halo length of every CP rank.

limit
int | NoneDefaults to None

Largest number of tokens a layer can use; None fetches the whole halo.

Returns: torch.Tensor

Tensor of shape [batch, channels, halo, …] in sequence order. Gradients flow back

nemo_automodel.components.models.lfm3.context_parallel.prepare_cp_inputs(
num_chunks: int = 1

Return the sharder for packed THD batches at any CP size and for unpacked batches under CP.

Parameters:

batch
CPBatch

Unused here; the sharder reads the batch, with token Tensors of shape [batch, sequence], when it shards.

num_chunks
intDefaults to 1

Unused; LFM3 shards whole rows.

Returns: dict[str, ContextParallelSharder]

Mapping with the cp_sharder whose shard_batch is shard_documents and

nemo_automodel.components.models.lfm3.context_parallel.shard_documents(
cp_mesh: torch.distributed.device_mesh.DeviceMesh | None,
tp_mesh: torch.distributed.device_mesh.DeviceMesh | None,
loss_mask: torch.Tensor | None = None,
padding_token_id: int = 0

Keep this CP rank’s contiguous shard and describe its documents to the model.

Parameters:

cp_mesh
DeviceMesh | None

CP mesh, or None; every rank of it receives the same batch.

tp_mesh
DeviceMesh | None

Unused TP mesh forwarded to the shared sharder.

batch
CPBatch

Token Tensors of shape [batch, sequence]. A THD batch carries the document spans of each row in seq_lens_padded, a batch packed for FlashAttention carries cu_seq_lens_q or the document indices _packed_seq_ids, and an unpacked batch holds one document per row, split where a 2D attention_mask marks padding (see _document_boundaries).

loss_mask
torch.Tensor | NoneDefaults to None

Optional Tensor of shape [batch, sequence] sharded with the batch.

padding_token_id
intDefaults to 0

Token used when the sequence is padded to a multiple of the CP size.

Returns: Callable[[], contextlib.AbstractContextManager]

A null context, the batch with Tensors of shape [batch, sequence / CP] plus

nemo_automodel.components.models.lfm3.context_parallel.CPBatch: TypeAlias = dict[str, torch.Tensor | DocumentHalo | str | int | None]