nemo_automodel.components.models.deepseek_v41.cp
nemo_automodel.components.models.deepseek_v41.cp
Contiguous context parallelism for packed V4.1 text and image tokens.
Module Contents
Classes
Functions
API
Bases: Function
Gather sequence-first storage once; backward sums all remote uses.
Reduce [batch, global_sequence, …] to its summed local sequence shard.
Parameters:
Gradient of every global token used by this rank.
Returns: tuple[torch.Tensor, None]
Local gradient [batch, local_sequence, …] and no group gradient.
Gather [batch, local_sequence, …] into [batch, global_sequence, …].
Parameters:
Equal-length local sequence shards with arbitrary trailing axes.
The group whose rank order defines the global sequence.
Returns: torch.Tensor
Independently allocated global sequence; batch>1 can be noncontiguous.
Gather rank-ordered sequence shards without a second full-sized concatenation.
Parameters:
Tensor of shape [batch, local_sequence, …], with arbitrary trailing dimensions. Equal sequence lengths are required on every rank.
CP process group, or None for an identity operation.
Returns: torch.Tensor
Tensor of shape [batch, global_sequence, …]. Floating activations retain
Gather short shard suffixes and retain only the required left halo.
Parameters:
Local KV tensor [batch, local_sequence, head_dim].
CP group, or None for a local sequence.
Causal window width, including the query’s own token.
Returns: torch.Tensor
KV [batch, halo_sequence + local_sequence, head_dim] and the physical
Prepare packed boundaries once, then keep one contiguous query shard.
Parameters:
Context-parallel mesh, or None for a local packed forward.
Optional tensor-parallel mesh.
Text tensors input_ids, labels, position_ids and optional binary attention_mask [batch, global_sequence]. Packed input adds seq_lens and optional seq_lens_padded [batch, documents]. Labels must already be shifted independently within each document. Replaced in place with local tensors and document IDs [batch, local_sequence].
Optional tensor of shape [batch, global_sequence].
Input token used for padding; None uses zero. Validity comes from metadata.
LCM of this model’s active compression ratios.
Document alignment for compression groups, independent of CP size.
Synchronize the physical packed length across WORLD for HybridEP’s uniform input.
Returns: Callable
Null context factory, the local batch with runtime cp_group, and its ShardLayout.