Skip to navigation

nemo_automodel.components.models.deepseek_v41.cp

View as MarkdownOpen in Claude

Contiguous context parallelism for packed V4.1 text and image tokens.

Module Contents

Classes

NameDescription
_GatherSequenceGather sequence-first storage once; backward sums all remote uses.

Functions

NameDescription
gather_sequenceGather rank-ordered sequence shards without a second full-sized concatenation.
gather_windowGather short shard suffixes and retain only the required left halo.
shard_cp_batchPrepare packed boundaries once, then keep one contiguous query shard.

API

class nemo_automodel.components.models.deepseek_v41.cp._GatherSequence()

Bases: Function

Gather sequence-first storage once; backward sums all remote uses.

nemo_automodel.components.models.deepseek_v41.cp._GatherSequence.backward(
ctx,
gradient: torch.Tensor
) -> tuple[torch.Tensor, None]
staticmethod

Reduce [batch, global_sequence, …] to its summed local sequence shard.

Parameters:

gradient
torch.Tensor

Gradient of every global token used by this rank.

Returns: tuple[torch.Tensor, None]

Local gradient [batch, local_sequence, …] and no group gradient.

nemo_automodel.components.models.deepseek_v41.cp._GatherSequence.forward(
ctx,
tensor: torch.Tensor,
group: torch.distributed.ProcessGroup
) -> torch.Tensor
staticmethod

Gather [batch, local_sequence, …] into [batch, global_sequence, …].

Parameters:

tensor
torch.Tensor

Equal-length local sequence shards with arbitrary trailing axes.

group
dist.ProcessGroup

The group whose rank order defines the global sequence.

Returns: torch.Tensor

Independently allocated global sequence; batch>1 can be noncontiguous.

nemo_automodel.components.models.deepseek_v41.cp.gather_sequence(
tensor: torch.Tensor,
group: torch.distributed.ProcessGroup | None
) -> torch.Tensor

Gather rank-ordered sequence shards without a second full-sized concatenation.

Parameters:

tensor
torch.Tensor

Tensor of shape [batch, local_sequence, …], with arbitrary trailing dimensions. Equal sequence lengths are required on every rank.

group
dist.ProcessGroup | None

CP process group, or None for an identity operation.

Returns: torch.Tensor

Tensor of shape [batch, global_sequence, …]. Floating activations retain

nemo_automodel.components.models.deepseek_v41.cp.gather_window(
tensor: torch.Tensor,
group: torch.distributed.ProcessGroup | None,
window_size: int
) -> tuple[torch.Tensor, int]

Gather short shard suffixes and retain only the required left halo.

Parameters:

tensor
torch.Tensor

Local KV tensor [batch, local_sequence, head_dim].

group
dist.ProcessGroup | None

CP group, or None for a local sequence.

window_size
int

Causal window width, including the query’s own token.

Returns: torch.Tensor

KV [batch, halo_sequence + local_sequence, head_dim] and the physical

nemo_automodel.components.models.deepseek_v41.cp.shard_cp_batch(
cp_mesh: torch.distributed.device_mesh.DeviceMesh | None,
tp_mesh: torch.distributed.device_mesh.DeviceMesh | None,
batch: dict[str, typing.Any],
loss_mask: torch.Tensor | None = None,
padding_token_id: int | None = 0,
pad_multiple: int = 1,
packed_alignment: int = 1,
sync_packed_length: bool = False
) -> tuple[collections.abc.Callable, dict[str, typing.Any], nemo_automodel.components.distributed.context_parallel.sharder.ShardLayout]

Prepare packed boundaries once, then keep one contiguous query shard.

Parameters:

cp_mesh
DeviceMesh | None

Context-parallel mesh, or None for a local packed forward.

tp_mesh
DeviceMesh | None

Optional tensor-parallel mesh.

batch
dict[str, Any]

Text tensors input_ids, labels, position_ids and optional binary attention_mask [batch, global_sequence]. Packed input adds seq_lens and optional seq_lens_padded [batch, documents]. Labels must already be shifted independently within each document. Replaced in place with local tensors and document IDs [batch, local_sequence].

loss_mask
torch.Tensor | NoneDefaults to None

Optional tensor of shape [batch, global_sequence].

padding_token_id
int | NoneDefaults to 0

Input token used for padding; None uses zero. Validity comes from metadata.

pad_multiple
intDefaults to 1

LCM of this model’s active compression ratios.

packed_alignment
intDefaults to 1

Document alignment for compression groups, independent of CP size.

sync_packed_length
boolDefaults to False

Synchronize the physical packed length across WORLD for HybridEP’s uniform input.

Returns: Callable

Null context factory, the local batch with runtime cp_group, and its ShardLayout.