> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/automodel/_mcp/server.

# nemo_automodel.components.models.deepseek_v41.cp

Contiguous context parallelism for the V4.1 text backbone.

## Module Contents

### Functions

| Name                                                                                   | Description                                                           |
| -------------------------------------------------------------------------------------- | --------------------------------------------------------------------- |
| [`gather_sequence`](#nemo_automodel-components-models-deepseek_v41-cp-gather_sequence) | Gather rank-ordered sequence shards, summing remote uses in backward. |
| [`shard_cp_batch`](#nemo_automodel-components-models-deepseek_v41-cp-shard_cp_batch)   | Prepare packed boundaries once, then keep one contiguous query shard. |

### API

```python
nemo_automodel.components.models.deepseek_v41.cp.gather_sequence(
    tensor: torch.Tensor,
    group: torch.distributed.ProcessGroup | None
) -> torch.Tensor
```

Gather rank-ordered sequence shards, summing remote uses in backward.

**Parameters:**

**`tensor`** `torch.Tensor`

Tensor of shape \[batch, local\_sequence, ...], with arbitrary
trailing dimensions. Equal sequence lengths are required on every rank.

---

**`group`** `dist.ProcessGroup | None`

CP process group, or None for an identity operation.

---

**Returns:** `torch.Tensor`

Tensor of shape \[batch, global\_sequence, ...]. Floating activations retain

```python
nemo_automodel.components.models.deepseek_v41.cp.shard_cp_batch(
    cp_mesh: torch.distributed.device_mesh.DeviceMesh | None,
    tp_mesh: torch.distributed.device_mesh.DeviceMesh | None,
    batch: dict[str, typing.Any],
    loss_mask: torch.Tensor | None = None,
    padding_token_id: int | None = 0,
    pad_multiple: int = 1,
    packed_alignment: int = 1,
    sync_packed_length: bool = False
) -> tuple[collections.abc.Callable, dict[str, typing.Any], nemo_automodel.components.distributed.context_parallel.sharder.ShardLayout]
```

Prepare packed boundaries once, then keep one contiguous query shard.

**Parameters:**

**`cp_mesh`** `DeviceMesh | None`

Context-parallel mesh, or None for a local packed forward.

---

**`tp_mesh`** `DeviceMesh | None`

Optional tensor-parallel mesh.

---

**`batch`** `dict[str, Any]`

Text tensors input\_ids, labels, position\_ids and optional binary attention\_mask
\[batch, global\_sequence]. Packed input adds seq\_lens and optional seq\_lens\_padded
\[batch, documents]. Labels must already be shifted independently within each document.
Replaced in place with local tensors and document IDs \[batch, local\_sequence].

---

**`loss_mask`** `torch.Tensor | None` — default: None

Optional tensor of shape \[batch, global\_sequence].

---

**`padding_token_id`** `int | None` — default: 0

Input token used for padding; None uses zero. Validity comes from metadata.

---

**`pad_multiple`** `int` — default: 1

LCM of this model's active compression ratios.

---

**`packed_alignment`** `int` — default: 1

Document alignment for compression groups, independent of CP size.

---

**`sync_packed_length`** `bool` — default: False

Synchronize the physical packed length across WORLD for HybridEP's uniform input.

---

**Returns:** `Callable`

Null context factory, the local batch with runtime cp\_group, and its ShardLayout.