> This page is for version 0.5.0 · 26.06.
> For other versions, use one of these documentation indexes:
> - Nightly (default): https://docs.nvidia.com/nemo/automodel/nightly/llms.txt
> - Latest: https://docs.nvidia.com/nemo/automodel/latest/llms.txt
> - 0.7.0 · 26.10: https://docs.nvidia.com/nemo/automodel/v0.7/llms.txt
> - 0.5.0 · 26.06: https://docs.nvidia.com/nemo/automodel/v0.5/llms.txt
> - 0.4.0 · 26.04: https://docs.nvidia.com/nemo/automodel/v0.4/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt.

# nemo_automodel.components.models.lfm3.context_parallel

Packed-document boundaries and context parallelism for LFM3.

Each CP rank owns one contiguous shard of the sequence. A convolution or attention layer that
needs tokens before the shard asks the earlier ranks for them: only the part of the document
that the shard starts in, and no more than the layer can reach (the kernel width minus one for a
convolution, the window for sliding attention, the whole document for full attention).

## Module Contents

### Classes

| Name                                                                                         | Description                                                                                               |
| -------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- |
| [`DocumentHalo`](#nemo_automodel-components-models-lfm3-context_parallel-DocumentHalo)       | For every CP rank, the number of tokens before its shard that belong to the document the shard starts in. |
| [`PackedDocuments`](#nemo_automodel-components-models-lfm3-context_parallel-PackedDocuments) | Document boundaries of the local tokens, rows flattened in order.                                         |
| [`_HaloExchange`](#nemo_automodel-components-models-lfm3-context_parallel-_HaloExchange)     | Send shard suffixes to later ranks and receive this rank's halo; backward returns the gradients.          |

### Functions

| Name                                                                                                             | Description                                                                                                    |
| ---------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- |
| [`_document_boundaries`](#nemo_automodel-components-models-lfm3-context_parallel-_document_boundaries)           | Return document boundaries over the flattened \[batch \* sequence] tokens and drop the entries they came from. |
| [`_halo_sources`](#nemo_automodel-components-models-lfm3-context_parallel-_halo_sources)                         | Return the (rank, token count) pairs that hold a rank's halo, in sequence order.                               |
| [`_join_group`](#nemo_automodel-components-models-lfm3-context_parallel-_join_group)                             | Run one collective on the CP group, once per process.                                                          |
| [`_transfer`](#nemo_automodel-components-models-lfm3-context_parallel-_transfer)                                 | Send and receive Tensors between CP ranks in one batch of point-to-point operations.                           |
| [`convolve_documents`](#nemo_automodel-components-models-lfm3-context_parallel-convolve_documents)               | Run the causal convolution without reading across documents.                                                   |
| [`document_attention_inputs`](#nemo_automodel-components-models-lfm3-context_parallel-document_attention_inputs) | Add this rank's halo to the keys and values and describe the documents to the attention backend.               |
| [`exchange_halo`](#nemo_automodel-components-models-lfm3-context_parallel-exchange_halo)                         | Fetch the tokens before this rank's shard that belong to its first document.                                   |
| [`prepare_cp_inputs`](#nemo_automodel-components-models-lfm3-context_parallel-prepare_cp_inputs)                 | Return the sharder for packed THD batches at any CP size and for unpacked batches under CP.                    |
| [`shard_documents`](#nemo_automodel-components-models-lfm3-context_parallel-shard_documents)                     | Keep this CP rank's contiguous shard and describe its documents to the model.                                  |

### Data

[`CPBatch`](#nemo_automodel-components-models-lfm3-context_parallel-CPBatch)

### API

```python
class nemo_automodel.components.models.lfm3.context_parallel.DocumentHalo(
    group: torch.distributed.ProcessGroup,
    lengths: tuple[int, ...]
)
```

Dataclass

For every CP rank, the number of tokens before its shard that belong to the document the shard starts in.

**`group`** `ProcessGroup`

---

**`lengths`** `tuple[int, ...]`

---

```python
class nemo_automodel.components.models.lfm3.context_parallel.PackedDocuments(
    cu_seq_lens: torch.Tensor,
    max_length: int,
    halo: nemo_automodel.components.models.lfm3.context_parallel.DocumentHalo | None = None
)
```

Dataclass

Document boundaries of the local tokens, rows flattened in order.

`cu_seq_lens` is an int32 Tensor of shape \[documents + 1] and `max_length` the longest
document, as FlashAttention's varlen interface takes them. `halo` is set under context
parallelism.

**`cu_seq_lens`** `Tensor`

---

**`halo`** `DocumentHalo | None = None`

---

**`max_length`** `int`

---

```python
nemo_automodel.components.models.lfm3.context_parallel.PackedDocuments.document_index(
    tokens: int
) -> torch.Tensor
```

Return the document of each token as a long Tensor of shape \[tokens].

```python
class nemo_automodel.components.models.lfm3.context_parallel._HaloExchange()
```

**Bases:** `Function`

Send shard suffixes to later ranks and receive this rank's halo; backward returns the gradients.

```python
nemo_automodel.components.models.lfm3.context_parallel._HaloExchange.backward(
    ctx: torch.autograd.function.FunctionCtx,
    grad: torch.Tensor
) -> tuple[torch.Tensor, None, None, None]
```

staticmethod

Return the halo's gradient to the ranks that own the tokens.

**Parameters:**

**`ctx`** `torch.autograd.function.FunctionCtx`

Autograd context holding the exchange plan of the forward.

---

**`grad`** `torch.Tensor`

Gradient of the halo, a Tensor of shape \[batch, channels, halo_tokens, ...].

---

**Returns:** `torch.Tensor`

Gradient of the local shard, a Tensor of shape \[batch, channels, sequence, ...] that

```python
nemo_automodel.components.models.lfm3.context_parallel._HaloExchange.forward(
    ctx: torch.autograd.function.FunctionCtx,
    tensor: torch.Tensor,
    group: torch.distributed.ProcessGroup,
    receives: list[tuple[int, int]],
    sends: list[tuple[int, int]]
) -> torch.Tensor
```

staticmethod

Exchange shard suffixes.

**Parameters:**

**`ctx`** `torch.autograd.function.FunctionCtx`

Autograd context that records the exchange plan.

---

**`tensor`** `torch.Tensor`

This rank's shard, a Tensor of shape \[batch, channels, sequence, ...];
axis 2 is this rank's contiguous interval of the global sequence.

---

**`group`** `dist.ProcessGroup`

Context-parallel process group.

---

**`receives`** `list[tuple[int, int]]`

Pairs of source rank and token count, in sequence order.

---

**`sends`** `list[tuple[int, int]]`

Pairs of destination rank and the number of trailing tokens it reads.

---

**Returns:** `torch.Tensor`

The received pieces in sequence order, a Tensor of shape

```python
nemo_automodel.components.models.lfm3.context_parallel._document_boundaries(
    batch: nemo_automodel.components.models.lfm3.context_parallel.CPBatch
) -> list[int]
```

Return document boundaries over the flattened \[batch \* sequence] tokens and drop the entries they came from.

**Parameters:**

**`batch`** `CPBatch`

Token Tensors of shape \[batch, sequence]. The boundaries come from the first of:
`seq_lens_padded` \[batch, documents] of a THD batch (`qkv_format` and `seq_lens`
are removed with it); `cu_seq_lens_q` \[documents + 1] of a batch packed for
FlashAttention; `_packed_seq_ids` \[batch, sequence], 1-based document indices with
0 at padding (removed); a 2D `attention_mask` \[batch, sequence], nonzero at real
tokens. Padding runs of the last two become segments of their own, so real tokens
never attend to or convolve over padding. Any other batch holds one document per row.

---

**Returns:** `list[int]`

Boundaries as ints from 0 to batch \* sequence; every row starts a new document.

```python
nemo_automodel.components.models.lfm3.context_parallel._halo_sources(
    lengths: tuple[int, ...],
    shard_length: int,
    rank: int,
    limit: int | None
) -> list[tuple[int, int]]
```

Return the (rank, token count) pairs that hold a rank's halo, in sequence order.

```python
nemo_automodel.components.models.lfm3.context_parallel._join_group(
    group: torch.distributed.ProcessGroup,
    device: torch.device
) -> None
```

Run one collective on the CP group, once per process.

A batch of point-to-point operations may be the first operation on a process group only if
every rank takes part in it, and a rank whose shard starts at a document boundary can have
nothing to send or receive.

```python
nemo_automodel.components.models.lfm3.context_parallel._transfer(
    group: torch.distributed.ProcessGroup,
    sends: list[tuple[int, torch.Tensor]],
    receives: list[tuple[int, torch.Tensor]]
) -> None
```

Send and receive Tensors between CP ranks in one batch of point-to-point operations.

**Parameters:**

**`group`** `dist.ProcessGroup`

Context-parallel process group; the ranks below are ranks within it.

---

**`sends`** `list[tuple[int, torch.Tensor]]`

Pairs of destination rank and the Tensor to send, of shape
\[batch, channels, piece_tokens, ...]. Each piece is copied before it is sent.

---

**`receives`** `list[tuple[int, torch.Tensor]]`

Pairs of source rank and the buffer that receives its piece, of shape
\[batch, channels, piece_tokens, ...] with the sender's dtype, device and trailing
dimensions. The buffers are filled in place.

---

```python
nemo_automodel.components.models.lfm3.context_parallel.convolve_documents(
    conv: torch.nn.Conv1d,
    gated: torch.Tensor,
    documents: nemo_automodel.components.models.lfm3.context_parallel.PackedDocuments
) -> torch.Tensor
```

Run the causal convolution without reading across documents.

Documents are laid out `kernel_size - 1` zeros apart, so each one gets the left padding
it has when it runs alone.

**Parameters:**

**`conv`** `nn.Conv1d`

Depthwise convolution padded by `kernel_size - 1` on both ends.

---

**`gated`** `torch.Tensor`

Tensor of shape \[batch, channels, sequence].

---

**`documents`** `PackedDocuments`

Boundaries of the flattened \[batch \* sequence] tokens.

---

**Returns:** `torch.Tensor`

Tensor of shape \[batch, channels, sequence].

```python
nemo_automodel.components.models.lfm3.context_parallel.document_attention_inputs(
    documents: nemo_automodel.components.models.lfm3.context_parallel.PackedDocuments,
    key: torch.Tensor,
    value: torch.Tensor,
    sliding_window: int | None,
    attn_implementation: str
) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor | None, dict[str, torch.Tensor | int]]
```

Add this rank's halo to the keys and values and describe the documents to the attention backend.

**Parameters:**

**`documents`** `PackedDocuments`

Boundaries of the flattened \[batch \* sequence] tokens.

---

**`key`** `torch.Tensor`

Tensor of shape \[batch, heads, sequence, head_dim].

---

**`value`** `torch.Tensor`

Tensor of shape \[batch, heads, sequence, head_dim].

---

**`sliding_window`** `int | None`

Number of keys a query sees, itself included; `None` for full attention.

---

**`attn_implementation`** `str`

Hugging Face attention backend of the layer.

---

**Returns:** `torch.Tensor`

Keys and values of shape \[batch, heads, halo + sequence, head_dim], the attention mask

```python
nemo_automodel.components.models.lfm3.context_parallel.exchange_halo(
    tensor: torch.Tensor,
    halo: nemo_automodel.components.models.lfm3.context_parallel.DocumentHalo,
    limit: int | None = None
) -> torch.Tensor
```

Fetch the tokens before this rank's shard that belong to its first document.

**Parameters:**

**`tensor`** `torch.Tensor`

This rank's shard, a Tensor of shape \[batch, channels, sequence, ...].

---

**`halo`** `DocumentHalo`

Halo length of every CP rank.

---

**`limit`** `int | None` — default: None

Largest number of tokens a layer can use; `None` fetches the whole halo.

---

**Returns:** `torch.Tensor`

Tensor of shape \[batch, channels, halo, ...] in sequence order. Gradients flow back

```python
nemo_automodel.components.models.lfm3.context_parallel.prepare_cp_inputs(
    batch: nemo_automodel.components.models.lfm3.context_parallel.CPBatch,
    num_chunks: int = 1
) -> dict[str, nemo_automodel.components.distributed.context_parallel.sharder.ContextParallelSharder]
```

Return the sharder for packed THD batches at any CP size and for unpacked batches under CP.

**Parameters:**

**`batch`** `CPBatch`

Unused here; the sharder reads the batch, with token Tensors of shape
\[batch, sequence], when it shards.

---

**`num_chunks`** `int` — default: 1

Unused; LFM3 shards whole rows.

---

**Returns:** `dict[str, ContextParallelSharder]`

Mapping with the `cp_sharder` whose `shard_batch` is `shard_documents` and

```python
nemo_automodel.components.models.lfm3.context_parallel.shard_documents(
    cp_mesh: torch.distributed.device_mesh.DeviceMesh | None,
    tp_mesh: torch.distributed.device_mesh.DeviceMesh | None,
    batch: nemo_automodel.components.models.lfm3.context_parallel.CPBatch,
    loss_mask: torch.Tensor | None = None,
    padding_token_id: int = 0
) -> tuple[collections.abc.Callable[[], contextlib.AbstractContextManager], nemo_automodel.components.models.lfm3.context_parallel.CPBatch, nemo_automodel.components.distributed.context_parallel.sharder.ShardLayout]
```

Keep this CP rank's contiguous shard and describe its documents to the model.

**Parameters:**

**`cp_mesh`** `DeviceMesh | None`

CP mesh, or `None`; every rank of it receives the same batch.

---

**`tp_mesh`** `DeviceMesh | None`

Unused TP mesh forwarded to the shared sharder.

---

**`batch`** `CPBatch`

Token Tensors of shape \[batch, sequence]. A THD batch carries the document
spans of each row in `seq_lens_padded`, a batch packed for FlashAttention
carries `cu_seq_lens_q` or the document indices `_packed_seq_ids`, and an
unpacked batch holds one document per row, split where a 2D `attention_mask`
marks padding (see `_document_boundaries`).

---

**`loss_mask`** `torch.Tensor | None` — default: None

Optional Tensor of shape \[batch, sequence] sharded with the batch.

---

**`padding_token_id`** `int` — default: 0

Token used when the sequence is padded to a multiple of the CP size.

---

**Returns:** `Callable[[], contextlib.AbstractContextManager]`

A null context, the batch with Tensors of shape \[batch, sequence / CP] plus

```python
nemo_automodel.components.models.lfm3.context_parallel.CPBatch: TypeAlias = dict[str, torch.Tensor | DocumentHalo | str | int | None]
```