bridge.models.inkling.modeling_inkling2.context_parallel#
Context parallelism for the Inkling2 decoder.
Megatron’s context-parallel batch split gives CP rank r two zigzag chunks of
every sequence, or of every packed document: chunk r and chunk
2 * CP - 1 - r. Inkling2 reads across tokens in two places. The attention
gathers K and V from every rank and puts them back in global order, while each
rank keeps its own queries. The causal short convolutions on K, V, the attention
output, and the MoE output need only the W - 1 tokens before each local
chunk; every rank shares the last W - 1 rows of its chunks in a small
all-gather (the halo). Everything else in the decoder works token by token.
Module Contents#
Classes#
Where one CP rank’s tokens sit in the full sequence, and how to reach the others. |
Functions#
Return the global index of every CP rank’s local tokens, |
|
Return this rank’s token layout, or |
Data#
API#
- bridge.models.inkling.modeling_inkling2.context_parallel.context_parallel_token_indices(
- total_tokens: int,
- cp_size: int,
- device: torch.device,
- cu_seqlens: torch.Tensor | None = None,
Return the global index of every CP rank’s local tokens,
[cp_size, total_tokens // cp_size].Follows Megatron’s split. Without
cu_seqlensthe sequence is cut into2 * cp_sizeequal chunks and rankrholds chunksrand2 * cp_size - 1 - r. With packed documents, every segment between consecutive physical boundaries is split the same way, segment by segment, as Transformer Engine’sthd_get_partitioned_indicesdoes.- Parameters:
total_tokens – Sequence length before the split.
cp_size – Context-parallel world size.
device – Device of the returned indices.
cu_seqlens – Physical document boundaries of a packed sequence, or
None.
- Returns:
Long tensor whose row
rlists rankr’s tokens in local order.
- class bridge.models.inkling.modeling_inkling2.context_parallel.ContextParallelLayout#
Where one CP rank’s tokens sit in the full sequence, and how to reach the others.
.. attribute:: group
Context-parallel process group.
.. attribute:: total_tokens
Sequence length before the CP split.
.. attribute:: local_index
Global index of each local token,
[S_local]... attribute:: gather_order
Row of the rank-major all-gather that holds each global token,
[S]... attribute:: local_positions
Offset of each local token inside its document,
[S_local, B]... attribute:: local_documents
Document of each local token,
[S_local]... attribute:: documents
Document of every global token,
[S]... attribute:: halo_rows
Local rows shared with the other ranks: the last
W - 1rows of every local chunk... attribute:: halo_sources
For each lag
< W, the row ofcat(local rows, gathered halos)that holdsx[t - lag]for every local tokent, or-1before the start of its document... attribute:: block_masks
FlexAttention block masks of this layout, by attention window.
- group: torch.distributed.ProcessGroup#
None
- total_tokens: int#
None
- local_index: torch.Tensor#
None
- gather_order: torch.Tensor#
None
- local_positions: torch.Tensor#
None
- local_documents: torch.Tensor#
None
- documents: torch.Tensor#
None
- halo_rows: torch.Tensor#
None
- halo_sources: torch.Tensor#
None
- block_masks: dict#
‘field(…)’
- gather(x: torch.Tensor) torch.Tensor#
All-gather local rows
[S_local, ...]into the full sequence[S, ...]in global order.The backward pass reduce-scatters, so every rank’s contribution to a gathered row reaches the rank that owns it.
- extend(x: torch.Tensor) torch.Tensor#
Append every rank’s halo rows to the local rows
[S_local, ...].
- shift(
- extended: torch.Tensor,
- x: torch.Tensor,
- lag: int,
Return
x[t - lag]for every local token, zero before the start of its document.- Parameters:
extended –
extend(x).x – Local rows
[S_local, ...].lag – Distance back in the global sequence, below the convolution width.
- bridge.models.inkling.modeling_inkling2.context_parallel._layout_cache: OrderedDict[tuple, ContextParallelLayout]#
‘OrderedDict(…)’
- bridge.models.inkling.modeling_inkling2.context_parallel.context_parallel_layout(
- group: torch.distributed.ProcessGroup,
- *,
- local_tokens: int,
- batch: int,
- device: torch.device,
- width: int,
- cu_seqlens: torch.Tensor | None = None,
Return this rank’s token layout, or
Nonewithout context parallelism.- Parameters:
group – Context-parallel process group.
local_tokens – Local sequence length (after the CP split, before sequence parallelism).
batch – Batch size (1 for packed sequences).
device – Device of the layout tensors.
width – Short-convolution width, which bounds the halo.
cu_seqlens – Physical document boundaries of a packed batch, or
None.
- Returns:
The cached layout for this sequence shape and document structure.