bridge.models.inkling.modeling_inkling2.context_parallel#

Context parallelism for the Inkling2 decoder.

Megatron’s context-parallel batch split gives CP rank r two zigzag chunks of every sequence, or of every packed document: chunk r and chunk 2 * CP - 1 - r. Inkling2 reads across tokens in two places. The attention gathers K and V from every rank and puts them back in global order, while each rank keeps its own queries. The causal short convolutions on K, V, the attention output, and the MoE output need only the W - 1 tokens before each local chunk; every rank shares the last W - 1 rows of its chunks in a small all-gather (the halo). Everything else in the decoder works token by token.

Module Contents#

Classes#

ContextParallelLayout

Where one CP rank’s tokens sit in the full sequence, and how to reach the others.

Functions#

context_parallel_token_indices

Return the global index of every CP rank’s local tokens, [cp_size, total_tokens // cp_size].

context_parallel_layout

Return this rank’s token layout, or None without context parallelism.

Data#

API#

bridge.models.inkling.modeling_inkling2.context_parallel.context_parallel_token_indices(
total_tokens: int,
cp_size: int,
device: torch.device,
cu_seqlens: torch.Tensor | None = None,
) → torch.Tensor#

Return the global index of every CP rank’s local tokens, [cp_size, total_tokens // cp_size].

Follows Megatron’s split. Without cu_seqlens the sequence is cut into 2 * cp_size equal chunks and rank r holds chunks r and 2 * cp_size - 1 - r. With packed documents, every segment between consecutive physical boundaries is split the same way, segment by segment, as Transformer Engine’s thd_get_partitioned_indices does.

Parameters:
  • total_tokens – Sequence length before the split.

  • cp_size – Context-parallel world size.

  • device – Device of the returned indices.

  • cu_seqlens – Physical document boundaries of a packed sequence, or None.

Returns:

Long tensor whose row r lists rank r’s tokens in local order.

class bridge.models.inkling.modeling_inkling2.context_parallel.ContextParallelLayout#

Where one CP rank’s tokens sit in the full sequence, and how to reach the others.

.. attribute:: group

Context-parallel process group.

.. attribute:: total_tokens

Sequence length before the CP split.

.. attribute:: local_index

Global index of each local token, [S_local].

.. attribute:: gather_order

Row of the rank-major all-gather that holds each global token, [S].

.. attribute:: local_positions

Offset of each local token inside its document, [S_local, B].

.. attribute:: local_documents

Document of each local token, [S_local].

.. attribute:: documents

Document of every global token, [S].

.. attribute:: halo_rows

Local rows shared with the other ranks: the last W - 1 rows of every local chunk.

.. attribute:: halo_sources

For each lag < W, the row of cat(local rows, gathered halos) that holds x[t - lag] for every local token t, or -1 before the start of its document.

.. attribute:: block_masks

FlexAttention block masks of this layout, by attention window.

group: torch.distributed.ProcessGroup#

None

total_tokens: int#

None

local_index: torch.Tensor#

None

gather_order: torch.Tensor#

None

local_positions: torch.Tensor#

None

local_documents: torch.Tensor#

None

documents: torch.Tensor#

None

halo_rows: torch.Tensor#

None

halo_sources: torch.Tensor#

None

block_masks: dict#

‘field(…)’

gather(x: torch.Tensor) → torch.Tensor#

All-gather local rows [S_local, ...] into the full sequence [S, ...] in global order.

The backward pass reduce-scatters, so every rank’s contribution to a gathered row reaches the rank that owns it.

extend(x: torch.Tensor) → torch.Tensor#

Append every rank’s halo rows to the local rows [S_local, ...].

shift(
extended: torch.Tensor,
x: torch.Tensor,
lag: int,
) → torch.Tensor#

Return x[t - lag] for every local token, zero before the start of its document.

Parameters:
  • extended – extend(x).

  • x – Local rows [S_local, ...].

  • lag – Distance back in the global sequence, below the convolution width.

bridge.models.inkling.modeling_inkling2.context_parallel._layout_cache: OrderedDict[tuple, ContextParallelLayout]#

‘OrderedDict(…)’

bridge.models.inkling.modeling_inkling2.context_parallel.context_parallel_layout(
group: torch.distributed.ProcessGroup,
*,
local_tokens: int,
batch: int,
device: torch.device,
width: int,
cu_seqlens: torch.Tensor | None = None,
) → bridge.models.inkling.modeling_inkling2.context_parallel.ContextParallelLayout | None#

Return this rank’s token layout, or None without context parallelism.

Parameters:
  • group – Context-parallel process group.

  • local_tokens – Local sequence length (after the CP split, before sequence parallelism).

  • batch – Batch size (1 for packed sequences).

  • device – Device of the layout tensors.

  • width – Short-convolution width, which bounds the halo.

  • cu_seqlens – Physical document boundaries of a packed batch, or None.

Returns:

The cached layout for this sequence shape and document structure.