bridge.models.inkling.modeling_inkling2.attention#

Inkling2 attention: learned relative logits, a query-conditioned sink, and K/V short convolutions.

Megatron Core’s attention modules cannot express this layer: scores add a query-dependent relative logit per causal distance, a per-query sink logit absorbs probability mass, Q/K/R are L2-clipped instead of normalized, K and V pass through short convolutions (with per-token dynamic taps in full-attention layers), and full-attention layers read Q, K, and V from three different residual mixtures. The core attention runs on PyTorch FlexAttention without the sink; the sink then scales each output row through the log-normalizer that FlexAttention returns, which keeps the sink out of the attention kernels.

Under context parallelism each rank keeps its own queries and gathers K and V from the whole CP group (see context_parallel), so cp_comm_type does not apply to this layer.

Module Contents#

Classes#

Inkling2Attention

One Inkling2 attention sublayer, including its output short convolution.

Functions#

_flex_attention

_flex_kernel_options

Return FlexAttention kernel options for device and head_dim, or None for PyTorch’s defaults.

_attend_with_sink

Run FlexAttention without the sink, then add the sink through the log-normalizer.

packed_cu_seqlens

Return the token-layout document boundaries of a packed batch, or None.

_block_mask

relative_sink_attention

Causal attention with relative logits and a zero-value sink.

_context_parallel_block_mask

context_parallel_relative_sink_attention

func:

relative_sink_attention for one context-parallel rank’s queries over the full sequence.

param query:

This rank’s queries [S_local, B, Hq, D], in local order.

param key:

Keys of the full sequence [S, B, Hkv, D], in global order.

param value:

Values of the full sequence [S, B, Hkv, D], in global order.

param rel_logits:

Relative logits of this rank’s queries [B, Hq, S_local, E].

param sink:

Sink logits of this rank’s queries [B, Hq, S_local].

param window:

Local attention span including the current token, or None.

param layout:

This rank’s token layout.

param out_dtype:

Dtype of the output; the dtype of query when None.

returns:

Attention output for this rank’s queries [S_local, B, Hq, D].

_eager_relative_sink_attention

Reference implementation of the relative-sink attention for CPU inputs.

_linear

_tp_sharded_parameter

Data#

API#

bridge.models.inkling.modeling_inkling2.attention._compiled_flex_attention#

None

bridge.models.inkling.modeling_inkling2.attention._block_mask_cache: OrderedDict[tuple, object]#

‘OrderedDict(…)’

bridge.models.inkling.modeling_inkling2.attention._BLACKWELL_BACKWARD_KERNEL_OPTIONS#

None

bridge.models.inkling.modeling_inkling2.attention._flex_attention()#
bridge.models.inkling.modeling_inkling2.attention._flex_kernel_options(
device: torch.device,
head_dim: int,
) → dict | None#

Return FlexAttention kernel options for device and head_dim, or None for PyTorch’s defaults.

bridge.models.inkling.modeling_inkling2.attention._attend_with_sink(
query,
key,
value,
sink,
*,
score_mod,
block_mask,
out_dtype,
)#

Run FlexAttention without the sink, then add the sink through the log-normalizer.

A zero-value sink adds exp(sink) to each softmax denominator, which scales the output row by exp(lse) / (exp(lse) + exp(sink)) = sigmoid(lse - sink). Keeping the sink out of the kernel matters for speed: as an extra key block that every query block attends to, it serializes the backward pass over all query blocks.

Parameters:
  • query – [S_q, B, Hq, D].

  • key – [S_kv, B, Hkv, D].

  • value – [S_kv, B, Hkv, D].

  • sink – Sink logits [B, Hq, S_q].

  • score_mod – FlexAttention score modification (the relative logits).

  • block_mask – FlexAttention block mask over S_q queries and S_kv keys.

  • out_dtype – Dtype of the returned output.

Returns:

Attention output [S_q, B, Hq, D].

bridge.models.inkling.modeling_inkling2.attention.packed_cu_seqlens(
packed_seq_params: megatron.core.packed_seq_params.PackedSeqParams | None,
) → torch.Tensor | None#

Return the token-layout document boundaries of a packed batch, or None.

bridge.models.inkling.modeling_inkling2.attention._block_mask(
seq_len: int,
window: int | None,
docs: torch.Tensor,
cache_key: tuple,
)#
bridge.models.inkling.modeling_inkling2.attention.relative_sink_attention(
query: torch.Tensor,
key: torch.Tensor,
value: torch.Tensor,
rel_logits: torch.Tensor,
sink: torch.Tensor,
*,
window: int | None,
cu_seqlens: torch.Tensor | None = None,
out_dtype: torch.dtype | None = None,
) → torch.Tensor#

Causal attention with relative logits and a zero-value sink.

score(i, j, h) = q_i . k_j / sqrt(D) + rel_logits[h, i, i - j] for 0 <= i - j < E, and the sink contributes exp(sink[h, i]) to the softmax denominator with a zero value, which equals scaling the ordinary output by sigmoid(lse - sink).

Parameters:
  • query – [S, B, Hq, D].

  • key – [S, B, Hkv, D].

  • value – [S, B, Hkv, D].

  • rel_logits – Relative logits [B, Hq, S, E].

  • sink – Sink logits [B, Hq, S].

  • window – Local attention span including the current token, or None.

  • cu_seqlens – Document boundaries of a packed batch (B == 1).

  • out_dtype – Dtype of the output; the dtype of query when None.

Returns:

Attention output [S, B, Hq, D].

bridge.models.inkling.modeling_inkling2.attention._context_parallel_block_mask(
layout: megatron.bridge.models.inkling.modeling_inkling2.context_parallel.ContextParallelLayout,
window: int | None,
query_len: int,
)#
bridge.models.inkling.modeling_inkling2.attention.context_parallel_relative_sink_attention(
query: torch.Tensor,
key: torch.Tensor,
value: torch.Tensor,
rel_logits: torch.Tensor,
sink: torch.Tensor,
*,
window: int | None,
layout: megatron.bridge.models.inkling.modeling_inkling2.context_parallel.ContextParallelLayout,
out_dtype: torch.dtype | None = None,
) → torch.Tensor#
Func:

relative_sink_attention for one context-parallel rank’s queries over the full sequence.

Parameters:
  • query – This rank’s queries [S_local, B, Hq, D], in local order.

  • key – Keys of the full sequence [S, B, Hkv, D], in global order.

  • value – Values of the full sequence [S, B, Hkv, D], in global order.

  • rel_logits – Relative logits of this rank’s queries [B, Hq, S_local, E].

  • sink – Sink logits of this rank’s queries [B, Hq, S_local].

  • window – Local attention span including the current token, or None.

  • layout – This rank’s token layout.

  • out_dtype – Dtype of the output; the dtype of query when None.

Returns:

Attention output for this rank’s queries [S_local, B, Hq, D].

bridge.models.inkling.modeling_inkling2.attention._eager_relative_sink_attention(
query,
key,
value,
rel_logits,
sink,
*,
window,
docs,
query_positions=None,
out_dtype=None,
)#

Reference implementation of the relative-sink attention for CPU inputs.

docs labels every key; query_positions gives each query’s index into the keys (all of them, in order, when None). The output has out_dtype, or the dtype of query when None.

bridge.models.inkling.modeling_inkling2.attention._linear(module: torch.nn.Module, inputs: torch.Tensor) → torch.Tensor#
bridge.models.inkling.modeling_inkling2.attention._tp_sharded_parameter(
tensor: torch.Tensor,
partition_dim: int,
) → torch.nn.Parameter#
class bridge.models.inkling.modeling_inkling2.attention.Inkling2Attention(
config,
layer_number: int,
cp_comm_type: str | None = None,
pg_collection: megatron.core.process_groups_config.ProcessGroupCollection | None = None,
pp_layer_offset: int | None = None,
name: str | None = None,
is_mtp_layer: bool = False,
)#

Bases: megatron.core.transformer.module.MegatronModule

One Inkling2 attention sublayer, including its output short convolution.

The module receives the normalized Q, K, and V sources (one shared tensor in local layers). Under sequence parallelism it gathers them, computes the projections for its tensor-parallel heads on the full sequence, and returns its sequence-parallel slice of the output.

Initialization

_column_linear(
input_size: int,
output_size: int,
) → megatron.core.extensions.transformer_engine.TEColumnParallelLinear#
sharded_state_dict(
prefix: str = '',
sharded_offsets: tuple = (),
metadata: dict | None = None,
) → megatron.core.dist_checkpointing.mapping.ShardedStateDict#

Declare the tensor-parallel axes of the directly owned parameters.

_gather(tensor: torch.Tensor) → torch.Tensor#
forward(
query_source: torch.Tensor,
key_source: torch.Tensor | None = None,
value_source: torch.Tensor | None = None,
*,
packed_seq_params: megatron.core.packed_seq_params.PackedSeqParams | None = None,
) → torch.Tensor#

Compute the attention sublayer output, including its output short convolution.

Parameters:
  • query_source – Normalized Q-side source [S(/TP), B, H]; it also feeds R, the output gate, and the sink.

  • key_source – Normalized K-side source; defaults to query_source.

  • value_source – Normalized V-side source; defaults to query_source.

  • packed_seq_params – Document boundaries of a packed (THD) batch; under context parallelism, the boundaries of the full sequence.

Returns:

Output of shape [S(/TP), B, H].

_dynamic_conv(
inputs: torch.Tensor,
source: torch.Tensor,
static_weight: torch.Tensor,
projection: torch.Tensor,
positions: torch.Tensor,
layout: megatron.bridge.models.inkling.modeling_inkling2.context_parallel.ContextParallelLayout | None,
) → torch.Tensor#