bridge.models.inkling.modeling_inkling2.attention#
Inkling2 attention: learned relative logits, a query-conditioned sink, and K/V short convolutions.
Megatron Core’s attention modules cannot express this layer: scores add a query-dependent relative logit per causal distance, a per-query sink logit absorbs probability mass, Q/K/R are L2-clipped instead of normalized, K and V pass through short convolutions (with per-token dynamic taps in full-attention layers), and full-attention layers read Q, K, and V from three different residual mixtures. The core attention runs on PyTorch FlexAttention without the sink; the sink then scales each output row through the log-normalizer that FlexAttention returns, which keeps the sink out of the attention kernels.
Under context parallelism each rank keeps its own queries and gathers K and V
from the whole CP group (see context_parallel), so cp_comm_type does not
apply to this layer.
Module Contents#
Classes#
One Inkling2 attention sublayer, including its output short convolution. |
Functions#
Return FlexAttention kernel options for |
|
Run FlexAttention without the sink, then add the sink through the log-normalizer. |
|
Return the token-layout document boundaries of a packed batch, or |
|
Causal attention with relative logits and a zero-value sink. |
|
|
|
Reference implementation of the relative-sink attention for CPU inputs. |
|
Data#
API#
- bridge.models.inkling.modeling_inkling2.attention._compiled_flex_attention#
None
- bridge.models.inkling.modeling_inkling2.attention._block_mask_cache: OrderedDict[tuple, object]#
‘OrderedDict(…)’
- bridge.models.inkling.modeling_inkling2.attention._BLACKWELL_BACKWARD_KERNEL_OPTIONS#
None
- bridge.models.inkling.modeling_inkling2.attention._flex_attention()#
- bridge.models.inkling.modeling_inkling2.attention._flex_kernel_options(
- device: torch.device,
- head_dim: int,
Return FlexAttention kernel options for
deviceandhead_dim, orNonefor PyTorch’s defaults.
- bridge.models.inkling.modeling_inkling2.attention._attend_with_sink(
- query,
- key,
- value,
- sink,
- *,
- score_mod,
- block_mask,
- out_dtype,
Run FlexAttention without the sink, then add the sink through the log-normalizer.
A zero-value sink adds
exp(sink)to each softmax denominator, which scales the output row byexp(lse) / (exp(lse) + exp(sink)) = sigmoid(lse - sink). Keeping the sink out of the kernel matters for speed: as an extra key block that every query block attends to, it serializes the backward pass over all query blocks.- Parameters:
query –
[S_q, B, Hq, D].key –
[S_kv, B, Hkv, D].value –
[S_kv, B, Hkv, D].sink – Sink logits
[B, Hq, S_q].score_mod – FlexAttention score modification (the relative logits).
block_mask – FlexAttention block mask over
S_qqueries andS_kvkeys.out_dtype – Dtype of the returned output.
- Returns:
Attention output
[S_q, B, Hq, D].
- bridge.models.inkling.modeling_inkling2.attention.packed_cu_seqlens(
- packed_seq_params: megatron.core.packed_seq_params.PackedSeqParams | None,
Return the token-layout document boundaries of a packed batch, or
None.
- bridge.models.inkling.modeling_inkling2.attention._block_mask(
- seq_len: int,
- window: int | None,
- docs: torch.Tensor,
- cache_key: tuple,
- bridge.models.inkling.modeling_inkling2.attention.relative_sink_attention(
- query: torch.Tensor,
- key: torch.Tensor,
- value: torch.Tensor,
- rel_logits: torch.Tensor,
- sink: torch.Tensor,
- *,
- window: int | None,
- cu_seqlens: torch.Tensor | None = None,
- out_dtype: torch.dtype | None = None,
Causal attention with relative logits and a zero-value sink.
score(i, j, h) = q_i . k_j / sqrt(D) + rel_logits[h, i, i - j]for0 <= i - j < E, and the sink contributesexp(sink[h, i])to the softmax denominator with a zero value, which equals scaling the ordinary output bysigmoid(lse - sink).- Parameters:
query –
[S, B, Hq, D].key –
[S, B, Hkv, D].value –
[S, B, Hkv, D].rel_logits – Relative logits
[B, Hq, S, E].sink – Sink logits
[B, Hq, S].window – Local attention span including the current token, or
None.cu_seqlens – Document boundaries of a packed batch (
B == 1).out_dtype – Dtype of the output; the dtype of
querywhenNone.
- Returns:
Attention output
[S, B, Hq, D].
- bridge.models.inkling.modeling_inkling2.attention._context_parallel_block_mask(
- layout: megatron.bridge.models.inkling.modeling_inkling2.context_parallel.ContextParallelLayout,
- window: int | None,
- query_len: int,
- bridge.models.inkling.modeling_inkling2.attention.context_parallel_relative_sink_attention(
- query: torch.Tensor,
- key: torch.Tensor,
- value: torch.Tensor,
- rel_logits: torch.Tensor,
- sink: torch.Tensor,
- *,
- window: int | None,
- layout: megatron.bridge.models.inkling.modeling_inkling2.context_parallel.ContextParallelLayout,
- out_dtype: torch.dtype | None = None,
- Func:
relative_sink_attentionfor one context-parallel rank’s queries over the full sequence.- Parameters:
query – This rank’s queries
[S_local, B, Hq, D], in local order.key – Keys of the full sequence
[S, B, Hkv, D], in global order.value – Values of the full sequence
[S, B, Hkv, D], in global order.rel_logits – Relative logits of this rank’s queries
[B, Hq, S_local, E].sink – Sink logits of this rank’s queries
[B, Hq, S_local].window – Local attention span including the current token, or
None.layout – This rank’s token layout.
out_dtype – Dtype of the output; the dtype of
querywhenNone.
- Returns:
Attention output for this rank’s queries
[S_local, B, Hq, D].
- bridge.models.inkling.modeling_inkling2.attention._eager_relative_sink_attention(
- query,
- key,
- value,
- rel_logits,
- sink,
- *,
- window,
- docs,
- query_positions=None,
- out_dtype=None,
Reference implementation of the relative-sink attention for CPU inputs.
docslabels every key;query_positionsgives each query’s index into the keys (all of them, in order, whenNone). The output hasout_dtype, or the dtype ofquerywhenNone.
- bridge.models.inkling.modeling_inkling2.attention._linear(module: torch.nn.Module, inputs: torch.Tensor) torch.Tensor#
- bridge.models.inkling.modeling_inkling2.attention._tp_sharded_parameter(
- tensor: torch.Tensor,
- partition_dim: int,
- class bridge.models.inkling.modeling_inkling2.attention.Inkling2Attention(
- config,
- layer_number: int,
- cp_comm_type: str | None = None,
- pg_collection: megatron.core.process_groups_config.ProcessGroupCollection | None = None,
- pp_layer_offset: int | None = None,
- name: str | None = None,
- is_mtp_layer: bool = False,
Bases:
megatron.core.transformer.module.MegatronModuleOne Inkling2 attention sublayer, including its output short convolution.
The module receives the normalized Q, K, and V sources (one shared tensor in local layers). Under sequence parallelism it gathers them, computes the projections for its tensor-parallel heads on the full sequence, and returns its sequence-parallel slice of the output.
Initialization
- _column_linear(
- input_size: int,
- output_size: int,
- sharded_state_dict(
- prefix: str = '',
- sharded_offsets: tuple = (),
- metadata: dict | None = None,
Declare the tensor-parallel axes of the directly owned parameters.
- _gather(tensor: torch.Tensor) torch.Tensor#
- forward(
- query_source: torch.Tensor,
- key_source: torch.Tensor | None = None,
- value_source: torch.Tensor | None = None,
- *,
- packed_seq_params: megatron.core.packed_seq_params.PackedSeqParams | None = None,
Compute the attention sublayer output, including its output short convolution.
- Parameters:
query_source – Normalized Q-side source
[S(/TP), B, H]; it also feeds R, the output gate, and the sink.key_source – Normalized K-side source; defaults to
query_source.value_source – Normalized V-side source; defaults to
query_source.packed_seq_params – Document boundaries of a packed (THD) batch; under context parallelism, the boundaries of the full sequence.
- Returns:
Output of shape
[S(/TP), B, H].
- _dynamic_conv(
- inputs: torch.Tensor,
- source: torch.Tensor,
- static_weight: torch.Tensor,
- projection: torch.Tensor,
- positions: torch.Tensor,
- layout: megatron.bridge.models.inkling.modeling_inkling2.context_parallel.ContextParallelLayout | None,