bridge.models.inkling.modeling_inkling2.layers#
Inkling2 decoder blocks as Hybrid layers joined by Block Attention Residuals.
Each decoder block is an attention layer (Hybrid symbol *) followed by a MoE
layer (E), so Hybrid layers 2 * i and 2 * i + 1 hold block i.
Each sublayer reads a learned softmax mixture over the completed residual blocks
and the current partial block instead of a single residual stream. Full-attention
blocks mix separate Q, K, and V inputs. Every attention and MoE output passes
through a residual short convolution before it joins the partial block.
HybridStack passes a single tensor from layer to layer, so every layer packs the AttnRes bank of completed blocks next to the partial block along the hidden dimension. The width grows by one hidden size per completed block, also across pipeline stages, whose communication exchanges tensor shapes.
Module Contents#
Classes#
Softcapped SwiGLU expert activation, installed through MCore’s custom-activation path. |
|
AttnRes bookkeeping shared by the attention and MoE layers of a decoder block. |
|
Attention layer of an Inkling2 decoder block (Hybrid symbol |
|
MoE layer of an Inkling2 decoder block (Hybrid symbol |
Functions#
Return the number of completed AttnRes blocks that decoder block |
|
Split a packed AttnRes state into its |
API#
- class bridge.models.inkling.modeling_inkling2.layers.Inkling2SiTU(config)#
Bases:
torch.nn.ModuleSoftcapped SwiGLU expert activation, installed through MCore’s custom-activation path.
Initialization
- forward(inputs: torch.Tensor) torch.Tensor#
Apply the activation to a fused
[gate, up]projection.
- bridge.models.inkling.modeling_inkling2.layers.num_attn_res_blocks_before(block_index: int, block_size: int) int#
Return the number of completed AttnRes blocks that decoder block
block_indexreads.
- bridge.models.inkling.modeling_inkling2.layers.unpack_attn_res_state(
- state: torch.Tensor,
- hidden_size: int,
- count: int,
Split a packed AttnRes state into its
counthidden-size slices.
- class bridge.models.inkling.modeling_inkling2.layers._Inkling2HybridLayer(*args, **kwargs)#
Bases:
megatron.core.transformer.transformer_layer.TransformerLayerAttnRes bookkeeping shared by the attention and MoE layers of a decoder block.
Initialization
- _add_float32_parameter(name: str, *shape: int) None#
Register an AttnRes query, which stays FP32 under BF16 training.
- _mix(
- sources: list[torch.Tensor],
- queries: torch.Tensor,
- **kwargs,
- _normalize(
- norm: torch.nn.Module,
- inputs: torch.Tensor,
Normalize an AttnRes mixture into the activation dtype.
- _check_supported(inference_context) None#
- class bridge.models.inkling.modeling_inkling2.layers.Inkling2AttentionLayer(*args, **kwargs)#
Bases:
bridge.models.inkling.modeling_inkling2.layers._Inkling2HybridLayerAttention layer of an Inkling2 decoder block (Hybrid symbol
*).Initialization
- forward(
- hidden_states: torch.Tensor,
- attention_mask: torch.Tensor | None = None,
- inference_context=None,
- packed_seq_params: megatron.core.packed_seq_params.PackedSeqParams | None = None,
- **kwargs,
Attend over the AttnRes mixtures and hand the block’s state to its MoE layer.
- Parameters:
hidden_states – Packed
[partial, *bank]state of the previous block.packed_seq_params – Document boundaries of a packed batch.
- Returns:
[attention, *bank, partial]when the block starts a new AttnRes block, otherwise[partial, attention, *bank]; and no context.
- class bridge.models.inkling.modeling_inkling2.layers.Inkling2MoELayer(*args, **kwargs)#
Bases:
bridge.models.inkling.modeling_inkling2.layers._Inkling2HybridLayerMoE layer of an Inkling2 decoder block (Hybrid symbol
E).Initialization
- _mlp_conv(
- output: torch.Tensor,
- packed_seq_params: megatron.core.packed_seq_params.PackedSeqParams | None,
- static _without_bias(
- output_with_bias: tuple[torch.Tensor, torch.Tensor | None],
- forward(
- hidden_states: torch.Tensor,
- attention_mask: torch.Tensor | None = None,
- inference_context=None,
- packed_seq_params: megatron.core.packed_seq_params.PackedSeqParams | None = None,
- padding_mask: torch.Tensor | None = None,
- **kwargs,
Run the MoE on its AttnRes mixture and update the partial block.
- Parameters:
hidden_states – Packed state from the block’s attention layer.
packed_seq_params – Document boundaries of a packed batch.
padding_mask – Optional MoE router padding mask.
- Returns:
The packed
[partial, *bank]state, or the final AttnRes output after the last block; and no context.