bridge.models.inkling.inkling2_config#

Builder-backed configuration for the Inkling2 HybridModel integration.

Module Contents#

Classes#

Inkling2TransformerConfig

Transformer settings read by the Inkling2 attention, AttnRes, and MoE layers.

Inkling2ModelConfig

Inkling2 HybridModel with an optional image encoder on the embedding stage.

API#

class bridge.models.inkling.inkling2_config.Inkling2TransformerConfig#

Bases: megatron.bridge.models.transformer_config.TransformerConfig

Transformer settings read by the Inkling2 attention, AttnRes, and MoE layers.

variable_seq_lengths: bool#

True

Each layer passes the AttnRes bank packed with its partial block, so the width of the tensor between pipeline stages depends on the boundary and pipeline communication exchanges shapes.

hetereogenous_dist_checkpoint: bool#

True

Save each layer’s parameters under its own checkpoint key. The layers differ: full-attention layers carry three AttnRes queries and key and value input norms, and the first layer has no attention query, so the parameters cannot be stacked along a common layer axis.

inkling2_local_layers: tuple[int, ...]#

()

0-based indices of the sliding-window decoder blocks; the remaining blocks use full attention. Block i is Hybrid layers 2 * i (attention) and 2 * i + 1 (MoE).

inkling2_sliding_window: int#

1024

Local attention span, including the current token; also the local relative extent.

inkling2_rel_dim: int#

16

Width of the per-head query feature that produces relative logits.

inkling2_rel_extent: int#

1024

Number of causal distances with a learned relative logit in full-attention layers.

inkling2_qk_clip_threshold: float#

22.627416997969522

inkling2_rel_clip_threshold: float#

8.0

inkling2_value_norm: bool#

False

Whether values pass through a weightless RMSNorm over the head dimension.

inkling2_log_scaling_n_floor: float#

128000.0

inkling2_log_scaling_alpha: float#

0.1

inkling2_sconv_kernel_size: int#

4

inkling2_attn_res_block_size: int#

6

Number of decoder blocks whose outputs form one AttnRes block.

inkling2_attn_res_normalize_values: bool#

False

inkling2_mlp_gate_softcap: float#

0.0

inkling2_mlp_up_softcap: float#

0.0

inkling2_final_logit_softcap: float | None#

None

inkling2_unpadded_vocab_size: int | None#

None

Logits at and above this token ID are masked out of the softmax.

inkling2_compile_ops: bool#

False

Run the AttnRes mixtures, short convolutions, L2 clips, SiTU activations, and residual additions through torch.compile as fused kernels. Dynamo compiles a graph for every new call signature, so this suits training at fixed sequence shapes; leave it off for inference over changing sequence lengths.

class bridge.models.inkling.inkling2_config.Inkling2ModelConfig#

Bases: megatron.bridge.models.hybrid.model_config.BridgeHybridModelConfig

Inkling2 HybridModel with an optional image encoder on the embedding stage.

Each decoder block is an attention layer followed by a MoE layer, so the Hybrid pattern repeats *E and num_layers is twice the block count.

builder: ClassVar[str]#

‘megatron.bridge.models.inkling.inkling2_builder.Inkling2ModelBuilder’

transformer_config_class: ClassVar[type[megatron.bridge.models.transformer_config.TransformerConfig]]#

None

vision_config: dict[str, Any]#

‘field(…)’

HF vision_config of the checkpoint; empty for a text-only model.

image_token_id: int | None#

None

Placeholder token ID replaced by image features: <|unused_200054|> (200054) in the release.

freeze_language_model: bool#

False

freeze_vision_model: bool#

False

freeze_vision_projection: bool#

False

property special_token_ids: dict[str, int]#

Return the media placeholder IDs for collators.

finalize() → None#

Reject execution paths that the Inkling2 layers do not implement.

_get_num_floating_point_operations_with_runtime_stats(
*,
batch_size: int,
seqlen_sum: int | None,
seqlen_squared_sum: int | None,
cross_seqlen_sum: int | None = None,
cross_seqlen_product_sum: int | None = None,
) → float#

Return the forward and backward FLOPs of one global batch of text.

Counts the matrix multiplications with the conventions of the standard GPT estimate: the query, key, value, relative-query, gate, sink, and output projections; the dynamic-convolution projections of full-attention layers; the relative logits; causal attention over the full context or the sliding window; the router and the routed experts; and the output layer. Normalization, short convolutions, and the AttnRes mixtures are elementwise or per-source reductions and, like norms in the standard estimate, are not counted; neither is the image encoder.

Parameters:
  • batch_size – Number of sequences in the global batch.

  • seqlen_sum – Total tokens in the global batch, or None for full-length sequences.

  • seqlen_squared_sum – Sum of squared subsequence lengths, or None for full-length sequences.

  • cross_seqlen_sum – Unused; Inkling2 has no cross-attention.

  • cross_seqlen_product_sum – Unused; Inkling2 has no cross-attention.

Returns:

Floating-point operations of one training step.