bridge.models.inkling.modeling_inkling2.spec#

MCore Hybrid stack specification for Inkling2.

Module Contents#

Functions#

_with_situ_activation

Clone an MLP or MoE builder and install the softcapped SwiGLU activation.

build_inkling2_hybrid_stack_spec

Compose Inkling2 attention and SiTU MoE layers for a repeated *E Hybrid stack.

API#

bridge.models.inkling.modeling_inkling2.spec._with_situ_activation(builder)#

Clone an MLP or MoE builder and install the softcapped SwiGLU activation.

bridge.models.inkling.modeling_inkling2.spec.build_inkling2_hybrid_stack_spec(
config,
) → megatron.core.transformer.spec_utils.ModuleSpec#

Compose Inkling2 attention and SiTU MoE layers for a repeated *E Hybrid stack.

Parameters:

config – The Inkling2 transformer config.

Returns:

A HybridStack spec whose attention and MoE layers form the decoder blocks.