Laguna

Laguna is Poolside’s hybrid-attention MoE language model family. The Automodel implementation targets full SFT with expert parallelism for large Laguna checkpoints such as Laguna S 2.1.

Architecture

• LagunaForCausalLM
• Layer-specific attention head counts, QK RMSNorm, and softplus attention output gating.
• Full and sliding-window attention layers can use separate RoPE settings.
• MoE blocks use nemo_automodel.components.moe.layers.MoE with sigmoid routing, top-k probability normalization, fp32 gate compute, correction bias loading, one shared expert, and grouped expert weights for EP.

Example Recipes

Run the Recipe

$
uv run automodel --nproc-per-node=8 examples/llm_finetune/laguna/laguna_s_2p1_hellaswag_ep16.yaml

For multi-node launches, submit through your cluster launcher with two 8-GPU nodes for the default ep_size: 16.