Qwen3.8-2.4T-A95B

Qwen3.8-2.4T-A95B is Alibaba Cloud’s sparse Qwen3.8 text backbone. NeMo AutoModel provides a native CausalLM implementation for full-parameter fine-tuning with FSDP2, expert parallelism, and pipeline parallelism.

Fine-Tune Qwen3.8-2.4T-A95B

Allocate 64 GB200 nodes with four GPUs per node through your cluster launcher, then run:

uv run automodel --nproc-per-node=4 examples/llm_finetune/qwen/qwen3_8_2_4t_a95b_hellaswag_ep32_pp8.yaml

See the Slurm Launcher Guide for multi-node setup and the Large MoE Fine-Tuning Guide for distributed MoE configuration.

Model Reference

Model Architecture

Architecture

• The text entry point uses the native Qwen3.8-2.4T-A95B CausalLM path.
• The decoder combines attention, GatedDeltaNet linear attention, routed experts, shared experts, and optional multi-token prediction.
• The HF state-dict adapter handles both grouped BF16 expert checkpoints and split block-FP8 per-expert checkpoints.
• GatedDeltaNet state tensors that require fp32 precision keep the expected checkpoint contract.

Validated Configuration

The checked-in recipe targets the public checkpoint, whose index at revision 207bd685a7e3696cfaff12ded7c6a7ea0f88c996 contains grouped BF16 expert tensors, not split-FP8 expert weights. The recipe disables MTP, so this configuration does not validate split-FP8 MTP loading or model convergence. Split-FP8 decoder and MTP loading are covered separately by checkpoint-loader regression tests.

Example Recipe

References

• Qwen3.8-2.4T-A95B model card
• Qwen3.8-2.4T-A95B announcement