Qwen3.8-2.4T-A95B

Qwen3.8-2.4T-A95B is Alibaba Cloud’s sparse Qwen3.8 text backbone. NeMo AutoModel provides a native CausalLM implementation for full-parameter fine-tuning with FSDP2, expert parallelism, and pipeline parallelism.

Fine-Tune Qwen3.8-2.4T-A95B

Allocate 64 GB200 nodes with four GPUs per node through your cluster launcher, then run:

uv run automodel --nproc-per-node=4 examples/llm_finetune/qwen/qwen3_8_2_4t_a95b_hellaswag_ep32_pp8.yaml

See the Slurm Launcher Guide for multi-node setup and the Large MoE Fine-Tuning Guide for distributed MoE configuration.

Model Reference

Model Architecture

Architecture

• The text entry point uses the native Qwen3.8-2.4T-A95B CausalLM path.
• The decoder combines attention, GatedDeltaNet linear attention, routed experts, shared experts, and optional multi-token prediction.
• The HF state-dict adapter handles both grouped BF16 expert checkpoints and split block-FP8 per-expert checkpoints.
• GatedDeltaNet state tensors that require fp32 precision keep the expected checkpoint contract.

Validated Configuration

The validated 64-node run loaded the public split-FP8 checkpoint and completed a full-model HellaSwag SFT run. This validates the training mechanics and checkpoint layout, not model convergence.

Example Recipe

References

• Qwen3.8-2.4T-A95B model card
• Qwen3.8-2.4T-A95B announcement