Model CoverageLarge Language Models

Qwen3.8-2.4T-A95B

View as Markdown

Qwen3.8-2.4T-A95B is Alibaba Cloud’s sparse Qwen3.8 text backbone. NeMo AutoModel provides a native CausalLM implementation for full-parameter fine-tuning with FSDP2, expert parallelism, and pipeline parallelism.

TaskText Generation (MoE, hybrid attention)
ArchitectureQwen3.8-2.4T-A95B CausalLM
Parameters2.4T total
Layers92
Experts512 routed experts, top-10 routing
Context Length262,144 tokens
HF OrgQwen

Architecture

  • The text entry point uses the native Qwen3.8-2.4T-A95B CausalLM path.
  • The decoder combines attention, GatedDeltaNet linear attention, routed experts, shared experts, and optional multi-token prediction.
  • The HF state-dict adapter handles both grouped BF16 expert checkpoints and split block-FP8 per-expert checkpoints.
  • GatedDeltaNet state tensors that require fp32 precision keep the expected checkpoint contract.

Validated Configuration

AreaValidated Configuration
FSDP2Full-parameter training
Expert ParallelismEP32
Pipeline ParallelismPP8
WorkloadHellaSwag next-token prediction
Scale64 nodes / 256 GPUs
Duration100 training steps

The validated 64-node run loaded the public split-FP8 checkpoint and completed a full-model HellaSwag SFT run. This validates the training mechanics and checkpoint layout, not model convergence.

Example Recipe

RecipeDescription
qwen3_8_2_4t_a95b_hellaswag_ep32_pp8.yaml100-step full-parameter SFT on HellaSwag with FSDP2, EP32, and PP8

Run the Recipe

Allocate 64 GB200 nodes with four GPUs per node through your cluster launcher, then run:

$uv run automodel --nproc-per-node=4 examples/llm_finetune/qwen/qwen3_8_2_4t_a95b_hellaswag_ep32_pp8.yaml

See the Slurm Launcher Guide for multi-node setup and the Large MoE Fine-Tuning Guide for distributed MoE configuration.

References