> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/automodel/_mcp/server.

# Qwen3.8-2.4T-A95B

> Fine-tune Qwen3.8-2.4T-A95B text checkpoints with FSDP2, expert parallelism, and pipeline parallelism.

[Qwen3.8-2.4T-A95B](https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B) is Alibaba Cloud's sparse Qwen3.8 text backbone. NeMo AutoModel provides a native CausalLM implementation for full-parameter fine-tuning with FSDP2, expert parallelism, and pipeline parallelism.

|                    |                                         |
| ------------------ | --------------------------------------- |
| **Task**           | Text Generation (MoE, hybrid attention) |
| **Architecture**   | Qwen3.8-2.4T-A95B CausalLM              |
| **Parameters**     | 2.4T total                              |
| **Layers**         | 92                                      |
| **Experts**        | 512 routed experts, top-10 routing      |
| **Context Length** | 262,144 tokens                          |
| **HF Org**         | [Qwen](https://huggingface.co/Qwen)     |

## Architecture

* The text entry point uses the native Qwen3.8-2.4T-A95B CausalLM path.
* The decoder combines attention, GatedDeltaNet linear attention, routed experts, shared experts, and optional multi-token prediction.
* The HF state-dict adapter handles both grouped BF16 expert checkpoints and split block-FP8 per-expert checkpoints.
* GatedDeltaNet state tensors that require fp32 precision keep the expected checkpoint contract.

## Validated Configuration

| Area                 | Validated Configuration         |
| -------------------- | ------------------------------- |
| FSDP2                | Full-parameter training         |
| Expert Parallelism   | EP32                            |
| Pipeline Parallelism | PP8                             |
| Workload             | HellaSwag next-token prediction |
| Scale                | 64 nodes / 256 GPUs             |
| Duration             | 100 training steps              |

The validated 64-node run loaded the public split-FP8 checkpoint and completed a full-model HellaSwag SFT run. This validates the training mechanics and checkpoint layout, not model convergence.

## Example Recipe

| Recipe                                                                                                                                                                      | Description                                                        |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ |
| [qwen3\_8\_2\_4t\_a95b\_hellaswag\_ep32\_pp8.yaml](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/llm_finetune/qwen/qwen3_8_2_4t_a95b_hellaswag_ep32_pp8.yaml) | 100-step full-parameter SFT on HellaSwag with FSDP2, EP32, and PP8 |

## Run the Recipe

Allocate 64 GB200 nodes with four GPUs per node through your cluster launcher, then run:

```bash
uv run automodel --nproc-per-node=4 examples/llm_finetune/qwen/qwen3_8_2_4t_a95b_hellaswag_ep32_pp8.yaml
```

See the [Slurm Launcher Guide](/job-launchers/slurm-cluster) for multi-node setup and the [Large MoE Fine-Tuning Guide](/recipes-e2e-examples/large-moe-fine-tuning) for distributed MoE configuration.

## References

* [Qwen3.8-2.4T-A95B model card](https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B)
* [Qwen3.8-2.4T-A95B announcement](https://qwen.ai/blog?id=qwen3.8)