> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/automodel/_mcp/server. # Qwen3.8-Flash-Next > Fine-tune the language-only Qwen3.8-Flash-Next Engram MoE checkpoint with FSDP2, expert parallelism, and context parallelism. Qwen3.8-Flash-Next is a hybrid-attention Mixture-of-Experts (MoE) model with four persistent HyperConnection streams and an Engram N-gram Embedding table (named `ple` in the checkpoint configuration). NeMo AutoModel supports the language-only `Qwen3_8_FlashNextForConditionalGeneration` path for supervised fine-tuning (SFT). All backbone parameters are trainable except the QSA indexers, whose reference path exposes discrete top-k IDs but no auxiliary loss or straight-through gradient. | | | | ------------------- | ------------------------------------------------------------------------------------------------------------------ | | **Task** | Text generation (MoE, hybrid attention, Engram N-gram Embedding) | | **Architecture** | `Qwen3_8_FlashNextForConditionalGeneration` | | **Parameters** | 125B backbone with 6B activated per token, plus the 51.2B Engram table; the checkpoint's 4B MTP head is not loaded | | **Decoder** | 48 layers: 36 GatedDeltaNet and 12 QSA sparse-attention layers | | **Experts** | 512 routed experts, top-10 routing, plus one shared expert per layer | | **Engram** | One 51.2B-parameter N-gram Embedding table in decoder layer 2 | | **Sequence length** | 262,144 native; validated here at 4,096 tokens | ## Architecture * The decoder repeats three GatedDeltaNet layers followed by one QSA sparse-attention layer. QSA compresses index keys in groups of four, selects up to 512 causal groups (2,048 tokens), appends the incomplete causal tail, and evaluates main GQA only on those selected token IDs. Every layer has separate learned HyperConnection reads and writes for attention and MoE. * The first decoder layer uses the normal Qwen3.8-Flash-Next GatedDeltaNet and MoE skeleton without Engram. The second decoder layer first adds Engram values derived from eight raw-token bigram hashes and eight raw-token trigram hashes. * The Engram table has 320,001,536 padded rows of width 160. Its 51.2 billion parameters are stored as 128 physical checkpoint shards. * The final learned HyperConnection read collapses four 2,560-wide streams to one 2,560-wide state before the untied language-model head. ## Validated Configuration | Area | Validated Configuration | | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | | FSDP2 | 47 decoder blocks use whole-layer activation checkpointing; the Engram owner-collective block remains eager; QSA indexers are frozen | | Expert parallelism | EP64 baseline; EP128 for the CP1/CP8 comparison | | Engram ownership | Global `Shard(0)` DTensor over the world group; 64 or 128 contiguous local row shards | | Context parallelism | CP1 and CP8 at world size 128; CP was disabled in the EP64 baseline run below | | Tensor / pipeline parallelism | Disabled | | Workload | Non-packed right-tail-padded HellaSwag; packed (THD) Tulu-3 full-token SFT | | Scale | 8 nodes / 64 H100 baseline; 16 nodes / 128 H100 CP comparison | | Duration | 100 optimizer steps | The 100-step run reduced training loss from 3.0734 at step 0 to 1.8195 at step 99 and finished with validation loss 2.0610. The run completed without NaNs or collective errors and published model, optimizer, dataloader, scheduler, and RNG state in a distributed checkpoint. The Engram table is a globally shaped DTensor with placement `Shard(0)`. Requesting ranks group hashed row IDs by owner; an equal-capacity All-to-All sends IDs to owners, and a second autograd-aware All-to-All returns values. Each rank looks up only its contiguous local DTensor shard, and the backward pass reverses the value route so only the owner stores and updates each row's gradient. The validated cluster runs used NCCL's internal IB transport (`NCCL_NET_PLUGIN=none` and `NCCL_NET=IB`). ## Numerical Validation The released checkpoint was compared with its SGLang reference on two fixed raw-token sequences, including an EOS-reset case. Token embeddings, N-gram hash IDs, and table values matched exactly. The first four decoder-layer states had cosine similarity from 0.9998 to 1.0000, and the corresponding logit-lens top-1 token matched at every layer. The Engram delta had relative L2 error below 0.0063 in both cases. The QSA routing path was also compared with SGLang on a fixed 4,096-token input. Rows through position 2,050 matched the reference routing exactly. Replaying the exact SGLang layer input through NeMo AutoModel produced the same selected-token sets at the first sparse row (2,051) and the last row (4,095), and relative L2 error 0.0041 for the projected sparse-attention output. Positional ordering after top-k is not compared because the reference CUDA kernel does not define a stable order. A full-model EP8 run executing sparse QSA in all 12 QSA layers produced final logits with cosine similarity 0.99952, relative L2 error 0.03175, and the same top-1 and top-5 tokens as the SGLang reference. The eight EP ranks were bitwise identical for every captured layer state, routing probe, final hidden state, and final logits. Two-rank distributed tests compare model-owned CP with CP1 for exact QSA routing IDs, QSA and Engram outputs, per-token input gradients, globally summed parameter gradients, Engram hashes across an EOS/shard boundary, and the Engram convolution's nine-token cross-rank halo. A full 128-GPU, EP128 comparison then trained CP1 and CP8 for the same 100 optimizer steps over the same padded 4,096-token samples. Both runs completed with finite losses and wrote model, optimizer, dataloader, scheduler, and RNG checkpoint payloads. Across all 100 steps, loss-curve cosine similarity was 0.9999992, mean absolute loss difference was 0.00054, and the maximum absolute difference was 0.00267. Final losses were 0.60801 for CP1 and 0.60800 for CP8. Packed (THD) context parallelism was validated the same way at world size 64 with EP64: CP1 (DP64/GA2) and CP8 (DP8/GA16) trained 100 optimizer steps over identical THD packs (packing is topology-independent for the model-owned CP path, so both runs consume byte-identical pack sequences). Loss-curve cosine similarity was 0.999999925, mean absolute difference 0.00024, and maximum absolute difference 0.00069; final losses were 0.56053 for CP1 and 0.56011 for CP8. A world-64 single-forward probe also matched CP8 against CP1 with exact per-row routing IDs and passing Engram/GDN/QSA activation gates. ## Example Recipe | Recipe | Description | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------- | | [qwen3\_8\_flash\_next\_180b\_hellaswag\_ep64.yaml](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/llm_finetune/qwen/qwen3_8_flash_next_180b_hellaswag_ep64.yaml) | 100-step SFT on HellaSwag with FSDP2, EP64, and world-sized Engram ownership | The recipe loads [Qwen/Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) directly from the Hugging Face Hub. Allocate 8 nodes with 8 GPUs per node and launch the recipe through your cluster launcher. See the [Slurm Launcher Guide](/job-launchers/slurm-cluster) for multi-node setup. ## Current Scope * QSA uses FlexAttention sparse GQA for CUDA BF16 training when `model.backend.attn=flex`. Route IDs are scattered into a boolean membership table of shape `[batch, queries, key/value tokens]` and executed through a `BlockMask`, avoiding materialized dense per-head attention scores; the same code path serves dense, packed, and context-parallel layouts. CPU execution retains a bounded-workspace PyTorch numerical oracle; tests can also call that oracle directly. CUDA execution requires the `flex` backend and never silently falls back. The discrete QSA indexers are frozen because the supplied inference reference and checkpoint define neither an indexer auxiliary loss nor a straight-through estimator; main attention Q/K/V and the rest of the backbone remain trainable. * Context parallelism uses contiguous rank-ordered sequence shards. QSA keeps queries local and gathers compressed index keys plus differentiable main K/V; GatedDeltaNet uses FLA's contiguous CP recurrence; Engram hashes replicated global raw IDs, looks up only local table rows, and exchanges an exact autograd-aware nine-token convolution halo. * Activation checkpointing wraps 47 of the 48 decoder blocks. The block that owns Engram remains eager because replaying its mutable owner-sharded All-to-All collectives during recomputation is not supported. * Native distributed checkpoints store the Engram parameter and optimizer tensors under their global DTensor FQNs, so DCP can reshard them when the owner world size changes. Checkpoints from the earlier rank-local owner format are not supported; resume from a native DTensor checkpoint or restart the optimizer. * CP is limited to language-only inputs with TP=PP=1. Non-packed right-tail-padded batches and packed (THD) batches with `cu_seqlens` (or loader `seq_lens`) are both supported; the same FlexAttention path serves packed CUDA training, and packing is performed topology-independently. Left padding, interior padding, TP, PP, MTP, and vision and video training remain unsupported. ## References * [DeepSeek Engram repository](https://github.com/deepseek-ai/Engram) * [Engram paper](https://arxiv.org/abs/2601.07372) * [Megatron-LM-FL trainable Engram implementation](https://github.com/flagos-ai/Megatron-LM-FL/blob/main/megatron/core/transformer/engram.py)