Qwen3.8-Flash-Next
Qwen3.8-Flash-Next is a hybrid-attention Mixture-of-Experts (MoE) model with four
persistent HyperConnection streams and an Engram N-gram Embedding table (named
ple in the checkpoint configuration). NeMo AutoModel supports the language-only
Qwen3_8_FlashNextForConditionalGeneration path for supervised fine-tuning
(SFT). All backbone parameters are trainable except the QSA indexers, whose
reference path exposes discrete top-k IDs but no auxiliary loss or
straight-through gradient.
Model Reference
Model Architecture
Architecture
- The decoder repeats three GatedDeltaNet layers followed by one QSA sparse-attention layer. QSA compresses index keys in groups of four, selects up to 512 causal groups (2,048 tokens), appends the incomplete causal tail, and evaluates main GQA only on those selected token IDs. Every layer has separate learned HyperConnection reads and writes for attention and MoE.
- The first decoder layer uses the normal Qwen3.8-Flash-Next GatedDeltaNet and MoE skeleton without Engram. The second decoder layer first adds Engram values derived from eight raw-token bigram hashes and eight raw-token trigram hashes.
- The Engram table has 320,001,536 padded rows of width 160. Its 51.2 billion parameters are stored as 128 physical checkpoint shards.
- The final learned HyperConnection read collapses four 2,560-wide streams to one 2,560-wide state before the untied language-model head.
Validated Configuration
The 100-step run reduced training loss from 3.0734 at step 0 to 1.8195 at step 99 and finished with validation loss 2.0610. The run completed without NaNs or collective errors and published model, optimizer, dataloader, scheduler, and RNG state in a distributed checkpoint.
The Engram table is a globally shaped DTensor with placement Shard(0).
Requesting ranks group hashed row IDs by owner; an equal-capacity All-to-All
sends IDs to owners, and a second autograd-aware All-to-All returns values.
Each rank looks up only its contiguous local DTensor shard, and the backward
pass reverses the value route so only the owner stores and updates each row’s
gradient. The
validated cluster runs used NCCL’s internal InfiniBand (IB) transport
(NCCL_NET_PLUGIN=none and NCCL_NET=IB).
Numerical Validation
The released checkpoint was compared with its SGLang reference on two fixed raw-token sequences, including an EOS-reset case. Token embeddings, N-gram hash IDs, and table values matched exactly. The first four decoder-layer states had cosine similarity from 0.9998 to 1.0000, and the corresponding logit-lens top-1 token matched at every layer. The Engram delta had relative L2 error below 0.0063 in both cases.
The QSA routing path was also compared with SGLang on a fixed 4,096-token input. Rows through position 2,050 matched the reference routing exactly. Replaying the exact SGLang layer input through NeMo AutoModel produced the same selected-token sets at the first sparse row (2,051) and the last row (4,095). The relative L2 error for the projected sparse-attention output was 0.0041. Positional ordering after top-k is not compared because the reference CUDA kernel does not define a stable order. A full-model EP8 run executing sparse QSA in all 12 QSA layers produced final logits with cosine similarity 0.99952, relative L2 error 0.03175, and the same top-1 and top-5 tokens as the SGLang reference. The eight EP ranks were bitwise identical for every captured layer state, routing probe, final hidden state, and final logits.
Two-rank distributed tests compare model-owned CP with CP1 for exact QSA routing IDs, QSA and Engram outputs, per-token input gradients, globally summed parameter gradients, Engram hashes across an EOS/shard boundary, and the Engram convolution’s nine-token cross-rank halo. A full 128-GPU, EP128 comparison then trained CP1 and CP8 for the same 100 optimizer steps over the same padded 4,096-token samples. Both runs completed with finite losses and wrote model, optimizer, dataloader, scheduler, and RNG checkpoint payloads. Across all 100 steps, loss-curve cosine similarity was 0.9999992, mean absolute loss difference was 0.00054, and the maximum absolute difference was 0.00267. Final losses were 0.60801 for CP1 and 0.60800 for CP8.
Packed (THD) context parallelism was validated the same way at world size 64 with EP64: CP1 (DP64/GA2) and CP8 (DP8/GA16) trained 100 optimizer steps over identical THD packs (packing is topology independent for the model-owned CP path, so both runs consume byte-identical pack sequences). Loss-curve cosine similarity was 0.999999925, mean absolute difference was 0.00024, and the maximum absolute difference was 0.00069; final losses were 0.56053 for CP1 and 0.56011 for CP8. A world-64 single-forward probe also matched CP8 against CP1 with exact per-row routing IDs, and the Engram, GatedDeltaNet, and QSA activation gates passed.
Example Recipe
The recipe loads Qwen/Qwen3.8-Flash-Next directly from the Hugging Face Hub. Allocate 8 nodes with 8 GPUs per node and launch the recipe through your cluster launcher. See the Slurm Launcher Guide for multi-node setup.
Current Scope
- QSA uses FlexAttention sparse GQA for CUDA BF16 training when
model.backend.attn=flex. Route IDs are scattered into a boolean membership table of shape[batch, queries, key/value tokens]and executed through aBlockMask, avoiding materialized dense per-head attention scores; the same code path serves dense, packed, and context-parallel layouts. CPU execution retains a bounded-workspace PyTorch numerical oracle; tests can also call that oracle directly. CUDA execution never silently falls back. The discrete QSA indexers are frozen because the supplied inference reference and checkpoint define neither an indexer auxiliary loss nor a straight-through estimator; main attention Q/K/V and the rest of the backbone remain trainable. model.backend.attn=cudnnselects experimental cuDNN Frontend indexed QSA forward and backward on Blackwell SM100/SM103. Install the optional runtime withuv pip install "nvidia-cudnn-frontend[cutedsl]>=1.32.1". It requires BF16, head dimension 256, 24Q/2KV (or 12Q/1KV),indexer_compress_ratio=4,indexer_budget<=2048, andCP=1. The existing frozen indexer, route IDs, softmax scale, and projection weights are retained. Packed document offsets, right-tail padding, and empty route rows are supported. The adapter pads route width to a multiple of four and packs batch-local IDs; it does not build a quadratic token-membership bitmap. The 24Q/2KV geometry runs two independent 12Q/1KV calls and includes their packing and copying costs. First-order autograd saves cuDNN’s original FP32 output and natural-log LSE; backward uses atomic accumulation and is not bitwise deterministic. This adapter explicitly uses the general backward path; the specializedcausal_topkbackward is not enabled. Numerical tests compare output and gradients against independent FP64 attention using the existing BF16 relative L2 bound of 0.004. Small-shape GB200 tests pass; target-length FP64 validation remains untested. Differences fromflexare diagnostic and are not an additional numerical acceptance criterion. Deterministic training requests andCP>1are rejected explicitly. Useflexfor context parallelism. Kernel support does not establish model FLOPs utilization (MFU) for the full model or training-convergence parity; validate those for the intended recipe before changing its backend.- Context parallelism uses contiguous rank-ordered sequence shards. QSA keeps queries local and gathers compressed index keys plus differentiable main K/V; GatedDeltaNet uses FLA’s contiguous CP recurrence; Engram hashes replicated global raw IDs, looks up only local table rows, and exchanges an exact autograd-aware nine-token convolution halo.
- Activation checkpointing wraps 47 of the 48 decoder blocks. The block that owns Engram remains eager because replaying its mutable owner-sharded All-to-All collectives during recomputation is not supported.
- Native distributed checkpoints store the Engram parameter and optimizer tensors under their global DTensor fully qualified names (FQNs), so PyTorch Distributed Checkpoint (DCP) can reshard them when the owner world size changes. Checkpoints from the earlier rank-local owner format are not supported; resume from a native DTensor checkpoint or restart the optimizer.
- CP is limited to language-only inputs with
TP=PP=1. Non-packed right-tail-padded batches and packed (THD) batches withcu_seqlens(or loaderseq_lens) are both supported; the same FlexAttention path serves packed CUDA training, and packing is performed topology-independently. Left padding, interior padding, TP, PP, MTP, and vision and video training remain unsupported.