Skip to navigation
Training RecipesSpeculative Drafter TrainingTrain a DSpark Drafter for Speculative Decoding

Train a DSpark Drafter for Speculative Decoding

View as MarkdownOpen in Claude

This guide shows you how to train a DSpark speculative-decoding drafter to accelerate LLM inference with NeMo AutoModel.

What is DSpark?

DSpark is a semi-autoregressive parallel drafter. A parallel backbone proposes every position of a block in a single forward pass, a lightweight serial Markov head injects intra-block token dependency (mitigating the acceptance decay of purely parallel drafters), and a confidence head predicts per-position acceptance probability for scheduled verification. The draft shares and freezes the target’s embed_tokens and lm_head, training only the backbone, the feature projection, the Markov head, and the confidence head.

It follows the same scaffolding as the EAGLE and DFlash recipes: online target hidden-state capture, gradient accumulation, and consolidated-safetensors checkpointing.

Objective

The draft is trained with a three-term, position-decay-weighted objective:

TermMeaning
L_ce (ce_loss_alpha, default 0.1)cross-entropy against the next target token
L_tv (l1_loss_alpha, default 0.9)total-variation distance to the target distribution (a direct acceptance proxy)
L_conf (confidence_head_alpha, default 1.0)BCE training the confidence head against measured acceptance

Positions are weighted by exp(-(k-1)/loss_decay_gamma).

Data

Use a chat dataset of OpenAI-format messages rows. As in the DSpark paper, use Open-PerfectBlend prompts with the responses regenerated by the target model (training is teacher-forced; regenerate before training to avoid a training and inference distribution mismatch). Point recipe_args.train_data_path at the regenerated JSONL or a Hugging Face dataset id.

Run It

Example configs live under examples/speculative/dspark/ (qwen3_0.6b_dspark.yaml, gemma4_12b_dspark.yaml, and deepseek_v41_flash_dspark.yaml). Multi-GPU defaults to FSDP2 (distributed.strategy: fsdp2); set it to ddp for simple data parallelism.

torchrun --standalone --nproc_per_node=2 \
-m nemo_automodel.recipes.speculative_decoding.train_dspark \
-c examples/speculative/dspark/qwen3_0.6b_dspark.yaml

Per-step metrics (loss, ce_loss, l1_loss, confidence_loss, lr, mem) are reduced across data-parallel ranks and written to <output_dir>/dspark_train_metrics.jsonl.

Key Config Fields

FieldMeaning
target_model_name_or_pathfrozen target (e.g. Qwen/Qwen3-4B)
draft_num_hidden_layersdraft backbone depth (paper: 5)
block_sizetokens drafted per block (paper: 7)
num_anchorsblocks sampled per sequence per step
target_layer_idstarget feature layers fed to the draft (defaults to an even spread)
mask_token_idreserved token id filling non-anchor block positions
markov_rank / markov_head_typeserial head size and variant (vanilla / gated / rnn)
confidence_head_alpha / confidence_head_with_markovconfidence-head weight and conditioning
confidence_head_stop_gradientDeepSeek V4.1 only: train the confidence head on detached inputs so its loss cannot back-propagate into the draft backbone (default false)
packed_sequence_sizepacks documents into rows of this many tokens when greater than 0 (Qwen3, DeepSeek V4, and DeepSeek V4.1 targets; online text-only data with cp_size: 1 and target_model_backend: colocated)

DeepSeek V4.1 Flash

DeepSeek V4.1 uses the native DSpark architecture stored under the checkpoint’s mtp.* namespace rather than the configurable dense draft used for DeepSeek V4. The implementation reuses NeMo AutoModel’s V4.1 mHC, MLA, RoPE, Mixture-of-Experts (MoE) routing, normalization, and distributed-training components. The target remains frozen; only the three draft stages, target-feature projection, Markov head, and confidence head receive DSpark gradients.

NeMo AutoModel feeds the final RMSNorm output into the V4.1 confidence head to reduce its sensitivity to residual magnitude. The example keeps confidence_head_alpha: 1.0 and confidence_head_stop_gradient: false so the confidence loss still trains the draft backbone. This differs from the released confidence head, which reads the raw residual. Serving checkpoints trained with this recipe requires the same normalized confidence input; loading released confidence weights does not preserve their original predictions. Full-scale training stability with this normalization still requires validation.

The recipe starts those trainable DSpark weights from a fresh initialization and copies only the frozen embedding and LM head from the target. Use checkpoint.restore_from to resume a DSpark checkpoint produced by NeMo AutoModel. The V4.1 technical report does not name the dataset used for the dedicated DSpark stage, so the example data path is illustrative rather than a claim about the released model’s training data.

The example tokenizes messages-format data with deepseek_v41_flash_training_chat_template.jinja, a copy of the checkpoint’s released chat_template.jinja that renders the same prompt text and adds {% generation %} spans, so only assistant output is supervised. The copy defaults to chat (non-thinking) mode instead of the released thinking default, and it rejects assistant turns that carry reasoning_content, which chat mode would drop. For such data, set chat_template_kwargs: {thinking_mode: thinking} in recipe_args so training matches thinking-mode serving; the single-process precompute_dspark takes the same mapping as --chat-template-kwargs '{"thinking_mode": "thinking"}'. The cache manifest records the template and these arguments, and an offline cache keeps the framing it was built with. Training from a cache rejects chat_template_kwargs that differ from the recorded ones (unset counts as none), and logs a warning when chat_template is set to a different template; rebuild the cache to change either.

The following settings are fixed by the released checkpoint and the recipe rejects mismatches:

FieldDeepSeek V4.1 value
draft_num_hidden_layers3
block_size5
target_layer_ids[37, 38, 39]
mask_token_id128799
markov_rank256
draft attentionbidirectional five-position block plus a 128-token target window
draft MoE128 routed experts, top-3

Use the production-scale deepseek_v41_flash_dspark.yaml as the starting point. It uses FSDP2 for the trainable draft and the model’s distributed EP and FSDP path for the frozen target. Pipeline parallelism, context parallelism, image inputs, and the serving-time dynamic verification scheduler are not part of this training path.

Sequence packing (packed_sequence_size) works for both DeepSeek V4 and V4.1. The frozen target starts every packed document on a compression-group boundary, the least common multiple of its nonzero compress_ratios (128 for V4-Flash, 2 for V4.1-Flash), so no compressed group or indexer score mixes documents and each document’s features match a forward on that document alone. The draft keeps every block inside its anchor’s document. Alignment padding lengthens the target forward by less than one alignment unit per document, plus one for the row; with the HybridEP dispatcher every rank also pads to the longest aligned row of all ranks.

MiniMax M3.1 Flash

minimax_m3_1_flash_dspark.yaml trains a draft in the layout of the checkpoint’s released dspark/ folder. When the target ships dspark/config.json, that file fixes the draft and the recipe rejects mismatches:

FieldMiniMax M3.1 Flash value
draft_num_hidden_layers5
block_size7
target_layer_ids[3, 17, 31, 45, 59]
mask_token_id200058
markov_rank / markov_head_type256 / vanilla
confidence_head_alpha / confidence_head_with_markovgreater than 0 / true
freeze_embeddingstrue
draft FFNdense, hidden width 12288
draft attentionsliding window of 4096 tokens

The consolidated checkpoint uses the released key names under language_model.model.dspark. and a copy of the released config.json without quantization_config. It stores unquantized weights and no shared embedding or LM head, so the draft relies on the target’s. MiniMax M3 has no dspark/ folder, so its drafts keep native keys and take the FFN width from intermediate_size. A local M3.1 snapshot needs its dspark/ folder too; without it the recipe logs a warning and builds that same draft.

Supported targets include Qwen3 (dense and MoE), Gemma4, DeepSeek V4 and V4.1, GLM-5.2, Kimi K3, MiniMax M3, and MiniMax M3.1 Flash.