Train a DSpark Drafter for Speculative Decoding
This guide shows you how to train a DSpark speculative-decoding drafter to accelerate LLM inference with NeMo AutoModel.
What is DSpark?
DSpark is a semi-autoregressive parallel drafter. A parallel backbone proposes
every position of a block in a single forward pass, a lightweight serial
Markov head injects intra-block token dependency (mitigating the acceptance
decay of purely parallel drafters), and a confidence head predicts
per-position acceptance probability for scheduled verification. The draft shares
and freezes the target’s embed_tokens and lm_head, training only the
backbone, the feature projection, the Markov head, and the confidence head.
It follows the same scaffolding as the EAGLE and DFlash recipes: online target hidden-state capture, gradient accumulation, and consolidated-safetensors checkpointing.
Objective
The draft is trained with a three-term, position-decay-weighted objective:
Positions are weighted by exp(-(k-1)/loss_decay_gamma).
Data
Use a chat dataset of OpenAI-format messages rows. As in the DSpark paper, use
Open-PerfectBlend prompts with the responses regenerated by the target model
(training is teacher-forced; regenerate before training to avoid a
training and inference distribution mismatch). Point recipe_args.train_data_path at
the regenerated JSONL or a Hugging Face dataset id.
Run It
Example configs live under examples/speculative/dspark/
(qwen3_0.6b_dspark.yaml, gemma4_12b_dspark.yaml, and
deepseek_v41_flash_dspark.yaml). Multi-GPU defaults to
FSDP2 (distributed.strategy: fsdp2); set it to ddp for simple data
parallelism.
Per-step metrics (loss, ce_loss, l1_loss, confidence_loss, lr, mem)
are reduced across data-parallel ranks and written to
<output_dir>/dspark_train_metrics.jsonl.
Key Config Fields
DeepSeek V4.1 Flash
DeepSeek V4.1 uses the native DSpark architecture stored under the checkpoint’s
mtp.* namespace rather than the configurable dense draft used for DeepSeek
V4. The implementation reuses NeMo AutoModel’s V4.1 mHC, MLA, RoPE,
Mixture-of-Experts (MoE) routing, normalization, and distributed-training
components. The target remains frozen;
only the three draft stages, target-feature projection, Markov head, and
confidence head receive DSpark gradients.
NeMo AutoModel feeds the final RMSNorm output into the V4.1 confidence head to
reduce its sensitivity to residual magnitude. The example keeps
confidence_head_alpha: 1.0 and confidence_head_stop_gradient: false so the
confidence loss still trains the draft backbone. This differs from the released
confidence head, which reads the raw residual. Serving checkpoints trained with
this recipe requires the same normalized confidence input; loading released
confidence weights does not preserve their original predictions. Full-scale
training stability with this normalization still requires validation.
The recipe starts those trainable DSpark weights from a fresh initialization
and copies only the frozen embedding and LM head from the target. Use
checkpoint.restore_from to resume a DSpark checkpoint produced by NeMo AutoModel.
The V4.1 technical report does not name the dataset used for the dedicated
DSpark stage, so the example data path is illustrative rather than a claim
about the released model’s training data.
The example tokenizes messages-format data with
deepseek_v41_flash_training_chat_template.jinja,
a copy of the checkpoint’s released chat_template.jinja that renders the same
prompt text and adds {% generation %} spans, so only assistant output is
supervised. The copy defaults to chat (non-thinking) mode instead of the
released thinking default, and it rejects assistant turns that carry
reasoning_content, which chat mode would drop. For such data, set
chat_template_kwargs: {thinking_mode: thinking} in recipe_args so training
matches thinking-mode serving; the single-process precompute_dspark takes
the same mapping as --chat-template-kwargs '{"thinking_mode": "thinking"}'.
The cache manifest records the template and these arguments, and an offline
cache keeps the framing it was built with. Training from a cache rejects
chat_template_kwargs that differ from the recorded ones (unset counts as
none), and logs a warning when chat_template is set to a different template;
rebuild the cache to change either.
The following settings are fixed by the released checkpoint and the recipe rejects mismatches:
Use the production-scale
deepseek_v41_flash_dspark.yaml
as the starting point. It uses FSDP2 for the trainable draft and the model’s
distributed EP and FSDP path for the frozen target. Pipeline parallelism, context
parallelism, image inputs, and the serving-time dynamic verification scheduler
are not part of this training path.
Sequence packing (packed_sequence_size) works for both DeepSeek V4 and V4.1.
The frozen target starts every packed document on a compression-group boundary,
the least common multiple of its nonzero compress_ratios (128 for V4-Flash, 2
for V4.1-Flash), so no compressed group or indexer score mixes documents and each
document’s features match a forward on that document alone. The draft keeps every
block inside its anchor’s document. Alignment padding lengthens the target forward
by less than one alignment unit per document, plus one for the row; with the
HybridEP dispatcher every rank also pads to the longest aligned row of all ranks.
MiniMax M3.1 Flash
minimax_m3_1_flash_dspark.yaml
trains a draft in the layout of the checkpoint’s released dspark/ folder. When
the target ships dspark/config.json, that file fixes the draft and the recipe
rejects mismatches:
The consolidated checkpoint uses the released key names under
language_model.model.dspark. and a copy of the released config.json without
quantization_config. It stores unquantized weights and no shared embedding or
LM head, so the draft relies on the target’s. MiniMax M3 has no dspark/ folder,
so its drafts keep native keys and take the FFN width from intermediate_size. A
local M3.1 snapshot needs its dspark/ folder too; without it the recipe logs a
warning and builds that same draft.
Supported targets include Qwen3 (dense and MoE), Gemma4, DeepSeek V4 and V4.1, GLM-5.2, Kimi K3, MiniMax M3, and MiniMax M3.1 Flash.