Skip to navigation

nemo_automodel.components.speculative.dflash.dflash2_core

View as MarkdownOpen in Claude

DFlash 2 online training wrapper.

DFlash 2 (https://inco.ai/blog/dflash2/) keeps DFlash’s parallel block draft and adds a two-tap in-block convolution to the backbone plus a pairwise path selector over each position’s top-k candidates (see nemo_automodel.components.speculative.dflash.draft_qwen3_dflash2). The convolution needs no new supervision — it is part of the backbone and is trained by the ordinary DFlash objective — so this wrapper differs from DFlashTrainerModule in exactly one place: it adds a second term that trains the selector::

loss = base_loss + selector_loss_weight * selector_loss

base_loss is DFlash’s decay-weighted block CE over the full vocabulary, i.e. what makes each position’s candidate list good. selector_loss is a CE over the selector_top_k candidates of that same position, scored against the ground-truth predecessor (the token the walk would have committed had every earlier position been right) and supervised with the index of the true token inside the candidate list. Positions whose true token missed the candidate list carry no selector signal — there is nothing there to select — and are excluded from the selector term; candidate_recall reports how often that happens.

Both terms share base_loss’s DFlashDecayLoss position-weighting scheme (fixed decay for loss_type="dflash", detached D-PACE confidence for the "dpace*" variants, arXiv:2605.18810) via weighted_mean, so a position’s importance is identical in the two objectives and switching loss_type rescales both consistently. The selector term’s weight schedule is always built from the full pred_mask — the same mask base_loss uses — and only narrowed to the positions with selector signal (has_target) afterward, via weighted_mean’s value_mask. D-PACE’s weight is a sequential cumprod/cumsum across the block, so building it from a narrower mask directly (e.g. pred_mask * has_target) would corrupt neighboring positions’ weights, not just the excluded one’s. loss_type="variable_prefix" is rejected: the selector teacher-forces the predecessor from the fixed-anchor block layout, which a variable visible prefix breaks.

Both terms also share the base objective’s total_blocks fix: D-PACE’s “mean” denominator is batch_size * num_anchors (the trainer’s configured block-sampling budget), not the batch’s achieved sampled-block count, which varies with each micro-batch’s own content and therefore differs across DP ranks — see _mean_denominator.

Module Contents

Classes

NameDescription
DFlash2StepMetricsPer-step training outputs for the DFlash 2 draft.
DFlash2TrainerModuleDFlash 2 online training wrapper: DFlash block CE + candidate-selection CE.

API

class nemo_automodel.components.speculative.dflash.dflash2_core.DFlash2StepMetrics(
loss: torch.Tensor,
loss_weight: torch.Tensor,
accuracy: torch.Tensor,
valid_tokens: torch.Tensor,
correct_tokens: torch.Tensor,
accept_len: torch.Tensor,
accept_len_sum: torch.Tensor,
valid_blocks: torch.Tensor,
base_loss: torch.Tensor,
selector_loss: torch.Tensor,
selector_loss_denominator: torch.Tensor,
base_accuracy: torch.Tensor,
base_correct_tokens: torch.Tensor,
base_accept_len: torch.Tensor,
base_accept_len_sum: torch.Tensor,
candidate_recall: torch.Tensor
)
Dataclass

Per-step training outputs for the DFlash 2 draft.

loss / accuracy / valid_tokens mirror DFlashStepMetrics so the shared DFlash training loop consumes them unchanged. The primary accuracy and acceptance-length fields describe the selector path — what the draft actually emits at decode time — and the base_* fields the backbone’s own top-1 picks, so the two are directly comparable on the same denominator.

accept_len
Tensor

Scalar tensor containing selector-path mean acceptance length.

accept_len_sum
Tensor

Scalar tensor containing selector-path additive acceptance length.

accuracy
Tensor

Scalar tensor containing selector-path greedy token accuracy.

base_accept_len
Tensor

Scalar tensor containing backbone mean acceptance length.

base_accept_len_sum
Tensor

Scalar tensor containing backbone additive acceptance length.

base_accuracy
Tensor

Scalar tensor containing backbone top-1 token accuracy.

base_correct_tokens
Tensor

Scalar tensor containing backbone correct-token count.

base_loss
Tensor

Scalar tensor containing the backbone block-CE term, normalized by loss_weight.

candidate_recall
Tensor

Scalar tensor containing the fraction of supervised positions whose true token is in the backbone’s top-k candidates — the ceiling the selector can reach.

correct_tokens
Tensor

Scalar tensor containing selector-path correct-token count.

loss
Tensor

Scalar tensor containing the differentiable training loss.

loss_weight
Tensor

Scalar tensor containing the effective denominator of base_loss.

selector_loss
Tensor

Scalar tensor containing the candidate-selection CE term, normalized by selector_loss_denominator.

selector_loss_denominator
Tensor

Scalar tensor containing the effective denominator of selector_loss.

valid_blocks
Tensor

Scalar tensor containing the number of evaluated draft blocks.

valid_tokens
Tensor

Scalar tensor containing the supervised-token count.

class nemo_automodel.components.speculative.dflash.dflash2_core.DFlash2TrainerModule(
target_lm_head: torch.nn.Module,
target_embed_tokens: torch.nn.Module,
mask_token_id: int,
block_size: int = 16,
attention_backend: str = 'flex_attention',
num_anchors: int = 512,
loss_decay_gamma: float | None = None,
selector_loss_weight: float = 1.0,
sliding_window: int | None = None,
loss_type: str = 'dflash',
dpace_alpha: float = 0.5
)

Bases: DFlashTrainerModule

DFlash 2 online training wrapper: DFlash block CE + candidate-selection CE.

selector_loss_weight
= float(selector_loss_weight)
nemo_automodel.components.speculative.dflash.dflash2_core.DFlash2TrainerModule._selector_scores(
hidden: torch.Tensor,
logits: torch.Tensor,
target_ids: torch.Tensor
) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]

Score each supervised position’s top-k candidates against its true predecessor.

Teacher-forces the predecessor: position k is scored after the token at anchor + k - 1, which is what the decode-time walk would have committed if every earlier position had been accepted. Position 1’s predecessor is the anchor token itself, exactly as at decode time.

Parameters:

hidden
torch.Tensor

Tensor of shape [batch, blocks, depth, hidden]; the draft hidden states of the predicted (non-anchor) block positions.

logits
torch.Tensor

Tensor of shape [batch, blocks, depth, vocab]; the backbone logits at those positions.

target_ids
torch.Tensor

Long tensor of shape [batch, blocks, block_size]; the ground-truth token at anchor + k for block position k, so depth == block_size - 1.

Returns: torch.Tensor

Tuple (scores, candidate_ids, target_index, has_target): scores

nemo_automodel.components.speculative.dflash.dflash2_core.DFlash2TrainerModule.forward(
input_ids: torch.Tensor,
hidden_states: torch.Tensor,
loss_mask: torch.Tensor,
position_ids: torch.Tensor | None = None,
seq_lens: torch.Tensor | None = None,
doc_remaining: torch.Tensor | None = None,
label_ids: torch.Tensor | None = None

Parallel block-wise training forward with the DFlash 2 path selector.

Sequence packing (position_ids [B, S] per-document reset positions, seq_lens [B, max_docs] document lengths, doc_remaining [B, S]) is handled by the shared DFlash prologue, which keeps every block inside one document.

Parameters:

input_ids
torch.Tensor

Long tensor of shape [batch, sequence]; the context tokens.

hidden_states
torch.Tensor

Tensor of shape [batch, sequence, layers * hidden]; the captured target-model context features.

loss_mask
torch.Tensor

Tensor of shape [batch, sequence]; the supervised-token mask.

position_ids
torch.Tensor | NoneDefaults to None

Long tensor of shape [batch, sequence] with per-document reset positions under packing, or None.

seq_lens
torch.Tensor | NoneDefaults to None

Long tensor of shape [batch, max_docs] with packed document lengths, or None when unpacked.

doc_remaining
torch.Tensor | NoneDefaults to None

Long tensor of shape [batch, sequence]; real tokens left in each position’s document, or None when unpacked.

label_ids
torch.Tensor | NoneDefaults to None

Optional long tensor with the shape, dtype and device of input_ids; label_ids[:, t] is the unshifted supervision token for position t. Every entry must be a valid vocabulary id, since in-block labels are embedded as selector predecessors; exclude positions with loss_mask, not -100. input_ids remains the context/anchor stream; None supervises input_ids.

Returns: DFlash2StepMetrics

DFlash2StepMetrics for this micro-batch.