nemo_automodel.components.models.deepseek_v41.model
nemo_automodel.components.models.deepseek_v41.model
DeepSeek V4.1 text and image backbone for AutoModel training.
Forward contract (reference Transformer.forward of inference/model.py):
- Embed tokens and expand the hidden state into
hc_multresidual streams. - Run the 40 blocks with single-pass mHC: each block receives the input mix produced by the previous block’s FFN site (a one-hot mix reads stream 0 at the start). Engram modules write into the streams before their layer.
- Collapse the streams with the last FFN-site mix, apply the final RMSNorm and
the fp32
lm_head.
Cross-layer CSA2 state (shared compressed KV, index keys, Top-K indices and the
hierarchical candidate pool) lives in per-layer snapshots of
DeepseekV41AttentionState.
Snapshots share tensors and preserve the state needed for activation recomputation.
The optional vision tower inserts projected image patches and learned image
delimiters into the text sequence. Text and image batches use full sequences
with two-dimensional token layouts. DSpark draft layers (mtp.*) are built
separately by nemo_automodel.components.models.deepseek_v41.dspark so
their objective cannot backpropagate into this backbone. Inference-time KV
caching and SWA bounded replay remain out of scope.
Module Contents
Classes
Data
API
Bases: Module
CSA2 and MoE sublayers with the single-pass mHC coefficient handoff.
Expose the shared parallelizer’s MoE interface without duplicate registration.
Expose the attention module to the shared CP parallelizer.
Execute one block while retaining differentiable shared KV ownership.
Parameters:
Tensor of shape [batch, sequence, streams, hidden].
FP32 tensor of shape [batch, sequence, streams].
Shared CSA2 tensors with layouts documented by DeepseekV41AttentionState; no tensor is mutated.
Integer tensor of shape [batch, sequence].
Optional binary tensor of shape [batch, sequence].
Optional boolean tensor of shape [batch, sequence].
Optional logical memory rows [batch, local_sequence, hash_heads].
CP group; all token axes are local and CSA2 keys are global.
Optional document IDs [batch, local_sequence], zero for padding.
Returns: torch.Tensor
Updated streams [batch, sequence, streams, hidden], the next pre-mix
Extend this block’s activation-checkpoint contexts with CSA2 selection replay.
The MoE parallelizer calls this model-owned hook when it wraps the block in a checkpoint
wrapper. With reuse_sparse_selection_on_recompute the attention layer records the frozen
indexer’s selection and the sparse-index plan during the checkpoint forward and takes them back
during the recompute instead of recomputing them; with the knob off the incoming contexts are
returned unchanged.
Parameters:
The parallelizer’s torch.utils.checkpoint context factory, or None.
Returns: Callable[[], tuple[AbstractContextManager, AbstractContextManager]] | None
The (possibly wrapped) context factory.
Bases: DeepseekV41PipelineMixin, HFCheckpointingMixin, PreTrainedModel, MoEFSDPSyncMixin
DeepSeek V4.1 causal LM with optional vision and an fp32 lm_head.
engram_process_group explicitly selects contiguous row owners for the
Engram tables. Distributed models default to WORLD, including a one-rank
WORLD. Without distributed initialization, tables remain local. FSDP’s
shard mesh must match the owner group exactly.
Register owner table DTensors before FSDP records ignored parameters.
Parameters:
One-dimensional shard mesh whose ranks and ordering must match the Engram owner group.
Returns: set[nn.Parameter]
Exact registered parameter identities to exclude from FSDP. Each
Compute language outputs, or the documented PP stage tensor contract.
Parameters:
Integer IDs [batch, local_sequence], or incoming hidden streams [batch, local_sequence, streams, hidden] on the non-first PP stage.
Only on a non-first stage: FP32 pre_mix [batch, local_sequence, streams] and int64 metadata [batch, local_sequence, 5], followed by live CSA2 state tensors, as documented by get_pipeline_stage_metas and _pack_pipeline_state.
Optional binary right-padding tensor [batch, sequence].
Optional integer tensor of shape [batch, sequence].
Optional targets of shape [batch, sequence], with -100 ignored.
Optional image patches [all_patches, 3, patch_size, patch_size].
Optional patch-grid sizes [images, 2].
Optional image/text markers [batch, sequence].
Number of final positions, or integer position indices [kept].
Attach the final normalized hidden states
[batch, sequence, hidden], in the backbone’s activation dtype, as
every other model does; the recipes’ linear losses project them
with lm_head.weight.
Attach the per-layer residual streams instead, a
tuple of [batch, sequence, streams, hidden] captured before each block,
for parity checks and debugging. Exclusive with output_hidden_states.
Optional real document lengths [batch, documents] for packed text.
Optional physical document spans [batch, documents].
Optional “thd” marker from packed_sequence_thd_collater.
Prepared document IDs [batch, local_sequence], zero for padding. The model-owned CP sharder supplies these after aligning and sharding a pack.
CP group; input/output token axes contain only the local contiguous shard. The recipe computes loss from globally shifted labels.
Returns: CausalLMOutputWithPast | tuple[torch.Tensor, ...] | torch.Tensor
CausalLMOutputWithPast containing logits [batch, kept_sequence, vocab],
Construct using the NeMo registry’s configuration entry point.
Return modules whose inputs are the released DSpark target features.
The reference implementation captures the residual streams immediately before attention in each selected layer, after that layer’s optional Engram update. The attention hyper-connection is the first module to consume those streams, so its forward input is the exact capture point.
Parameters:
Strictly increasing decoder-layer indices in
[0, num_hidden_layers).
Returns: nn.Module
Modules ordered like layer_ids. Each receives a tensor of shape
Raises:
ValueError: If the indices are duplicated, unsorted, or outside the active decoder depth.
Return the untied token embedding module.
Return the independent vocabulary projection.
Initialize every trainable backbone weight after meta materialization.
Select contiguous text sharding with complete compression groups.
Parameters:
Full text tensors of shape [batch, global_sequence]. The hook leaves tensors intact; sharding happens through the returned strategy.
Framework chunk count; this model uses one chunk.
Returns: dict[str, Any]
Model-owned CP sharder. Its local token tensors have shape
Bases: Module
DeepSeek V4.1 decoder stack: embeddings, hyper-connected blocks, final norm.
Fuse globally ordered images into the local contiguous language shard.
Parameters:
Integer local token IDs [batch, local_sequence].
Global patches [patches, 3, patch_size, patch_size], replicated within CP; None for a text-only batch.
Global image grids [images, 2], in batch-row order.
Local markers [batch, local_sequence]; -1 for text, 0/1/2/3 for image start/content/newline/end.
CP-only group, or None. Complete spans are validated after gathering marker metadata, so a shard may start inside an image.
Local document IDs [batch, local_sequence], zero for padding. An image must lie wholly inside one document.
Returns: torch.Tensor
Local fused embeddings [batch, local_sequence, hidden]. Vision
Compute all positions without inference-only prefill shortcuts.
Parameters:
Integer tensor of shape [batch, sequence], also used for Engram.
Optional integer tensor of shape [batch, sequence].
Optional binary right-padding tensor [batch, sequence].
Optional boolean tensor [batch, sequence] with True at padding, as the standard
collator emits it; used when attention_mask is absent. Non-first PP stages take the
validity of each token from the stage metadata instead.
Optional mask of the supervised positions, as the standard collator emits it. The recipe loss consumes it; the model does not read it, and accepts it so that the standard batch reaches the pipeline stages.
Optional boolean image-span tensor [batch, sequence].
Optional image patches [all_patches, 3, patch_size, patch_size].
Optional patch grids [images, 2].
Optional image/text markers [batch, sequence].
Optional projected multimodal embeddings [batch, sequence, hidden].
Whether to retain the residual streams before each block (parity and debugging; not the recipe interface).
Optional document IDs [batch, local_sequence], zero for padding.
CP group; token axes above are local sequence shards. Engram hashes use global token history, with only local rows looked up.
Incoming streams [batch, local_sequence, streams, hidden], required only on a stage without token embeddings.
Incoming FP32 coefficients [batch, local_sequence, streams], required with pipeline_hidden_states; gradients must not be detached.
Incoming immutable CSA2 state, with layouts in DeepseekV41AttentionState, belonging to this microbatch only.
Returns: DeepseekV41BackboneOutput
Stage values with layouts documented by DeepseekV41BackboneOutput.