nemo_automodel.components.models.deepseek_v41.config

Hugging Face checkpoint configuration for DeepSeek-V4.1-Flash.

Module Contents

Classes

Name	Description
`DeepseekV41Config`	Nested checkpoint configuration for `DeepseekV41ForCausalLM`.
`DeepseekV41DSparkConfig`	Validated declarative configuration for the released V4.1 DSpark module.
`DeepseekV41DSparkTargetConfig`	Construction settings for the frozen, text-only V4.1 DSpark target.
`DeepseekV41TextConfig`	Declarative configuration of the CSA2, single-pass mHC and Engram backbone.
`DeepseekV41VisionConfig`	Configuration of the released 2D-RoPE vision encoder and image sizing.
`_DSparkDraftOptions`	Declarative DSpark settings supplied by the generic training recipe.

API

class nemo_automodel.components.models.deepseek_v41.config.DeepseekV41Config(
    text_config: dict[str, typing.Any] | nemo_automodel.components.models.deepseek_v41.config.DeepseekV41TextConfig | None = None,
    vision_config: dict[str, typing.Any] | nemo_automodel.components.models.deepseek_v41.config.DeepseekV41VisionConfig | None = None,
    image_token_id: int = 129264,
    dtype: str = 'bfloat16',
    pad_token_id: int | None = 2,
    bos_token_id: int = 0,
    eos_token_id: int = 1,
    tie_word_embeddings: bool = False,
    architectures: list[str] | None = None,
    kwargs: typing.Any = {}
)

Bases: PretrainedConfig

Nested checkpoint configuration for DeepseekV41ForCausalLM.

Hugging Face dictionaries are materialized at this boundary; model components receive the typed text_config and vision_config. Quantization metadata, when present in a checkpoint, is preserved by the base class rather than imposed on checkpoint-free BF16 configurations.

keys_to_ignore_at_inference

= ['past_key_values']

model_type

= 'deepseek_v41'

sub_configs

nemo_automodel.components.models.deepseek_v41.config.DeepseekV41Config.build_dspark_draft(
    options: nemo_automodel.components.models.deepseek_v41.config._DSparkDraftOptions
) -> nemo_automodel.components.models.deepseek_v41.dspark.DeepseekV41DSparkModel

Build the model-owned DSpark training adapter.

Parameters:

options

_DSparkDraftOptions

Declarative draft settings supplied by the DSpark recipe.

Returns: DeepseekV41DSparkModel

Native DeepSeek V4.1 DSpark model ready for device placement.

nemo_automodel.components.models.deepseek_v41.config.DeepseekV41Config.build_tokenizer() -> transformers.PreTrainedTokenizerFast

Load the checkpoint’s fast tokenizer for deterministic Engram hashing.

Returns: PreTrainedTokenizerFast

The original fast tokenizer from this configuration’s checkpoint

Raises:

ValueError: The configuration has no checkpoint source. Callers constructing a tiny model can supply a tokenizer directly.
TypeError: The checkpoint resolves to a slow tokenizer.

class nemo_automodel.components.models.deepseek_v41.config.DeepseekV41DSparkConfig(
    text_config: nemo_automodel.components.models.deepseek_v41.config.DeepseekV41TextConfig,
    num_draft_layers: int,
    target_layer_ids: tuple[int, ...],
    block_size: int,
    num_anchors: int,
    mask_token_id: int,
    markov_rank: int,
    markov_head_type: str,
    confidence_head_alpha: float,
    confidence_head_with_markov: bool,
    confidence_head_stop_gradient: bool = False,
    quantization_config: dict[str, typing.Any] | None = None
)

Dataclass

Validated declarative configuration for the released V4.1 DSpark module.

block_size

int

confidence_head_alpha

float

confidence_head_stop_gradient

bool = False

confidence_head_with_markov

bool

markov_head_type

str

markov_rank

int

mask_token_id

int

num_anchors

int

num_draft_layers

int

quantization_config

dict[str, Any] | None = None

target_layer_ids

tuple[int, ...]

text_config

DeepseekV41TextConfig

nemo_automodel.components.models.deepseek_v41.config.DeepseekV41DSparkConfig.build() -> nemo_automodel.components.models.deepseek_v41.dspark.DeepseekV41DSparkModel

Validate the released contract and build its training adapter.

Returns: DeepseekV41DSparkModel

Newly initialized DeepSeek V4.1 DSpark training model.

Raises:

ValueError: If recipe settings disagree with the released checkpoint.

class nemo_automodel.components.models.deepseek_v41.config.DeepseekV41DSparkTargetConfig(
    target_path: str,
    trust_remote_code: bool = False,
    target_num_hidden_layers: int | None = None,
    attn_backend: str = 'tilelang',
    dispatcher: str = 'hybridep',
    experts: str = 'torch_mm',
    enable_fsdp_optimizations: bool = True
)

Dataclass

Construction settings for the frozen, text-only V4.1 DSpark target.

The released feature contract requires the full target depth. The native MTP tensors belong to the separately trained draft.

attn_backend

str = 'tilelang'

dispatcher

str = 'hybridep'

enable_fsdp_optimizations

bool = True

experts

str = 'torch_mm'

target_num_hidden_layers

int | None = None

target_path

str

trust_remote_code

bool = False

nemo_automodel.components.models.deepseek_v41.config.DeepseekV41DSparkTargetConfig.build(
    device: torch.device,
    compute_dtype: torch.dtype,
    distributed_setup: nemo_automodel.components.distributed.config.DistributedSetup
) -> nemo_automodel.components.models.deepseek_v41.model.DeepseekV41ForCausalLM

Load the text target through the supplied EP/FSDP infrastructure.

Parameters:

device

torch.device

Resolved execution device; the sharded target requires CUDA.

compute_dtype

torch.dtype

Precision used to load and compute the frozen target.

distributed_setup

DistributedSetup

Runtime parallelism configuration composed by the recipe.

Returns: DeepseekV41ForCausalLM

The pretrained target with its vision tower disabled.

class nemo_automodel.components.models.deepseek_v41.config.DeepseekV41TextConfig(
    vocab_size: int = 129280,
    hidden_size: int = 5120,
    moe_intermediate_size: int = 2304,
    num_hidden_layers: int = 40,
    num_attention_heads: int = 64,
    num_key_value_heads: int = 1,
    head_dim: int = 512,
    qk_rope_head_dim: int = 64,
    q_lora_rank: int = 1280,
    o_lora_rank: int = 1024,
    o_groups: int = 8,
    hidden_act: str = 'silu',
    swiglu_limit: float = 10.0,
    rms_norm_eps: float = 1e-20,
    attention_bias: bool = False,
    attention_dropout: float = 0.0,
    initializer_range: float = 0.02,
    use_cache: bool = True,
    tie_word_embeddings: bool = False,
    max_position_embeddings: int = 1048576,
    rope_theta: float = 10000.0,
    rope_scaling: dict[str, typing.Any] | None = None,
    n_routed_experts: int = 384,
    n_shared_experts: int = 1,
    num_experts_per_tok: int = 6,
    scoring_func: str = 'sqrtsoftplus',
    topk_method: str = 'noaux_tc',
    norm_topk_prob: bool = True,
    routed_scaling_factor: float = 1.5,
    sliding_window: int = 128,
    compress_ratios: list[int] | None = None,
    compress_rope_theta: float = 160000.0,
    kv_source_layer_ids: list[int] | None = None,
    index_source_layer_ids: list[int] | None = None,
    index_n_heads: int = 32,
    index_head_dim: int = 128,
    index_topk: int = 512,
    candidate_source_layer_id: int = 20,
    candidate_topk_blocks: int = 2048,
    candidate_block_size: int = 8,
    hc_mult: int = 4,
    hc_sinkhorn_iters: int = 20,
    hc_eps: float = 1e-06,
    engram_layer_ids: list[int] | None = None,
    engram_num_embeddings: list[int] | None = None,
    engram_max_ngram_size: int = 4,
    engram_vocab_size: int = 16000000,
    engram_n_heads: int = 8,
    engram_head_dim: int = 256,
    engram_pad_token_id: int = 2,
    engram_compressed_vocab_size: int = 99092,
    num_nextn_predict_layers: int = 3,
    dspark_block_size: int = 5,
    dspark_noise_token_id: int = 128799,
    dspark_target_layer_ids: list[int] | None = None,
    dspark_markov_rank: int = 256,
    dspark_n_routed_experts: int = 128,
    dspark_num_experts_per_tok: int = 3,
    dtype: str = 'bfloat16',
    pad_token_id: int | None = 2,
    bos_token_id: int = 0,
    eos_token_id: int = 1,
    kwargs: typing.Any = {}
)

Bases: PretrainedConfig

Declarative configuration of the CSA2, single-pass mHC and Engram backbone.

Defaults match the released Flash text configuration. Reducing num_hidden_layers retains the complete source-layer and compression schedules: a pretrained prefix must keep its original sharing and hashing identities. Explicit smaller schedules support independent tiny models. Unknown checkpoint metadata is retained by the Hugging Face base class.

base_config_key

= 'text_config'

compress_ratios

dspark_target_layer_ids

engram_layer_ids

engram_num_embeddings

index_source_layer_ids

keys_to_ignore_at_inference

= ['past_key_values']

kv_source_layer_ids

model_type

= 'deepseek_v41_text'

rope_scaling

nemo_automodel.components.models.deepseek_v41.config.DeepseekV41TextConfig._validate_dimensions() -> None

nemo_automodel.components.models.deepseek_v41.config.DeepseekV41TextConfig._validate_layer_schedule() -> None

class nemo_automodel.components.models.deepseek_v41.config.DeepseekV41VisionConfig(
    num_hidden_layers: int = 32,
    hidden_size: int = 1024,
    num_attention_heads: int = 16,
    intermediate_size: int = 2816,
    patch_size: int = 14,
    rope_theta: float = 10000.0,
    downsample_ratio: int = 3,
    max_image_tokens: int = 1024,
    min_pixels: int = 295936,
    max_wh_ratio: float | None = None,
    kwargs: typing.Any = {}
)

Bases: PretrainedConfig

Configuration of the released 2D-RoPE vision encoder and image sizing.

base_config_key

= 'vision_config'

model_type

= 'deepseek_v41_vision'

class nemo_automodel.components.models.deepseek_v41.config._DSparkDraftOptions()

Protocol

Declarative DSpark settings supplied by the generic training recipe.

block_size

int

confidence_head_alpha

float

confidence_head_stop_gradient

bool

confidence_head_with_markov

bool

markov_head_type

str

markov_rank

int

mask_token_id

int

num_anchors

int

num_draft_layers

int

target_layer_ids

list[int]