nemo_automodel.components.models.deepseek_v41.config

Hugging Face checkpoint configuration for DeepSeek-V4.1-Flash.

Module Contents

Classes

Name	Description
`DeepseekV41Config`	Nested checkpoint configuration for `DeepseekV41ForCausalLM`.
`DeepseekV41TextConfig`	Declarative configuration of the CSA2, single-pass mHC and Engram backbone.
`DeepseekV41VisionConfig`	Configuration of the released 2D-RoPE vision encoder and image sizing.

API

class nemo_automodel.components.models.deepseek_v41.config.DeepseekV41Config(
    text_config: dict[str, typing.Any] | nemo_automodel.components.models.deepseek_v41.config.DeepseekV41TextConfig | None = None,
    vision_config: dict[str, typing.Any] | nemo_automodel.components.models.deepseek_v41.config.DeepseekV41VisionConfig | None = None,
    image_token_id: int = 129264,
    dtype: str = 'bfloat16',
    pad_token_id: int | None = 2,
    bos_token_id: int = 0,
    eos_token_id: int = 1,
    tie_word_embeddings: bool = False,
    architectures: list[str] | None = None,
    kwargs: typing.Any = {}
)

Bases: PretrainedConfig

Nested checkpoint configuration for DeepseekV41ForCausalLM.

Hugging Face dictionaries are materialized at this boundary; model components receive the typed text_config and vision_config. Quantization metadata, when present in a checkpoint, is preserved by the base class rather than imposed on checkpoint-free BF16 configurations.

keys_to_ignore_at_inference

= ['past_key_values']

model_type

= 'deepseek_v41'

sub_configs

nemo_automodel.components.models.deepseek_v41.config.DeepseekV41Config.build_tokenizer() -> transformers.PreTrainedTokenizerFast

Load the checkpoint’s fast tokenizer for deterministic Engram hashing.

Returns: PreTrainedTokenizerFast

The original fast tokenizer from this configuration’s checkpoint

Raises:

ValueError: The configuration has no checkpoint source. Callers constructing a tiny model can supply a tokenizer directly.
TypeError: The checkpoint resolves to a slow tokenizer.

class nemo_automodel.components.models.deepseek_v41.config.DeepseekV41TextConfig(
    vocab_size: int = 129280,
    hidden_size: int = 5120,
    moe_intermediate_size: int = 2304,
    num_hidden_layers: int = 40,
    num_attention_heads: int = 64,
    num_key_value_heads: int = 1,
    head_dim: int = 512,
    qk_rope_head_dim: int = 64,
    q_lora_rank: int = 1280,
    o_lora_rank: int = 1024,
    o_groups: int = 8,
    hidden_act: str = 'silu',
    swiglu_limit: float = 10.0,
    rms_norm_eps: float = 1e-20,
    attention_bias: bool = False,
    attention_dropout: float = 0.0,
    initializer_range: float = 0.02,
    use_cache: bool = True,
    tie_word_embeddings: bool = False,
    max_position_embeddings: int = 1048576,
    rope_theta: float = 10000.0,
    rope_scaling: dict[str, typing.Any] | None = None,
    n_routed_experts: int = 384,
    n_shared_experts: int = 1,
    num_experts_per_tok: int = 6,
    scoring_func: str = 'sqrtsoftplus',
    topk_method: str = 'noaux_tc',
    norm_topk_prob: bool = True,
    routed_scaling_factor: float = 1.5,
    sliding_window: int = 128,
    compress_ratios: list[int] | None = None,
    compress_rope_theta: float = 160000.0,
    kv_source_layer_ids: list[int] | None = None,
    index_source_layer_ids: list[int] | None = None,
    index_n_heads: int = 32,
    index_head_dim: int = 128,
    index_topk: int = 512,
    candidate_source_layer_id: int = 20,
    candidate_topk_blocks: int = 2048,
    candidate_block_size: int = 8,
    hc_mult: int = 4,
    hc_sinkhorn_iters: int = 20,
    hc_eps: float = 1e-06,
    engram_layer_ids: list[int] | None = None,
    engram_num_embeddings: list[int] | None = None,
    engram_max_ngram_size: int = 4,
    engram_vocab_size: int = 16000000,
    engram_n_heads: int = 8,
    engram_head_dim: int = 256,
    engram_pad_token_id: int = 2,
    engram_compressed_vocab_size: int = 99092,
    num_nextn_predict_layers: int = 3,
    dspark_block_size: int = 5,
    dspark_noise_token_id: int = 128799,
    dspark_target_layer_ids: list[int] | None = None,
    dspark_markov_rank: int = 256,
    dspark_n_routed_experts: int = 128,
    dspark_num_experts_per_tok: int = 3,
    dtype: str = 'bfloat16',
    pad_token_id: int | None = 2,
    bos_token_id: int = 0,
    eos_token_id: int = 1,
    kwargs: typing.Any = {}
)

Bases: PretrainedConfig

Declarative configuration of the CSA2, single-pass mHC and Engram backbone.

Defaults match the released Flash text configuration. Reducing num_hidden_layers retains the complete source-layer and compression schedules: a pretrained prefix must keep its original sharing and hashing identities. Explicit smaller schedules support independent tiny models. Unknown checkpoint metadata is retained by the Hugging Face base class.

base_config_key

= 'text_config'

compress_ratios

dspark_target_layer_ids

engram_layer_ids

engram_num_embeddings

index_source_layer_ids

keys_to_ignore_at_inference

= ['past_key_values']

kv_source_layer_ids

model_type

= 'deepseek_v41_text'

rope_scaling

nemo_automodel.components.models.deepseek_v41.config.DeepseekV41TextConfig._validate_dimensions() -> None

nemo_automodel.components.models.deepseek_v41.config.DeepseekV41TextConfig._validate_layer_schedule() -> None

class nemo_automodel.components.models.deepseek_v41.config.DeepseekV41VisionConfig(
    num_hidden_layers: int = 32,
    hidden_size: int = 1024,
    num_attention_heads: int = 16,
    intermediate_size: int = 2816,
    patch_size: int = 14,
    rope_theta: float = 10000.0,
    downsample_ratio: int = 3,
    max_image_tokens: int = 1024,
    min_pixels: int = 295936,
    max_wh_ratio: float | None = None,
    kwargs: typing.Any = {}
)

Bases: PretrainedConfig

Configuration of the released 2D-RoPE vision encoder and image sizing.

base_config_key

= 'vision_config'

model_type

= 'deepseek_v41_vision'