nemo_automodel.components.models.qwen3_5_moe.state_dict_adapter

State-dict adapter for Qwen3.5-MoE.

HF Qwen3.5-MoE stores expert weights as aggregated 3-D tensors:

model.language_model.layers.{L}.mlp.experts.gate_up_proj # [n_experts, 2*moe_inter, hidden] model.language_model.layers.{L}.mlp.experts.down_proj # [n_experts, hidden, moe_inter]

NeMo uses a different naming convention and transposed layout (x @ weight):

model.language_model.layers.{L}.mlp.experts.gate_and_up_projs # [n_experts, hidden, 2*moe_inter] model.language_model.layers.{L}.mlp.experts.down_projs # [n_experts, moe_inter, hidden]

Both expert tensors require .transpose(1, 2) when converting between formats.

Additionally, the shared expert uses singular in HF and plural in NeMo:

HF: .mlp.shared_expert.{gate,up,down}_proj.weight NeMo: .mlp.shared_experts.{gate,up,down}_proj.weight

All other keys (attention, linear_attn/GatedDeltaNet, norms, embeddings, vision encoder) pass through unchanged. The HF VLM checkpoint stores the language model head as model.lm_head while Automodel registers it on the outer model as lm_head.

Module Contents

Classes

Name	Description
`Qwen3_5MoeStateDictAdapter`	Converts between HF Qwen3.5-MoE checkpoints and the NeMo native format.

Functions

Name	Description
`_block_scale_placeholder`	Create a regular 128x128 block-scale load target for a 2-D weight.
`_dequantize_block_fp8`	Dequantize one local 2-D 128x128 block-scaled FP8 expert weight.
`_filter_excluded`	Remove exported HF entries matched by `exclude_key_regex`.
`_get_local_safetensors_keys`	Read key metadata from a local HF safetensors checkpoint without loading tensors.
`_infer_base_expert_hf_layout`	Infer the decoder expert layout from checkpoint key names.
`_infer_mtp_expert_hf_layout`	Infer the MTP expert layout from checkpoint key names.
`_route_fp32_params`	Route bare GDN fp32 params into the holder used by the native module.
`_slice_ep_shard_dim1`	Slice dim 1 of a local expert tensor for the `ep_shard` mesh dimension.
`_strip_fp32_params`	Strip the fp32 holder segment from GDN state-dict keys.

Data

_BASE_GROUPED_EXPERT_KEY

_BASE_SPLIT_FP8_EXPERT_KEY

_BASE_SPLIT_FP8_EXPERT_PARAM

_FP8_BLOCK_SIZE

_MTP_GROUPED_EXPERT_KEY

_MTP_SPLIT_EXPERT_KEY

API

class nemo_automodel.components.models.qwen3_5_moe.state_dict_adapter.Qwen3_5MoeStateDictAdapter(
    config: typing.Any,
    moe_config: nemo_automodel.components.moe.layers.MoEConfig,
    backend: nemo_automodel.components.models.common.BackendConfig,
    dtype: torch.dtype = torch.float32,
    pretrained_model_name_or_path: str | None = None,
    mtp_expert_hf_layout: str | None = None,
    text_only: bool = False
)

Bases: StateDictAdapter

Converts between HF Qwen3.5-MoE checkpoints and the NeMo native format.

HF Qwen3.5-MoE stores expert weights as aggregated 3-D tensors:

model.language_model.layers.{L}.mlp.experts.gate_up_proj # [n_experts, 2*moe_inter, hidden] model.language_model.layers.{L}.mlp.experts.down_proj # [n_experts, hidden, moe_inter]

NeMo uses a different naming convention and transposed layout (x @ weight):

model.language_model.layers.{L}.mlp.experts.gate_and_up_projs # [n_experts, hidden, 2*moe_inter] model.language_model.layers.{L}.mlp.experts.down_projs # [n_experts, moe_inter, hidden]

Both expert tensors require .transpose(1, 2) when converting between formats.

Additionally, the shared expert uses singular in HF and plural in NeMo:

HF: .mlp.shared_expert.{gate,up,down}_proj.weight NeMo: .mlp.shared_experts.{gate,up,down}_proj.weight

_expert_hf_layout

str | None = None

_inferred_mtp_expert_hf_layout

str | None = None

hf_to_internal_map

= {'.mlp.shared_expert.': '.mlp.shared_experts.'}

internal_to_hf_map

nemo_automodel.components.models.qwen3_5_moe.state_dict_adapter.Qwen3_5MoeStateDictAdapter._apply_key_mapping(
    state_dict: dict[str, typing.Any],
    mapping: dict[str, str]
) -> dict[str, typing.Any]

Apply key substring mappings to state dict keys.

nemo_automodel.components.models.qwen3_5_moe.state_dict_adapter.Qwen3_5MoeStateDictAdapter._get_expert_hf_layout(
    checkpoint_keys: typing.Iterable[str] | None = None
) -> str

Resolve and remember whether decoder experts use grouped BF16 or split block-FP8 HF keys.

nemo_automodel.components.models.qwen3_5_moe.state_dict_adapter.Qwen3_5MoeStateDictAdapter._get_mtp_expert_hf_layout(
    checkpoint_keys: typing.Iterable[str] | None = None
) -> str

Resolve and remember whether MTP experts use split or grouped HF keys.

nemo_automodel.components.models.qwen3_5_moe.state_dict_adapter.Qwen3_5MoeStateDictAdapter.convert_single_tensor_to_hf(
    fqn: str,
    tensor: typing.Any,
    kwargs = {}
) -> list[tuple[str, typing.Any]]

Rename a single native key to HF format and transpose expert tensors.

nemo_automodel.components.models.qwen3_5_moe.state_dict_adapter.Qwen3_5MoeStateDictAdapter.forced_hf_dtype_mapping(
    state_dict: dict[str, typing.Any]
) -> dict[str, str]

Return HF export dtype overrides for intrinsically-fp32 GDN tensors.

nemo_automodel.components.models.qwen3_5_moe.state_dict_adapter.Qwen3_5MoeStateDictAdapter.from_hf(
    hf_state_dict: dict[str, typing.Any],
    device_mesh: typing.Optional[torch.distributed.device_mesh.DeviceMesh] = None,
    kwargs = {}
) -> dict[str, typing.Any]

Rename HF keys to native keys and transpose expert tensors.

DTensors (DCP path): rename + transpose; split MTP experts are locally reassembled without applying EP-shard slicing again. Plain tensors (init path): slice to local EP shard, transpose, create DTensor.

nemo_automodel.components.models.qwen3_5_moe.state_dict_adapter.Qwen3_5MoeStateDictAdapter.to_hf(
    state_dict: dict[str, typing.Any],
    exclude_key_regex: str | None = None,
    quantization: bool = False,
    kwargs = {}
) -> dict[str, typing.Any]

Rename native keys to HF keys and transpose expert tensors.

nemo_automodel.components.models.qwen3_5_moe.state_dict_adapter._block_scale_placeholder(
    weight: typing.Any
) -> torch.Tensor

Create a regular 128x128 block-scale load target for a 2-D weight.

Parameters:

weight

Any

FP8 weight Tensor or DTensor of shape [out, in].

Returns: torch.Tensor

Tensor of shape [ceil(out / 128), ceil(in / 128)] with one scale for each 128x128 block.

nemo_automodel.components.models.qwen3_5_moe.state_dict_adapter._dequantize_block_fp8(
    weight: typing.Any,
    scale_inv: typing.Any,
    dtype: torch.dtype
) -> torch.Tensor

Dequantize one local 2-D 128x128 block-scaled FP8 expert weight.

Parameters:

weight

Any

FP8 weight Tensor or DTensor of shape [out, in].

scale_inv

Any

Block-scale Tensor or DTensor of shape [ceil(out / 128), ceil(in / 128)], with one scale for each 128x128 block.

dtype

torch.dtype

Output dtype.

Returns: torch.Tensor

Dequantized Tensor of shape [out, in] in dtype.

nemo_automodel.components.models.qwen3_5_moe.state_dict_adapter._filter_excluded(
    result: list[tuple[str, typing.Any]],
    exclude_key_regex: str | None
) -> list[tuple[str, typing.Any]]

Remove exported HF entries matched by exclude_key_regex.

nemo_automodel.components.models.qwen3_5_moe.state_dict_adapter._get_local_safetensors_keys(
    model_path: str | None
) -> set[str]

Read key metadata from a local HF safetensors checkpoint without loading tensors.

nemo_automodel.components.models.qwen3_5_moe.state_dict_adapter._infer_base_expert_hf_layout(
    checkpoint_keys: typing.Iterable[str]
) -> str | None

Infer the decoder expert layout from checkpoint key names.

nemo_automodel.components.models.qwen3_5_moe.state_dict_adapter._infer_mtp_expert_hf_layout(
    checkpoint_keys: typing.Iterable[str]
) -> str | None

Infer the MTP expert layout from checkpoint key names.

nemo_automodel.components.models.qwen3_5_moe.state_dict_adapter._route_fp32_params(
    key: str
) -> str

Route bare GDN fp32 params into the holder used by the native module.

nemo_automodel.components.models.qwen3_5_moe.state_dict_adapter._slice_ep_shard_dim1(
    local_tensor: torch.Tensor,
    ep_shard_rank: int,
    ep_shard_size: int,
    tensor_name: str
) -> torch.Tensor

Slice dim 1 of a local expert tensor for the ep_shard mesh dimension.

nemo_automodel.components.models.qwen3_5_moe.state_dict_adapter._strip_fp32_params(
    key: str
) -> str

Strip the fp32 holder segment from GDN state-dict keys.

nemo_automodel.components.models.qwen3_5_moe.state_dict_adapter._BASE_GROUPED_EXPERT_KEY = re.compile('(?:model\\.)?(?:language_model\\.)?layers\\.\\d+\\.mlp\\.experts\\.(...

nemo_automodel.components.models.qwen3_5_moe.state_dict_adapter._BASE_SPLIT_FP8_EXPERT_KEY = re.compile('(?:model\\.)?(?:language_model\\.)?layers\\.\\d+\\.mlp\\.experts\\.\...

nemo_automodel.components.models.qwen3_5_moe.state_dict_adapter._BASE_SPLIT_FP8_EXPERT_PARAM = re.compile('(?:model\\.)?(?:language_model\\.)?layers\\.(\\d+)\\.mlp\\.experts\\...

nemo_automodel.components.models.qwen3_5_moe.state_dict_adapter._FP8_BLOCK_SIZE = 128

nemo_automodel.components.models.qwen3_5_moe.state_dict_adapter._MTP_GROUPED_EXPERT_KEY = re.compile('mtp\\.layers\\.\\d+\\.mlp\\.experts\\.(?:gate_up_proj|down_proj)')

nemo_automodel.components.models.qwen3_5_moe.state_dict_adapter._MTP_SPLIT_EXPERT_KEY = re.compile('mtp\\.layers\\.\\d+\\.mlp\\.experts\\.\\d+\\.(?:gate_proj|up_proj|do...