nemo_automodel.components.moe.quantized_experts
nemo_automodel.components.moe.quantized_experts
MXFP4-resident expert storage and model application for frozen MoE experts.
GroupedExpertsMXFP4 keeps the frozen routed-expert base weights packed as
fp4-e2m1 + e8m0 block scales (the DeepSeek V4 Flash checkpoint format) and
dequantizes on the fly inside the grouped GEMM, instead of holding them in
bf16. This is the storage win for LoRA / frozen-base training of large MoE
models, where the routed experts dominate parameter memory.
MXFP4ExpertStorageMixin shares packed storage and base GEMMs between the
frozen and LoRA experts, with Torch or DeepEP token dispatch.
Module Contents
Classes
Functions
Data
API
Bases: MXFP4ExpertStorageMixin, GroupedExpertsDeepEP
Frozen routed experts with mxfp4-resident base weights under DeepEP dispatch.
Drop-in replacement for GroupedExpertsDeepEP when the experts are frozen
(e.g. LoRA on attention only). The DeepEP fused all-to-all token dispatch is reused
unchanged — mxfp4 only changes the two post-dispatch grouped GEMMs, which read the
packed base weights via MXFP4GroupedMM instead of bf16 torch._grouped_mm.
The DeepEP parent always uses native grouped MM.
Forward over mxfp4 base weights with DeepEP dispatch.
Preserves the tensor and EP contract of GroupedExpertsDeepEP.forward, replacing the two base
torch._grouped_mm calls with MXFP4GroupedMM over the packed weights.
Parameters:
Tensor of shape [local_tokens, hidden].
Boolean tensor of shape [local_tokens], marking valid tokens.
Routing probabilities of shape [local_tokens, top_k].
Global expert IDs of shape [local_tokens, top_k].
Returns: torch.Tensor
Combined expert output of shape [local_tokens, hidden].
Bases: NVFP4ExpertStorageMixin, GroupedExpertsDeepEP
Frozen routed experts with NVFP4-resident base weights under DeepEP dispatch.
Drop-in replacement for GroupedExpertsDeepEP when the experts are frozen
(e.g. LoRA on attention only). The DeepEP fused all-to-all token dispatch is reused
unchanged — NVFP4 only changes the two post-dispatch grouped GEMMs, which read the
packed base weights via NVFP4GroupedMM instead of bf16 torch._grouped_mm.
The DeepEP parent always uses native grouped MM.
Forward over NVFP4 base weights with DeepEP dispatch.
Preserves the tensor and EP contract of GroupedExpertsDeepEP.forward, replacing the two base
torch._grouped_mm calls with NVFP4GroupedMM over the packed weights.
Bases: MXFP4ExpertStorageMixin, GroupedExperts
Frozen routed experts with mxfp4-resident base weights and no adapter.
Drop-in replacement for GroupedExperts when the experts are frozen (e.g.
LoRA training that targets only attention). Forward mirrors
GroupedExperts._forward_grouped_mm but reads the packed base weights.
Compute the frozen local experts’ contribution before the EP reduction.
Parameters:
Tensor of shape [tokens, hidden], gathered across the EP group.
Boolean tensor of shape [tokens] selecting valid tokens.
Tensor of shape [tokens, top_k] with differentiable routing probabilities.
Integer tensor of shape [tokens, top_k] with global expert IDs.
Number of experts on this rank.
First global expert ID on this rank.
Returns: torch.Tensor
FP32 tensor of shape [tokens, hidden], before the EP reduction.
Compute packed experts with the tensor and EP contract of GroupedExperts.forward.
Bases: NVFP4ExpertStorageMixin, GroupedExperts
Frozen routed experts with NVFP4-resident base weights and no adapter.
Drop-in replacement for GroupedExperts when the experts are frozen (e.g.
LoRA training that targets only attention). Forward mirrors
GroupedExperts._forward_grouped_mm but reads the packed base weights.
Compute the frozen local experts’ contribution before the EP reduction.
Parameters:
Tensor of shape [tokens, hidden], gathered across the EP group.
Boolean tensor of shape [tokens] selecting valid tokens.
Tensor of shape [tokens, top_k] with differentiable routing probabilities.
Integer tensor of shape [tokens, top_k] with global expert IDs.
Number of experts on this rank.
First global expert ID on this rank.
Returns: torch.Tensor
FP32 tensor of shape [tokens, hidden], before the EP reduction.
Compute packed experts with the tensor and EP contract of GroupedExperts.forward.
Packed-mxfp4 base-weight storage and grouped GEMM for routed experts.
Mixed into a GroupedExperts (or GroupedExpertsLoRA) subclass. The base
projections gate_and_up_projs / down_projs are stored as packed fp4
(int8, two e2m1 nibbles per byte) plus float8_e8m0fnu block scales, in
checkpoint orientation [n_experts, out_dim, in_dim] so the block scales run
along the contraction dim. Floating-point base parameters are dropped once packed.
Meta weights become packed placeholders for direct checkpoint loading. Materialized weights are quantized immediately during construction.
Replace floating-point bases with packed placeholders or quantized weights.
Register packed meta parameters in checkpoint orientation [experts, out, in].
Multiply routed activations by a frozen packed base projection.
Parameters:
Tensor of shape [tokens, in_dim], grouped contiguously by local expert.
Base projection parameter name.
Int32 tensor of shape [local_experts], holding cumulative token counts.
Returns: torch.Tensor
Tensor of shape [tokens, out_dim] with the activation dtype and device.
Dequantize expert 0 of base weight name to compute layout [in, out].
Quantize materialized base projections and release their floating-point storage.
Replace one floating-point base projection with frozen packed parameters.
Parameters:
Base projection parameter name.
Int8 tensor of shape [experts, out_dim, in_dim // 2].
E8M0 tensor of shape [experts, out_dim, in_dim // 32]. DTensors must retain the base parameter’s mesh and expert-axis placement.
Reject execution modes that the packed expert computation does not implement.
Names of the packed codes and block scales that replace the floating-point base projections.
Packed-NVFP4 base-weight storage and grouped GEMM for routed experts (ModelOpt NVFP4 checkpoints).
Mixed into a GroupedExperts (or GroupedExpertsLoRA) subclass. The base
projections gate_and_up_projs / down_projs are stored as packed fp4-e2m1 codes
(uint8, two per byte), one float8_e4m3fn scale per NVFP4_BLOCK_SIZE elements and
one float32 scale per output row, in checkpoint orientation [n_experts, out_dim, in_dim]
so the blocks run along the contraction dim: 0.5625 bytes per parameter plus 4 bytes per row.
Floating-point base parameters are dropped once packed.
Meta weights become packed placeholders in the shapes and dtypes of a ModelOpt NVFP4 per-expert HF
checkpoint; the standard base-checkpoint load fills them through the model’s state-dict adapter
(set_expert_storage_format("nvfp4") maps the per-expert weight / weight_scale /
weight_scale_2 tensors onto them), so no quantization runs at load. Materialized
(floating-point) weights are quantized immediately during construction.
Register packed meta parameters in checkpoint orientation [experts, out, in].
Replace floating-point bases with packed placeholders or quantized weights.
Multiply routed activations by a frozen packed base projection.
Parameters:
Tensor of shape [tokens, in_dim], grouped contiguously by local expert.
Base projection parameter name.
Int32 tensor of shape [local_experts], holding cumulative token counts.
Returns: torch.Tensor
Tensor of shape [tokens, out_dim] with the activation dtype and device.
Dequantize expert 0 of base weight name to compute layout [in, out].
Quantize materialized base projections and release their floating-point storage.
Replace one floating-point base projection with frozen packed parameters.
Parameters:
Base projection parameter name.
uint8 tensor of shape [experts, out_dim, in_dim // 2].
float8_e4m3fn tensor of shape [experts, out_dim, in_dim // NVFP4_BLOCK_SIZE].
float32 tensor of shape [experts, out_dim].
DTensors must retain the base parameter’s mesh and expert-axis placement.
Reject execution modes that the packed expert computation does not implement.
Names of the packed codes, e4m3 block scales and fp32 row scales replacing base projection name.
Names of the packed codes, block scales and row scales that replace the floating-point base projections.
Record a packed-storage parameter’s dtype as its compute dtype so the frozen-parameter casts keep it.
prepare_frozen_parameters_for_sharding and fully_shard_by_dtype honour _hf_compute_dtype;
the frozen-parameter casts also receive the parameters as keep_storage (packed_expert_parameters).
Record the storage dtypes of packed parameters on module so whole-model casts keep them.
cast_model_to_dtype consults _preserve_parameter_dtypes. Recording the packed names when they are
registered makes the contract independent of prepare_frozen_parameters_for_sharding, which only FSDP2
PEFT runs call before initialize_weights.
Return the local shard of a DTensor, or the tensor unchanged.
Check the dtype and shape of every routed-expert tensor in the checkpoint’s safetensors headers.
Apply MXFP4-resident storage to the model’s common routed-expert modules.
Call this after LoRA injection and before distributed sharding. LoRA-targeted
experts that are already MXFP4-resident are preserved; remaining plain
GroupedExperts and GroupedExpertsDeepEP modules are replaced in place.
Meta experts receive packed placeholders for direct checkpoint loading, and their model state-dict adapter must opt into the MXFP4 expert storage format. Already materialized expert weights are quantized without changing the adapter’s checkpoint loading mode. Model-specific adapters own checkpoint layouts.
Apply NVFP4-resident storage to the model’s common routed-expert modules.
Call this after LoRA injection and before distributed sharding. LoRA-targeted
experts that are already NVFP4-resident are preserved; remaining plain
GroupedExperts and GroupedExpertsDeepEP modules are replaced in place.
Transformer Engine experts are rejected: the packed storage implements the
torch_mm experts backend (torch, DeepEP or HybridEP dispatch) only.
Meta experts receive packed placeholders for direct checkpoint loading, and their model state-dict adapter must opt into the NVFP4 expert storage format. Already materialized expert weights are quantized without changing the adapter’s checkpoint loading mode. Model-specific adapters own checkpoint layouts.
Raises:
NotImplementedError: If a Transformer Engine expert module is present, or the model state-dict adapter cannot select the NVFP4 expert storage format.
Return the packed codes and scales of every mxfp4 / nvfp4 expert module in model.
Their storage dtype is the weight format, not a master-weight dtype, so callers pass them to the
frozen-parameter casts as keep_storage.
Parameters:
Model or model part, before or after distributed wrapping.
Returns: set[nn.Parameter]
The currently registered packed parameters; empty when model has no packed experts.
Check a checkpoint’s quantization metadata and expert tensor layout before its packed experts are loaded.
ModelOpt writes hf_quant_config.json (quantization.quant_algo / group_size) and mirrors it in
config.json’s quantization_config. The packed storage implements NVFP4 with 16-column blocks and
experts-only quantization (the dense projections, attention, router, norms and embeddings must stay in
floating point: they are loaded by the ordinary base load). The safetensors headers of a local checkpoint are
then checked against the per-expert layout the state-dict adapters map onto the packed parameters: every
routed-expert projection must carry weight (U8 or I8 codes, [out, in / 2]), weight_scale
(F8_E4M3, [out, in / NVFP4_BLOCK_SIZE]) and a scalar F32 weight_scale_2. The checkpoint load
enforces shapes only and copies with dtype conversion, so a wrongly typed export would otherwise load silently.
A checkpoint that also quantized the dense layers fails later in the base load.
Parameters:
Local HF checkpoint directory (a snapshot holding only the JSON files skips the header check).
Raises:
ValueError: If the metadata names another algorithm or block size, or an expert tensor has another dtype or an inconsistent shape.
Reject distributed layouts the packed-expert storage formats do not implement.
Packed experts (mxfp4 / nf4 / nvfp4) are owned by their module as plain integer codes plus scale
tensors, sharded only along the expert axis. They are supported on one rank, and with expert
parallelism that spans the whole non-pipeline mesh (ep_size == world_size // pp_size) where the
experts are excluded from FSDP. Two layouts are refused: ep_size == 1 on several ranks (FSDP would
shard the packed tensors by compute dtype and all-gather the scales in bf16) and an additional
expert-shard axis ep_shard > 1 (FSDP would shard the codes on dim 1 and cast them to the policy
dtype, which breaks the packed layout).
Parameters:
PeftConfig.expert_weight_format; anything but a packed format is accepted.
Number of ranks.
Expert-parallel size.
Pipeline-parallel size (tensor and context parallelism are part of the non-pipeline mesh,
as in mesh_utils, and do not change the expert-shard arithmetic).
Raises:
ValueError: If the layout is unsupported for packed experts.