Skip to navigation

nemo_automodel.components.moe.quantized_experts

View as MarkdownOpen in Claude

MXFP4-resident expert storage and model application for frozen MoE experts.

GroupedExpertsMXFP4 keeps the frozen routed-expert base weights packed as fp4-e2m1 + e8m0 block scales (the DeepSeek V4 Flash checkpoint format) and dequantizes on the fly inside the grouped GEMM, instead of holding them in bf16. This is the storage win for LoRA / frozen-base training of large MoE models, where the routed experts dominate parameter memory.

MXFP4ExpertStorageMixin shares packed storage and base GEMMs between the frozen and LoRA experts, with Torch or DeepEP token dispatch.

Module Contents

Classes

NameDescription
GroupedExpertsDeepEPMXFP4Frozen routed experts with mxfp4-resident base weights under DeepEP dispatch.
GroupedExpertsDeepEPNVFP4Frozen routed experts with NVFP4-resident base weights under DeepEP dispatch.
GroupedExpertsMXFP4Frozen routed experts with mxfp4-resident base weights and no adapter.
GroupedExpertsNVFP4Frozen routed experts with NVFP4-resident base weights and no adapter.
MXFP4ExpertStorageMixinPacked-mxfp4 base-weight storage and grouped GEMM for routed experts.
NVFP4ExpertStorageMixinPacked-NVFP4 base-weight storage and grouped GEMM for routed experts (ModelOpt NVFP4 checkpoints).

Functions

NameDescription
_pin_storage_dtypeRecord a packed-storage parameter’s dtype as its compute dtype so the frozen-parameter casts keep it.
_preserve_packed_storage_dtypesRecord the storage dtypes of packed parameters on module so whole-model casts keep them.
_to_localReturn the local shard of a DTensor, or the tensor unchanged.
_validate_nvfp4_expert_tensor_headersCheck the dtype and shape of every routed-expert tensor in the checkpoint’s safetensors headers.
apply_mxfp4_to_moe_expertsApply MXFP4-resident storage to the model’s common routed-expert modules.
apply_nvfp4_to_moe_expertsApply NVFP4-resident storage to the model’s common routed-expert modules.
packed_expert_parametersReturn the packed codes and scales of every mxfp4 / nvfp4 expert module in model.
validate_nvfp4_checkpoint_configCheck a checkpoint’s quantization metadata and expert tensor layout before its packed experts are loaded.
validate_packed_expert_topologyReject distributed layouts the packed-expert storage formats do not implement.

Data

_NVFP4_EXPERT_TENSOR

_NVFP4_EXPERT_TENSOR_DTYPES

_PACKED_EXPERT_FORMATS

logger

API

class nemo_automodel.components.moe.quantized_experts.GroupedExpertsDeepEPMXFP4(
)

Bases: MXFP4ExpertStorageMixin, GroupedExpertsDeepEP

Frozen routed experts with mxfp4-resident base weights under DeepEP dispatch.

Drop-in replacement for GroupedExpertsDeepEP when the experts are frozen (e.g. LoRA on attention only). The DeepEP fused all-to-all token dispatch is reused unchanged — mxfp4 only changes the two post-dispatch grouped GEMMs, which read the packed base weights via MXFP4GroupedMM instead of bf16 torch._grouped_mm.

The DeepEP parent always uses native grouped MM.

dispatch_rows_never_empty
= orig_module.dispatch_rows_never_empty
dispatcher_capacity_factor
= orig_module.dispatcher_capacity_factor
dispatcher_equal_token_counts
= orig_module.dispatcher_equal_token_counts
dispatcher_hybridep_compact_routing
= orig_module.dispatcher_hybridep_compact_routing
dispatcher_hybridep_max_tokens_per_rank
dispatcher_hybridep_num_blocks_permute
= orig_module.dispatcher_hybridep_num_blocks_permute
dispatcher_hybridep_num_blocks_unpermute
dispatcher_hybridep_num_sms_preprocessing
dispatcher_hybridep_permute_fusion
= orig_module.dispatcher_hybridep_permute_fusion
dispatcher_recompute_cache
= orig_module.dispatcher_recompute_cache
nccl_ep
= orig_module.nccl_ep
nemo_automodel.components.moe.quantized_experts.GroupedExpertsDeepEPMXFP4.forward(
x: torch.Tensor,
token_mask: torch.Tensor,
weights: torch.Tensor,
indices: torch.Tensor
) -> torch.Tensor

Forward over mxfp4 base weights with DeepEP dispatch.

Preserves the tensor and EP contract of GroupedExpertsDeepEP.forward, replacing the two base torch._grouped_mm calls with MXFP4GroupedMM over the packed weights.

Parameters:

x
torch.Tensor

Tensor of shape [local_tokens, hidden].

token_mask
torch.Tensor

Boolean tensor of shape [local_tokens], marking valid tokens.

weights
torch.Tensor

Routing probabilities of shape [local_tokens, top_k].

indices
torch.Tensor

Global expert IDs of shape [local_tokens, top_k].

Returns: torch.Tensor

Combined expert output of shape [local_tokens, hidden].

class nemo_automodel.components.moe.quantized_experts.GroupedExpertsDeepEPNVFP4(
)

Bases: NVFP4ExpertStorageMixin, GroupedExpertsDeepEP

Frozen routed experts with NVFP4-resident base weights under DeepEP dispatch.

Drop-in replacement for GroupedExpertsDeepEP when the experts are frozen (e.g. LoRA on attention only). The DeepEP fused all-to-all token dispatch is reused unchanged — NVFP4 only changes the two post-dispatch grouped GEMMs, which read the packed base weights via NVFP4GroupedMM instead of bf16 torch._grouped_mm.

The DeepEP parent always uses native grouped MM.

dispatch_rows_never_empty
= orig_module.dispatch_rows_never_empty
dispatcher_capacity_factor
= orig_module.dispatcher_capacity_factor
dispatcher_equal_token_counts
= orig_module.dispatcher_equal_token_counts
dispatcher_hybridep_compact_routing
= orig_module.dispatcher_hybridep_compact_routing
dispatcher_hybridep_max_tokens_per_rank
dispatcher_hybridep_num_blocks_permute
= orig_module.dispatcher_hybridep_num_blocks_permute
dispatcher_hybridep_num_blocks_unpermute
dispatcher_hybridep_num_sms_preprocessing
dispatcher_hybridep_permute_fusion
= orig_module.dispatcher_hybridep_permute_fusion
dispatcher_recompute_cache
= orig_module.dispatcher_recompute_cache
nemo_automodel.components.moe.quantized_experts.GroupedExpertsDeepEPNVFP4.forward(
x: torch.Tensor,
token_mask: torch.Tensor,
weights: torch.Tensor,
indices: torch.Tensor
) -> torch.Tensor

Forward over NVFP4 base weights with DeepEP dispatch.

Preserves the tensor and EP contract of GroupedExpertsDeepEP.forward, replacing the two base torch._grouped_mm calls with NVFP4GroupedMM over the packed weights.

class nemo_automodel.components.moe.quantized_experts.GroupedExpertsMXFP4(
)

Bases: MXFP4ExpertStorageMixin, GroupedExperts

Frozen routed experts with mxfp4-resident base weights and no adapter.

Drop-in replacement for GroupedExperts when the experts are frozen (e.g. LoRA training that targets only attention). Forward mirrors GroupedExperts._forward_grouped_mm but reads the packed base weights.

use_torch_mm
= orig_module.use_torch_mm
nemo_automodel.components.moe.quantized_experts.GroupedExpertsMXFP4._forward_grouped_mm_mxfp4(
x: torch.Tensor,
token_mask: torch.Tensor,
weights: torch.Tensor,
indices: torch.Tensor,
n_local_experts: int,
experts_start_idx: int
) -> torch.Tensor

Compute the frozen local experts’ contribution before the EP reduction.

Parameters:

x
torch.Tensor

Tensor of shape [tokens, hidden], gathered across the EP group.

token_mask
torch.Tensor

Boolean tensor of shape [tokens] selecting valid tokens.

weights
torch.Tensor

Tensor of shape [tokens, top_k] with differentiable routing probabilities.

indices
torch.Tensor

Integer tensor of shape [tokens, top_k] with global expert IDs.

n_local_experts
int

Number of experts on this rank.

experts_start_idx
int

First global expert ID on this rank.

Returns: torch.Tensor

FP32 tensor of shape [tokens, hidden], before the EP reduction.

nemo_automodel.components.moe.quantized_experts.GroupedExpertsMXFP4.forward(
x: torch.Tensor,
token_mask: torch.Tensor,
weights: torch.Tensor,
indices: torch.Tensor
) -> torch.Tensor

Compute packed experts with the tensor and EP contract of GroupedExperts.forward.

class nemo_automodel.components.moe.quantized_experts.GroupedExpertsNVFP4(
)

Bases: NVFP4ExpertStorageMixin, GroupedExperts

Frozen routed experts with NVFP4-resident base weights and no adapter.

Drop-in replacement for GroupedExperts when the experts are frozen (e.g. LoRA training that targets only attention). Forward mirrors GroupedExperts._forward_grouped_mm but reads the packed base weights.

use_torch_mm
= orig_module.use_torch_mm
nemo_automodel.components.moe.quantized_experts.GroupedExpertsNVFP4._forward_grouped_mm_nvfp4(
x: torch.Tensor,
token_mask: torch.Tensor,
weights: torch.Tensor,
indices: torch.Tensor,
n_local_experts: int,
experts_start_idx: int
) -> torch.Tensor

Compute the frozen local experts’ contribution before the EP reduction.

Parameters:

x
torch.Tensor

Tensor of shape [tokens, hidden], gathered across the EP group.

token_mask
torch.Tensor

Boolean tensor of shape [tokens] selecting valid tokens.

weights
torch.Tensor

Tensor of shape [tokens, top_k] with differentiable routing probabilities.

indices
torch.Tensor

Integer tensor of shape [tokens, top_k] with global expert IDs.

n_local_experts
int

Number of experts on this rank.

experts_start_idx
int

First global expert ID on this rank.

Returns: torch.Tensor

FP32 tensor of shape [tokens, hidden], before the EP reduction.

nemo_automodel.components.moe.quantized_experts.GroupedExpertsNVFP4.forward(
x: torch.Tensor,
token_mask: torch.Tensor,
weights: torch.Tensor,
indices: torch.Tensor
) -> torch.Tensor

Compute packed experts with the tensor and EP contract of GroupedExperts.forward.

class nemo_automodel.components.moe.quantized_experts.MXFP4ExpertStorageMixin()

Packed-mxfp4 base-weight storage and grouped GEMM for routed experts.

Mixed into a GroupedExperts (or GroupedExpertsLoRA) subclass. The base projections gate_and_up_projs / down_projs are stored as packed fp4 (int8, two e2m1 nibbles per byte) plus float8_e8m0fnu block scales, in checkpoint orientation [n_experts, out_dim, in_dim] so the block scales run along the contraction dim. Floating-point base parameters are dropped once packed.

Meta weights become packed placeholders for direct checkpoint loading. Materialized weights are quantized immediately during construction.

_MXFP4_BASE_NAMES
tuple[str, ...] = ('gate_and_up_projs', 'down_projs')
nemo_automodel.components.moe.quantized_experts.MXFP4ExpertStorageMixin._init_mxfp4_storage() -> None

Replace floating-point bases with packed placeholders or quantized weights.

nemo_automodel.components.moe.quantized_experts.MXFP4ExpertStorageMixin._init_packed_placeholders() -> None

Register packed meta parameters in checkpoint orientation [experts, out, in].

nemo_automodel.components.moe.quantized_experts.MXFP4ExpertStorageMixin._mxfp4_base_mm(
x: torch.Tensor,
name: str,
offs: torch.Tensor
) -> torch.Tensor

Multiply routed activations by a frozen packed base projection.

Parameters:

x
torch.Tensor

Tensor of shape [tokens, in_dim], grouped contiguously by local expert.

name
str

Base projection parameter name.

offs
torch.Tensor

Int32 tensor of shape [local_experts], holding cumulative token counts.

Returns: torch.Tensor

Tensor of shape [tokens, out_dim] with the activation dtype and device.

nemo_automodel.components.moe.quantized_experts.MXFP4ExpertStorageMixin._mxfp4_dequant_expert0(
name: str,
dtype: torch.dtype
) -> torch.Tensor

Dequantize expert 0 of base weight name to compute layout [in, out].

nemo_automodel.components.moe.quantized_experts.MXFP4ExpertStorageMixin._pack_base_weights() -> None

Quantize materialized base projections and release their floating-point storage.

nemo_automodel.components.moe.quantized_experts.MXFP4ExpertStorageMixin._register_packed_base_weight(
name: str,
packed: torch.Tensor,
scales: torch.Tensor
) -> None

Replace one floating-point base projection with frozen packed parameters.

Parameters:

name
str

Base projection parameter name.

packed
torch.Tensor

Int8 tensor of shape [experts, out_dim, in_dim // 2].

scales
torch.Tensor

E8M0 tensor of shape [experts, out_dim, in_dim // 32]. DTensors must retain the base parameter’s mesh and expert-axis placement.

nemo_automodel.components.moe.quantized_experts.MXFP4ExpertStorageMixin._validate_mxfp4_config() -> None

Reject execution modes that the packed expert computation does not implement.

nemo_automodel.components.moe.quantized_experts.MXFP4ExpertStorageMixin.packed_parameter_names() -> tuple[str, ...]

Names of the packed codes and block scales that replace the floating-point base projections.

class nemo_automodel.components.moe.quantized_experts.NVFP4ExpertStorageMixin()

Packed-NVFP4 base-weight storage and grouped GEMM for routed experts (ModelOpt NVFP4 checkpoints).

Mixed into a GroupedExperts (or GroupedExpertsLoRA) subclass. The base projections gate_and_up_projs / down_projs are stored as packed fp4-e2m1 codes (uint8, two per byte), one float8_e4m3fn scale per NVFP4_BLOCK_SIZE elements and one float32 scale per output row, in checkpoint orientation [n_experts, out_dim, in_dim] so the blocks run along the contraction dim: 0.5625 bytes per parameter plus 4 bytes per row. Floating-point base parameters are dropped once packed.

Meta weights become packed placeholders in the shapes and dtypes of a ModelOpt NVFP4 per-expert HF checkpoint; the standard base-checkpoint load fills them through the model’s state-dict adapter (set_expert_storage_format("nvfp4") maps the per-expert weight / weight_scale / weight_scale_2 tensors onto them), so no quantization runs at load. Materialized (floating-point) weights are quantized immediately during construction.

_NVFP4_BASE_NAMES
tuple[str, ...] = ('gate_and_up_projs', 'down_projs')
nemo_automodel.components.moe.quantized_experts.NVFP4ExpertStorageMixin._init_nvfp4_placeholders() -> None

Register packed meta parameters in checkpoint orientation [experts, out, in].

nemo_automodel.components.moe.quantized_experts.NVFP4ExpertStorageMixin._init_nvfp4_storage() -> None

Replace floating-point bases with packed placeholders or quantized weights.

nemo_automodel.components.moe.quantized_experts.NVFP4ExpertStorageMixin._nvfp4_base_mm(
x: torch.Tensor,
name: str,
offs: torch.Tensor
) -> torch.Tensor

Multiply routed activations by a frozen packed base projection.

Parameters:

x
torch.Tensor

Tensor of shape [tokens, in_dim], grouped contiguously by local expert.

name
str

Base projection parameter name.

offs
torch.Tensor

Int32 tensor of shape [local_experts], holding cumulative token counts.

Returns: torch.Tensor

Tensor of shape [tokens, out_dim] with the activation dtype and device.

nemo_automodel.components.moe.quantized_experts.NVFP4ExpertStorageMixin._nvfp4_dequant_expert0(
name: str,
dtype: torch.dtype
) -> torch.Tensor

Dequantize expert 0 of base weight name to compute layout [in, out].

nemo_automodel.components.moe.quantized_experts.NVFP4ExpertStorageMixin._pack_base_weights_nvfp4() -> None

Quantize materialized base projections and release their floating-point storage.

nemo_automodel.components.moe.quantized_experts.NVFP4ExpertStorageMixin._register_nvfp4_base_weight(
name: str,
packed: torch.Tensor,
scales: torch.Tensor,
global_scale: torch.Tensor
) -> None

Replace one floating-point base projection with frozen packed parameters.

Parameters:

name
str

Base projection parameter name.

packed
torch.Tensor

uint8 tensor of shape [experts, out_dim, in_dim // 2].

scales
torch.Tensor

float8_e4m3fn tensor of shape [experts, out_dim, in_dim // NVFP4_BLOCK_SIZE].

global_scale
torch.Tensor

float32 tensor of shape [experts, out_dim]. DTensors must retain the base parameter’s mesh and expert-axis placement.

nemo_automodel.components.moe.quantized_experts.NVFP4ExpertStorageMixin._validate_nvfp4_config() -> None

Reject execution modes that the packed expert computation does not implement.

nemo_automodel.components.moe.quantized_experts.NVFP4ExpertStorageMixin.nvfp4_parameter_names(
name: str
) -> tuple[str, str, str]
staticmethod

Names of the packed codes, e4m3 block scales and fp32 row scales replacing base projection name.

nemo_automodel.components.moe.quantized_experts.NVFP4ExpertStorageMixin.packed_parameter_names() -> tuple[str, ...]

Names of the packed codes, block scales and row scales that replace the floating-point base projections.

nemo_automodel.components.moe.quantized_experts._pin_storage_dtype(
param: torch.nn.Parameter
) -> torch.nn.Parameter

Record a packed-storage parameter’s dtype as its compute dtype so the frozen-parameter casts keep it.

prepare_frozen_parameters_for_sharding and fully_shard_by_dtype honour _hf_compute_dtype; the frozen-parameter casts also receive the parameters as keep_storage (packed_expert_parameters).

nemo_automodel.components.moe.quantized_experts._preserve_packed_storage_dtypes(
module: torch.nn.Module,
names: tuple[str, ...]
) -> None

Record the storage dtypes of packed parameters on module so whole-model casts keep them.

cast_model_to_dtype consults _preserve_parameter_dtypes. Recording the packed names when they are registered makes the contract independent of prepare_frozen_parameters_for_sharding, which only FSDP2 PEFT runs call before initialize_weights.

nemo_automodel.components.moe.quantized_experts._to_local(
t
)

Return the local shard of a DTensor, or the tensor unchanged.

nemo_automodel.components.moe.quantized_experts._validate_nvfp4_expert_tensor_headers(
checkpoint_dir: str
) -> None

Check the dtype and shape of every routed-expert tensor in the checkpoint’s safetensors headers.

nemo_automodel.components.moe.quantized_experts.apply_mxfp4_to_moe_experts(
model: torch.nn.Module
) -> torch.nn.Module

Apply MXFP4-resident storage to the model’s common routed-expert modules.

Call this after LoRA injection and before distributed sharding. LoRA-targeted experts that are already MXFP4-resident are preserved; remaining plain GroupedExperts and GroupedExpertsDeepEP modules are replaced in place.

Meta experts receive packed placeholders for direct checkpoint loading, and their model state-dict adapter must opt into the MXFP4 expert storage format. Already materialized expert weights are quantized without changing the adapter’s checkpoint loading mode. Model-specific adapters own checkpoint layouts.

nemo_automodel.components.moe.quantized_experts.apply_nvfp4_to_moe_experts(
model: torch.nn.Module
) -> torch.nn.Module

Apply NVFP4-resident storage to the model’s common routed-expert modules.

Call this after LoRA injection and before distributed sharding. LoRA-targeted experts that are already NVFP4-resident are preserved; remaining plain GroupedExperts and GroupedExpertsDeepEP modules are replaced in place. Transformer Engine experts are rejected: the packed storage implements the torch_mm experts backend (torch, DeepEP or HybridEP dispatch) only.

Meta experts receive packed placeholders for direct checkpoint loading, and their model state-dict adapter must opt into the NVFP4 expert storage format. Already materialized expert weights are quantized without changing the adapter’s checkpoint loading mode. Model-specific adapters own checkpoint layouts.

Raises:

  • NotImplementedError: If a Transformer Engine expert module is present, or the model state-dict adapter cannot select the NVFP4 expert storage format.
nemo_automodel.components.moe.quantized_experts.packed_expert_parameters(
model: torch.nn.Module
) -> set[torch.nn.Parameter]

Return the packed codes and scales of every mxfp4 / nvfp4 expert module in model.

Their storage dtype is the weight format, not a master-weight dtype, so callers pass them to the frozen-parameter casts as keep_storage.

Parameters:

model
nn.Module

Model or model part, before or after distributed wrapping.

Returns: set[nn.Parameter]

The currently registered packed parameters; empty when model has no packed experts.

nemo_automodel.components.moe.quantized_experts.validate_nvfp4_checkpoint_config(
checkpoint_dir: str
) -> None

Check a checkpoint’s quantization metadata and expert tensor layout before its packed experts are loaded.

ModelOpt writes hf_quant_config.json (quantization.quant_algo / group_size) and mirrors it in config.json’s quantization_config. The packed storage implements NVFP4 with 16-column blocks and experts-only quantization (the dense projections, attention, router, norms and embeddings must stay in floating point: they are loaded by the ordinary base load). The safetensors headers of a local checkpoint are then checked against the per-expert layout the state-dict adapters map onto the packed parameters: every routed-expert projection must carry weight (U8 or I8 codes, [out, in / 2]), weight_scale (F8_E4M3, [out, in / NVFP4_BLOCK_SIZE]) and a scalar F32 weight_scale_2. The checkpoint load enforces shapes only and copies with dtype conversion, so a wrongly typed export would otherwise load silently. A checkpoint that also quantized the dense layers fails later in the base load.

Parameters:

checkpoint_dir
str

Local HF checkpoint directory (a snapshot holding only the JSON files skips the header check).

Raises:

  • ValueError: If the metadata names another algorithm or block size, or an expert tensor has another dtype or an inconsistent shape.
nemo_automodel.components.moe.quantized_experts.validate_packed_expert_topology(
expert_weight_format: str,
world_size: int,
ep_size: int,
pp_size: int = 1
) -> None

Reject distributed layouts the packed-expert storage formats do not implement.

Packed experts (mxfp4 / nf4 / nvfp4) are owned by their module as plain integer codes plus scale tensors, sharded only along the expert axis. They are supported on one rank, and with expert parallelism that spans the whole non-pipeline mesh (ep_size == world_size // pp_size) where the experts are excluded from FSDP. Two layouts are refused: ep_size == 1 on several ranks (FSDP would shard the packed tensors by compute dtype and all-gather the scales in bf16) and an additional expert-shard axis ep_shard > 1 (FSDP would shard the codes on dim 1 and cast them to the policy dtype, which breaks the packed layout).

Parameters:

expert_weight_format
str

PeftConfig.expert_weight_format; anything but a packed format is accepted.

world_size
int

Number of ranks.

ep_size
int

Expert-parallel size.

pp_size
intDefaults to 1

Pipeline-parallel size (tensor and context parallelism are part of the non-pipeline mesh, as in mesh_utils, and do not change the expert-shard arithmetic).

Raises:

  • ValueError: If the layout is unsupported for packed experts.
nemo_automodel.components.moe.quantized_experts._NVFP4_EXPERT_TENSOR = re.compile('\\.experts\\.\\d+\\.(?:gate_proj|up_proj|down_proj)\\.(weight|weight...
nemo_automodel.components.moe.quantized_experts._NVFP4_EXPERT_TENSOR_DTYPES = {'weight': ('U8', 'I8'), 'weight_scale': ('F8_E4M3',), 'weight_scale_2': ('F32',...
nemo_automodel.components.moe.quantized_experts._PACKED_EXPERT_FORMATS = ('mxfp4', 'nf4', 'nvfp4')
nemo_automodel.components.moe.quantized_experts.logger = logging.getLogger(__name__)