nemo_automodel.components.quantization
nemo_automodel.components.quantization
Submodules
nemo_automodel.components.quantization.fp8nemo_automodel.components.quantization.mxfp4nemo_automodel.components.quantization.nvfp4nemo_automodel.components.quantization.qatnemo_automodel.components.quantization.qlora
Package Contents
Classes
Functions
Data
API
Configuration for FP8 quantization settings.
List of fully qualified names of modules to skip applying float8 training to. nn.Linear modules with any dim size not divisible by 16 are always skipped due to hardware requirements. Example: [“attention.wq”, “attention.wk”, “attention.wv”, “lm_head”]
Apply FP8 quantization to a PyTorch model using torchao.
This function can be called in two ways:
- With an FP8Config object: apply_fp8_to_model(model, config=fp8_config)
- With individual parameters: apply_fp8_to_model(model, filter_fqns=…, recipe_name=…, etc.)
Parameters:
The model to convert
FP8Config object containing all configuration. If provided, individual parameters are ignored.
List of module names to exclude from FP8 conversion
Recipe name for FP8 configuration (“tensorwise”, “rowwise”, etc.)
Whether to force recompute FP8 weight in backward pass
Whether to enable FSDP FP8 all-gather
Use emulation instead of hardware acceleration (for testing on older GPUs)
Whether FP8 quantization is enabled (only used when config is None)
Whether to precompute float8 scales dynamically
Returns: nn.Module
The model with FP8 linear layers (modified in-place)
Raises:
ImportError: If torchao is not installedValueError: If hardware doesn’t support FP8 and emulation is disabled
Build a FP8 config from configuration.
Parameters:
Configuration source for FP8 quantization. None returns a
disabled config.
Returns: FP8Config
FP8Config instance.
Create the BitsAndBytes configuration used for QLoRA.
Parameters:
The recipe’s quantization section. A ConfigNode satisfies
this protocol; only the two load_in_* flags and the four
bnb_4bit_* settings are read.
Returns: BitsAndBytesConfig | None
The BitsAndBytes configuration, or None when neither 4-bit nor
Raises:
ImportError: If bitsandbytes or transformers is unavailable.ValueError: If the quantization section omits both load flags.
Create a FP8Config from a mapping-like configuration source.
Parameters:
Configuration source for FP8 quantization. A ConfigNode
satisfies this protocol.
Returns: FP8Config
FP8Config instance.
Verify that FP8 conversion was successful by counting converted modules.
Parameters:
The model to verify
Returns: _FP8ConversionStats
Dict with conversion statistics
Return whether any module carries a BitsAndBytes quantized state.
Parameters:
Model to inspect.
Returns: bool
True when at least one module has a quant_state of type