Skip to navigation

nemo_automodel.components.quantization

View as MarkdownOpen in Claude

Submodules

Package Contents

Classes

NameDescription
FP8ConfigConfiguration for FP8 quantization settings.

Functions

NameDescription
apply_fp8_to_modelApply FP8 quantization to a PyTorch model using torchao.
build_fp8_configBuild a FP8 config from configuration.
create_bnb_configCreate the BitsAndBytes configuration used for QLoRA.
create_fp8_config_from_dictCreate a FP8Config from a mapping-like configuration source.
verify_fp8_conversionVerify that FP8 conversion was successful by counting converted modules.
verify_qlora_quantizationReturn whether any module carries a BitsAndBytes quantized state.

Data

HAVE_TORCHAO

API

class nemo_automodel.components.quantization.fp8.FP8Config(
enabled: bool = False,
enable_fsdp_float8_all_gather: bool = False,
precompute_float8_dynamic_scale_for_fsdp: bool = False,
force_recompute_fp8_weight_in_bwd: bool = False,
filter_fqns: list[str] | None = None,
emulate: bool = False
)
Dataclass

Configuration for FP8 quantization settings.

filter_fqns
list[str] = filter_fqns or []

List of fully qualified names of modules to skip applying float8 training to. nn.Linear modules with any dim size not divisible by 16 are always skipped due to hardware requirements. Example: [“attention.wq”, “attention.wk”, “attention.wv”, “lm_head”]

nemo_automodel.components.quantization.fp8.FP8Config.to_dict() -> nemo_automodel.components.quantization.fp8._FP8ConfigDict
nemo_automodel.components.quantization.fp8.apply_fp8_to_model(
model: torch.nn.Module,
filter_fqns: list[str] | None = None,
force_recompute_fp8_weight_in_bwd: bool = False,
enable_fsdp_float8_all_gather: bool = False,
emulate: bool = False,
enabled: bool = True,
precompute_float8_dynamic_scale_for_fsdp: bool = False
) -> torch.nn.Module

Apply FP8 quantization to a PyTorch model using torchao.

This function can be called in two ways:

  1. With an FP8Config object: apply_fp8_to_model(model, config=fp8_config)
  2. With individual parameters: apply_fp8_to_model(model, filter_fqns=…, recipe_name=…, etc.)

Parameters:

model
nn.Module

The model to convert

config
FP8Config | NoneDefaults to None

FP8Config object containing all configuration. If provided, individual parameters are ignored.

filter_fqns
list[str] | NoneDefaults to None

List of module names to exclude from FP8 conversion

recipe_name
FP8RecipeName | NoneDefaults to None

Recipe name for FP8 configuration (“tensorwise”, “rowwise”, etc.)

force_recompute_fp8_weight_in_bwd
boolDefaults to False

Whether to force recompute FP8 weight in backward pass

enable_fsdp_float8_all_gather
boolDefaults to False

Whether to enable FSDP FP8 all-gather

emulate
boolDefaults to False

Use emulation instead of hardware acceleration (for testing on older GPUs)

enabled
boolDefaults to True

Whether FP8 quantization is enabled (only used when config is None)

precompute_float8_dynamic_scale_for_fsdp
boolDefaults to False

Whether to precompute float8 scales dynamically

Returns: nn.Module

The model with FP8 linear layers (modified in-place)

Raises:

  • ImportError: If torchao is not installed
  • ValueError: If hardware doesn’t support FP8 and emulation is disabled
nemo_automodel.components.quantization.fp8.build_fp8_config(

Build a FP8 config from configuration.

Parameters:

cfg
_FP8ConfigSource | None

Configuration source for FP8 quantization. None returns a disabled config.

Returns: FP8Config

FP8Config instance.

nemo_automodel.components.quantization.qlora.create_bnb_config(
) -> transformers.BitsAndBytesConfig | None

Create the BitsAndBytes configuration used for QLoRA.

Parameters:

config
_QuantizationConfig

The recipe’s quantization section. A ConfigNode satisfies this protocol; only the two load_in_* flags and the four bnb_4bit_* settings are read.

Returns: BitsAndBytesConfig | None

The BitsAndBytes configuration, or None when neither 4-bit nor

Raises:

  • ImportError: If bitsandbytes or transformers is unavailable.
  • ValueError: If the quantization section omits both load flags.
nemo_automodel.components.quantization.fp8.create_fp8_config_from_dict(

Create a FP8Config from a mapping-like configuration source.

Parameters:

config_dict
_FP8ConfigSource

Configuration source for FP8 quantization. A ConfigNode satisfies this protocol.

Returns: FP8Config

FP8Config instance.

nemo_automodel.components.quantization.fp8.verify_fp8_conversion(
model: torch.nn.Module

Verify that FP8 conversion was successful by counting converted modules.

Parameters:

model
nn.Module

The model to verify

Returns: _FP8ConversionStats

Dict with conversion statistics

nemo_automodel.components.quantization.qlora.verify_qlora_quantization(
model: torch.nn.Module
) -> bool

Return whether any module carries a BitsAndBytes quantized state.

Parameters:

model
nn.Module

Model to inspect.

Returns: bool

True when at least one module has a quant_state of type

nemo_automodel.components.quantization.fp8.HAVE_TORCHAO = True