> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/automodel/v0.4/nemo-automodel/nemo_automodel/components/quantization/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/automodel/_mcp/server.

# nemo_automodel.components.quantization

## Submodules

* **[`nemo_automodel.components.quantization.fp8`](/nemo-automodel/nemo_automodel/components/quantization/fp8)**
* **[`nemo_automodel.components.quantization.mxfp4`](/nemo-automodel/nemo_automodel/components/quantization/mxfp4)**
* **[`nemo_automodel.components.quantization.qat`](/nemo-automodel/nemo_automodel/components/quantization/qat)**
* **[`nemo_automodel.components.quantization.qlora`](/nemo-automodel/nemo_automodel/components/quantization/qlora)**

## Package Contents

### Classes

| Name                                                                 | Description                                  |
| -------------------------------------------------------------------- | -------------------------------------------- |
| [`FP8Config`](#nemo_automodel-components-quantization-fp8-FP8Config) | Configuration for FP8 quantization settings. |

### Functions

| Name                                                                                                     | Description                                                              |
| -------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------ |
| [`apply_fp8_to_model`](#nemo_automodel-components-quantization-fp8-apply_fp8_to_model)                   | Apply FP8 quantization to a PyTorch model using torchao.                 |
| [`build_fp8_config`](#nemo_automodel-components-quantization-fp8-build_fp8_config)                       | Build a FP8 config from configuration.                                   |
| [`create_bnb_config`](#nemo_automodel-components-quantization-qlora-create_bnb_config)                   | Create BitsAndBytes config for quantization.                             |
| [`create_fp8_config_from_dict`](#nemo_automodel-components-quantization-fp8-create_fp8_config_from_dict) | Create a FP8Config from a dictionary.                                    |
| [`verify_fp8_conversion`](#nemo_automodel-components-quantization-fp8-verify_fp8_conversion)             | Verify that FP8 conversion was successful by counting converted modules. |
| [`verify_qlora_quantization`](#nemo_automodel-components-quantization-qlora-verify_qlora_quantization)   | Verify that the model has been properly quantized.                       |

### Data

[`HAVE_TORCHAO`](#nemo_automodel-components-quantization-fp8-HAVE_TORCHAO)

### API

```python
class nemo_automodel.components.quantization.fp8.FP8Config(
    enabled: bool = False,
    recipe_name: typing.Literal['tensorwise', 'rowwise', 'rowwise_with_gw_hp'] | None = None,
    enable_fsdp_float8_all_gather: bool = False,
    precompute_float8_dynamic_scale_for_fsdp: bool = False,
    force_recompute_fp8_weight_in_bwd: bool = False,
    filter_fqns: typing.List[str] = None,
    emulate: bool = False
)
```

Dataclass

Configuration for FP8 quantization settings.

**`filter_fqns`** `List[str] = filter_fqns or []`

List of fully qualified names of modules to skip applying float8 training to.
nn.Linear modules with any dim size not divisible by 16 are always skipped due to hardware requirements.
Example: \["attention.wq", "attention.wk", "attention.wv", "lm\_head"]

---

```python
nemo_automodel.components.quantization.fp8.FP8Config.to_dict()
```

```python
nemo_automodel.components.quantization.fp8.apply_fp8_to_model(
    model: torch.nn.Module,
    config: nemo_automodel.components.quantization.fp8.FP8Config | None = None,
    filter_fqns: typing.List[str] | None = None,
    recipe_name: str | None = None,
    force_recompute_fp8_weight_in_bwd: bool = False,
    enable_fsdp_float8_all_gather: bool = False,
    emulate: bool = False,
    enabled: bool = True,
    precompute_float8_dynamic_scale_for_fsdp: bool = False
) -> torch.nn.Module
```

Apply FP8 quantization to a PyTorch model using torchao.

This function can be called in two ways:

1. With an FP8Config object: apply\_fp8\_to\_model(model, config=fp8\_config)
2. With individual parameters: apply\_fp8\_to\_model(model, filter\_fqns=..., recipe\_name=..., etc.)

**Parameters:**

**`model`** `nn.Module`

The model to convert

---

**`config`** `FP8Config | None` — default: None

FP8Config object containing all configuration. If provided, individual
parameters are ignored.

---

**`filter_fqns`** `List[str] | None` — default: None

List of module names to exclude from FP8 conversion

---

**`recipe_name`** `str | None` — default: None

Recipe name for FP8 configuration ("tensorwise", "rowwise", etc.)

---

**`force_recompute_fp8_weight_in_bwd`** `bool` — default: False

Whether to force recompute FP8 weight in backward pass

---

**`enable_fsdp_float8_all_gather`** `bool` — default: False

Whether to enable FSDP FP8 all-gather

---

**`emulate`** `bool` — default: False

Use emulation instead of hardware acceleration (for testing on older GPUs)

---

**`enabled`** `bool` — default: True

Whether FP8 quantization is enabled (only used when config is None)

---

**`precompute_float8_dynamic_scale_for_fsdp`** `bool` — default: False

Whether to precompute float8 scales dynamically

---

**Returns:** `nn.Module`

The model with FP8 linear layers (modified in-place)

**Raises:**

* `ImportError`: If torchao is not installed
* `ValueError`: If hardware doesn't support FP8 and emulation is disabled

```python
nemo_automodel.components.quantization.fp8.build_fp8_config(
    cfg: typing.Dict[str, typing.Any] | None
) -> nemo_automodel.components.quantization.fp8.FP8Config
```

Build a FP8 config from configuration.

**Parameters:**

**`cfg`** `Dict[str, Any] | None`

Configuration dictionary for FP8 quantization.

---

**Returns:** `FP8Config`

FP8Config instance.

```python
nemo_automodel.components.quantization.qlora.create_bnb_config(
    config: dict
)
```

Create BitsAndBytes config for quantization.

```python
nemo_automodel.components.quantization.fp8.create_fp8_config_from_dict(
    config_dict: typing.Dict[str, typing.Any]
) -> nemo_automodel.components.quantization.fp8.FP8Config
```

Create a FP8Config from a dictionary.

**Parameters:**

**`config_dict`** `Dict[str, Any]`

Dictionary containing FP8 configuration.

---

**Returns:** `FP8Config`

FP8Config instance.

```python
nemo_automodel.components.quantization.fp8.verify_fp8_conversion(
    model: torch.nn.Module
) -> dict
```

Verify that FP8 conversion was successful by counting converted modules.

**Parameters:**

**`model`** `nn.Module`

The model to verify

---

**Returns:** `dict`

Dict with conversion statistics

```python
nemo_automodel.components.quantization.qlora.verify_qlora_quantization(
    model
) -> bool
```

Verify that the model has been properly quantized.

```python
nemo_automodel.components.quantization.fp8.HAVE_TORCHAO = True
```

## Docs

- [nemo_automodel.components.quantization.fp8](https://docs.nvidia.com/nemo/automodel/v0.4/nemo-automodel/nemo_automodel/components/quantization/fp8.md)
- [nemo_automodel.components.quantization.mxfp4](https://docs.nvidia.com/nemo/automodel/v0.4/nemo-automodel/nemo_automodel/components/quantization/mxfp4.md)
- [nemo_automodel.components.quantization.qat](https://docs.nvidia.com/nemo/automodel/v0.4/nemo-automodel/nemo_automodel/components/quantization/qat.md)
- [nemo_automodel.components.quantization.qlora](https://docs.nvidia.com/nemo/automodel/v0.4/nemo-automodel/nemo_automodel/components/quantization/qlora.md)