bridge.models.inkling.inkling2_quantization#

NVFP4 and MXFP8 expert weight formats of Inkling2 checkpoints.

NVFP4 experts store packed E2M1 values (uint8, low nibble first), one FP8 E4M3 scale per 16 input channels, and an outer scale amax[expert] / outer_scale_denominator. MXFP8 experts store FP8 E4M3 values with one UE8M0 power-of-two scale per 32 input channels. Bridge imports both as BF16 and, on export, quantizes them back against the source scales, which reproduces the source bytes for unchanged weights. A plain export (an explicit export dtype) writes the experts unquantized instead, and align_precision_config then lists them under their written dtype in hf_quant_config.json.

Module Contents#

Classes#

LazyQuantizedExperts

Fused [E, N, K] expert weights that dequantize one expert at a time.

Functions#

_e2m1_table

dequantize_nvfp4

Dequantize one NVFP4 matrix to FP32.

quantize_nvfp4_like

Quantize [N, K] weights to packed NVFP4 codes against existing scales.

dequantize_mxfp8

Dequantize one MXFP8 matrix ([N, K] E4M3 values, [N, K / 32] UE8M0 scales) to FP32.

quantize_mxfp8_like

Quantize [N, K] weights to E4M3 against existing UE8M0 block scales.

expert_format

Return "nvfp4", "mxfp8", or None for an expert weight key.

companion_keys

Return the scale tensors stored next to a quantized expert weight.

align_precision_config

Make an hf_quant_config.json precision config match the dtypes an export wrote.

Data#

API#

bridge.models.inkling.inkling2_quantization.NVFP4_BLOCK#

16

bridge.models.inkling.inkling2_quantization.MXFP8_BLOCK#

32

bridge.models.inkling.inkling2_quantization._E2M1_MAGNITUDES#

(0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0)

bridge.models.inkling.inkling2_quantization._e2m1_table(device: torch.device) → torch.Tensor#
bridge.models.inkling.inkling2_quantization.dequantize_nvfp4(
packed: torch.Tensor,
scale: torch.Tensor,
amax: torch.Tensor,
denominator: torch.Tensor,
) → torch.Tensor#

Dequantize one NVFP4 matrix to FP32.

Parameters:
  • packed – [N, K / 2] uint8 codes, element 2i in the low nibble.

  • scale – [N, K / 16] FP8 E4M3 block scales.

  • amax – Scalar FP32 outer amax of this expert.

  • denominator – Scalar FP32 outer-scale denominator.

Returns:

[N, K] FP32 weights.

bridge.models.inkling.inkling2_quantization.quantize_nvfp4_like(
weight: torch.Tensor,
scale: torch.Tensor,
amax: torch.Tensor,
denominator: torch.Tensor,
) → torch.Tensor#

Quantize [N, K] weights to packed NVFP4 codes against existing scales.

Each value is divided by its block’s dequantization factor and rounded to the nearest E2M1 magnitude (saturating at 6), preserving the sign of zero.

bridge.models.inkling.inkling2_quantization.dequantize_mxfp8(
values: torch.Tensor,
scale: torch.Tensor,
) → torch.Tensor#

Dequantize one MXFP8 matrix ([N, K] E4M3 values, [N, K / 32] UE8M0 scales) to FP32.

bridge.models.inkling.inkling2_quantization.quantize_mxfp8_like(
weight: torch.Tensor,
scale: torch.Tensor,
) → torch.Tensor#

Quantize [N, K] weights to E4M3 against existing UE8M0 block scales.

bridge.models.inkling.inkling2_quantization.expert_format(
name: str,
keys: collections.abc.Mapping[str, object] | set[str],
) → str | None#

Return "nvfp4", "mxfp8", or None for an expert weight key.

bridge.models.inkling.inkling2_quantization.companion_keys(name: str, fmt: str | None) → tuple[str, ...]#

Return the scale tensors stored next to a quantized expert weight.

bridge.models.inkling.inkling2_quantization.align_precision_config(
config: dict,
written_dtypes: collections.abc.Mapping[str, str],
) → bool#

Make an hf_quant_config.json precision config match the dtypes an export wrote.

The config assigns every tensor to a parameter group whose output_dtype is its storage dtype. A tensor written in another dtype, such as an NVFP4 expert exported in BF16, moves to the plain group of that dtype; plain groups carry no quantization settings, so its scale entries go too. Groups left empty are removed, and the mtp section, which describes the separate draft file, is left unchanged.

Parameters:
  • config – Parsed hf_quant_config.json, updated in place.

  • written_dtypes – Torch dtype name ("bfloat16") of every exported tensor.

Returns:

Whether config changed.

Raises:

ValueError – If a tensor was written in a dtype other than BF16 or FP32, which a plain parameter group cannot describe.

class bridge.models.inkling.inkling2_quantization.LazyQuantizedExperts(source, name: str, fmt: str)#

Fused [E, N, K] expert weights that dequantize one expert at a time.

Importing a whole fused expert tensor at once would hold every expert of every layer in memory. Indexing reads one expert’s slice from the safetensors shard and returns it in BF16.

Initialization

ndim#

3

_slice(key: str)#
_small(key: str) → torch.Tensor#
property shape: torch.Size#

Return the dequantized fused shape [E, N, K].

__getitem__(expert: int) → torch.Tensor#