bridge.models.inkling.inkling2_quantization#
NVFP4 and MXFP8 expert weight formats of Inkling2 checkpoints.
NVFP4 experts store packed E2M1 values (uint8, low nibble first), one
FP8 E4M3 scale per 16 input channels, and an outer scale
amax[expert] / outer_scale_denominator. MXFP8 experts store FP8 E4M3
values with one UE8M0 power-of-two scale per 32 input channels. Bridge imports
both as BF16 and, on export, quantizes them back against the source scales,
which reproduces the source bytes for unchanged weights. A plain export
(an explicit export dtype) writes the experts unquantized instead, and
align_precision_config then lists them under their written dtype in
hf_quant_config.json.
Module Contents#
Classes#
Fused |
Functions#
Dequantize one NVFP4 matrix to FP32. |
|
Quantize |
|
Dequantize one MXFP8 matrix ( |
|
Quantize |
|
Return |
|
Return the scale tensors stored next to a quantized expert weight. |
|
Make an |
Data#
API#
- bridge.models.inkling.inkling2_quantization.NVFP4_BLOCK#
16
- bridge.models.inkling.inkling2_quantization.MXFP8_BLOCK#
32
- bridge.models.inkling.inkling2_quantization._E2M1_MAGNITUDES#
(0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0)
- bridge.models.inkling.inkling2_quantization._e2m1_table(device: torch.device) torch.Tensor#
- bridge.models.inkling.inkling2_quantization.dequantize_nvfp4(
- packed: torch.Tensor,
- scale: torch.Tensor,
- amax: torch.Tensor,
- denominator: torch.Tensor,
Dequantize one NVFP4 matrix to FP32.
- Parameters:
packed –
[N, K / 2]uint8codes, element2iin the low nibble.scale –
[N, K / 16]FP8 E4M3 block scales.amax – Scalar FP32 outer amax of this expert.
denominator – Scalar FP32 outer-scale denominator.
- Returns:
[N, K]FP32 weights.
- bridge.models.inkling.inkling2_quantization.quantize_nvfp4_like(
- weight: torch.Tensor,
- scale: torch.Tensor,
- amax: torch.Tensor,
- denominator: torch.Tensor,
Quantize
[N, K]weights to packed NVFP4 codes against existing scales.Each value is divided by its block’s dequantization factor and rounded to the nearest E2M1 magnitude (saturating at 6), preserving the sign of zero.
- bridge.models.inkling.inkling2_quantization.dequantize_mxfp8(
- values: torch.Tensor,
- scale: torch.Tensor,
Dequantize one MXFP8 matrix (
[N, K]E4M3 values,[N, K / 32]UE8M0 scales) to FP32.
- bridge.models.inkling.inkling2_quantization.quantize_mxfp8_like(
- weight: torch.Tensor,
- scale: torch.Tensor,
Quantize
[N, K]weights to E4M3 against existing UE8M0 block scales.
- bridge.models.inkling.inkling2_quantization.expert_format(
- name: str,
- keys: collections.abc.Mapping[str, object] | set[str],
Return
"nvfp4","mxfp8", orNonefor an expert weight key.
- bridge.models.inkling.inkling2_quantization.companion_keys(name: str, fmt: str | None) tuple[str, ...]#
Return the scale tensors stored next to a quantized expert weight.
- bridge.models.inkling.inkling2_quantization.align_precision_config(
- config: dict,
- written_dtypes: collections.abc.Mapping[str, str],
Make an
hf_quant_config.jsonprecision config match the dtypes an export wrote.The config assigns every tensor to a parameter group whose
output_dtypeis its storage dtype. A tensor written in another dtype, such as an NVFP4 expert exported in BF16, moves to the plain group of that dtype; plain groups carry no quantization settings, so its scale entries go too. Groups left empty are removed, and themtpsection, which describes the separate draft file, is left unchanged.- Parameters:
config – Parsed
hf_quant_config.json, updated in place.written_dtypes – Torch dtype name (
"bfloat16") of every exported tensor.
- Returns:
Whether
configchanged.- Raises:
ValueError – If a tensor was written in a dtype other than BF16 or FP32, which a plain parameter group cannot describe.
- class bridge.models.inkling.inkling2_quantization.LazyQuantizedExperts(source, name: str, fmt: str)#
Fused
[E, N, K]expert weights that dequantize one expert at a time.Importing a whole fused expert tensor at once would hold every expert of every layer in memory. Indexing reads one expert’s slice from the safetensors shard and returns it in BF16.
Initialization
- ndim#
3
- _slice(key: str)#
- _small(key: str) torch.Tensor#
- property shape: torch.Size#
Return the dequantized fused shape
[E, N, K].
- __getitem__(expert: int) torch.Tensor#