NVFP4 Attention QAT Backward
This is an experimental API and subject to change.
Overview
nvfp4_attention_qat_backward computes explicit Q, K, and V gradients for
scaled dot-product attention trained with NVFP4 fake quantization. Its Triton
reference backend is a port of FastVideo’s attention QAT backward at commit
e9bbaca07d511b2ee7e16474dae6f923426223dc:
The operation fake-quantizes Q, K, and V to the NVFP4 E2M1 data format with an E4M3 scale for every 16 values, then immediately dequantizes them for the attention computation. The probability matrix follows two paths:
- dQ and dK use the unquantized softmax probability, implementing the straight-through estimator (STE).
- dV uses the NVFP4 fake-quantized probability.
The backend="triton" implementation launches four kernels: fused Q fake-quantization/delta
preprocessing, K/V fake-quantization, dQ, and dK/dV. Causal backward skips
fully masked tiles while retaining the elementwise mask on the diagonal.
The local-scale conversion uses precise division so exact E2M1 midpoints
preserve round-to-nearest-even; no tolerance relaxation is required.
The production
non-causal SM100 and SM103 configurations use 64 by 64 tiles; other supported
Blackwell configurations use 32 by 32 tiles.
Installation
From a source checkout, install the CuTe DSL base dependencies, Triton, and the torch dependency group:
For a published wheel, install a CUDA-enabled torch build separately:
Triton 3.7 or newer is supported on Linux with Python 3.10 or newer for this API.
High-level wrapper
The result keys are dq_tensor, dk_tensor, and dv_tensor. Optional
preallocated tensors with those names can be passed to the wrapper. Pass a
cuda.CUstream as current_stream to order wrapper allocations and all
kernel launches on an explicit stream.
high_precision_o is not the probability-quantized user-visible QAT output.
It must be the matching softmax(Q_fake K_fake^T) @ V_fake value saved before
probability fake quantization. lse is the corresponding natural-log
log-sum-exp statistic. Supplying forward auxiliaries from a different
quantization recipe produces incorrect gradients.
Class API
Nvfp4AttentionQatBackward exposes explicit validation, compilation, and
execution. execute performs no allocations; the caller supplies contiguous
gradient buffers and a one-dimensional CUDA torch.uint8 workspace.
compile() materializes the selected backend’s shape- and architecture-specialized
kernels without launching them. execute() reuses those artifacts.
Automatic backend selection
Both the class constructor and wrapper default to backend="auto": prefer
FROST when its dependency, architecture and shape requirements are met;
otherwise select Triton during check_support(). Explicit backend="frost"
and backend="triton" force their respective implementations. Forced FROST
raises with the rejection reason rather than silently falling back.
FROST is the backend name; CuTe DSL is its kernel implementation technology.
Inspect op.selected_backend, op.fallback_reason, and
op.selected_head_chunk after check_support() to see the resolved plan.
Before selection these are None, None, and 0, respectively. op.backend
retains the requested policy. Selection is fixed for the lifetime of the plan;
construct a new object to change its declaration or policy. No selection,
compilation or fallback occurs inside execute(). Invalid arguments and
compiler, allocation or execution failures propagate; they are not support
rejections and do not trigger fallback. This is support-based dispatch,
not timing-based autotuning or a guarantee of a speedup for every shape.
The normal package dependencies remain required, including the base CuTe DSL
dependency used by APIBase. An installed DSL below FROST’s 4.7.0 floor (for
example 4.6.2) selects Triton without importing the FROST kernel; this does
not promise the API works with the base cutlass package completely absent.
FROST backend (SM100/SM103)
The FROST backend requires CuTe DSL >= 4.7.0, SM100 or SM103, BF16, D128,
B=1, equal query/KV head counts, noncausal attention, and equal positive
sequence lengths divisible by 256. With auto, unsupported FROST declarations
select Triton, which has wider support including tails. Neither route adds
padding or layout-conversion adapters.
This backend reuses the Triton Q/delta and KV quantizers, then launches two
two-CTA FROST kernels implemented in CuTe DSL: a dK/dV kernel that tiles over
KV (P fake-quantized for dV, dS consumed from shared memory for dK) and a dQ
kernel that tiles over Q and recomputes dS. Like the Triton backend it never
materializes an S-by-S intermediate, and every gradient is accumulated in a
fixed order, so results are bitwise reproducible. The quantizers write their
BSHD intermediates directly; the kernels address caller-owned BHSD dO/dQ/dK/dV
natively. All stages honor current_stream. Compilation is plan-time-only; no
mutable global configuration is switched between plans. The wrapper’s bounded
cache separates requested backend, head-chunk and workspace-limit plans.
The FROST workspace is the same as Triton’s and independent of head_chunk,
which only sets the launch granularity (head_chunk=0 launches all heads at
once; a positive divisor of H launches H/head_chunk times per kernel). With
auto, a valid chunk is unused if Triton is selected; forced triton
requires head_chunk=0. Total FROST workspace, in bytes, is:
Both APIs accept workspace_limit_bytes=None (no explicit scratch budget) or
a nonnegative integer. If the workspace exceeds the limit, support checking
raises before compilation or allocation (both backends need the same scratch).
The limit covers this API’s explicit scratch only, not inputs, outputs,
compiler runtime storage or total process memory. Selection never queries
fluctuating free GPU memory or recovers from an OOM by switching backend.
Outputs, workspace and inputs must not overlap; concurrent executions need separate output/workspace storage. The wrapper allocates workspace each call; use the class API for explicit reuse and CUDA Graph capture.
Both backends implement the same local-scale-floor NVFP4/STE contract.
The FROST path folds softmax_scale into dS before its BF16 store;
the Triton path applies it after gradient accumulation. Expect small BF16
rounding differences in dQ/dK, not bitwise identity.
benchmark/nvfp4_attention_qat/benchmark_backends.py checks correctness and
changed-input graph replay before timing complete backward with alternating
backend order. These are backward-component measurements, not model E2E.
Tensor contract
All tensors must use contiguous, 16-byte-aligned BHSD storage and reside on one
CUDA device. softmax_scale defaults to 1 / sqrt(128) and must match the
forward pass.
Current support and limitations
The following describes the overall API coverage provided by Triton, which
auto selects outside FROST’s narrower initial coverage listed above.
- GPU: SM100, SM103, SM120, and SM121 Blackwell.
- Attention: MHA with equal query and KV head counts; head dimension 128.
- Sequence lengths: self-attention and non-causal cross-attention, including non-aligned tails.
- Causal mode: self-attention only.
- Dtype: BF16 activations and FP32 LSE.
- Not implemented: GQA/MQA, dropout, padding or packed variable-length sequences, bias, local masks, and deterministic-mode selection.