nemo_automodel.components.models.minimax_m3_vl.kernels.msa_backward_preprocess_sm100
nemo_automodel.components.models.minimax_m3_vl.kernels.msa_backward_preprocess_sm100
Delta preprocess for the MiniMax M3 MSA SM100 backward.
Computes delta[t, h] = sum_d out[t, h, d] * grad_out[t, h, d] in FP32 from
BF16 inputs, which the backward subtracts from dP to form dS. One CTA of
256 threads reduces a [128, 128] tile of one head; rows past T are
predicated off.
Adapted from FlashAttention’s CuTe DSL backward preprocess
(flash_attn/cute/flash_bwd_preprocess.py, Copyright (c) 2025, Jay Shah,
Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, Tri Dao;
BSD-3-Clause), specialized to the fixed [T, 64, 128] MSA layout.
Module Contents
Classes
Functions
Data
API
Row-wise sum(O * dO) over head_dim for the THD contract.
Compile the delta reduction once per process with a dynamic token count.
Enqueue delta[t, h] = sum_d out[t, h, d] * grad_out[t, h, d].
Parameters:
Contiguous BF16 [T, 64, 128] forward output.
Contiguous BF16 [T, 64, 128] output gradient, on the same device.
Contiguous FP32 [T, 64] written in place.