Skip to navigation
DevelopmentAPI ReferenceFull Library ReferenceNemo AutomodelNemo AutomodelComponentsModelsKimi K3Kda Fusednemo_automodel.components.models.kimi_k3.kda_fused.chunk_kda_bwd_triton

nemo_automodel.components.models.kimi_k3.kda_fused.chunk_kda_bwd_triton

View as Markdown

Chunked Kimi Delta Attention BACKWARD in Triton for the Kimi-K3 call. From the raw forward inputs q, k, v, g, beta, cu_seqlens and the output gradient do it recomputes the forward intermediates and produces dq, dk, dv (bf16) and dg, dbeta (fp32) — nothing is saved from the forward. Measured on GB200 against FLA 0.4.2’s autograd backward (headline mode with saved intermediates) at the Kimi-K3 layer-microbatch shape: 4.25-4.39 ms vs 4.89-4.95 ms, every output within per-output tolerances (dq/dk/dv/dg atol 2e-2 rtol 2e-2, dbeta atol 2e-1 rtol 5e-2, matched ratio >= 0.99), identical numerics on document layouts it was not tuned on. Entry point: run.

Module Contents

Functions

NameDescription
_bwd_local-
_bwd_qkg-
_inv32(I + X)^-1 for a strictly-lower-triangular 32x32 X, via the doubling
_l2n-
_prep-
_scan-
runBackward of the K3 chunk_kda call (destination-passing): fills dq, dk, dv (bf16) and dg, dbeta (fp32).

Data

CLAMP

HAVE_TRITON

LOG2E

API

nemo_automodel.components.models.kimi_k3.kda_fused.chunk_kda_bwd_triton._bwd_local(
DO,
VN,
DVN,
VV,
KE,
PM,
GM,
BETA,
HS,
DHT,
CS,
CL,
DV,
DBETA,
DQG,
DKG,
DKE,
DAQ,
DGM,
DGL,
H: triton.language.constexpr,
K: triton.language.constexpr,
V: triton.language.constexpr,
BT: triton.language.constexpr,
BV: triton.language.constexpr
)
nemo_automodel.components.models.kimi_k3.kda_fused.chunk_kda_bwd_triton._bwd_qkg(
Q,
KK,
GC,
GB,
GLF,
RQ,
RK,
DQG,
DKG,
DKE,
DAQ,
DGM,
DGL,
CS,
CL,
DQ,
DK,
DG,
scale,
H: triton.language.constexpr,
K: triton.language.constexpr,
BT: triton.language.constexpr,
BC: triton.language.constexpr,
BK: triton.language.constexpr,
QSCALE: triton.language.constexpr
)
nemo_automodel.components.models.kimi_k3.kda_fused.chunk_kda_bwd_triton._inv32(
Xb,
i32,
BC: triton.language.constexpr
)

(I + X)^-1 for a strictly-lower-triangular 32x32 X, via the doubling identity (I+X)^-1 = (I-X)(I+X^2)(I+X^4)(I+X^8)(I+X^16).

FOUR squarings, not five. After s steps the running product equals sum_{j=0}^{2^(s+1)-1} (-X)^j; a 32x32 strictly-lower X has X^32 = 0, so s=4 (terms 0..31) is EXACT and a 5th step would multiply by I+X^32 = I. The 64x64 version needs terms 0..63, hence the 5 steps in the current code. Verified: s=4 -> err 1.6e-14 vs exact, s=3 -> 5.8e-5 (insufficient).

nemo_automodel.components.models.kimi_k3.kda_fused.chunk_kda_bwd_triton._l2n(
Q,
KK,
DQ,
DK,
RQ,
RK,
N,
BR: triton.language.constexpr,
K: triton.language.constexpr
)
nemo_automodel.components.models.kimi_k3.kda_fused.chunk_kda_bwd_triton._prep(
Q,
KK,
G,
BETA,
CS,
CL,
QG,
KE,
KG,
W,
PM,
GM,
AQ,
GL,
RQ,
RK,
GC,
GB,
scale,
H: triton.language.constexpr,
K: triton.language.constexpr,
V: triton.language.constexpr,
BT: triton.language.constexpr,
BC: triton.language.constexpr,
QSCALE: triton.language.constexpr
)
nemo_automodel.components.models.kimi_k3.kda_fused.chunk_kda_bwd_triton._scan(
W,
PM,
BETA,
VV,
KG,
GL,
HS,
VN,
QG,
AQ,
DO,
DHT,
DVN,
CS,
CL,
DF,
DN,
H: triton.language.constexpr,
K: triton.language.constexpr,
V: triton.language.constexpr,
BT: triton.language.constexpr,
BV: triton.language.constexpr
)
nemo_automodel.components.models.kimi_k3.kda_fused.chunk_kda_bwd_triton.run(
q,
k,
v,
g,
beta,
cu_seqlens,
do,
dq,
dk,
dv,
dg,
dbeta,
cu_seqlens_cpu = None
)

Backward of the K3 chunk_kda call (destination-passing): fills dq, dk, dv (bf16) and dg, dbeta (fp32).

q, k: [1, T, H, 128] bf16; v: [1, T, H, 128] bf16; g: [1, T, H, 128] fp32 log gate in (-5, 0); beta: [1, T, H] fp32; cu_seqlens: [N + 1] int32 document offsets; do: [1, T, H, 128] bf16 gradient of the output. Every forward intermediate is recomputed here. The per-chunk metadata needs the document offsets on the host: pass them as cu_seqlens_cpu (a sequence of ints, e.g. derived from a dense batch’s shape) to avoid the device->host sync that cu_seqlens.tolist() otherwise costs — the same contract as FLA’s own cu_seqlens_cpu.

nemo_automodel.components.models.kimi_k3.kda_fused.chunk_kda_bwd_triton.CLAMP = tl.constexpr(126.0)
nemo_automodel.components.models.kimi_k3.kda_fused.chunk_kda_bwd_triton.HAVE_TRITON = True
nemo_automodel.components.models.kimi_k3.kda_fused.chunk_kda_bwd_triton.LOG2E = tl.constexpr(1.4426950408889634)