Kimi Delta Attention in JAX
cudnn.linear_attention.jax_api.kimi_delta_attention returns (output, final_state_or_None) and
supports jax.jit and first-order reverse-mode differentiation. It uses the
existing Frost KDA kernels, including their backward, checkpoint recompute,
split scheduler, gate gradients, and grouped-head reductions.
This implementation is pinned to Frost. Other engines have different launch/runtime contracts; accepting their graph plans would not make them executable through this bridge. The existing torch API retains its engine routing.
Inputs and outputs
T is the packed token count, N the sequence count, and HO = max(HQ, HV).
cu_seqlens must start at zero, be nondecreasing, and end at T. Values are
runtime device data: changing sequence lengths without changing tensor shapes
does not require recompilation. The wrapper validates metadata, not boundary
values; it never copies boundaries to the host. Callers must supply valid
boundaries. Empty individual sequences preserve their initial state (or zero).
Enable jax_enable_x64 before constructing INT64 offsets; otherwise JAX may
downcast them to INT32. INT64 storage does not remove the signed-INT32 limits
on total tokens and scheduler counts.
The state is V-major. It is a differentiable array, not a mutable cache. Pass the
returned state explicitly to the next call, including inside jax.lax.scan.
The native engine’s paged-state and in-place state options remain supported for
torch; this JAX API does not expose state_indices or overwrite_initial_state.
Usage
Other GPUs may be visible. Concrete arrays run on the GPU that holds them, and all of
them must be on one supported GPU. Under jax.jit the arrays are tracers with no
device, so KDA targets JAX’s default device (jax.default_device(...), or the first
local device). That is where uncommitted computations run: keep the inputs on it.
CuTeDSL compiles for the first visible GPU unless CUTE_DSL_ARCH is set, so on a
node with mixed GPU types either list a supported GPU first in CUDA_VISIBLE_DEVICES
or set CUTE_DSL_ARCH (for example sm_100a). The tested combination is JAX 0.11.1, CuTeDSL 4.7.1,
SM100; broader version/architecture qualification is still required.
Safe-gate parameters are ordinary differentiable arguments:
For explicit backward:
An omitted final-state cotangent means zero. Gradients for absent initial state
or parameters are None. Boundaries and static options are not differentiable.
The explicit APIs do not themselves provide autodiff rules; the high-level API
installs custom_vjp using the existing backward kernels. Higher derivatives and
forward-mode differentiation are unsupported.
Pass gate_domain="linear" to either forward API when g contains decay
factors such as 0.9, rather than their natural logarithms. Backward and
jax.grad then return gradients with respect to those factors. Linear gates
cannot be combined with safe_gate=True.
checkpoint_every_n_tokens=64, for example, saves fewer recurrent states than
cadence 16. Backward uses Frost’s existing seeded recomputation to reconstruct
the intermediate states. This reduces saved residual storage, but backward
still needs workspace for the reconstructed states. Cadence 0 saves no
checkpoints; any positive multiple of 16 fitting signed INT32 is supported.
Bounded support
For a positive cadence C, checkpoint capacity is
[max(T // C + N, 1), HO, V, K] in q dtype. The valid
prefix contains sum(ceil(sequence_length / C)) rows; padding is initialized to
zero. The residual holds primals, static configuration and optional checkpoints.
It owns no graph handle, workspace or executable. Do not fabricate or edit
residuals: checkpoints must correspond to their saved primals and configuration.
Deferred: cuTile/other engine routing, SM90/SM120/SM107, indexed state pools,
zero total tokens, arbitrary physical strides, vmap, JVP, higher derivatives,
sharding/multiple GPUs, portable executable export, and full FLA compatibility.
JAX supplies compact row-major operands to the custom call; upstream transposes
may require materialization.
Execution
Frost chooses the schedule and workspace layout. JAX uses the same kernel configuration builders and multi-kernel hosts as the native executor, with XLA-owned outputs, per-call workspace and stream. Compilation metadata is cached; workspace is never shared between calls. Execution needs no torch import or Python callback.
SM103, additional CUDA/JAX/CuTeDSL versions, independent-stream stress and customer workloads still need qualification. The 10–20 us CPU dispatch target and JAX/native performance parity have not been established by numerical tests.
Tests
Run JAX coverage without the torch-based parent fixtures:
The suite covers forward/backward parity, state and gate gradients, repeated calls, packed boundaries, recurrent scans, command-buffer replay, concurrent dispatch and torch-free execution. Tests skip when JAX, a supported GPU or CuTeDSL >=4.7.0 is unavailable.
Run native cache and output-stride regressions from test/python: