Operations Overview
The API reference groups graph operations and experimental Python kernels by operation. Use the sidebar to find an operation and its related APIs.
- Attention
- Sparse Attention
- Linear Attention
- Matmul and GEMM Fusions
- Convolutions and Causal Conv1d
- Normalizations
- Block Scaling
- RoPE
- Pointwise and Reduction
Experimental Python APIs
FE-OSS APIs are experimental and subject to change.
The GEMM CuTeDSL APIs are type-erased and torch-lazy: torch is imported only when torch tensors are passed. JAX arrays are additionally accepted wherever the kernel’s tensor layouts are expressible as row-major arrays (each API’s page has a “JAX support” section with its exact contract):
- Dense fusions (amax, swiglu, srelu, dsrelu): full JAX eager support, plus
jax.jit-compatible XLA custom-call entry points for all four (built oncutlass.jax.cutlass_call/ CuTeDSL’s nativecutlass.jaxbridge; seegemm_amax.md“Using JAX arrays”). - Grouped / discrete-grouped: JAX eager support in discrete (pointer-array) weight modes — unfused grouped GEMM, glu/dglu (BF16), dsrelu (FP8), wgrad (BF16), and discrete-grouped swiglu/dswiglu (FP8) — plus a
jax.jit-compatible*_jax_sm100entry point for each of those same families (built oncutlass.jax.cutlass_call; each API page documents its exact jit contract). Contiguous grouped MXFP8 SwiGLU/dSwiGLU wrappers accept Torch tensors and canonical JAX arrays, including underjax.jit, with explicit FP8 output dtype andsf_vec_size=32for JAX. The unified GLU wrapper likewise dispatches canonical MXFP8 JAX inputs togrouped_gemm_glu_jax_sm100(SwiGLU/GeGLU), which runs the Rubin GLU kernel on SM107. The unified dGLU, wgrad and quant wrappers likewise dispatch canonical JAX inputs togrouped_gemm_dglu_jax_sm100(dSwiGLU/dGeGLU),grouped_gemm_wgrad_jax_sm100andgrouped_gemm_quant_jax_forward, which run the Rubin kernels on SM107. The grouped SReLU, dSReLU, GLU+Hadamard and GLU+Hadamard+quant wrappers dispatch togrouped_gemm_srelu_jax_forward,grouped_gemm_dsrelu_jax_sm100,grouped_gemm_glu_hadamard_jax_forwardandgrouped_gemm_glu_hadamard_quant_jax_forward(SM100+, as in Torch; GLU+Hadamard+quant runs its Rubin kernel on SM107). These canonical entries take dense weights in MXFP8 or packed FP4 (asuint8, with E4M3 scales per 16 for NVFP4), and GLU, dGLU and wgrad also take BF16; each page lists its formats. Existing Torch behavior and wrapper defaults are unchanged. Discrete block-scaled weights, column-major bias layouts and the op classes reject JAX with clear errors. - proj_rope_mxfp8: JAX eager support on both input paths with
w_out_in=True(the transposed [in, out] weight view is torch-only), plus thejax.jit-compatiblegemm_proj_rope_mxfp8_jax_sm100entry point.
New APIs follow the Python API Naming Conventions. The experimental API catalog includes:
-
Fused GDN Decode / Verify Steps as a FlashInfer Backend (SM100 / SM120) — the fused Gated DeltaNet decode step and MTP verify step of a Qwen 3.5-architecture dense model’s GDN sub-layer as one FROST launch each, registered as the
frost_febackend of FlashInfer’s experimental Qwen 3.5 GDN block (bf16 operands, fp32 state, FP8 output) -
Shared-Expert Decode Pair (SM100-class) — the bf16 shared expert next to a routed MoE at decode-time token counts (1 .. 24) as two FROST launches (gate_up + SiLU-mul with the gate logit, down + sigmoid gate), bitwise repeatable and batch-invariant
-
Attention Gate + Quantize Decode Step (SM100-class) — the sigmoid output gate of an attention block fused with the static FP8 (E4M3) quantization of the following projection’s input, at decode-time row counts (1 .. 160), as one FROST launch (bf16 in, E4M3 out), bitwise repeatable and batch-invariant
-
FP8 Projection for Decode Rows (SM100-class) — an FP8 E4M3 weight times FP8 E4M3 activations to bf16 at decode-time row counts (1 .. 32) as one FROST launch with split-K clusters (or one K slice per weight tile, without a cluster) and the weight stream ahead of the dependent-launch wait; the split is a numerics attribute fixed per entry, bitwise repeatable and batch-invariant
-
Add + RMSNorm + Quantize Decode Step (SM100-class) — the residual add, the RMSNorm and the quantization of the next projection’s input at decode-time row counts (1 .. 1024) as one FROST launch (bf16 in; bf16, FP8 E4M3 or NVFP4 out), bitwise repeatable and batch-invariant
-
SwiGLU FP4 Gate / Up Product for Decode Rows (SM100-class) — the gate / up product of a block-scaled FP4 (NVFP4) MLP with SiLU(gate) x up and the FP4 re-quantization of the result at decode-time row counts (1 .. 64) as one FROST launch, with the weight stream ahead of the dependent-launch wait; no numerics knob, bitwise repeatable and batch-invariant
-
Gated Attention Block (SM107) — projection, QK-norm + RoPE, SDPA, sigmoid gate, out projection as one FROST block (bf16 / FP8 / MXFP8, optional MXFP4 weights and NVFP4 / MXFP4 output); index-list sparse attention (
QsaSpec) with the in-block indexer’s decode and extend steps over the paged side caches, and write-through into a paged KV cache -
Grouped GEMM + sReLU (Unified) — optionally tanh soft-clamped via
tanh_clamp_scale -
Simplified Sparse Attention: attention and trainable indexer
-
Query Sparse Attention (QSA): attention and indexer entry points
-
SDPA Backward, d = 128, dense and THD MHA/GQA/MQA (SM100 / SM103)
-
[SDPA Backward, d = 256 and d in (256, 512] (SM107 / Rubin; SM100 / SM103 via the 2x2-datapath body)](/operations/Attention/sdpa_bwd_sm107)
-
Zero-centered RMSNorm training: deterministic Cake forward and backward.
-
Cake LM-head log-probabilities: prepared execution and JAX JIT/gradients.
-
Cake mHC training: explicit stages and JAX branch composition/gradients.
Installation and setup
All Frontend OSS APIs come installed with the nvidia-cudnn-frontend package. Except on Windows ARM64, nvidia-cutlass-dsl[cu13]>=4.8.0 and apache-tvm-ffi are required dependencies, so the CuTeDSL runtime comes with the base install (and the DSL pulls cuda-python in transitively):
On Windows ARM64, these two runtime dependencies are excluded from the base installation, so the base install does not enable the CuTeDSL APIs there.
(pip install nvidia-cudnn-frontend[cutedsl] still works; the cutedsl extra now names only cuda-python. A few APIs still want extras of their own — the cuTile linear-attention engines need [cutile].)
The Triton NVFP4 attention QAT backward API additionally requires the
triton extra. Its API page documents the framework and GPU requirements.
Those required dependencies are framework-neutral. Install your tensor framework separately — from a checkout, the PEP 735 dependency groups pin the right companion packages:
For the published wheel, install PyTorch with pip install torch torch-c-dlpack-ext.
All CuTeDSL JAX entry points require JAX >=0.9.1. Install GPU-enabled JAX separately; for CUDA 13:
After installation, you can import the APIs directly from the cudnn package, i.e. from cudnn import {your_operation}
API Usage
The causal-convolution family exposes semantic PyTorch operations only through
cudnn.ops.causal_conv1d and cudnn.ops.causal_conv1d_update; they return
ordinary tensors and keep prepared kernel objects private. The wrapper and
class conventions below apply to prepared frontend-only kernel APIs.
Most compiler-style operations expose the following two APIs. Functional autograd integrations such as BSA and Flex Attention instead document their own wrapper and reusable-plan lifecycle on their operation pages.
1. High-level wrapper
- Single pythonic function call
- Allocates and returns output tensors
- Returns outputs as a
TupleDict(supports both dictionary-style key access and tuple unpacking) - No explicit compilation step – internally caches compiled kernels via a simple dictionary lookup
- When to use:
- Fast prototyping and common cases
- You want automatic allocation and minimal boilerplate
- You are okay with the library managing the compiled-kernel cache
2. Class API
- Explicit lifecycle with compile and execute steps
- Reusable object with underlying compiled kernel for multiple executions
- Requires preallocated output tensors
- When to use:
- You need to reuse a compiled kernel across many calls
- You want explicit control over compilation and lifecycle management
Methods:
check_support()– validates target problem configuration (i.e. tensor shapes, tensor strides, dtypes, tiling/cluster/kernel configurations, environment, etc.)compile()– compiles the kernel with the provided sample tensors and parameters.execute(inputs, ..., outputs, ..., current_stream)– runs the kernel with the provided inputs and outputs.
Common Parameters and Conventions
-
CUDA stream (
current_streamin class API,streamin wrapper)- The cuda stream to use for operation kernel execution.
- Default: None (uses default stream)
-
Determinism
- Several kernels reduce with cross-CTA atomics, so those outputs are not bit-exact run to
run. Kernels react to
torch.use_deterministic_algorithms(True): they switch to a deterministic path where one exists, and otherwise raise (or warn, underwarn_only) rather than silently return a non-reproducible result. - Where a deterministic path exists it can also be selected per call with
deterministic=True, independent of the torch setting. See Grouped GEMM + dsReLU.
- Several kernels reduce with cross-CTA atomics, so those outputs are not bit-exact run to
run. Kernels react to
File structure and examples
-
All FE OSS APIs are implemented in the
python/cudnndirectory. -
Correctness tests/samples are implemented under
test/python/<operation>/cutedsl(JAX bindings undertest/python/<operation>/jax). -
Cake GDN training: opt-in packed BF16 forward and backward.
-
Cake chunk-64 KDA training: opt-in serial forward and backward.