> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/cudnn/latest/operations/Matmul/llms.txt. # Matmul The Matmul operation computes: $$ C[M, N] = A[M, K] * B[K, N] $$ Last two dimensions of input dimensions are interpreted as M, N, K. All other preceding dimensions are interpreted as batch dimensions. The operation also has broadcasting capabilities which are described in [cudnn backend's matmul operation](https://docs.nvidia.com/deeplearning/cudnn/backend/latest/api/cudnn-graph-library.html#cudnn-backend-operation-matmul-descriptor). ### Tensor rank Which ranks build depends on the engine that serves the graph: - **cuDNN backend engines** take exactly rank-3 tensors: A `[B, M, K]`, B `[B, K, N]` and C `[B, M, N]`. A batch extent of 1 on A or B broadcasts across the other operand's batch. Rank-2 and rank-4 or higher graphs are declined when the plans are built with `cudnnGraphNotSupportedError`. Write the plain product `C[M, N] = A[M, K] * B[K, N]` as `[1, M, K] x [1, K, N] -> [1, M, N]`, and fold several batch dimensions into one before building the graph. C must be rank 3 as well: on SM90, rank-3 inputs with a rank-2 C were declined. - **The FROST GEMM engine** (opt-in with `CUDNN_FRONTEND_ENABLE_FROST_ENGINES=1` on the GPUs it serves) supports rank-2 graphs and folds leading batch dimensions for rank-4 or higher graphs when they can be represented by one affine batch stride. After padding shorter batch prefixes with leading ones, each input's batch prefix must either match the output batch prefix or consist entirely of ones. Partial or crossed broadcasting, such as A `[2, 1, M, K]` and B `[1, 3, K, N]`, is not supported. Non-affine batch layouts are also declined rather than copied. Measured with cuDNN 9.27 and fp16 inputs. On SM90 and SM103 the backend engines served only rank 3. On SM103 with FROST enabled, ranks 2 through 5 were served. ## C++ API ``` std::shared_ptr Matmul(std::shared_ptr a, std::shared_ptr b, Matmul_attributes); ``` Matmul attributes is a lightweight structure with setters: ``` Matmul_attributes& set_name(std::string const&) Matmul_attributes& set_compute_data_type(DataType_t value) ``` ## Python API - matmul - A - B - name - compute_data_type ## Docs - [Discrete Grouped GEMM + dSwiGLU (SM100)](https://docs.nvidia.com/cudnn/latest/operations/Matmul/gemm_fusions/discrete_grouped_gemm_dswiglu.md) - [Discrete Grouped GEMM + SwiGLU (SM100)](https://docs.nvidia.com/cudnn/latest/operations/Matmul/gemm_fusions/discrete_grouped_gemm_swiglu.md) - [FP8 Projection for Decode Rows (SM100-class)](https://docs.nvidia.com/cudnn/latest/operations/Matmul/gemm_fusions/fp8_projection_decode.md) - [SwiGLU FP4 Gate / Up Product for Decode Rows (SM100-class)](https://docs.nvidia.com/cudnn/latest/operations/Matmul/gemm_fusions/swiglu_fp4_decode.md) - [GEMM + Amax (SM100)](https://docs.nvidia.com/cudnn/latest/operations/Matmul/gemm_fusions/gemm_amax.md) - [GEMM + dsReLU (SM100)](https://docs.nvidia.com/cudnn/latest/operations/Matmul/gemm_fusions/gemm_dsrelu.md) - [GEMM + RoPE + MXFP8 Projection (SM100)](https://docs.nvidia.com/cudnn/latest/operations/Matmul/gemm_fusions/gemm_proj_rope_mxfp8.md) - [GEMM + sReLU (SM100)](https://docs.nvidia.com/cudnn/latest/operations/Matmul/gemm_fusions/gemm_srelu.md) - [GEMM + SwiGLU (SM100)](https://docs.nvidia.com/cudnn/latest/operations/Matmul/gemm_fusions/gemm_swiglu.md) - [Grouped GEMM (SM100 BF16)](https://docs.nvidia.com/cudnn/latest/operations/Matmul/gemm_fusions/grouped_gemm.md) - [Grouped GEMM + dGLU (SM100)](https://docs.nvidia.com/cudnn/latest/operations/Matmul/gemm_fusions/grouped_gemm_dglu.md) - [Grouped GEMM + dsReLU (SM100)](https://docs.nvidia.com/cudnn/latest/operations/Matmul/gemm_fusions/grouped_gemm_dsrelu.md) - [Grouped GEMM + dSwiGLU (SM100)](https://docs.nvidia.com/cudnn/latest/operations/Matmul/gemm_fusions/grouped_gemm_dswiglu.md) - [Grouped GEMM + GLU (SM100)](https://docs.nvidia.com/cudnn/latest/operations/Matmul/gemm_fusions/grouped_gemm_glu.md) - [Grouped GEMM + GLU + Hadamard (SM100)](https://docs.nvidia.com/cudnn/latest/operations/Matmul/gemm_fusions/grouped_gemm_glu_hadamard.md) - [Grouped GEMM + GLU + Hadamard + Quant (SM100)](https://docs.nvidia.com/cudnn/latest/operations/Matmul/gemm_fusions/grouped_gemm_glu_hadamard_quant.md) - [Grouped GEMM + Quant (SM100)](https://docs.nvidia.com/cudnn/latest/operations/Matmul/gemm_fusions/grouped_gemm_quant.md) - [Grouped GEMM + Quant -- Unified (SM100)](https://docs.nvidia.com/cudnn/latest/operations/Matmul/gemm_fusions/grouped_gemm_quant_unified.md) - [Grouped GEMM + sReLU (SM100)](https://docs.nvidia.com/cudnn/latest/operations/Matmul/gemm_fusions/grouped_gemm_srelu.md) - [Grouped GEMM + SwiGLU (SM100)](https://docs.nvidia.com/cudnn/latest/operations/Matmul/gemm_fusions/grouped_gemm_swiglu.md) - [Grouped GEMM + WGrad (Unified)](https://docs.nvidia.com/cudnn/latest/operations/Matmul/gemm_fusions/grouped_gemm_wgrad.md) - [DSv4.1 mHC Projection/RMS Backward (SM100/SM103)](https://docs.nvidia.com/cudnn/latest/operations/Matmul/gemm_fusions/mhc_projection_bwd.md)