> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/cudnn/latest/operations/RoPE/llms.txt. # RoPE (Rotary Position Embedding) RoPE is a first-class backend operation (`CUDNN_BACKEND_OPERATION_ROPE_FWD_DESCRIPTOR` / `_BWD_DESCRIPTOR`), not an SDPA attribute. The backend op-graph contains `RoPE(Q) + RoPE(K) + SDPA(Q', K', V)`; the SDPA fusion engine pattern-matches and runs the RoPE kernels around the attention kernel as a single execution unit. The kernel takes **raw `freqs`** (angles), not pre-computed cos+sin. It computes `sincosf(freqs)` once per position into shared memory, amortized across heads. One tensor instead of two; matches the TE/Mcore convention where `RotaryEmbedding` already caches raw angles. ## Math Non-interleaved (halved) rotation. Head dim `D` is split into a leading nope segment of size `D − rope_dim` (scaled passthrough) and a trailing rope segment of size `rope_dim`: $$ y_{\text{nope}} = \alpha \cdot x_{\text{nope}} $$ $$ y_{\text{lo}} = \alpha \cdot (x_{\text{lo}} \cos\theta - x_{\text{hi}} \sin\theta) $$ $$ y_{\text{hi}} = \alpha \cdot (x_{\text{hi}} \cos\theta + x_{\text{lo}} \sin\theta) $$ where `x_lo, x_hi` are the two halves of the rope segment and `α = output_scale`. The same scale folds into both rotated halves *and* the passthrough so the entire output is uniformly scaled. The backward op is rotation by `−θ` (transpose of the forward rotation) — implemented as the same kernel with `sin_sign = −1`. cos is unchanged. ## Forward graph User builds: ```python q_rot = graph.rope(input=q, freqs=freqs, output_scale=α_q, rope_dim=...) k_rot = graph.rope(input=k, freqs=freqs, output_scale=α_k, rope_dim=...) o, stats = graph.sdpa(q=q_rot, k=k_rot, v=v, attn_scale=...) ``` ```mermaid flowchart LR Q[Q
BHSD]:::user K[K
BHSD]:::user V[V
BHSD]:::user F["freqs
raw angles
S × 1 × 1 × D"]:::user Q --> RQ F --> RQ F --> RK K --> RK RQ["RoPE_fwd_Q
───────
output_scale_q
rope_dim_q"]:::op RK["RoPE_fwd_K
───────
output_scale_k
rope_dim_k"]:::op RQ -->|"Q_rot
real, user-bound"| S RK -->|"K_rot
real, user-bound"| S V --> S S["SDPA_fwd
───────
attn_scale"]:::op S --> O[O]:::user S --> ST[stats]:::user classDef user fill:#e3f2fd,stroke:#1976d2 classDef op fill:#fff3e0,stroke:#f57c00,font-weight:bold ``` `Q_rot` and `K_rot` are **real (user-bound) outputs**, not workspace, so they survive into the backward graph as inputs to SDPA bwd. ## Backward graph ```mermaid flowchart LR QR["Q_rot
saved from fwd"]:::saved KR["K_rot
saved from fwd"]:::saved V2[V]:::user O2[O]:::saved dO[dO]:::user ST2[stats]:::saved F2[freqs]:::user QR --> SB KR --> SB V2 --> SB O2 --> SB dO --> SB ST2 --> SB SB["SDPA_bwd
───────
attn_scale"]:::op SB -->|dQ_rot| RBQ SB -->|dK_rot| RBK SB --> dV[dV]:::user F2 --> RBQ F2 --> RBK RBQ["RoPE_bwd_Q
───────
output_scale_q
rope_dim_q"]:::newop RBK["RoPE_bwd_K
───────
output_scale_k
rope_dim_k"]:::newop RBQ --> dQ[dQ]:::user RBK --> dK[dK]:::user classDef user fill:#e3f2fd,stroke:#1976d2 classDef saved fill:#e8f5e9,stroke:#388e3c classDef op fill:#fff3e0,stroke:#f57c00,font-weight:bold classDef newop fill:#fce4ec,stroke:#c2185b,font-weight:bold ``` `dfreqs` is not produced — frequencies are constant during training. User builds: ```python dq_rot, dk_rot, dv = graph.sdpa_backward(q=q_rot, k=k_rot, v=v, o=o, dO=dO, stats=stats, attn_scale=...) dq = graph.rope_backward(dY=dq_rot, freqs=freqs, output_scale=α_q, rope_dim=...) dk = graph.rope_backward(dY=dk_rot, freqs=freqs, output_scale=α_k, rope_dim=...) ``` ## Folding `attn_scale` into RoPE Standard scaled attention multiplies `QKᵀ` by `α = 1/√D` inside the softmax. Because RoPE is linear, that scale can be hoisted into the RoPE Q output without changing the math: $$ \langle \alpha \cdot R(\theta) Q, R(\theta) K \rangle = \alpha \cdot \langle R(\theta) Q, R(\theta) K \rangle $$ ```mermaid flowchart TB subgraph Before["Before: scale lives inside SDPA"] direction LR Qb[Q] --> RQb["RoPE_fwd
output_scale=1"] Kb[K] --> RKb["RoPE_fwd
output_scale=1"] RQb -->|Q_rot| SDPAb["SDPA
attn_scale=α"] RKb -->|K_rot| SDPAb SDPAb --> Ob[O] end subgraph After["After: scale folded into RoPE Q"] direction LR Qa[Q] --> RQa["RoPE_fwd
output_scale=α"]:::scaled Ka[K] --> RKa["RoPE_fwd
output_scale=1"] RQa -->|"α · Q_rot"| SDPAa["SDPA
attn_scale=1"] RKa -->|K_rot| SDPAa SDPAa --> Oa[O] end classDef scaled fill:#fce4ec,stroke:#c2185b,font-weight:bold ``` The fwd kernel applies `output_scale` uniformly to both rotated halves and the nope-dim passthrough so every position of the Q output carries the α factor. SDPA then runs with `attn_scale = 1`. ### Backward: where α actually flows This is subtle. Standard scaled SDPA bwd has α between the softmax bwd and the matmul bwds: $$ dV = P^\top dO, \quad dS = P \odot \big(dP - \text{rowSum}(O \odot dO)\big), \quad dQ' = \alpha \cdot dS \cdot K', \quad dK' = \alpha \cdot dS^\top \cdot Q' $$ α appears explicitly in *both* `dQ'` and `dK'`. In the folded case (`Q_sdpa = α·Q'`, `K_sdpa = K'`, `attn_scale = 1`), `S = Q_sdpa · K_sdpa^T` has the same value (α moved across the dot product), so `P`, `O`, `dS`, `dV` are unchanged. But the engine no longer multiplies by α between softmax bwd and the matmuls: $$ dQ_{\text{sdpa}} = 1 \cdot dS \cdot K_{\text{sdpa}} = dS \cdot K' \quad \textbf{(missing α)} $$ $$ dK_{\text{sdpa}} = 1 \cdot dS^\top \cdot Q_{\text{sdpa}} = dS^\top \cdot (\alpha Q') = \alpha \cdot dS^\top \cdot Q' \quad \textbf{(α came in through } Q_{\text{sdpa}}\textbf{)} $$ So K is already correct out of SDPA bwd; only Q needs RoPE_bwd to re-introduce the α factor. ```mermaid flowchart TB subgraph BBefore["Bwd before fold (standard)"] direction LR QRb["Q_rot"] --> SBb KRb["K_rot"] --> SBb Vb[V] --> SBb Ob2[O] --> SBb dOb[dO] --> SBb SBb["SDPA_bwd
attn_scale=α"]:::op SBb -->|"dQ_rot = α · dS · K'"| RBQb["RoPE_bwd_Q
output_scale=1"] SBb -->|"dK_rot = α · dSᵀ · Q'"| RBKb["RoPE_bwd_K
output_scale=1"] SBb --> dVb[dV] RBQb --> dQb[dQ] RBKb --> dKb[dK] end subgraph BAfter["Bwd after fold (Q-only)"] direction LR QRa["α · Q_rot"] --> SBa KRa["K_rot"] --> SBa Va[V] --> SBa Oa2[O] --> SBa dOa[dO] --> SBa SBa["SDPA_bwd
attn_scale=1"]:::op SBa -->|"dQ_sdpa = dS · K'
(no α)"| RBQa["RoPE_bwd_Q
output_scale=α"]:::scaled SBa -->|"dK_sdpa = α · dSᵀ · Q'
(α via Q_sdpa)"| RBKa["RoPE_bwd_K
output_scale=1"] SBa --> dVa[dV] RBQa --> dQa[dQ] RBKa --> dKa[dK] end classDef op fill:#fff3e0,stroke:#f57c00,font-weight:bold classDef scaled fill:#fce4ec,stroke:#c2185b,font-weight:bold ``` ### General rule The fold setting on each side is the *same* in fwd and bwd. The bwd correctness then drops out of the chain rule mechanically: whatever `output_scale` you used on `RoPE_fwd_X`, use the same on `RoPE_bwd_X`. | Setup | RoPE_fwd_Q | RoPE_fwd_K | SDPA attn_scale | RoPE_bwd_Q | RoPE_bwd_K | |--------------------|------------|------------|-----------------|------------|------------| | No fold (standard) | 1 | 1 | α | 1 | 1 | | Q-only fold | α | 1 | 1 | α | 1 | | Symmetric (mscale) | √α | √α | 1 | √α | √α | Why the asymmetry of the Q-only row "just works" for K bwd: the SDPA bwd computes `dK_sdpa = dS^T · Q_sdpa`, and `Q_sdpa` already carries α — so K's gradient gets α "for free" without an explicit scale on RoPE_bwd_K. ### Why bother One fewer pass over Q/K data: the α multiply happens during the same load that does the rotation, instead of being a separate op. Also unblocks future TE-block fusion where attention scaling needs to be a property of the Q/K producer, not the matmul consumer. ### Folding `inv_ln2` (extension) The same fold knob can absorb the `1/ln(2)` constant the softmax kernel uses to bridge `exp` ↔ `exp2`. GPU hardware only has `exp2`, so the kernel internally does: $$ \exp(z) = \exp_2(z \cdot \log_2 e) = \exp_2(z \cdot \text{inv\_ln2}) $$ — a per-element `inv_ln2` multiply against the score matrix `S` (the largest intermediate tensor in attention). If the producer pre-scales by `inv_ln2` instead, the kernel can use `exp2` directly. The chain reasoning: $$ \exp_2(\text{inv\_ln2} \cdot S - \max) = \exp_2(\text{inv\_ln2} \cdot (S - \max)) = \exp(S - \max) $$ — same correct softmax, but the multiply moved from `O(BHS_qS_{kv})` (per-S-cell) to `O(BHS_qD)` (per-Q-element) and is amortized into the RoPE load. Note: softmax is **not** scale-invariant — `softmax(c·x) ≠ softmax(x)` in general. So `attn_scale = 1/√D` stays mandatory (it controls the softmax shape). `inv_ln2` is different: it's a base-conversion artifact, removable only because the kernel switches `exp ↔ exp2` to match. #### Bwd contract The trick has two halves that move together. The kernel needs an `softmax_input_is_log2` flag that toggles **both**: - **Fwd**: skip the internal `S *= inv_ln2`; run `exp_2(S - \max)` directly. - **Bwd**: emit `dS_sdpa = ∂L/∂(kernel's S)` — the gradient w.r.t. the already-log2-scaled S, not the natural-log S. Math for the bwd half. With `P_i = \exp_2(S_i - m) / Z`, the chain rule gives $$ \frac{\partial P_j}{\partial S_i} = \ln 2 \cdot P_j \cdot (\delta_{ij} - P_i) $$ which propagates to $$ dS_\text{sdpa} = \ln 2 \cdot \big( P \odot (dP - \text{rowSum}(P \odot dP)) \big) = \ln 2 \cdot \text{standard\_dS} $$ So the bwd output is bigger by a factor of `ln 2` than today's kernel. That extra factor cancels with the user's `output_scale = α · inv_ln2` on RoPE_bwd: $$ dQ = (\alpha \cdot \text{inv\_ln2}) \cdot R(-\theta) \cdot dQ_\text{sdpa} = (\alpha \cdot \text{inv\_ln2}) \cdot R(-\theta) \cdot (\ln 2 \cdot \text{standard\_dS} \cdot K) = \alpha \cdot R(-\theta) \cdot \text{standard\_dS} \cdot K $$ Same on K via Q_sdpa. **Net effect: no change to the FE-side rule.** RoPE_bwd uses the same `output_scale` as RoPE_fwd, whatever you folded into it. The kernel-internal `ln 2` factor cancels with the `inv_ln2` you put in `output_scale`, automatically, by the chain rule. | Setup | RoPE_fwd_Q | RoPE_fwd_K | SDPA attn_scale | softmax flag | RoPE_bwd_Q | RoPE_bwd_K | |----------------------------------|------------------|------------|-----------------|---------------------|------------------|------------| | No fold | 1 | 1 | α | (kernel handles) | 1 | 1 | | Q-only fold of α | α | 1 | 1 | (kernel handles) | α | 1 | | Q-only fold of α + inv_ln2 | α · inv_ln2 | 1 | 1 | input_is_log2 | α · inv_ln2 | 1 | | Symmetric mscale + inv_ln2 | √α · inv_ln2 | √α | 1 | input_is_log2 | √α · inv_ln2 | √α | ## API ```cpp // Forward auto Q_rot = graph.rope(Q, freqs, RoPE_attributes() .set_output_scale(α_q) .set_rope_dim(rope_dim_q)); // Backward auto dQ = graph.rope_backward(dQ_rot, freqs, RoPE_backward_attributes() .set_output_scale(α_q) .set_rope_dim(rope_dim_q)); ``` Python: ```python q_rot = graph.rope(input=q, freqs=freqs, output_scale=α_q, rope_dim=rope_dim_q) dq = graph.rope_backward(dY=dq_rot, freqs=freqs, output_scale=α_q, rope_dim=rope_dim_q) ``` ## Support matrix | Arch | Datatype | Layout | Head dim | rope_dim | Backend version | |------------|-----------|--------|----------|----------------|-----------------| | SM80+ | f16, bf16 | BHSD | even | even, ≤ D | ≥ 9.24 | `freqs` is `[S, 1, 1, rope_dim]` in f32. For a partial-rotation config (e.g. DeepSeek-V3 MLA with `qk_rope_head_dim = 64` inside `D = 192`), the user supplies the smaller `freqs` tensor and sets `rope_dim = 64`; the leading `D − rope_dim` positions become a scaled passthrough. ## Docs - [Tail RoPE + Microscaled QDQ](https://docs.nvidia.com/cudnn/latest/operations/RoPE/rope-qdq.md) - [Prepared BF16 Tail RoPE](https://docs.nvidia.com/cudnn/latest/operations/RoPE/rope-tail.md) - [DSv4.1 Vision RoPE Backward](https://docs.nvidia.com/cudnn/latest/operations/RoPE/vision_rope_backward.md)