Skip to navigation

Python-native cudnn.pygraph and pluggable execution backends

View as MarkdownOpen in Claude

Execute-time shape overrides (override_uids / override_shapes / override_strides) are passed directly to native backend plans. Python engines whose compiled plan has takes_variant_pack=True receive the normalized override geometry in their VariantPack; legacy map-based Python engines reject these overrides before execution. Each engine determines which geometries it supports. See Execute-Time Shape Overrides.

What this is

cudnn.pygraph is a Python-native graph class: graph structure (nodes, tensors, op parameters) lives in Python with full introspection, and execution dispatches over python DSL engines and the cuDNN C++ backend alike. The C++ graph builder is internal (cudnn._pybind_module.backend_graph) and is reached exclusively through lowering.

cudnn.pygraph (Python IR) → create_execution_plans() → rank → ONE ranked plan list
nodes / tensors / params (freeze, analyze, (the graph's PlanConfig(engine_id,
fully introspectable query the backend) family) knobs) — python engines
AND backend entries

Why: python-DSL engines (CuTe-DSL / cuTile style GEMM and attention fusions) need to see the graph to decide whether and how to run it. Previously that required monkey-patching the pybind class and recording calls; now the graph is natively introspectable and an engine is one file implementing BaseEngine.

“Pluggable” means the library ships a table, not that callers hand engines over at runtime. There is no registration call: engines/manifest.py is the only way a python engine exists. An engine handed over at runtime could not be ranked anyway — it declares no Capabilities, so nothing could enumerate its configs or place it against the backend.

The dispatch tree

create_execution_plans([heur_mode.A, ...]) _pygraph.py
│
├─ validate() family declares a validator AND a python engine is
│ a candidate → native semantic validation (no lowering);
│ otherwise lowers + freezes any graph the backend CAN lower
├─ _finalize_backend_layout() backend layout inference lands, or records a decline
├─ _freeze() whole public surface sealed
├─ _attach_facts() family_for → resolve_analyzer → ONE parse, hung on the graph
│
└─ heuristics.rank(graph, _candidate_engines(), backend_plan_entries(), modes)
│ │ │
│ │ └─ _create_backend_plans(): one C++
│ │ create_execution_plans PER MODE, spans
│ │ recorded → every entry carries its mode
│ │ (+ the untagged delegating entry)
│ └─ manifest.engines_for(graph) — the family's offered slots, nothing else
│
├─ family_for(graph) → resolve_heuristics(family)
│ declares none → _unranked: accepting engines, then the backend
│
└─ _assemble(modes, <family>.recommend(kind, facts, offered), backend_plans)
│ e.g. sdpa/fwd/heuristics.py (propose)
├─ A → per eligible cell: a measured rule (_sm120_tiles) names the config,
│ runners-up behind it; a cell with one point per axis contributes one
│ entry. The backend's A block goes where the family's BACKEND marker
│ sits (sdpa/fwd/placement.py per shard; ours first without a marker).
├─ FALLBACK → the config expected to build, + the backend's FALLBACK block
├─ OPENSOURCE → our candidates, then the delegating entry (see below)
└─ dedup by (engine_id, knobs), first position wins
= graph.plans, position for position. build_plans() walks it.

Architecture

Graph IR

  • graph_types.Tensor, nodes.Node, _pygraph.pygraph — an engine-agnostic op DAG. Input/output port names equal the C++ pybind kwarg names, everywhere.
  • Three declarative op mechanisms cover 100% of the C++ op surface: _POINTWISE_TENSOR_ARGS (54 uniform pointwise ops; mode == method name), _STRUCTURED_OPS (25 ops: norms, reduction, block-scale, MoE, conv, structural — one table entry each: ports, attrs, outputs, shape-infer), _CAPTURED_OPS (6 SDPA variants, ~130 kwargs: generic capture over an explicit per-op schema carrying positional order, output-direction kwargs, and conditional outputs). matmul is explicit for positional ergonomics.

Backend contract (engines/)

  • BaseEngine: check_support(graph) (accept, or decline by raising), build_plan(graph, plan, ctx) → CompiledPlan (the expensive JIT step, once per graph/plan, cached on the graph), CompiledPlan.execute(graph, operands, ExecutionContext) with explicit handle/stream/workspace. Runtime shape/stride overrides are supported by the cuDNN backend and compatible VariantPack plans. A legacy uid-map plan rejects overrides rather than silently ignoring them. Simple eager engines implement execute() only.

Ordered bindings for repeated execution

graph.execute accepts either its existing tensor mapping or a tuple/list of buffers paired with a tensor_uids tuple/list. Both forms use the selected plan, workspace, and handle in the same way for the cuDNN backend and Python engines, including FROST. There is no additional preparation call:

# Q, K, V, O are graph tensors; q, k, v, out are runtime buffers.
uids = (Q.get_uid(), K.get_uid(), V.get_uid(), O.get_uid())
graph.execute((q, k, v, out), workspace, handle=handle, tensor_uids=uids)

UIDs must be distinct integers, with one UID per buffer; their order need not be sorted. As with mappings, buffers for unused UIDs are ignored, explicit bindings take precedence over graph-bound inputs, and all required operands must be supplied. The parameter name tensor_dict is retained for existing keyword callers; it accepts the buffer sequence when tensor_uids is present. execute_plan_at_index accepts the same ordered form.

Existing override_uids, override_shapes, and override_strides arguments also work with ordered bindings. Supply all three together, using the graph’s axis order and element units. Each override UID must name an operand, and geometry must be supported by the selected plan. This form does not extend an engine’s supported layouts or dynamic-shape envelope.

FE prepares the graph’s binding layout internally and reuses the most recent UID/override metadata by value. Changing a list in place is observed on the next call. Every execution reads the current buffers and workspace and creates its own pointer pack; the cache retains no runtime tensors, addresses, or stream. Buffer descriptions, capacity, and device information remain available to engine validation. Bare addresses retain the existing caller-responsibility contract; use real buffer objects when metadata must be validated.

Buffer and workspace lifetimes remain the caller’s responsibility through GPU completion, and through all replays of a captured CUDA graph. Changing a later call’s bindings does not update an already captured graph’s addresses.

The variant pack is normalized once

graph.execute() converts whatever the caller passed — a torch tensor, a DeviceView, any __dlpack__ / __cuda_array_interface__ producer, or a bare device address — into a VariantPack at the top, and everything below reads that. Do not add a branch on what the caller’s object is. This exists because there used to be two such branches: the backend path accepted a bare address (_native_var_pack’s if type(d) is int: return d) while an engine got the object untouched and frost.buffers.probe refused it. One public call, two answers, decided by which plan the heuristics happened to pick — which the caller does not control.

VariantPack carries the caller-filled uids ascending and the operands themselves, held in a C type (pygraph/variant_pack.cpp) as one DLTensor each. That type both consumes __dlpack_c_exchange_api__ — the C function table a producer publishes on its type, which is how one crossing reads the whole pack — and implements it, so a kernel reads an operand through the same fast path it has for a framework tensor. address is the pointer array _execute_with_raw_ptrs takes.

The pack’s vocabulary distinguishes the position from the thing at it: pack.index_of(tensor_or_uid) gives an operand’s POSITION, and pack.operands(indices) turns positions into OperandBuffers — one caller buffer described (pointer, shape, stride, dtype), non-owning. Resolve positions once; ask for buffers per call.

The declaration is the contract; the buffer supplies pointer, bytes and alignment. That is how the cuDNN backend has always read a variant pack — it takes the pointer and nothing else — and callers program against it: FlashInfer binds a 2-D matrix to a [1, m, k] tensor, a flat quantizer blob to a [b, bs_m, bs_k] F8_128x4-reordered scale tensor, a 0-d scalar to (1, 1, 1). So _normalize describes a slot from the graph when the buffer is a DENSE run of other extents that covers the declared bytes (the 2-D matrix, the flat blob, the 0-d scalar) — exactly what a bare address gets — and records the slot in VariantPack.graph_described, so an engine that reads the pack answers the way the backend does and one call cannot get two answers by plan selection. Two kinds of buffer keep their own description: one with the DECLARED extents under its own strides (a padded or transposed view of this very tensor — the strides carry information, and an engine that reads the pack honours them, which is how the linear-attention engines serve strided inputs), and one that is SMALLER than the declaration (a packed THD buffer under a padded declaration, say): the engine decides, and an engine that needs the declared extent refuses it at execute naming the operand (the backend would read past the allocation). A bare address lends the declaration outright — it carries no extent, so there is nothing to compare against: the caller guarantees the allocation covers the declared bytes and meets the engine’s alignment. Rules that follow:

  • The declaration supplies the dtype too. A buffer re-described from the declaration takes its dtype with the extents (a byte blob covering a bf16 output becomes that output), and a buffer of the declared extents whose slots are as wide as the declaration’s is read AS the declared dtype (FlashInfer binds packed fp4 data and the e4m3 scale blob as uint8; the backend read a pointer and never knew). A buffer too small for the declaration, or whose slots are not as wide as the declaration’s, keeps its own dtype with its own description.
  • override_shapes / override_strides speak cuDNN element units in the graph’s axis order, like every declaration. They are written INTO the slot (in the buffer’s axis order, _in_axis_order_of), never carried around it, so an engine honours them without knowing the concept exists.
  • The pack speaks storage slots. fp4 packs two elements per slot along the unit-stride axis, so a declared or overridden fp4 geometry is converted with _storage_geometry (extent halves, the other strides halve) before it lands in a slot; an odd unit-stride extent has no slot spelling and is refused. An engine’s per-slot packing factor (frost_gemm’s kpack) therefore applies to every slot alike — this is what stopped frost_gemm from doubling K on an overridden fp4 operand.
  • The pack still carries the shape about to RUN: read the IR port for the shape the plan was built for; read the pack for the shape this call runs (frost_gemm reads its M/N/K there, so one plan serves many problem sizes).

The rule is on every execute’s critical path, backend plans included, so the declaration side (storage_geometry of every slot) is computed once per graph into a native DeclaredLayout and the comparison runs in one crossing per pack (VariantPackNative.describe_from). Measured at 128×256×128 bf16, host enqueue per execute on SM100: backend plan 11.7 → 12.0 µs, frost_gemm 21.9 → 22.0 µs, and a FlashInfer-shaped 2-D binding costs what a declared one does (12.1). The first, per-operand Python form of the same rule was +7.5 µs on both paths and +10 µs more for the 2-D binding — two crossings and a storage_geometry per operand per call.

Two rules that are easy to break by accident:

  • The pointer array is per call. Two threads may execute one graph concurrently with different buffers. A shared array hands each thread the other’s pointers, and each pointer in it is individually valid, so the failure is a wrong number rather than a raise.
  • The operand order has exactly one source, never a union. The lowered graph’s variant-pack template when the graph has one — only C++ sees every user slot, since a tensor’s ragged_offset is an operand but hangs off the Tensor rather than off a node port, and the slots the graph fills itself (pass-by-value scalars, slice replacement destinations, workspace modifications) must be excluded. The python IR only for the python-only ops that never lower. The two sides do not have to agree: each indexes the layout it was handed.

is_virtual on a python-only graph does not mean “the caller supplies nothing” — it is a statement about the backend’s lowering, and a gdn graph marks its own O virtual while the caller passes a buffer for it. So the layout there is every wired port, and an unfilled slot is an optional port the caller did not request.

CompiledPlan.takes_variant_pack is the migration flag. An engine that has not set it still receives the caller’s {uid: buffer} map and reaches ports through resolve_node_buffers; the flag and that function both go once the last engine has moved. execute() builds the Tensors only for a plan that sets the flag — measured, normalizing for a plan that will not read the result costs more than it saves.

What a per-execute path costs

An engine owns its internals, and this section does not change that. It exists because the default outcome is expensive: an engine that re-derives its per-call facts lands around 40 µs of host time per execute, and for a single-kernel op that is most of what the caller pays. The same kernel with those facts read once is 20. Both numbers are frost_gemm at 256×256×128 bf16, host enqueue, min over 25 reps of a 64-call burst from a drained queue.

The budget it has to fit in, all measured on SM100:

µs
cuLaunchKernelEx, untraced1.85
one CuTe-DSL entry~3.6
graph.execute() entry + _normalize~8
everything else is the engine’s

Do not read a per-call cost out of an nsys trace: CUPTI adds ~2.2 µs per traced API call, which is more than the call.

Split the facts by when they are decided. Operand roles and majors, packing factors, alignment requirements, output shape rules, which outputs need a seed — all fixed when the kernel compiled. M/N/K, strides and pointers arrive per call. Read the first set into a table at build (gemm/frost/recipe.py is the worked example) and let the call read the table. That alone is 44 → 35.

Then lower the table into one closure per plan, with its constants captured and the operand structure flattened into the loop headers, so the call does no attribute lookup and takes no branch the build already settled. That is 35 → 20. Two rules make it safe:

  • The lowered path never raises, and it is the only path that runs. What it refuses it hands to a checker that reads the same table, names the rule and raises — it launches nothing. A graph the closure cannot serve at all is declined when the engine is asked to support it, so it goes to the backend rather than to a second executor.
  • So a refusal is the answer, not a slower route. The set of calls the closure refuses should equal the set of illegal calls; a legal call it will not serve is a bug. Keeping a reference executor instead would buy a differential that catches divergence but never a misconception the two share — which is exactly how an axis-order bug survived one here. The tests that matter are against intended semantics and against the BACKEND, at the shapes where two encodings coincide.

A loop over a flat table gets almost all of it, so do not hand-unroll per flavor. Measured three ways on the same plan and buffers: interpreting the table 35.8, looping over it flattened 19.7, a hand-written straight line with the structure unrolled 17.5. The loop is worth 45%; unrolling adds 12% and costs one closure body per operand shape — six flavors, six bodies to keep in agreement. One loop over arg_plan (the launch argument order as data) serves aux, extra outputs, multi-GEMM and block scale at 22 µs each, down from 39–50. Source codegen off the same table is how to buy the last 12% back later, for every flavor at once rather than for the one that was worth hand-writing.

This is a pattern to copy, not a framework to import. Sharing the code across engines would couple their kernels’ ABIs, which is the thing engine autonomy buys; sharing the shape of the solution costs nothing.

Costs that are easy to miss, each measured:

  • A from x import y inside a per-call function: 1.1 µs. It was 65% of what _check_plan_device cost.
  • torch.Tensor.permute(): 1.4 µs per call, per operand.
  • Rebuilding a {id(tensor): buffer} map to look operands back up, when the operand order was settled at build and a list index would do.
  • Recomputing a pure function of values every call. tensor_alignment’s layout half is 1.5 µs and memoizes on (shape, stride, elem_bytes) — values, so there is nothing to invalidate; only the pointer half is per call.
  • Reading an operand through the exchange vtable is 0.08 µs against 1.5 for the python attribute walk. Framework neutrality is not what costs.

Measure from a drained queue, and sweep the burst size. A number that is flat in the burst size is host-bound; one that climbs with it is the device rate, and back-to-back timing reads the device rate whenever host and device are close.

  • An engine does not propose its own plans. Which configs to try, in what order, and where the backend’s entries belong is one comparison across every candidate, and no engine can make it from the inside — it sees neither its siblings nor the backend. That decision lives in engines/heuristics.py and the per-family hook it dispatches to (see Ranking and the one plan list).
  • Every engine has a stable engine_id in one flat id space (engines/engine_ids.py: backend [0, 10_000), C++ OSS [10_000, 20_000), python [20_000, …) with a FAMILY_BLOCK-wide block per family) — reproducible pinning/autotune. Engines do not DECLARE their id: the manifest holds every slot and instantiate() hands each factory the ids its engines are to use, so an engine cannot claim a number it was not given, the whole space is readable in one file, and any id decodes back to a family and a slot (manifest.engine_for_id) with nothing registered first — which is what lets create_execution_plan(engine_id, knobs) replay an autotune result.
  • An engine declines a graph ONLY via NotImplementedError, cudnn.cudnnGraphNotSupportedError, or ImportError; anything else is an engine bug and propagates. ImportError counts because lowering imports are deferred past check_support() (see Import boundaries), so a missing optional dependency can only surface at build time — without it, a host lacking the cutedsl extra would lose graphs the backend could have served.
  • The contract is proven end to end without a GPU in test/python/core/test_dispatch.py, with stand-in engines injected through the manifest — the same path production uses. Those engines do no arithmetic: what dispatch is responsible for is reaching the engine and resolving the caller’s buffers, and checking a result against torch.matmul would put a torch reference implementation of matmul inside a dispatch test.
  • The linear-attention nodes (GDN, KDA, GDN-2, GDP, Mamba-2 and their summary nodes) are python-engine-only ops: they have no cuDNN backend lowering, so routing them to the backend entry raises cudnnGraphNotSupportedError at lowering. Their engines, attributes and state semantics are in Linear Attention (docs, GitHub).

The manifest: classify, then let the engine decide

engines/manifest.py answers only what an engine cannot answer about itself without being imported. Everything else is the engine’s own check_support().

  • A family is a KIND OF GRAPH — roughly the backend’s operation-graph mode, at a granularity of our choosing — not a group of engines that ship together. Every graph belongs to exactly one family or to none, so engines within a family compete and engines across families never do.
  • Classification is a lookup. _ANCHOR_NODE_TO_FAMILY maps the node types that NAME a family; family_for(graph) is a function, so “two families claimed this graph” is not a case that can arise. A graph naming two (a matmul and an sdpa together) belongs to neither and goes to the backend.
  • The table holds anchors, not an envelope. Node types absent from it — POINTWISE, REDUCTION, anything added tomorrow — are ignored when classifying, so matmul + pointwise is a gemm graph. Whether a family can serve the WHOLE graph is its analyzer’s judgment. A coarser copy of that judgment here is what closed_under was, and it promised RESHAPE support nothing implemented.
  • family_for(graph) is a pure property of the graph — no sm, no environment. What kind of graph something is cannot depend on which machine is asking. Availability is separate (EngineFamily.offered_ids).
  • What the manifest does NOT decide: architecture. An engine’s Capabilities declares an arch RANGE (sm_lo/sm_hi, major*10 + minor), because an sm100 kernel serves the sm100 LINE — enumerating the members that exist today silently declines the ones that ship later.
  • Maturity is per engine (EngineSlot.opt_in, gated by CUDNN_FRONTEND_ENABLE_FROST_ENGINES), so one implementation can graduate while a sibling matures. It lives in the manifest rather than on the engine class because the gate must answer without importing the engine.
  • A family may name a heuristics hook — like analyzer, a ("module", "callable") pair kept as strings so the coarse key stays import-free. It is handed the facts and the family’s offered ids and returns its proposals; a BACKEND marker in that list says where the backend’s own block goes inside each mode block (ours first when absent). The SDPA-forward family decides that per measured shard (sdpa/fwd/placement.py).
  • A family may name a validator hook — the same import-free pair, validate_graph(graph) -> bool. When the manifest offers a python engine for the graph, validate() runs it instead of the eager C++ lowering: it applies the family’s version- and arch-agnostic semantic rules with the classic error types, and returns False (classic lowering) for a graph holding a node it does not cover. It exists because the eager lowering coupled a graph a python engine fully serves to the installed backend’s version and per-arch gates (issue #704); the backend’s own verdict is not lost, only deferred to planning (_finalize_backend_layout records the decline, plan() raises it if no python engine proposes a plan either). A family without one validates classically. cudnn/_sdpa_validate.py and cudnn/_gemm_validate.py are the two today; both import only the IR.
  • There is no registration call. The manifest is the only way a python engine exists, and _candidate_engines() is the graph’s family and nothing else. An engine handed over at runtime could never be ranked anyway: it declares no Capabilities, so nothing can enumerate its configs or place it against the backend — it was an entry point into the plan list, not into the decision. Tests inject their fakes as a manifest family, so they reach dispatch the way real engines do.

Facts: one description per graph, shared

A family may name an analyzer — a ("module", "callable") pair, kept as strings so matching stays import-free. Planning resolves it and attaches the record to the graph; engines read that record back rather than parsing again.

  • Attached after the freeze. Planning runs _finalize_backend_layout() → _freeze() → _attach_facts(). Analyzing a graph that can still change means chasing every mutation point — the layout the backend infers, a dtype set between two validate() calls — and missing one leaves facts describing a graph that is gone. _facts_for() memoises only a frozen graph, so there is no invalidation rule to get wrong.
  • Keyed by the analyzer itself, so the ranking (which resolves it from EngineFamily.analyzer) and the engine (which passes the callable it already imports) reach ONE record with no name to keep in sync. Two families MAY share an analyzer; SDPA forward and backward do.
  • Facts describe, capabilities judge. SdpaGraphFacts records has_bias=True as a fact, never an error; each engine’s Capabilities row does the rejecting in mismatch(). A shared parser that starts rejecting becomes an if-ladder that must know every kernel.
  • Caller hints are facts too. cudnn.pygraph(..., is_cuda_graph_replay_expected=True) (keyword-only; cudnn.graph(...) forwards it) states that the caller will capture graph.execute into a CUDA graph and replay it. The SDPA analyzer records it as cuda_graph_replay, and a family’s heuristics may then lead with the plan that is fastest on the GPU alone, since per-execute host costs are paid once at capture (the SDPA forward d256 decode tile leads with its split-KV plan instead of listing it as the runner-up). The hint changes no numerics and no support, and it is never forwarded to the backend.
  • Framework-neutral vocabulary: cudnn.data_type, not torch.dtype; device from cudnn.create_device_properties(), the backend’s own descriptor. Facts are what every engine of a family reads, so expressing them in one framework’s types would make dispatch require that framework.

Handle, stream, and device

There is no ambient device state in dispatch. Three separate things:

  • Handle — execute(..., handle=h) if given, else the graph’s own. The handle is what carries the stream.
  • Stream — _resolve_stream(handle) is cudnn.get_stream(handle). With no handle it is None, and kernel wrappers then use the default stream. A failed query on a SUPPLIED handle raises rather than silently falling back to another stream: running on the wrong stream is a correctness bug, not a degradation. So it is a fallback for no handle, never a fallback for a handle whose stream could not be read.
  • Device — the analyzer reads compute capability and SM count from cudnn.create_device_properties(), the same serialisable descriptor the C++ deviceless-AoT path uses, rather than from torch.cuda.current_device(). Engines re-check the arch in check_support() regardless.

Import boundaries

Deciding whether an engine COULD serve a graph must not cost the machinery that would serve it — importing the CuTe DSL is ~1 s and 357 modules, and paying it only to decline is why closed_under existed.

  • Package __init__s under cudnn/sdpa are lazy (PEP 562) and EngineSpec.lower resolves its DSL adapter at build time, so import cudnn.sdpa.graph_analyzer costs 9 ms and 2 modules rather than 1059 ms and 381.
  • The graph API pulls no framework at all: describing and validating a graph imports neither torch nor cutlass — the family validators included (they see only the IR).
  • A missing or too-old DSL is a DECLINE at check_support(), probed without executing the module (importlib.util.find_spec, importlib.metadata). CUTEDSL_MIN_VERSION in frost/buffers.py is the floor these engines want; pyproject requires >=4.8.0. The runtime checks still protect environments that bypass dependency resolution and kernels with newer API requirements.
  • test/python/core/test_import_boundaries.py holds all of this, in a fresh interpreter, measuring the delta against an empty one.

Ranking and the one plan list

  • create_execution_plans() gathers the parsed facts, eligible engine ids, and the backend’s (engine_id, knobs) entries. engines/heuristics.py::rank calls the family’s hook with (kind, facts, offered). The hook returns its own proposals plus an optional internal cudnn.engines.heuristics.BACKEND marker. _assemble() expands that marker into the mode’s backend block, then strips mode annotations and deduplicates to form the final graph.plans list.
  • The backend’s entries arrive tagged with the mode that produced them. _create_backend_plans() asks C++ one heuristic mode at a time and records get_execution_plan_count() after each, so a family can say “the backend’s mode-A entries ahead of ours, its fallbacks behind”. C++ appends each query to the same plan list, which is exactly what makes the boundaries readable — no C++ change was needed.
  • heur_mode.OPENSOURCE is mode A without the backend’s recommendation: the python engines ARE the open-source implementation. Combine it to measure coverage — [OPENSOURCE, A, FALLBACK] tries every python config first and still has the backend behind it, so a graph that runs on a backend plan is one no python engine covers.
  • The list IS graph.plans, and the classic at-index APIs (get_execution_plan_count(), get_plan_name_at_index(), build_plan_at_index(), execute_plan_at_index(), get_workspace_size_plan_at_index()) address it, so code that loops over the plan count picks up python engines with no change.
  • A backend entry carries the cpp_index it holds in the lowered graph’s own plan list, so building it is one build_plan_at_index. Backend engine sets are still never statically enumerated: they are discovered per graph at plan time, and backend_plan_entries() returns [] when the backend declines the graph or is not installed — the backend participates in the ranking, it is not a hard dependency.
  • A plan’s identity is (engine_id, knobs), never its cpp_index. The index only says where one backend query happened to put it, so deduplicating on it lets [A, A], or one config both modes return, through twice — and an autotuner would build and time the same config twice.
  • Whether a mode SUCCEEDED is tracked per call, not read off the plan spans. An OPENSOURCE query registers a C++ OSS candidate without adding a plan, so it contributes no span; judging by spans would rethrow a later mode’s failure and discard the delegate that successful query earned.
  • The family positions the backend block with its marker; without a marker, its proposals precede the backend. Ranked indices address the assembled list, never the marker or an unexpanded family proposal. BACKEND_HEURISTIC_ENGINE_ID names one thing only: the delegating entry backend_plan_entries() appends, where the backend picks among OSS candidates it never exposes as plans and which therefore cannot be enumerated. It carries no mode. It leads the BACKEND’s entries, because Graph::build_plans tries it before its own engine configs — but it does NOT lead the family’s, because it is not a pure OSS entry: if the C++ OSS engine declines, that same call falls through to the native configs already enqueued. Ahead of the family’s OPENSOURCE block it would answer an OSS-coverage question with a native kernel.
  • build_plans() walks the list from the selected index and takes the first entry that builds; a decline advances to the next. select_plan(i) pins, and a pinned decline raises instead of running something else.
  • Ranking policy has ONE home: the family’s heuristics hook, replaceable per family. Which side leads is meant to be a measurement — a cell timed slower than the backend follows it — not a default. Any cell that has not been timed keeps the historical order, and the code says so where it is written.
  • A recommendation always names a concrete config. A knob field is None only where the capability row declares no domain for that axis. None never means “engine, pick for me” — that reading is what let the same choice be made in two places, once in the ranking and once in the DSL adapter, and drift. sdpa/fwd/heuristics.py::_sm120_tiles is the worked example of a rule: it reads facts only, names its choice first, and puts the rest of the domain behind it for a caller that autotunes. To add one — write the function, list the cell in _TILE_RULE_CELLS, and put the measurement in the commit.

Consumed by FlashInfer: the public surface, kept stable

FlashInfer’s cuDNN backend (flashinfer/cudnn/{prefill,decode,utils}.py) programs against this graph API and probes a handful of frontend facts before it routes work here. Those names are a caller boundary whatever their spelling (python/cudnn/AGENTS.md Rule 9: “a private method consumed outside FE is still a caller boundary”), so every one of them is public API, kept as spelled, with a public twin where the spelling was private. test/python/sdpa/frost/test_flashinfer_consumed_surface.py (host-only, L0) pins each name with its signature, so a rename fails this repository’s CI instead of silently degrading FlashInfer to its fallback path.

what the caller asksspelled today (kept)public spelling
does the native SDPA binder honor a packed-split / stride contract?cudnn._pybind_module._SdpaThdBinder.supports_stats_stride_override, .supports_nonpaged_packed_split, .supports_paged_split_sink, … (static read-only properties, True)"<cap>" in cudnn.sdpa_capabilities() — the frozenset of every supports_<cap> flag the build provides; append-only, never renamed
which engine serves the selected plan?graph.selected_engine (None = the cuDNN backend; else .name)the same property, documented: .name is the engine’s manifest name, a stable identifier (sdpa_fwd_prefill_sm100 / sdpa_fwd_prefill_sm107 / sdpa_fwd_prefill_sm120, …; manifest names and slots are append-only)
can the cuDNN backend still serve this graph?graph.backend_plan_entries() after create_execution_plans ([] = declined, guarded or not installed)the same method, documented above
can the FROST engines run on this install, for this GPU?cudnn.frost.buffers.cutedsl_state() / cutedsl_too_old(version) / CUTEDSL_MIN_VERSION — the version half onlycudnn.frost.cutedsl_runtime_available(device_cc=None): the version half, plus — given a (major, minor) cc — the target half (cutedsl_arch_requirement_error). The floor is 4.7.0 everywhere and 4.8.0 on cc 10.7 (the first public wheel whose Arch knows sm_107a), and a part below cc 8.0 has no target in any DSL (its Arch starts at sm_80): a device-blind probe on a cc 10.7 part with a 4.7.x DSL says True while every FROST row declines, so pass the GPU you route to
the cuDNN dtype of a torch dtypecudnn.datatypes._torch_to_cudnn_data_type(dtype)cudnn.datatypes.torch_to_cudnn_data_type(dtype) (the private name is an alias of it)
the unified attention implementation, the graph helpers, the caller hintscudnn.attention_implementation.UNIFIED, cudnn.graph_cache(key_fn=), cudnn.jit(heur_modes=), cudnn.graph(handle, **graph_kwargs), cudnn.pygraph(is_override_shape_enabled=, is_cuda_graph_replay_expected=)already public; listed so the pin covers them

Nothing here changes a Capabilities row: the probes report what the engines already decide, and import cudnn stays as cheap as before (the device-blind probe reads package metadata; only the cc 10.7 target check imports the DSL).

Knobs: one public vocabulary, and every python plan lists the knobs it will build

  • KnobType_t (include/cudnn_frontend/knobs.h, mirrored as cudnn.knob_type) is the ONE vocabulary for backend and python plans. Backend values 0..32 are frozen; python-only axes live in the band from FRONTEND_KNOB_TYPE_BASE (1000: SCHED_POLICY, PACK_GQA, SPLIT_KV, PIPELINE_ARCH, MMA_TILE_*, CTA_GROUP, WARPS_*). Both bands are append-only; a frontend-band value never reaches the backend (convert_to_backend_knob_type refuses it). Reuse a backend type when the meaning matches (TILE_M, TILE_CGA_M, SPLIT_K_SLC, SWAP_AB); mint a frontend one only for an axis the backend has no word for.
  • A python engine keeps whatever native knob object it likes in PlanConfig.knobs (SdpaFwdKnobs, SdpaBwdKnobs, GemmKnobs) and converts at the public boundary through BaseEngine.knobs_to_public / knobs_from_public. get_engine_and_knobs_at_index always returns a dict ({} when the plan has no axes, never None); create_execution_plan( engine_id, {cudnn.knob_type: int}) replays it; get_plan_name_at_index prints engine[SPLIT_KV=2, TILE_M=128, ...], sorted by knob name.
  • Every python plan is listed WITH the knobs it will build. A family’s recommend(kind, facts, offered) proposes complete PlanConfig(engine_id, knobs) entries from facts alone; the engine builds what was listed. Families with a real tuning axis — SDPA forward, SDPA backward (the SM120 row’s per-head-dim default tile, the MXFP8 row’s sole point), frost GEMM (the tile config as knobs) — all do this, so a recorded (engine_id, knobs) pins the kernel across releases even when a default table moves. A family whose kernels take no tuning decision (linear attention) lists {}, which IS its complete record. A knob-less plan whose engine picks inside build_plan is a bug: the record would replay a different kernel after the pick changes.
  • Knobs are performance-only: a plan computes the same function under any knob value, so an autotuner may pick freely. Anything numerics-changing (softmax_precision) or a contract on the inputs (attn_scale_prefolded: Q already carries the softmax scale) is an op attribute declared in the op spec’s python_only_attrs: never forwarded to C++, a SET value (a bool attribute set to False counts as unset) makes the node backend-unlowerable (serialize() and key() refuse it), and it surfaces as a graph fact the capability rows gate on.

One kernel per layout class, not per shape (SDPA THD)

A compile() of an SDPA template used to pin batch, head extents and every stride in its fakes, so FlashInfer’s serving shapes minted one 2.6 s kernel per (b, qh, kh, strides): 126 forward kernels in FlashInfer’s cuDNN attention tests. The kernels never needed that — _host reads B / QH / KH from problem_size at run time and the THD token totals were already sym_int. Under compile(dynamic_bhk=True) (THD only) the template rebinds b, qh, kh (and a padded LSE’s s_max) to cute.sym_int() right after the cache key is taken, keeps plain ints for the problem_size fake, and gives the metadata / O-descriptor fakes fresh symbols (SymInt has no __add__); a packed declared stride is passed as None so the compact fake derives it from the dynamic extents, a padded LSE in a compact dim order passes that order (lse_padded_order), and anything else keeps its static key. The adapter canonicalizes the key (b = qh = kh = 0, lse_padded_rows = 1) when the module offers dynamic_bhk. What stays static: d, dtypes, masks, the GQA ratio (CFG.QH_PER_KH), paged pool strides, dense (non-THD) shapes. Two DSL facts shaped this: and is staged, so const_expr(CFG.PACK_GQA and q.shape[2] != ...) must nest its constant test outside; a const_expr on a dynamic extent or stride is an error, which is why the padded-Stats store selects on the fake’s RANK (rank-4) and not on shape[0] > 1.

Per-port token origins from the bound ragged offsets (SDPA THD backward, SM80)

A ragged graph binds one ragged-offset tensor per port, and a padded THD layout (TE’s cu_seqlens_padded) puts a sequence’s rows somewhere other than prefix(lengths). The SM80 backward reads those values on device (issue #737). Its existing setup launch, which turns the lengths into the compact cu_seqlens, also writes one token origin per port and sequence, org_p[b] = ro_p[b] × M_p / ts_p (M_p the port’s ragged_offset_multiplier, ts_p its token stride in the CALLER’s buffer, Int64 arithmetic), into an Int64 [rows, B] workspace region with one row per port that binds offsets. No extra launch; the host only binds the current offset pointers, so the origins follow the bound values on every execute and CUDA-graph replay.

Two kinds of row origin then coexist. The internal packed buffers (the fp32 dQ accumulator, do_dot, the GQA dK/dV partials) stay at prefix(lengths), so their sizing and the masks and loop bounds do not change. The caller’s ports are addressed at their origins: the Q/dO and K/V loads, the Stats read, the direct-bound dK/dV stores, and the O·dO dot, dQ cast and dK/dV fold, which walk (sequence, row) over the envelope and touch origin + row for row \< len[b] only. Gap rows are never read or written. A staged port (a head dim inside the flavor envelope) keeps its origins: the staging copy moves every row in place, which is why origins are computed from the caller’s token stride.

tile_dsl.thd.write_thd_port_origins / thd_port_origin are the shared helpers; each path materializes only the ports it needs. Whole-token offsets are a supported-input precondition of the FROST engines, not a restriction of the cuDNN tensor API, and are not checked on device. When the backend cannot lower a graph, the Python-side variant-pack order includes each port’s ragged offset tensor, so the offsets reach the engine as bound slots.

Accept means run

For a python plan, check_support() accepted ⇒ build_plans() and execute() succeed on the buffers a caller binds as the graph declares them, and — when the graph has a backend plan — the result matches that plan; a python-only graph (GDN/KDA/…) has no cuDNN reference and is held to its engine’s own numerics tests instead. Nothing an engine can decide from the DECLARATION (operand rank, blob size, scalar shape, layout) may refuse a call after acceptance: it is decided at check_support, and the pack hands the engine the declaration. Buffer capacity is not decidable before execute — the buffers arrive with the call — so a buffer smaller than its declaration is a caller error refused at execute, not a decline. test/python/gemm/frost/test_flashinfer_shaped_gemm.py and test/python/sdpa/frost/test_flashinfer_shaped_sdpa.py re-declare FlashInfer’s graphs and buffers byte for byte and assert exactly this; a decline at check_support is reported as xfail with the row’s reason, a refusal after acceptance fails. They are the acceptance gate for turning the opt-in rows on by default.

The compiled-plan cache

cute.compile runs the whole backend once per process for every distinct kernel — 0.7 s for a FROST GEMM, 2.6 s for an SDPA prefill on SM100 — and the DSL’s own file cache stores only MLIR bytecode, so a process that warms tens of plans pays minutes at start-up. cudnn.frost.compiled_cache keeps the exported tvm-ffi object of every kernel compiled with --enable-tvm-ffi and reloads it in milliseconds. compile_cached(fn, *args, cache_key=, symbol=, **kwargs) is the drop-in for cute.compile at a kernel’s compile site. Two families route through it today: the GEMM templates, with the digest of their generated source as the key (the full GEMM suites: 5655 tests, 20:53 cold, 1:35 warm), and the SDPA forward / backward templates, with template_key(globals(), locals(), "<function>") as the FIRST statement of each function that compiles — the file + params digest template_loader records as FROST_SOURCE_DIGEST, joined with the function’s name and its arguments, which pin shapes, strides and flags the way the params pin dtypes and masks (SDPA + GEMM suites: 5867 tests, 53:06 cold, 2:53 warm). An argument that is not a plain value (a tensor, a device, a stream) makes the key None and the call compiles as before. The linear-attention hosts the FROST engines build (GDN/GDP/Mamba-2 warmup, GDN2/KDA uncut and every family’s piece-chain hosts: forward, backward and summary) also use this cache, with their existing static specialization tuples as keys. Those tuples include dtype, rank/layout specialization, device, schedule and gate/state flags; packed token totals remain dynamic. The stand-alone linear-attention chunk_* launchers and the SDPA adapters’ helper kernels in api_dsl still compile without persistent artifacts. A hit is the same object a miss produced, so the reloaded kernel is called exactly as the in-process one.

The rules, borrowed from FlashInfer’s autotune cache v2 so that a stale artifact can never be reused by accident:

  • Identity is the whole manifest, hashed. Frontend version AND a digest of every .py in the cudnn package (a kernel is compiled from the shared helpers it imports as much as from its template, and the version does not move in a checkout someone is editing — an uncommitted edit lands in another directory; a wheel hashes the same every process), cutlass-dsl and tvm-ffi versions, the CUDA driver, and the device’s name, compute capability, SM count and L2 size (FROST bakes the last two into kernels) name the directory <root>/v2/<env_hash>/. Any change lands elsewhere; an unreadable field is hashed as "unknown", never skipped.
  • An entry is reused only under its own embedded key. entry.json carries the full key, symbol and signature and is compared on load; the object is written first and the record after it (the commit marker), both via temp file + os.replace, so a crash or a concurrent writer never yields a loadable entry without its key.
  • Anything doubtful is a miss: missing, malformed, mismatched, or an object load_module refuses (an arch the device cannot run). Never an error.
  • Concurrent cold callers share a compilation on POSIX. A cache miss takes an advisory lock for its environment and artifact key, then checks the cache again before compiling. Different keys can compile concurrently; cache hits do not acquire the lock. The OS releases it when a compiler exits, including after a crash. If locking is unavailable, compilation and best-effort export continue without deduplication. Empty lock files live in <root>/.locks/, outside the prunable environments, and are retained to keep one stable lock inode per key while other processes may be waiting. Do not unlink them while the cache is in use. Filesystems must support POSIX advisory locking to deduplicate callers sharing that cache.
  • A hit and a miss run the same thing. A reloaded tvm-ffi function is positional-only, so it is wrapped with the kwargs wrapper the DSL itself uses, rebuilt from the RUNTIME spec the record carries (arg_names, defaults, keyword-only names) — the Python signature minus cutlass.Constexpr and env-stream parameters, which do not exist at run time. A wrapper built from the Python signature would shift every argument after a constexpr one (the fp8 SDPA kernels have one). The miss path exports and then reloads, so a bad artifact fails at build time, not at the next start-up. Kernels whose in-process object converts raw pointer arguments, takes a runtime dataclass argument, or has a default JSON cannot carry are not persisted. Constexpr dataclasses are exportable when the DSL executor can establish that they are absent from the runtime arguments; their configuration must still be part of the caller’s specialization key.
  • One object per kernel per process. In front of the files sits an in-process memo keyed by the device a build targets, the live CUDA device the compile is issued under (the DSL ties a compiled object’s executor to a device context; the two agree unless a handle scoped the build to another GPU), the key, the symbol and the compile options: a second plan over the same kernel in the same process gets the object the first plan got — no trace, no compile, no file touched — whatever the on-disk cache’s state. A graph that builds several plans over one kernel, or a test suite that builds one block instance per case, otherwise pays a full JIT per plan (cute.compile has no memo of its own; with the on-disk cache off, the gated-attention-block backward suite — 109 cases — went from 855 s to 261 s and from 730 to 100 compiles on a Rubin development part). A kernel without a key is never memoised; a failed compile is not memoised; clear_memo() forgets the objects. CUDNN_FRONTEND_COMPILED_CACHE_INPROCESS_MEMO=0 turns the memo off — for a test that exercises the reload path itself, or an A/B of the memo.
  • Location: CUDNN_FRONTEND_COMPILED_CACHE, else $XDG_CACHE_HOME/cudnn_frontend/compiled_plans; set_cache_dir() for a caller that owns a workspace (FlashInfer); CUDNN_FRONTEND_DISABLE_COMPILED_CACHE=1 turns it off; stats() reports hits / misses / bypassed / invalid / pruned / memo_hits / seeded / seed_rejected per process. Bump _SCHEMA on any incompatible change.
  • Dead environments are retired. The manifest hashes the package’s source, so every edited checkout and every CI commit mints an environment directory that will never be hit again — a few hundred MB per commit on a runner with a persistent home. A process’s first write runs prune(): whole environment directories go (never single entries), dead schema roots first, then oldest first, until the root is under CUDNN_FRONTEND_COMPILED_CACHE_MAX_BYTES (4 GiB by default; 0 disables); the process’s own environment is never a candidate. CI should still point CUDNN_FRONTEND_COMPILED_CACHE at a job-local directory: nothing there is ever warm across commits, and the cap is a backstop, not a policy.

Not yet routed: kernels compiled from real tensors at call time (the remaining linear-attention chunk_* launchers, the SDPA adapters’ _dot_fn / _reduce_fn / _setup_fn helpers): a key for them has to spell the traced tensors’ dtype, rank and dynamic marks; same hook once it does.

Serializing a python-engine plan

serialize() / deserialize() keep their classic signatures and follow the SELECTED plan. A backend plan is the cuDNN backend’s blob, as before. A python engine’s plan is not a blob: it is a deterministic function of the graph and its (engine_id, knobs), the pair create_execution_plan replays. So its blob (cudnn._plan_serdes, magic CUDNN_FE_PYPLAN) carries:

  • the graph: every node and tensor of the IR, tensors by uid. A value the format cannot represent exactly (a score_mod callback, a framework object) refuses serialization; nothing is dropped. Buffers bound at build time (_data_bindings) are not carried; pass them to execute().
  • the plan: engine id, engine name (checked on load) and public knobs. The current frontend must still admit the saved graph/configuration; retaining an engine id does not promise unchanged eligibility across releases.
  • the kernels (include_kernels=True, the default): the compiled-cache entries the plan’s build used (compiled_cache.record_entries() around build_plan), each with the environment manifest it was compiled under.

pygraph().deserialize(blob) rebuilds the IR into an empty graph, installs each kernel with compiled_cache.seed_entry only when its manifest equals this process’s exactly (the same identity as the cache directory, so a stale object is never loaded), then replays create_execution_plan, pins it and builds it: compile_cached hits the seeded entries and still checks each record’s key. A refused or missing kernel compiles as usual; enforce_precompiled=True raises instead. Measured on B200 (bf16 SDPA prefill b2 h8 s1024 d128): build 3.1 s; deserialize into another process with an empty cache 33 ms, with imports already done (a backend plan takes 23-30 ms). 3 kernels seeded, zero compiles, O bitwise equal.

A retired graph/configuration is a recipe cache miss, not a request to substitute another implementation silently. Deserialization raises and restores the empty destination graph; discard the saved recipe and reconstruct the graph with heuristics. Namespace persisted recipes by the frontend build/version as well as the target environment. For example, exact SM100/SM103 half D256 recipes using engine 20511 must be regenerated for the explicit 20519/20520 engines. Exact D512/V512 and D576/V512 half recipes likewise move from 20511 to the explicit role-split (20521) or 2x2 (20522) implementation. These new wide engines report physical cluster width through TILE_CGA_M: 4 for D512, 2 for D576. Their MMA group is fixed at two CTAs; the old 20511 pair encoding is not a valid D512 cluster assignment for the new engines. Regenerate the complete engine/knob recipe through heuristics instead of rewriting its engine id. Exact SM100/SM103 per-tensor FP8 D512/V512 recipes similarly move from 20513 to the explicit role-split engine 20523, with physical TILE_CGA_M=4 and a fixed two-CTA MMA group. Exact D256/V256 per-tensor FP8 recipes move to the single-Q engine 20524, with TILE_CGA_M=1 and a one-CTA MMA group. Exact D192/V128 per-tensor FP8 recipes move to the multi-Q engine 20525, with matching physical-cluster/MMA widths of one or two CTAs. Exact D64/D128 per-tensor FP8 recipes move to multi-Q (20527), dual-KV (20528) or swap-AB (20529). The new swap-AB record exposes its physical TILE_M=16 and NATURAL scheduler; SWAP_AB is no longer a knob on these three engines because the engine id already fixes the implementation. Regenerate the full recipe instead of changing an old id in place. Dense head envelopes keep their legacy owner. Exact SM120-family half D512/V512 and D576/V512 recipes move from 20505 to the explicit paired-warp engine 20526 with unchanged knobs; the padded head envelopes keep 20505. Exact SM120-family per-tensor FP8 and MXFP8 D512/V512 recipes likewise move from 20507 and 20518 to 20533 and 20534. Exact SM120-family half D256/V256 recipes move from 20505 to 20537 (TILE_M 64 / 128) or 20538 (TILE_M 16 / 32) with unchanged knobs. Exact SM120-family per-tensor FP8 and MXFP8 general-tile recipes (both dims multiples of 32 up to 256) move from 20507 / 20518 to 20542 / 20543. Exact SM120-family half recipes on the other 16-wide head tiles (both dims multiples of 16 up to 256) move from 20505 to 20544 (TILE_M 64 / 128) or 20538 (TILE_M 16 / 32) with unchanged knobs. SM80 recipes on 20508 move to the owner of their head dims with unchanged knobs: 20546 (D_QK <= 192, D_V <= 128), 20547 (other dims up to 256) or 20548 (wider). The blob’s frontend_version is provenance, not an automatic fallback switch; the compiled-kernel manifest protects binary reuse, not recipe eligibility.

The kernels are native code: deserialize only blobs from a trusted source. The C++ Graph::deserialize does not read this format. Running a python-engine plan from C++ is a separate problem: the engine’s host logic is Python.

Key invariants

  • uid ownership: the Python IR owns the whole uid namespace; every uid is pushed explicitly to C++ and a post-build assertion fails loudly on violation (C++ auto-assignment never runs for Python-built graphs — its enumeration order is nondeterministic for multi-output ops). A user uid landing on an auto-assigned one steals it (the holder is renumbered); user-user collisions raise.
  • Pure-python or pure-C++: a graph routed to a python engine never touches C++ on the execute path; mixed construction is unsupported. (Explicitly querying the backend plan space lowers the backend entry on demand — that is the caller asking for the backend.)
  • One-shot planning: a second create_execution_plans() raises (the classic C++ graph never supported re-planning — it appends engine configs by accident). Switch plans with select_plan(); plan differently by building a new graph.
  • Whole-surface freeze: after lowering/planning, every public mutation path raises — op builders and fluent setters, direct attribute writes on Tensor/Node/GraphContext, dict writes on node ports/params (MappingProxy), in-place dim/stride edits (sealed to tuples). Inspection stays fully readable. validate() lowers and freezes any graph the backend CAN lower (classic error timing) unless the graph’s family validated it natively (see The manifest), so the mutable-after-validate window is the ops with no backend node (GDN/KDA/…) plus natively validated graphs; a mutation in it invalidates the validation and drops the cached candidate list. Planning freezes BEFORE it analyses, so facts never describe a graph that can still change.
  • Output layout contract: only USER-assigned output dim/stride are pushed to the lowered graph; IR-inferred strides are provisional (row-major) and the backend keeps its classic per-op layout inference (e.g. channels-last conv). A unified layout resolver across python/cuDNN candidates belongs to the heuristics follow-up.
  • Classic parity: the public cudnn.pygraph surface behaves as before — cudnnGraphNotSupportedError at validate() for a semantically invalid graph (natively or via the backend; with a python engine candidate, a rejection the backend alone would raise — version or arch gate — surfaces at plan() instead, and only if no python engine proposes a plan), conditional outputs return None, torch dtypes/torch.Size accepted, ragged (THD) offsets and multipliers on outputs, serialize/deserialize passthrough, plan queries delegate to the lowered graph.

Naming

  • cudnn.pygraph — THE public graph class (Python IR), implemented in cudnn/_pygraph.py.
  • cudnn._pybind_module.backend_graph — the internal C++ builder the IR lowers to (renamed from its pre-flip public name to avoid two things called pygraph).

Testing the backend path

The test_native_backend_lowering.py suite builds graphs natively, lowers, executes on GPU, and checks numerics against torch references. Dispatch-level assertions (selected_engine is None, backend plans created, lowered graph present) prove the execution went through the cuDNN backend plan path rather than a python engine; kernel identity below the backend API is deliberately not asserted (kernel names are backend-internal and version-dependent).

What each machine actually covers

A dispatch change needs three runs, because the suites SKIP rather than fail on the wrong arch — a green sweep on one box says nothing about the others:

targetcovers
any GPU (CPU-only logic)test_dispatch.py, test_graph_native.py, test_import_boundaries.py, sdpa/frost/test_flashinfer_consumed_surface.py — the ranking contract, one-shot planning, the at-index APIs, manifest classification, the surface FlashInfer consumes
SM100every FROST SDPA-forward cell except sm120; the FROST GEMM family; linear_attention (GDN / KDA / GDN-2), which is where the cuTile-vs-FROST pin lives
SM120sdpa_fwd_prefill_sm120 and its tile rule; test_mhas_v2 routing tallies; the FROST GEMM family’s sm120 tree (warp-level mma.sync; dense, block-scale and MoE grouped matmul with fused epilogues): gemm/frost/sm120/ — the parent suites’ sm120 copies, where a case still pinned to an sm100 config skips — plus the parents’ own sm120 cases

Enumerate the device with torch.cuda.get_device_properties(i).major and pin it with CUDA_VISIBLE_DEVICES — CUDA’s device order is not nvidia-smi’s, and defaulting to device 0 is how an SM100 suite silently skips in full.

Follow-ups (separate MRs)

  • More per-family tuning rules on top of the ranking frame, each with the measurements behind it. sdpa/fwd/heuristics.py::_sm120_tiles and _pack_gqa_wins are the two today; every other cell falls back to its capability row’s sole point per axis, which is the honest answer while nobody has timed it.
  • FALLBACK is one config per cell today — the smallest tile the row admits, the config that asks least of the device. Picking the handful that between them cover the plane needs measurements; the TODO is in _mode_fallback.
  • sdpa/fwd/placement.py places the SDPA-forward family using B200 / RTX PRO 6000 measurements; an untimed row keeps the order this dispatch has always had. Rankings are tuned and evaluated offline; unit tests check the planner’s marker contract independently of workload winners. A cost model that can compare a python config against a cuDNN engine on a common currency (predicted time) would replace the table with a number.
  • DSL engine integration (the cuTile matmul engine lives in this track).
  • Structural cleanup: lifecycle state objects, a CudnnBackendAdapter to remove selected_engine is None branching, lowering extracted to its own module, op-identity dedup (NodeType vs registry keys), longer-term a typed OpSpec as the single per-op source for builder/validation/lowering.
  • SDPA forward THD, padded Stats: the -inf seed of the (b, s_max, h) buffer is a separate D32 memset ahead of the kernel; fold the tail-row write into the kernel’s persistent schedule to save the launch on that path. The dense padded-Q .item() read in sdpa/fwd/engines.py runs at execute and breaks CUDA-graph capture; decide it at check_support or drop it.

Plan-specific Stats stride override note

After selecting the final plan, query graph.get_behavior_notes() (or get_behavior_notes_for_plan_at_index(i)). cudnn.behavior_note.SUPPORTS_SDPA_STATS_STRIDE_OVERRIDE declares that the plan accepts legal execution-time Stats stride overrides, including changes to the head stride of packed HN Stats. It does not permit changing the packing, dtype, or storage capacity, or bypass the plan’s ragged-offset and override bounds.

The declaration covers admitted override-enabled FP16/BF16 THD plans with packed HN Stats through the prepared THD binder on qualified SM100/SM103 devices, plus qualified SM107 and SM120 (cc 12.0) half THD plans. SM121 remains unqualified. The note follows the admitted graph/config and the prepared binder, including the new explicit implementation owners; it does not apply to every plan on those architectures. This includes the explicit D256 lookahead engine and eligible legacy half engines. Dense, token-major/padded Stats, no-Stats, override-disabled, quantized and ragged-Q dense-decode plans do not declare it. D256 swap-AB does not serve THD. Backend plans retain their existing notes and do not inherit this frontend declaration. RUNTIME_COMPILATION remains present on FROST plans.

Callers should use the selected plan’s note to choose direct HN execution or retain their existing NH fallback. Do not infer support from the engine name or from FROST membership. Selecting/deselecting behavior notes uses the same per-plan declaration, but integrations should not filter the candidate pool just to choose a Stats layout. Consumers of this new enum must require an FE release that supplies it.

The note has frontend-only integer value 1000, in the band starting at FRONTEND_BEHAVIOR_NOTE_BASE; existing values 0–5 are unchanged. It serializes as SUPPORTS_SDPA_STATS_STRIDE_OVERRIDE. Backend enum conversion rejects it; it is never passed as a cudnnBackendBehaviorNote_t.