> This page is for version Latest (default).
> For other versions, use one of these documentation indexes:
> - Latest (default): https://docs.nvidia.com/cudnn/latest/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/cudnn/llms.txt.

# C++ launcher: repeated calls launched from C++

A frontend Python API spends host time on every call before its kernels launch:
argument checks, cache keys, graph and engine dispatch, workspace carving, and the
DSL's argument conversion. For small and medium problems that host time is longer
than the kernels, so the GPU waits on Python. The *C++ launcher* records what one call
does and replays it from C++ for the later calls with the same arguments, so the
API's Python runs once per argument signature instead of once per call.

It lives in `cudnn.cpp_launcher` (Python) and `_compiled_module._Launcher`
(C++, `python/cpp_launcher.cpp`). It needs no tvm-ffi or framework headers; kernels
are called through the tvm-ffi C ABI of the `libtvm_ffi.so` the process loaded, which
has to report the version (0.1) that ABI was written for. Python hands each kernel to
C++ through a tvm-ffi call, so a recording holds its own reference to it. Where that
cannot work (another tvm-ffi ABI, another platform than Linux, a CuTeDSL whose call
path is not the one recordings watch), nothing is recorded and every call runs the
function.

## Using it in an API

```python
from cudnn.cpp_launcher import cpp_launched, launch


@cpp_launched
def my_op(x, *, alpha):
    out = torch.empty_like(x)
    scratch = torch.zeros(n, device=x.device)  # dropped at return: per-call scratch
    launch(compiled, x, scratch, alpha, stream)  # every kernel launch goes through launch()
    launch(compiled2, scratch, out, stream)
    return out
```

- `@cpp_launched` wraps the function. Calls are keyed by each tensor argument's dtype,
  shape, strides and device and by the other arguments' values and types; stream arguments are
  bound per call, not keyed. A `CUstream` or an object with `__cuda_stream__` is
  recognized anywhere; a parameter that may carry a plain integer handle must be
  named: `@cpp_launched(stream="current_stream")`. The kernel itself must receive the
  stream as a `CUstream` (or `__cuda_stream__` object): an `int` kernel argument is
  recorded as a constant, since an integer cannot be told apart from a stream by value.
- `launch(fn, *args)` is `fn(*args)` for a compiled kernel (a `tvm_ffi.Function`, or a
  CuTeDSL function compiled with `--enable-tvm-ffi`). Outside a recording it adds one
  thread-local check.
- `scratch(buffer)` declares a buffer the API keeps across calls, such as a workspace it
  allocated once, as the call's scratch: no kernel reads what an earlier call left in it,
  and nothing reads it once the call's kernels are done. A replay gives the kernels a
  buffer of its own in its place. Without the declaration such a buffer declines the
  recording.

## What happens per call

| Call for a signature | What runs |
|---|---|
| The first 15 | The function, as before (the first one compiles and fills its caches). The first 8 are counted by the first argument's shape alone; from the 9th the call is keyed. |
| The 16th | The function, recorded: each `launch()` (the tvm-ffi function and the positional arguments it receives), each tensor allocated and filled, and the result. |
| Later | Python allocates the returned tensors and one scratch buffer and passes their addresses, the argument tensors' addresses and the stream to `_Launcher.execute`, which runs the fills (`cuMemsetD*Async`) and kernel calls in the recorded order. The function's Python does not run. |

`cpp_launched` functions compose: one that already has a recording, called while another
call is being recorded, runs its Python for that call, so the outer recording sees its
launches and allocations.

A recording costs about as much host time as 16 replayed calls save
(`cpp_launcher.HOT_CALLS`), so a signature that is not called that often, as with
sequence lengths that keep changing, never pays for one. Nor does it pay for a
key: until its first argument's shape has been seen 8 times, a call only counts
that shape (under a microsecond).

Every kernel argument of the recording is bound to where it comes from in the call:

- an argument tensor, or a view into one (by byte offset, dims, strides and dtype);
- a tensor the call allocates: one it returns, or scratch it drops or declares with
  `scratch()` (placed in a per-call arena);
- the stream (the call's stream argument, or the current stream);
- a constant value: an int, float or bool.

No argument is bound by an address of the recorded call.

## When a call is not recorded

The recording declines, and the signature keeps running the function, when the call
does anything a replay could not reproduce:

- it takes a host tensor (its values could decide the launches unseen);
- a torch operation on a CUDA tensor other than an allocation, a fill, or a view
  (for example arithmetic, a copy, or `.item()`);
- a CuTeDSL kernel launched other than through `launch()`;
- a kernel argument that is none of the above: a tuple, or a tensor or address the
  call was neither passed nor allocated (a buffer the API keeps across calls and did
  not declare with `scratch()`);
- a tensor the call allocates and keeps;
- an allocation on a stream other than the current one or a stream argument.

A kernel reloaded by the persistent compile cache (`cudnn.frost.compiled_cache`) is
watched (`cpp_launcher.watch`), so a direct call of it declines too; any other raw
`tvm_ffi.Function` called directly cannot be seen. A `cpp_launched` function must launch
every kernel through `launch()`, and what it does must follow from its arguments
(and the environment variables it names): a replay does not rerun its Python.

A later call runs the function instead of the recording whenever the function could
have behaved differently on it, since a replay skips the API's checks and choices:

- its argument tensors are less aligned than the recorded call's (up to 256 bytes);
- its argument tensors overlap each other differently from the recorded call's, or
  overlapping ones sit at another byte offset from each other (an API may reject an
  output that aliases an input, pick a packed layout, or address one argument
  through another).

The key also holds the current CUDA device and the values of the environment
variables the function reads per call (`cpp_launched(env=("NAME", ...))`). A call keys the
same however its arguments are passed (by position, by keyword, or left at their
defaults); JAX arrays, tensors inside lists, and functions taking `*args` or
`**kwargs` are not recorded.

Each function keeps its 256 most recently used signatures
(`CUDNN_FRONTEND_CPP_LAUNCHER_MAX`, read at import), so a workload whose shapes keep
changing cannot grow the recordings without bound.

## Checking a `cpp_launched` function

`CUDNN_FRONTEND_CPP_LAUNCHER=verify` (read at import) checks recordings instead of using
them. Every call of a `cpp_launched` function runs the function, then records it and
replays the recording on copies of the arguments at other addresses. The replay has
to return, and leave in its arguments, bit for bit what the call did, or the call
raises. New tensors are zero-filled in this mode so bytes no kernel writes compare
equal. A call inside a CUDA graph capture is not checked. At exit the process prints
how many calls matched and why others were not recorded. Run a `cpp_launched` API's tests in this mode: it is how a launch the recording
misses, or a result that does not follow from the arguments, shows up.

`CUDNN_FRONTEND_CPP_LAUNCHER=0` (read at import) turns the launcher off.