> This page is for version 5.0 (Latest) (default).
> For other versions, use one of these documentation indexes:
> - 5.0 (Latest) (default): https://docs.nvidia.com/holoscan/sdk-user-guide/5-0-latest/llms.txt
> - ltsb-2.0: https://docs.nvidia.com/holoscan/sdk-user-guide/ltsb-2-0/llms.txt
> - 4.6: https://docs.nvidia.com/holoscan/sdk-user-guide/4-6/llms.txt
> - 4.5: https://docs.nvidia.com/holoscan/sdk-user-guide/4-5/llms.txt
> - 4.4: https://docs.nvidia.com/holoscan/sdk-user-guide/4-4/llms.txt
> - 4.3: https://docs.nvidia.com/holoscan/sdk-user-guide/4-3/llms.txt
> - Archives: https://docs.nvidia.com/holoscan/sdk-user-guide/archives/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/holoscan/sdk-user-guide/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/holoscan/sdk-user-guide/_mcp/server.

CUDA Graph regions allow you to keep ordinary Holoscan operators while replacing
their steady-state, per-operator dispatch with one CUDA Graph launch. They are
most useful for a short GPU-only chain whose repeated kernel-launch and channel
overhead is significant compared with the GPU work itself.

CUDA Graph regions are explicitly enabled. Adding
`RegionConstraint::cuda_graph()` means "compile this region as a CUDA Graph."
If the compiler is unable to infer whether the region is safe to capture and replay,
the compilation fails. It never silently falls back to ordinary execution.

> **You will learn**
>
> * How to make an operator capture eligible and request CUDA Graph lowering
> * Which graph shapes, triggers, Tensor declarations, and placements are supported
> * What runs once during capture and what repeats on every activation
> * How fixed Tensor slots affect backpressure and output lifetime
> * How to read common compile and runtime failures

> **Note**
>
> Python exposes `RegionConstraint.cuda_graph()`. Native Python operators
> currently cannot declare `cuda_graph_capture_eligible()`.

## What changes at runtime

Without a CUDA Graph region, the scheduler activates each operator and each
internal channel carries a Tensor. With a CUDA Graph region, startup calls each
member's `compute()` once to record its CUDA work. Normal activations then
dispatch the region source as one fused operator and launch the recorded graph.

<title id="cuda-graph-fusion-title">
  Ordinary operator dispatch compared with CUDA Graph region replay
</title>

<desc id="cuda-graph-fusion-description">
  Ordinary execution dispatches multiply and add separately. CUDA Graph execution captures those operators once and replays them as one fused region between the unchanged source and sink.
</desc>

<defs>
  <path d="M 0 0 L 10 5 L 0 10 z" fill="currentColor" />
</defs>

Same operators, different steady-state dispatch

Ordinary execution

Source

Multiply

Add

Sink

scheduler activation

scheduler activation

CUDA Graph replay

Source

Fused CUDA Graph region

Multiply + Add, one graph launch

Sink

The source and sink remain ordinary operators.

The logical graph is still inspectable as the original operators. Fusion is a
physical execution decision in the compiled plan, not a change to the graph
you authored.

## Minimal authoring pattern

Three pieces are required: capture-safe Tensor ports, explicit operator
eligibility, and a constrained region.

### 1. Declare exact, Runtime-owned Tensor outputs

Every output that feeds a captured edge must use Runtime-pool storage and a
fully specified, contiguous Tensor representation. In a CUDA Graph region,
the declared maximum shape becomes the **exact** shape used for capture and
every replay.

```cpp
constexpr std::array<std::int64_t, 2> kShape{1080, 1920};
constexpr DLDataType kUint8{.code = kDLUInt, .bits = 8U, .lanes = 1U};

const holoscan::TensorRepresentation representation{
    .memory_kind = holoscan::MemoryKind::kCudaDevice,
    .dtype = kUint8,
    .rank = 2U,
    .strides = holoscan::TensorStridePolicy::kContiguousOnly,
};

holoscan::TensorOutputSpec output_spec() {
  return holoscan::TensorOutputSpec{
      .representation = representation,
      .bounds = holoscan::tensor_bounds(
          kShape,
          holoscan::TensorBoundsOptions{.max_byte_span = 1080U * 1920U}),
      .storage = holoscan::TensorOutputStorage::kRuntimePool,
  };
}

holoscan::TensorInputSpec input_spec() {
  return holoscan::TensorInputSpec{.representation = representation};
}
```

Set the rank explicitly. `tensor_bounds(kShape)` supplies bounds; it does not
infer `TensorRepresentation::rank`. A loan request with a different shape is
rejected, even when it is smaller than the declared maximum.

See [Tensors in Depth](/holoscan/sdk-user-guide/data-and-payloads/tensors-in-depth)
for the full Tensor allocation and stream-access model.

### 2. Mark each region member capture eligible

The declaration is an explicit promise by the operator author:

```cpp
void setup(holoscan::OperatorSpec& spec) override {
  spec.input(input, "input")
      .queue_depth(1U)
      .expects_tensor(input_spec());
  spec.output(output, "output")
      .max_emits_per_compute(1U)
      .produces_tensor(output_spec());
  spec.cuda_graph_capture_eligible();
}
```

Eligibility alone does not enable capture. The same operator still runs
normally unless it is placed in a CUDA Graph-constrained region.

### 3. Add the linear chain to a constrained region

```cpp
graph.add_flow(source->output, multiply->input,
               holoscan::ConnectionOptions{.queue_depth = 1U});
graph.add_flow(multiply->output, add->input,
               holoscan::ConnectionOptions{.queue_depth = 1U});
graph.add_flow(add->output, sink->input,
               holoscan::ConnectionOptions{.queue_depth = 1U});

graph.region("transforms")
    .add(multiply)
    .add(add)
    .constrain(holoscan::RegionConstraint::cuda_graph());
```

`cuda_graph()` must be the region's only `RegionConstraint`. A separate region
`profile(...)` is permitted because a profile is not a constraint.

## Supported region shape

The first-release model is deliberately narrow: a nonempty linear chain of
directly listed operators in the root graph, with at most one Tensor ingress
and at most one Tensor egress.

<title id="cuda-graph-shape-title">
  Supported CUDA Graph region topology and unsupported examples
</title>

<desc id="cuda-graph-shape-description">
  A supported region is one linear chain with at most one ingress and one egress. Branches, joins, side outputs, and boundaries attached to an intermediate member are not supported.
</desc>

<defs>
  <path d="M 0 0 L 10 5 L 0 10 z" fill="currentColor" />
</defs>

Region boundary rules

Supported

Producer

CUDA Graph region

Source member → ... → Sink member

Consumer

Not supported

<rect x="42" y="188" width="210" height="112" rx="8" fill="currentColor" opacity="0.04" />

Branch or join

fan-out, fan-in, parallel edges

<rect x="305" y="188" width="210" height="112" rx="8" fill="currentColor" opacity="0.04" />

Intermediate boundary

side input or output on a middle member

<rect x="568" y="188" width="210" height="112" rx="8" fill="currentColor" opacity="0.04" />

Embedded graph

nested members are not flattened here

The source may be input-driven or inputless. Boundary-free, ingress-only, and
egress-only chains are also supported as long as the remaining rules are met.
Multiple disjoint CUDA Graph regions may exist in one application.

## Admission checklist

Use this table before enabling the constraint.

| Area           | Required for a CUDA Graph region                                                                                                                                                                                                                                                                                      |
| -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Membership     | Every member is a directly listed root-graph operator, belongs to only one CUDA Graph region, and calls `cuda_graph_capture_eligible()` in `setup()`.                                                                                                                                                                 |
| Topology       | Members form one connected linear chain. There is at most one ingress into the first member and at most one egress from the last member. All region edges carry `Tensor`.                                                                                                                                             |
| Placement      | The complete region, its ingress producer, and its egress consumer are in one process group. The region resolves to one CUDA device and one selected CUDA context. [CUDA Green Context](/holoscan/sdk-user-guide/advanced/cuda-green-contexts) placement is supported when the complete region resolves consistently. |
| Tensor shape   | Dtype and rank are exact; the maximum shape is complete and becomes the exact replay shape. Non-scalar dimensions have positive extents. A scalar region ingress is not supported.                                                                                                                                    |
| Tensor layout  | Captured edges are contiguous. Bounded-strided capture buffers are not supported. Internal edges use CUDA-device memory; boundary edges may use CUDA-device or pinned-host memory. Pageable-host and managed memory are rejected where they cannot preserve the fixed capture contract.                               |
| Tensor storage | Captured outputs use `TensorOutputStorage::kRuntimePool`, `allocate_tensor()`, and the CUDA write path. Each edge has one fixed slot, queue depth one, one emit per compute, and no extra retained/history sample. Pass-through Tensor storage is not supported.                                                      |
| Scheduling     | An input-driven member uses default/`OnAnyInput` behavior or `OnEach` for its exact topology input. An inputless source has exactly one explicit `OnClock`, `OnReadable`, `OnNotified`, `OnDoorbell`, or `OnSpin` trigger.                                                                                            |
| Completion     | `complete_after(ComputeCalls{.limit = N})` is supported only on a zero-input, output-producing region source. Capture does not consume one of those compute calls.                                                                                                                                                    |

The compiler rejects explicit selectors, member `max_start_delay()` budgets,
independent triggers on input-driven members, branches, joins, parallel edges,
non-Tensor edges, cross-process boundaries, and cross-device regions.

## Capture and replay lifecycle

All member operators complete normal lifecycle startup before capture. The
runtime then performs one cold capture on a private, nonblocking stream in the
source operator's CUDA context. After capture, it restores ordinary bindings.
Every application activation launches the graph on the source operator's
already planned stream. The capture stream and replay stream are intentionally
different.

<title id="cuda-graph-lifecycle-title">
  CUDA Graph region startup capture and steady-state replay
</title>

<desc id="cuda-graph-lifecycle-description">
  Operators start normally, their compute callbacks run once on a private capture stream, ordinary bindings are restored, and later activations launch the graph on the source operator stream before publishing the egress asynchronously.
</desc>

<defs>
  <path d="M 0 0 L 10 5 L 0 10 z" fill="currentColor" />
</defs>

One cold capture, then repeated launches

<circle cx="105" cy="107" r="11" fill="currentColor" />

Start

all members

Capture once

member compute() callbacks

private stream, source context

<circle cx="487" cy="107" r="11" fill="currentColor" />

Restore

ordinary bindings

<rect x="576" y="62" width="178" height="90" rx="10" fill="currentColor" opacity="0.04" />

Replay N times

source operator stream

asynchronous egress

CPU member code runs in the capture box, not once per replay.

Only CUDA work recorded during capture repeats.

Egress publication remains asynchronous. The runtime submits the graph and
publishes the output Tensor with normal CUDA readiness; downstream access
performs the necessary ordering without a host synchronization in the region.

## Rules for capture-eligible `compute()`

The key mental model is simple: **the C++ callback runs once, but the captured
CUDA work runs on every activation**.

| Safe inside the region                                                              | Keep outside the region                                        |
| ----------------------------------------------------------------------------------- | -------------------------------------------------------------- |
| Receive the fixed Tensor input                                                      | Per-frame logging, counters, or CPU decisions                  |
| Allocate the declared exact-shape output with `allocate_tensor()`                   | Host writes to captured output memory                          |
| Obtain a CUDA writer and enqueue capture-compatible work on `context.cuda_stream()` | Direct `emit(Tensor)` or pass-through publication              |
| Commit the writer and emit the Tensor loan                                          | Runtime parameter updates or lifecycle restart requests        |
| Use constant configuration frozen before capture                                    | Per-frame changes to launch shape, Tensor address, or topology |

During capture, activation time and rate queries are unavailable, parameter
submissions are disabled, lifecycle restart has no active binding, and
`request_completion()` / `request_stop()` have no effect. Calling
`cuda_graph_capture_eligible()` means the operator author has verified that
the callback does not depend on those services during capture.

Publication metadata follows the fused region boundary. `EmitOptions` emitted
by the final member are recorded during capture and reused on every region
egress. Options produced by the source or internal members are capture-time
intermediates and do not become per-activation metadata. Put dynamically
changing metadata outside the region.

Captured members must also have immutable runtime and lifecycle parameter
state. Compilation rejects a capture region containing an admitted restart
target **or a member of that target's restart closure**. Keep restartable or
dynamically reconfigured operators, including their lifecycle dependents,
outside the region. See
[restart membership](/holoscan/sdk-user-guide/advanced/lifecycle-in-depth#derive-restart-membership-before-its-entry-stage).

## One fixed slot and backpressure

CUDA Graph replay requires stable Tensor addresses, so the current runtime
owns exactly one fixed Tensor slot for each captured edge. A new activation
cannot reuse a slot while the previous activation or its downstream consumer
still owns it.

An outside ingress producer must therefore treat `kResourceExhausted` from
`allocate_tensor()` and `kBackpressured` from publication as normal retry
conditions. Advance producer sequence state only after a successful emit.

```cpp
auto loan = output.allocate_tensor(kShape);
if (!loan) {
  if (loan.error().code == holoscan::ErrorCode::kResourceExhausted ||
      loan.error().code == holoscan::ErrorCode::kBackpressured) {
    return {};  // Keep the same logical item and retry on a later activation.
  }
  return holoscan::make_unexpected(std::move(loan.error()));
}

// Write and commit on context.cuda_stream(), then publish.
auto emitted = output.emit(std::move(*loan));
if (!emitted && emitted.error().code == holoscan::ErrorCode::kBackpressured) {
  return {};  // Do not advance producer state.
}
```

This is ordinary capacity-one backpressure applied to fixed CUDA Graph
storage. A slow consumer that retains the egress can delay the next launch;
the region does not add hidden buffering.

## Diagnosing rejection or failure

Check the compiled plan before starting the session:

```cpp
const holoscan::ExecutionPlan plan = holoscan::compile(graph, options);
if (!plan.ok()) {
  std::cerr << plan.json() << '\n';
  return 1;
}
```

CUDA Graph admission errors identify the region and the violated rule. The
most common causes are:

| Symptom                         | What to check                                                                                                                                                                            |
| ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Region topology rejected        | Confirm one directly listed, connected root-operator chain with no branch, join, side port, parallel edge, or embedded graph.                                                            |
| Member not capture eligible     | Call `cuda_graph_capture_eligible()` in every member's `setup()`. Do not add the outside producer or consumer unless they are intended region members.                                   |
| Tensor shape or layout rejected | Declare exact dtype and rank, complete positive shape bounds, contiguous layout, Runtime-pool storage, and one emit per compute. The requested loan shape must equal the declared shape. |
| Fixed storage rejected          | Use queue depth one, avoid pass-through outputs and retained history, and ensure the ingress producer output has no second consumer.                                                     |
| Scheduling rejected             | Use the exact topology input for `OnEach`, remove member `max_start_delay` contracts and selectors, or give an inputless source exactly one supported explicit trigger.                  |
| Placement rejected              | Keep the region and its immediate boundary operators in one process group and resolve all members to one CUDA device/context.                                                            |

Construction, capture, or launch failures fail the run with region-attributed
diagnostics. CUDA errors are asynchronous and can appear at a later readiness
boundary. Re-run the same operators without the CUDA Graph constraint to
isolate the member operation, then enable the region again.

## Measure before and after

CUDA Graph fusion reduces repeated CPU scheduling, channel traversal, and
kernel-launch overhead inside the region. It does not guarantee lower
end-to-end latency. Small kernels often benefit most, while a slow downstream
consumer or the one-slot lifetime can dominate the result.

The shipped `cuda_graph_region` example runs the same pipeline in both modes:

```bash
./cuda_graph_region --normal
./cuda_graph_region --cuda-graph
```

Its topology is `DeviceSource -> [Multiply -> Add] -> ResultSink`, and its
source demonstrates the required retry behavior. See
[Holoscan by Example](/holoscan/sdk-user-guide/examples/holoscan-by-example)
for the source link and build instructions. Profile both modes on the target
GPU and use [Nsight Systems](/holoscan/sdk-user-guide/performance/nsight-profiling)
to confirm that the fused launch improves the workload that matters.

## Recap

* Opt in twice: mark each member capture eligible, then apply the hard
  `RegionConstraint::cuda_graph()` request to the region.
* Keep the region a direct root-operator linear Tensor chain with at most one
  ingress and one egress, one process group, and one CUDA device/context.
* Use exact-shape, contiguous, Runtime-owned Tensor loans with one fixed slot
  per captured edge.
* Put only capture-compatible CUDA work and fixed publication plumbing inside
  captured callbacks. CPU side effects and control actions run only at capture.
* Expect normal capacity-one backpressure, preserve producer state across
  retries, and compare the result with ordinary execution on your hardware.