CUDA Graph Regions

CUDA Graph regions allow you to keep ordinary Holoscan operators while replacing their steady-state, per-operator dispatch with one CUDA Graph launch. They are most useful for a short GPU-only chain whose repeated kernel-launch and channel overhead is significant compared with the GPU work itself.

CUDA Graph regions are explicitly enabled. Adding RegionConstraint::cuda_graph() means “compile this region as a CUDA Graph.” If the compiler is unable to infer whether the region is safe to capture and replay, the compilation fails. It never silently falls back to ordinary execution.

What changes at runtime

Without a CUDA Graph region, the scheduler activates each operator and each internal channel carries a Tensor. With a CUDA Graph region, startup calls each member’s compute() once to record its CUDA work. Normal activations then dispatch the region source as one fused operator and launch the recorded graph.

The logical graph is still inspectable as the original operators. Fusion is a physical execution decision in the compiled plan, not a change to the graph you authored.

Minimal authoring pattern

Three pieces are required: capture-safe Tensor ports, explicit operator eligibility, and a constrained region.

1. Declare exact, Runtime-owned Tensor outputs

Every output that feeds a captured edge must use Runtime-pool storage and a fully specified, contiguous Tensor representation. In a CUDA Graph region, the declared maximum shape becomes the exact shape used for capture and every replay.

constexpr std::array<std::int64_t, 2> kShape{1080, 1920};
constexpr DLDataType kUint8{.code = kDLUInt, .bits = 8U, .lanes = 1U};
const holoscan::TensorRepresentation representation{
    .memory_kind = holoscan::MemoryKind::kCudaDevice,
    .dtype = kUint8,
    .rank = 2U,
    .strides = holoscan::TensorStridePolicy::kContiguousOnly,
};
holoscan::TensorOutputSpec output_spec() {
  return holoscan::TensorOutputSpec{
      .representation = representation,
      .bounds = holoscan::tensor_bounds(
          kShape,
          holoscan::TensorBoundsOptions{.max_byte_span = 1080U * 1920U}),
      .storage = holoscan::TensorOutputStorage::kRuntimePool,
  };
}
holoscan::TensorInputSpec input_spec() {
  return holoscan::TensorInputSpec{.representation = representation};
}

Set the rank explicitly. tensor_bounds(kShape) supplies bounds; it does not infer TensorRepresentation::rank. A loan request with a different shape is rejected, even when it is smaller than the declared maximum.

See Tensors in Depth for the full Tensor allocation and stream-access model.

2. Mark each region member capture eligible

The declaration is an explicit promise by the operator author:

void setup(holoscan::OperatorSpec& spec) override {
  spec.input(input, "input")
      .queue_depth(1U)
      .expects_tensor(input_spec());
  spec.output(output, "output")
      .max_emits_per_compute(1U)
      .produces_tensor(output_spec());
  spec.cuda_graph_capture_eligible();
}

Eligibility alone does not enable capture. The same operator still runs normally unless it is placed in a CUDA Graph-constrained region.

3. Add the linear chain to a constrained region

graph.add_flow(source->output, multiply->input,
               holoscan::ConnectionOptions{.queue_depth = 1U});
graph.add_flow(multiply->output, add->input,
               holoscan::ConnectionOptions{.queue_depth = 1U});
graph.add_flow(add->output, sink->input,
               holoscan::ConnectionOptions{.queue_depth = 1U});
graph.region("transforms")
    .add(multiply)
    .add(add)
    .constrain(holoscan::RegionConstraint::cuda_graph());

cuda_graph() must be the region’s only RegionConstraint. A separate region profile(...) is permitted because a profile is not a constraint.

Supported region shape

The first-release model is deliberately narrow: a nonempty linear chain of directly listed operators in the root graph, with at most one Tensor ingress and at most one Tensor egress.

The source may be input-driven or inputless. Boundary-free, ingress-only, and egress-only chains are also supported as long as the remaining rules are met. Multiple disjoint CUDA Graph regions may exist in one application.

Admission checklist

Use this table before enabling the constraint.

The compiler rejects explicit selectors, member max_start_delay() budgets, independent triggers on input-driven members, branches, joins, parallel edges, non-Tensor edges, cross-process boundaries, and cross-device regions.

Capture and replay lifecycle

All member operators complete normal lifecycle startup before capture. The runtime then performs one cold capture on a private, nonblocking stream in the source operator’s CUDA context. After capture, it restores ordinary bindings. Every application activation launches the graph on the source operator’s already planned stream. The capture stream and replay stream are intentionally different.

Egress publication remains asynchronous. The runtime submits the graph and publishes the output Tensor with normal CUDA readiness; downstream access performs the necessary ordering without a host synchronization in the region.

Rules for capture-eligible compute()

The key mental model is simple: the C++ callback runs once, but the captured CUDA work runs on every activation.

During capture, activation time and rate queries are unavailable, parameter submissions are disabled, lifecycle restart has no active binding, and request_completion() / request_stop() have no effect. Calling cuda_graph_capture_eligible() means the operator author has verified that the callback does not depend on those services during capture.

Publication metadata follows the fused region boundary. EmitOptions emitted by the final member are recorded during capture and reused on every region egress. Options produced by the source or internal members are capture-time intermediates and do not become per-activation metadata. Put dynamically changing metadata outside the region.

Captured members must also have immutable runtime and lifecycle parameter state. Compilation rejects a capture region containing an admitted restart target or a member of that target’s restart closure. Keep restartable or dynamically reconfigured operators, including their lifecycle dependents, outside the region. See restart membership.

One fixed slot and backpressure

CUDA Graph replay requires stable Tensor addresses, so the current runtime owns exactly one fixed Tensor slot for each captured edge. A new activation cannot reuse a slot while the previous activation or its downstream consumer still owns it.

An outside ingress producer must therefore treat kResourceExhausted from allocate_tensor() and kBackpressured from publication as normal retry conditions. Advance producer sequence state only after a successful emit.

auto loan = output.allocate_tensor(kShape);
if (!loan) {
  if (loan.error().code == holoscan::ErrorCode::kResourceExhausted ||
      loan.error().code == holoscan::ErrorCode::kBackpressured) {
    return {};  // Keep the same logical item and retry on a later activation.
  }
  return holoscan::make_unexpected(std::move(loan.error()));
}
// Write and commit on context.cuda_stream(), then publish.
auto emitted = output.emit(std::move(*loan));
if (!emitted && emitted.error().code == holoscan::ErrorCode::kBackpressured) {
  return {};  // Do not advance producer state.
}

This is ordinary capacity-one backpressure applied to fixed CUDA Graph storage. A slow consumer that retains the egress can delay the next launch; the region does not add hidden buffering.

Diagnosing rejection or failure

Check the compiled plan before starting the session:

const holoscan::ExecutionPlan plan = holoscan::compile(graph, options);
if (!plan.ok()) {
  std::cerr << plan.json() << '\n';
  return 1;
}

CUDA Graph admission errors identify the region and the violated rule. The most common causes are:

Construction, capture, or launch failures fail the run with region-attributed diagnostics. CUDA errors are asynchronous and can appear at a later readiness boundary. Re-run the same operators without the CUDA Graph constraint to isolate the member operation, then enable the region again.

Measure before and after

CUDA Graph fusion reduces repeated CPU scheduling, channel traversal, and kernel-launch overhead inside the region. It does not guarantee lower end-to-end latency. Small kernels often benefit most, while a slow downstream consumer or the one-slot lifetime can dominate the result.

The shipped cuda_graph_region example runs the same pipeline in both modes:

./cuda_graph_region --normal
./cuda_graph_region --cuda-graph

Its topology is DeviceSource -> [Multiply -> Add] -> ResultSink, and its source demonstrates the required retry behavior. See Holoscan by Example for the source link and build instructions. Profile both modes on the target GPU and use Nsight Systems to confirm that the fused launch improves the workload that matters.

Recap

• Opt in twice: mark each member capture eligible, then apply the hard RegionConstraint::cuda_graph() request to the region.
• Keep the region a direct root-operator linear Tensor chain with at most one ingress and one egress, one process group, and one CUDA device/context.
• Use exact-shape, contiguous, Runtime-owned Tensor loans with one fixed slot per captured edge.
• Put only capture-compatible CUDA work and fixed publication plumbing inside captured callbacks. CPU side effects and control actions run only at capture.
• Expect normal capacity-one backpressure, preserve producer state across retries, and compare the result with ordinary execution on your hardware.