Skip to navigation

Optimize an Application

View as Markdown

Improve application performance and make critical workloads more predictable with Holoscan SDK. Use profiling to identify bottlenecks, partition CPU and GPU resources, and reduce delays from memory allocation, synchronization, and data dependencies.

✨ Agent prompt: Optimize an application

Optimize the host

Ensure that your host has been configured for optimal performance. Work through the instructions for your platform in Optimize your host.

Profile your application

Start by identifying where your application spends time and which paths miss their latency budgets. Profile an Application shows how to measure source-to-sink latency and message rates, including tail latency and budget exceedances on critical paths.

To investigate delays around GPU work, capture additional CUDA detail with Nsight Systems. Pass --trace=cuda,nvtx,osrt through the run command’s --nsys-args to see kernel launches, transfers, synchronization, and thread waits. If the delay occurs before a CPU thread runs, add CPU sampling or context-switch tracing.

Use the Nsight diagnostic queries to inspect CUDA API calls, synchronization and kernel activity in a SQLite export.

Identify areas of improvement

The following sections address resource sharing, processing costs, recurring execution overhead, and dependencies between workloads. Use your profiling results to choose where to start:

What needs improvementRelevant guidance
Critical workloads slow down when other workloads runPartition CPU and GPU resources to control placement and resource allocation. For GPU interference that remains, investigate work-queue contention and bound outstanding GPU work.
Image, tensor, or other processing takes a substantial share of execution timeAccelerate processing stages using an implementation suited to the operations and data layout.
Memory management or synchronization adds delays during executionReuse allocated memory and remove unintended synchronization.
Launching or scheduling many small operations takes a substantial share of execution timeUse CUDA Graphs or group small sequential operations.
A critical workload waits for a slower producer or processes stale inputsReview the architectural patterns for different-rate workloads, including publishing completed results and selecting the latest usable input.

Partition CPU and GPU resources

To isolate critical workloads from competing work, place them in separate Holoscan execution partitions. Operators within a partition share a serial execution thread, while different partitions can run concurrently. Through deployment bindings, each partition can have CPU placement, scheduling policies, and GPU SM allocations suited to its workload.

Isolate CPU execution

To keep competing work from delaying critical callbacks, combine two layers of CPU isolation:

  • CPU affinity controls where a partition runs. Assign critical workloads CPU sets separate from competing work. Combine affinity with the host CPU isolation settings to move background work and interrupts away from those CPUs; affinity alone does not reserve those CPUs exclusively.
  • Scheduling policy and priority control which runnable thread runs first. SCHED_FIFO and SCHED_RR give real-time threads precedence over ordinary SCHED_OTHER work. Both use priorities from 1 to 99, with larger values taking precedence. FIFO runs until it blocks, yields, or is preempted by higher-priority work; round-robin also time-slices threads at the same priority.

For the most responsive enforcement of Linux real-time scheduling priorities, use a PREEMPT_RT kernel. It makes most kernel work preemptible, moves most interrupt handling into schedulable threads, and uses priority inheritance for kernel locks, allowing a ready, high-priority thread to run sooner. Standard kernels support the same scheduling policies, but non-preemptible kernel work can delay their effect. On AGX Thor, follow the host guide to install and select the real-time kernel.

To give a critical workload its own CPU affinity and scheduling priority, create a named partition and bind its thread placement before compiling the graph. For example, to isolate a critical control workload from inference, pin control to CPU 8 with FIFO priority 40 and inference to CPU 9 with the default policy. The example assumes controller and inference are operator handles already created in the graph; auto_partition_remaining() partitions any other operators:

graph.partition("control").add(controller);
graph.partition("inference").add(inference);
graph.auto_partition_remaining();
holoscan::CompileOptions options;
options.deployment
.bind_thread_placement(
"control",
holoscan::ThreadPlacement{
.affinity = holoscan::CPUAffinity{{8}},
.scheduling = {.policy = holoscan::SchedulingPolicy::kFIFO,
.priority = 40},
.probe_failure_policy = holoscan::ProbeFailurePolicy::kFailAdmission})
.bind_thread_placement(
"inference",
holoscan::ThreadPlacement{.affinity = holoscan::CPUAffinity{{9}},
.probe_failure_policy =
holoscan::ProbeFailurePolicy::kFailAdmission});

Choose CPU IDs from the sets available to your container, leaving CPUs for OS and runtime support work. To place the runtime controller or Channel pump separately, use bind_support_thread_affinity(). If workloads share CPUs, keep continuously runnable background work below critical workloads’ FIFO priorities to avoid starving them.

To select a different scheduling policy, use Holoscan’s kOther, kFIFO, kRoundRobin, or kDeadline, corresponding to the Linux policies above and SCHED_DEADLINE. Deadline scheduling takes runtime, deadline, and period budgets in nanoseconds instead of a fixed priority.

These placement requests require permission from the host and container. The example uses kFailAdmission to reject requests whose placement cannot be proven; check plan.ok() and its diagnostics for admission failures. Once running, inspect the threads with ps -L -p <pid> -o tid,cls,rtprio,psr,comm, then check their effective affinity and policy with taskset -pc <tid> and chrt -p <tid>. For a complete application configuration, see GR00T/WBC CPU placement.

Isolate GPU work with CUDA Green Contexts

Separate CPU threads can still compete for the same GPU streaming multiprocessors (SMs). To keep other GPU work from occupying all available SMs, use Holoscan’s CUDA Green Context placement to reserve subsets for critical workloads.

Continuing the control-and-inference example, reserve SMs for the critical control partition as follows:

  1. Find a supported allocation size by querying the deployment GPU with holoscan::query_cuda_green_context_capability(holoscan::DeviceId{0}) and check the result. Choose a positive multiple of structured_sm_quantum, within the reported structured capacity, leaving capacity for other CUDA work.

  2. Assign that allocation to the named control partition before compilation. Here control_sms is the selected count, and options is the same CompileOptions used for CPU placement:

    options.deployment.bind_cuda_green_context(
    "control", holoscan::CudaGreenContextPlacement{
    .device = holoscan::DeviceId{0},
    .sm_count = control_sms});
  3. Compile with those options and check the plan’s diagnostics, cuda_green_context_requests(), and cuda_green_context_assignments(). The capability query does not reserve resources; runtime startup also validates the actual device and creates the contexts and streams.

  4. Submit operator CUDA work on context.cuda_stream().get(). Configure external libraries to use that stream too. Work launched on an unrelated library or default stream is outside this placement contract.

In this example, the critical control workload uses its reserved SMs, and other Holoscan CUDA work in the same process on that device uses a residual Green Context. Holoscan SDK currently requires one Green-Context-owning runtime process per physical GPU; it does not coordinate SM reservations with unrelated CUDA applications.

Workloads still share memory bandwidth, caches, copy engines, and other GPU scheduling resources. If a critical workload stalls while its SMs are idle, investigate allocations, synchronization, and queued GPU work in the following sections. For an application using both CPU and GPU placement, see the GR00T/WBC placement example and its measured results.

Accelerate processing stages

Use CPU and GPU profiles to identify processing steps whose computation or intermediate memory traffic adds latency or jitter, such as image conversion, tensor indexing, or filtering.

Consider existing operators and optimized libraries suited to the workload. For regular tensor operations, MatX provides GPU implementations of element-wise arithmetic, type conversion, permutation, and indexed gather/scatter in C++. Its expression fusion combines compatible element-wise operations within one expression into a kernel without intermediate tensors.

LibTorch (PyTorch’s C++ API) also provides GPU tensor operations with CUDA stream and graph support, with an additional runtime dependency to package and maintain. Consider a custom CUDA kernel when available implementations do not fit a specialized transformation or layout.

See GR00T GPU execution for MatX joint indexing and a fused image kernel captured with TensorRT inference.

Programming guidelines for improving determinism

Resource isolation limits competition, but allocation, synchronization, and kernel submission can still introduce variable delays within a workload. To make recurring GPU execution more predictable, reuse allocated memory, keep synchronization scoped to the work that needs it, and use CUDA Graphs to reduce repeated launch overhead. The following guidelines address each source of variation.

Eliminate CUDA allocations from steady-state execution

CUDA allocation and deallocation can introduce synchronization or long host stalls that affect other workloads sharing the GPU. If the trace shows recurring cudaMalloc, cudaFree, host-pinning calls, or framework allocator misses, move the corresponding model setup, workspace allocation, and buffer creation into initialization.

Avoid unnecessary buffer initialization, such as filling a buffer with zeros when the next operation writes every value before reading it.

For outputs that need fresh storage on every callback, use Holoscan’s runtime-owned Tensor pools. Declare bounded output shapes and TensorOutputStorage::kRuntimePool, then borrow storage with allocate_tensor(). Size bounds and queues for the intended number of in-flight messages. Scratch buffers that remain private to an operator can instead be allocated once and reused on its assigned stream. For asynchronous host/device transfers, preallocate pinned host buffers too.

Use the SDK’s Tensor leases and stream ordering to avoid overwriting buffers still in use downstream. Release imported DLPack objects according to their ownership contract. If a pool gradually exhausts, inspect retained messages and leases before increasing its capacity.

Remove unintended synchronization

Legacy-default-stream operations and synchronous copies can couple otherwise independent GPU workloads. To preserve their independence, submit kernels and asynchronous copies on the Holoscan-provided stream. With MatX, pass that stream to matx::cudaExecutor when running expressions. When using PyTorch, establish a stream guard for that external stream around Torch operations in each callback according to its stream semantics; a stream selected on another host thread does not apply there.

Host readbacks are another source of waits: tensor.item(), CPU transfers, and data-dependent shape operations can force the CPU to wait for GPU results. Keep intermediate results on the GPU where possible. When the CPU needs a result, copy it into a preallocated pinned buffer and wait for completion before reading it. Pageable host memory can make even an asynchronous copy block the calling thread; see CUDA synchronization behavior.

When the result stays on the GPU, Holoscan’s Tensor stream contract provides readiness and release fences that order access without blocking the host.

Investigate CUDA work-queue contention

Independent streams can still share CUDA hardware work queues, creating false dependencies. If interference remains after fixing stream dependencies and allocations, compare the default CUDA_DEVICE_MAX_CONNECTIONS value with 32. Set it in the application process environment before CUDA initialization, including inside the container.

More connections can reduce queue sharing, but do not reserve an exclusive queue for each stream or guarantee concurrency. Applications with few active streams may see no benefit. See the CUDA environment-variable reference.

Use CUDA Graphs to reduce launch overhead

When a repeated GPU chain submits many small kernels, capture its work once and replay it as a CUDA Graph to reduce host launch overhead and interference with other workloads.

To capture a chain of Holoscan operators, place it in a CUDA Graph region. The chain must use linear Tensor connections, fixed shapes, runtime-pooled storage, queue depth one, and one consistent CUDA device/context. Mark each member with spec.cuda_graph_capture_eligible(), then apply the region constraint:

graph.region("transforms")
.add(preprocess)
.add(transform)
.constrain(holoscan::RegionConstraint::cuda_graph());

The runtime captures these callbacks once and replays their GPU work on subsequent activations. CPU side effects therefore occur only during capture; keep per-cycle CPU logic outside the region.

If an operator captures a library or inference engine internally, prepare that capture so each replay uses valid buffers, state, and placement:

  • Warm up the engine and allocate its workspaces before capture.
  • Keep captured input/output addresses stable; stage new inputs into those buffers and refresh changing state, such as random inputs, before each replay.
  • Capture in the assigned CUDA context. Launching a graph captured in the ordinary context on a Green Context stream does not retroactively place captured work. Holoscan regions manage capture and replay streams in the selected context.
  • Remove data-dependent host readbacks only when an equivalent fixed-shape computation is possible. Bake a value into an exported model only if it is constant for every supported input in that deployment.

Submit capture-compatible MatX operations through its CUDA executor on the capturing stream. The GR00T implementation captures joint gather/scatter together with inference. For LibTorch-managed capture, use at::cuda::CUDAGraph. PyTorch’s capture guidance covers warmup and long-lived input/output tensors; its graph memory management describes private memory pools configured by PyTorch’s capture APIs.

Architectural patterns for different-rate workloads

Resource isolation reduces competition, but data dependencies can still make a critical workload wait for other work. Choose how results cross each dependency boundary based on where they are consumed and how fresh they must be.

Publish completed results

To keep a GPU-to-CPU transfer wait out of a critical consumer’s callback, complete the transfer in the producing workload before publishing its CPU-readable result. Place the producer in its own execution partition and signal readiness with a completion event, or wait on its stream when a blocking callback is acceptable. If the consumer also runs on the GPU, use Holoscan’s asynchronous Tensor path instead.

Consume the latest completed result

Publishing completed results avoids transfer waits, but requiring a new result on every activation still ties a consumer to its producers’ rates. If a critical workload can use older results, let it run at its own cadence using the latest completed inputs. For example, a critical control workload can consume the latest completed inference result instead of waiting for the next inference.

Use Holoscan temporal contracts to define the consumer’s cadence, sample-age limits, and fallbacks for missing or stale inputs. Preserve capture timestamps, since processing delays and dropped updates can make results older than a producer’s expected period.

To keep only the newest arrival when intermediate results may be dropped, use a depth-one QueuePolicy::kDropOldest connection. This sets queue retention; the temporal contract still determines sample selection. Larger queues retain more history but can increase output age under overload.

Bound work submitted ahead of GPU completion

A callback can finish submitting GPU work while that work is still running. A small message queue therefore does not by itself bound the amount of queued GPU work. If a workload runs ahead and delays critical kernels, gate its next submission on completion of the previous one, or enforce another measured finite limit.

To enforce the limit, use a completion event to control readiness for the next submission, or wait on that workload’s stream before returning from its callback. The blocking form occupies that partition’s thread. Apply the same limit to CUDA Graph replays if their submissions can outpace GPU completion.

Group small sequential operations

If scheduling and message transfers dominate tiny computations, group related sequential work into one operator or use an eligible CUDA Graph region. Retain separate partitions for workloads that need concurrency; a larger callback delays other work in its partition.