Performance Considerations

Holoscan 5 compiles graphs into execution partitions, activation rules, and bounded channels. Tune those resources after verifying the pipeline’s delivery and termination behavior on the target machine.

Operator granularity and execution placement

A small compute() can cost less than its dispatch and data movement. Compare one combined operator with reusable operators placed in a suitable partition; measure both under the same workload. The compiler can optimize eligible linear chains, but eligibility depends on the contracts, payloads, and placement. Contract omission is supported; synthesized OnAnyInput has different fast-path and rate-sizing eligibility from explicit OnEach.

Use Graph composition for automatic partitioning, explicit partitions, and deployment thread placement. Measure dispatch and placement costs with the shipped benchmarks under the workload and deployment configuration you intend to use.

Bounded queues and overload

Increasing a queue can absorb a finite burst, but does not make sustained producer/consumer imbalance disappear. kBlocking preserves accepted samples; it does not guarantee that an emit waits until it succeeds. A refusal can return kBackpressured, and propagating it from compute() normally fails the run. kDropOldest changes which samples survive and must be an intentional choice. A slower nominal source is not a guarantee against scheduling stalls.

See Understanding failures for publication ownership and retry constraints, and Performance model for the costs of dispatch, channels, and retention.

Measure comparable workloads

Start with the Core Runtime benchmarks and Channel benchmarks. Keep payload, offered rate, work, backend, queue geometry, partition cover, stream policy, affinity, and termination criteria fixed when comparing authoring choices. Report warmup, hardware, build revision, and measurement scope. A change in fast-path eligibility is not a measured speedup by itself.

Use Nsight profiling to distinguish CPU scheduling, channel pressure, GPU work, and synchronization. A CUDA Graph region can reduce launch overhead for eligible GPU chains; compare ordinary and captured execution with the same downstream Tensor lifetime obligations.

Host CPU Isolation for Real-Time Schedulers

Deployment thread placement controls where execution partitions and configured support threads run. It does not stop unrelated work from using those CPUs. Consider isolation only after traces show that host contention causes material jitter; reserve housekeeping CPUs for the OS and unpinned helpers.

The following kernel command-line parameters work together:

• isolcpus=<list>
  • What it does: Removes the listed cores from the kernel scheduler’s general load-balancing domain so threads land on them only when explicitly pinned (via deployment affinity, taskset, or sched_setaffinity).
  • Cost / tradeoff: Those cores no longer participate in general workload balancing — your housekeeping cores absorb everything that is not explicitly pinned, so size the unisolated pool for the rest of the system’s load.
  • How to tune: Apply to exactly the cores your pinned execution partitions and configured support threads use. Skip it if your pipeline already runs comfortably on a shared core pool; isolation only helps when measurable jitter from co-scheduled work is the bottleneck.
• nohz_full=<list>
  • What it does: Disables the periodic scheduler tick on the listed cores while only one task is runnable on the core, typically reducing timer-interrupt jitter for long-running RT workers.
  • Cost / tradeoff: Tick offload work shifts to the timekeeping (housekeeping) core, and on most kernels the tick is only suppressed when a single task is runnable on the core — additional pinned threads on the same core re-enable it.
  • How to tune: Use the same list as isolcpus, but ensure each nohz_full core hosts only one runnable RT task. Skip it if you have not observed tick-driven jitter in profiling traces; the benefit is small for workloads already dominated by GPU or I/O latency.
• rcu_nocbs=<list>
  • What it does: Offloads RCU (read-copy-update) callback processing from the listed cores onto dedicated kernel threads that run elsewhere.
  • Cost / tradeoff: The offloaded RCU callbacks consume cycles on your housekeeping cores instead, so a heavily loaded RCU subsystem can add load there.
  • How to tune: Match the list to isolcpus / nohz_full. This is generally cheap to enable and worth doing whenever you isolate cores.

These three parameters are typically used with the same core list.

CPU Frequency Governor

• What it does: The performance governor pins each selected core to its maximum operating frequency, eliminating frequency-scaling and wake-from-idle latency that the default ondemand / schedutil governors introduce.
• Cost / tradeoff: Idle power draw and thermal output rise on the affected cores; on battery- or thermally-constrained systems this is a real cost.
• How to tune: Apply only to the cores hosting your real-time execution partitions. Skip it if you have not seen frequency-scaling-induced jitter in your traces — on many server-class systems the default governor is fast enough that pinning to performance is unnecessary.

sudo cpupower frequency-set -g performance

To target only specific cores:

sudo cpupower -c 1-6 frequency-set -g performance

Worked Example

For an application that explicitly places its execution partitions on CPUs 1–6, a matching isolation configuration could be:

isolcpus=1-6 nohz_full=1-6 rcu_nocbs=1-6

On x86 hosts, this is set in /etc/default/grub (the GRUB_CMDLINE_LINUX line) followed by update-grub. On IGX and Jetson, it is set in /boot/extlinux/extlinux.conf under the APPEND line of the active boot entry. Refer to your platform’s documentation for the exact procedure and reboot requirements.

After rebooting, set the governor for the selected cores, then run the application with its graph/deployment affinity configuration:

sudo cpupower -c 1-6 frequency-set -g performance
./my_holoscan_app

Verification

Confirm the kernel sees the isolated set:

cat /sys/devices/system/cpu/isolated
# Expected: 1-6

Confirm a running thread is pinned where you expect (replace <pid> with a worker thread ID from ps -eLo pid,tid,comm | grep my_app):

taskset -c -p <pid>
# Expected: pid <pid>'s current affinity list: 2

Holoscan keeps its own unpinned helper threads off the isolated set. The HoloMQ and Python Tensor release workers run on the process’s CPUs minus the isolated ones. Before it pins any thread, the Runtime creates the CUDA context of each device a Graph uses, and launches one host function on it, from a thread on those CPUs. CUDA’s own threads take the CPU set of the thread whose call starts them: context creation starts its event-handler threads, and the first host-function launch starts the thread that runs host functions. If your application initializes CUDA or launches host functions itself before running the Graph, do it from an unpinned thread. A CUDA thread started on an isolated, pinned thread stays there, which can delay CUDA host functions, and the HoloMQ buffer releases that depend on them, by milliseconds.

Recap

• Measure operator boundaries and partitions on the target workload.
• Queue size, loss policy, and producer pacing are separate decisions.
• Keep experimental controls fixed and record provenance for performance claims.
• Use the SDK’s current benchmarks and deployment placement APIs.