Robot control benchmark
This benchmark measures latency and command timing of a 1 kHz control loop for a simulated
7-DoF arm, using the latest target from an independent 137 Hz producer. GPU runs solve
inverse kinematics through a parallel search on the GPU, comparing clean, contended, and
CUDA Green Context-isolated execution. CPU runs solve inverse kinematics on the CPU,
comparing clean execution with host-side stress-ng load.
CPU and GPU IK perform different workloads, so compare conditions within each backend.
These measurements exclude physical sensors, robot communication, and actuator response.
The primary metric, scheduled activation → joint-command completion, starts when the control tick is due and ends when its command is written to the simulated robot. It captures both the delay before execution and the time spent computing the command.
Methodology
Control loop
Scenarios and workloads
The gpu-contended and gpu-isolated modes run the same operators, rates, and workloads. Their
difference is GPU placement: isolated mode reserves one structured group of streaming
multiprocessors (SMs) for control and assigns the remaining structured SMs to the background
load. This requires a GPU with sufficient CUDA Green Context resources.
On AGX Thor, isolated mode assigns two SMs to control and the remaining 18 to the background load.
Both backends use the same arm model. The CPU backend updates a single starting configuration, while the GPU backend searches many candidates in parallel:
- CPU IK: four double-precision damped least-squares iterations starting from the measured joint configuration, with fixed damping and per-iteration joint-step limits.
- GPU IK: 1,024 reproducible starting configurations, each with four single-precision iterations. The solver selects a bounded command based on pose error and joint movement, or holds the bounded measured configuration if no candidate improves its score.
GPU IK replays a CUDA graph containing input upload, candidate solves, selection, and result download. The control thread spin-polls a CUDA completion event until the result reaches host memory. This trades CPU utilization for lower completion latency. Graph preparation happens on the active stream, including the Green Context stream; keep warmup enabled.
The background GPU operator repeatedly runs bursts of 42 BF16 matrix multiplications (4096 × 4096), reusing fixed-seed pseudorandom inputs and overwriting one output buffer. A 10 µs clock coalesces missed ticks; each burst finishes before the next starts, leaving small scheduling gaps between bursts.
CPU clean and CPU contended use the same CPU IK application. CPU contention comes from an
external workload, such as normal-priority stress-ng arithmetic workers pinned to the CPUs
under test. The control thread retains FIFO priority. Record the stressor method, CPU placement,
and load level with the results: arithmetic load exercises a different kind of interference
from memory or I/O stress. The application does not launch or supervise CPU stressors.
Metrics
The application reports sample count, median (P50), P99.99, and maximum in microseconds for each distribution. The three bold metrics below lead the report. These runs use no Nsight Systems capture or trace analysis.
Read latency together with the correctness counters:
- Late commands: commands completed after their scheduled deadline.
- Coalesced ticks: scheduled activations skipped by runtime coalescing.
- Dropped ticks: control samples lost downstream, tracked separately from coalescing.
The application can finish successfully with late commands or skipped ticks. Those counters show whether it sustained the requested cadence; the latency distributions show how quickly completed commands were produced.
P50 describes typical latency, while P99.99 and maximum describe the tail observed during the run. Include the sample count, run length, source revision, and host configuration when sharing results. The slower target producer yields fewer response samples than control samples, which limits the resolution of its tail percentiles. Empty distributions report N/A. Component costs must be calculated from matching observations rather than differences between percentiles.
❓ How are timestamps and raw results recorded?
The scheduled activation comes from ExecutionContext::activation_time(). Callback entry and
command completion use the same runtime clock. Target history is tracked through warmup: a
target already answered during warmup is excluded from measured target-response samples.
--raw-observations-file PATH saves integer nanosecond timestamps, target versions, and counters
for every measured tick in bench-observations/v3, allowing independent recomputation.
Choose a path writable inside the application container.
Each run also emits a BENCHCSV row with 29 fields:
BENCHCSV, label, scenario, control rate in Hz, target rate in Hz.- Five groups of sample count, P50, P99.99, and maximum, in the metric order above.
Durations are microseconds; a zero-count group is
0,,,. - Late commands, coalesced ticks, dropped ticks, and the schema identifier
bench-latency/v6.
Results
Jetson AGX Thor Benchmark Configuration
- Run date: October 15, 2026 (UTC), one trial per condition on AGX Thor.
- Platform: NVIDIA Jetson AGX Thor Developer Kit; Jetson Linux R39.2.1
(
nvidia-l4t-core39.2.1-20260806224157), NVIDIA driver 595.78; PREEMPT_RT kernel6.8.12-1021-rt-tegra. - Isaac OS commit:
920ba3dd73c2a0c336b17c649fdb7fdcbef34903(internal mirror). - Tools: CUDA 13.2.51; Holoscan SDK 5.0.0, commit
971292c5fb460aa6a1355df4e1864e6a94eedb2e; hoststress-ng0.17.06. - Container image ID:
sha256:be9afbcfa3d1e0b2f029af66ddf8651adfdbedb56c499328958e24da5b617dd8. - Host settings: 120 W power mode (mode 1); CPU 2.601 GHz, GPU GPC 1.386 GHz, GPU NVD 1.557 GHz and EMC 4.266 GHz locked; display manager inactive; CPU idle states disabled.
- RT CPU budget:
kernel.sched_rt_runtime_us=-1(RT bandwidth throttling disabled). - CPU isolation: CPUs 8–13 isolated; housekeeping/IRQ CPUs 0–7. Tickless/RCU boot
options are set, but the running kernel lacks
CONFIG_NO_HZ_FULLandCONFIG_RCU_NOCB_CPU; tickless operation and RCU offload are not enabled. - Configuration:
config/robot_control_benchmark_config.yamlat the commit above, with the explicit overrides below. Target CPU 12, control CPU 13 withSCHED_FIFOpriority 10, GPU load CPU 10, and requested pump CPU 11. Main and runtime-controller threads retain inherited affinity and scheduling policies. Live snapshots confirmed control, target, and GPU-load placement. - Measurement: one trial per condition; 20,000 warmup ticks followed by 3,600,000 measured commands at 1 kHz (nominally one measured hour); target rate 137 Hz. Coalesced or dropped ticks extend the measured schedule span, listed below.
- GPU workload: 1,024 parallel IK candidates with four DLS iterations each. Contended and isolated add the same repeated bursts of 42 BF16 4096 × 4096 GEMMs. Isolated allocates two SMs to control and 18 to contention, as reported by the application.
- CPU workload: four double-precision DLS IK iterations per tick. The stress condition
adds one host
stress-ngCPU worker per CPU 8–13, usingint64at 100% requested load withSCHED_OTHER, nice 0. Once-per-minute samples on housekeeping CPUs 0–7 confirmed stress-worker affinity, scheduling policy, and CPU activity throughout the sampled interval. - Run order: GPU clean → contended → isolated, then CPU clean → with
stress-ng. Each column represents one trial; results are not pooled and there are no repeated trials. - Reporting: application
bench-latency/v6output. All five distributions and the late/coalesced/dropped counters were independently checked againstbench-observations/v3raw observations. No Nsight Systems capture was used.
The schedule span runs from the first measured scheduled activation through one control period after the last; it excludes warmup and result export.
GPU results
Clean runs GPU IK alone. Contended adds GPU load on the shared GPU; isolated runs those same workloads in separate CUDA Green Context SM allocations.
Scheduled-to-completion P99.99 was 6,598.34 µs contended and 266.83 µs isolated, compared with 64.00 µs clean. Only the contended trial recorded late commands, coalesced ticks, or dropped ticks.
Full Results
CPU results
Both conditions use CPU IK in the application’s clean scenario. The stress condition
adds external CPU load, including on the target and control CPUs.
Scheduled-to-completion P99.99 was 18.34 µs clean and 16.74 µs with stress-ng. Both trials recorded zero late commands, coalesced ticks, and dropped ticks.
Full Results
Late commands exceed their scheduled deadline (1 ms at 1 kHz). Coalesced ticks and downstream-dropped ticks are separate outcomes; the application applies no automatic pass/fail policy. Long joint-read intervals retain skipped or dropped ticks and are not a jitter metric. Each distribution has its own sample count. Target-response counts are usually smaller than command counts; retain counts and maxima alongside P99.99. Empty distributions are reported as zero samples and N/A, not zero duration.
Run the benchmark
Build and run
Run the following commands from the repository root. Complete the build prerequisites, then follow Optimize your host to apply the AGX Thor reference configuration. Keep host settings consistent when comparing results.
On Thor, inspect the setup before running:
The checker compares the observed host settings with the Thor reference configuration and reports differences and inspection errors. Review its findings before measuring performance; application placement is checked separately at startup.
Build and test:
Start with CPU IK:
The default run collects 20,000 commands after 1,000 warmup ticks. At 1 kHz, this takes about
21 seconds plus startup, or longer if ticks are skipped. On completion, the terminal shows the
latency distributions, correctness counters, and a machine-readable BENCHCSV row.
To compare the GPU scenarios, run them sequentially:
For a short smoke test:
Configuration and placement
config/robot_control_benchmark_config.yaml supplies rates, sample counts, timeout, and CPU
assignments. CLI arguments override YAML values. The run modes select the backend and scenario.
Main and runtime-controller threads retain inherited affinity and scheduling policies. For hosts with a different CPU layout, change the YAML assignments or pass overrides:
The control partition requires permission to enter FIFO scheduling. Its requested placement is checked in the compiled execution plan, and control-path operators verify their CPU at startup. The application does not configure host clocks or CPU isolation.
What should I check when a run fails?
- Host inspection: follow the diagnostic command printed by the checker. If clock inspection
reports that a password is required, run
sudo -vin the same terminal and retry. - CPU placement or scheduling: verify that the configured CPUs are available to the container and that the process has permission to enter FIFO scheduling.
- GPU isolation: check the reported Green Context requirement against the device’s available resources.
- Timeout: the application did not collect the requested samples within its time budget. Inspect its log before increasing the timeout or reducing the sample count.
Measurement configuration
To reproduce the five conditions in Results, use the same build and host
settings for every run. Run the following commands in Bash on the host from the repository
root. CPU assignments assume the Thor reference configuration: confirm CPUs 8–13 are
available and the control thread can use SCHED_FIFO priority 10. The Isaac OS launcher
supplies the container scheduling capability. Record the observed host configuration with
each new set of results.
Keep this shell for the commands below. Use a new trial directory for each set of runs. The sample count requests nominally one measured hour per condition, following 20,000 warmup ticks (nominally 20 seconds). The 10,880-second timeout provides headroom for missed ticks under contention. Ensure sufficient memory and output storage for 3,600,000 observations per run; the application retains observations in memory and exports raw timestamps after measurement.
GPU: clean, contended, and isolated
Run the three modes sequentially. The application starts its own GPU contention operator for the contended and isolated modes. Check the isolated run’s log for the SM allocation.
CPU: clean
Run with no stress workers active:
CPU: with stress-ng
Install stress-ng on the host before the stress run:
Start one normal-priority worker on each CPU 8–13, including target CPU 12 and control CPU 13. Only the control partition has real-time priority; target and support threads compete with the stress workers. The subshell below records the stress tool version and logs, checks that the load remains running, and stops its workers when the benchmark exits. Run it from a normal user shell with nice level 0.
Inspect the stress logs and verify CPU activity during the run. The process checks establish liveness, not sustained utilization. If the load exits early or the benchmark times out, retain the logs and rerun the condition before publishing a complete comparison.
Read the results
Each successful run prints its distributions and counters in application.log, followed
by one machine-readable BENCHCSV row with schema bench-latency/v6. Collect those rows
from the five output directories:
Verify that every run completed its requested 3,600,000 measured commands and produced
one complete row. The distribution order is the same as the Full Results tables. Both CPU
rows have scenario clean; their labels distinguish the stress treatment.
Keep the logs and observations.csv with the recorded host configuration. The raw file
uses bench-observations/v3 and integer nanosecond timestamps, allowing the reported
microsecond distributions to be independently recomputed. See
Metrics for the full row contract.
NVMe interrupts observed during a run
NVMe interrupts on CPUs used by the benchmark can cause occasional increases in control-loop latency.
The benchmark compares NVMe interrupt counters on the configured target, control, pump, and (when enabled) GPU-load CPUs. Its observation window includes graph setup, warmup, and execution, and ends before writing raw observations and printing results. The warning does not stop the application or establish that an interrupt caused a particular latency spike.
The managed_irq flag in the host guide’s isolcpus setting is best effort; matching the host
reference configuration does not guarantee an interrupt-free run. Retain the warning
with your results and inspect /proc/interrupts before and after a repeat run to
identify the active NVMe queues. Stop unrelated disk activity before comparing runs.
If both snapshots were captured successfully, no activity warning means the monitored
counters did not increase. An “Unable to check interrupt activity” warning means the
evidence is unavailable. Neither outcome establishes that interrupt routing changed.