SO-101 Reach Task - Part 4: Benchmarking the Pipeline
Latency and cadence tell you whether the reach pipeline keeps up with the arm. This part profiles a mock-mode run with Nsight Systems, reports both for two paths through the graph, and records the conditions behind the numbers.
Prerequisites
Before starting, ensure you have:
- The Isaac OS repository cloned and its Docker image built. See Getting Started.
- A CUDA-capable GPU. The policy runs TensorRT inference.
- Familiarity with the reach graph from Part 3: Deployment.
You do not need to install Nsight Systems. The Isaac OS application container runs both the capture and the report.
What You Will Measure
Mock mode runs the same policy and motion controller as the real arm. A mock state source publishes a constant pose at 100 Hz, and a mock sink receives the commands. With no robot I/O, the measurement isolates the graph and TensorRT inference. The arm never moves, so mock mode does not measure task performance, and every goal times out by design.
This part measures mock mode only. The sim and real modes stop at startup when
Nsight Systems traces NVTX, so they cannot be profiled with this procedure.
The data_flow_tracking report follows messages along a chain of operator ports. It
measures each response from the start of the source operator’s callback to the end of
the sink operator’s callback, so waits between operators count toward the latency.
The solid edges form the command path. The policy path follows the dotted edges into the motion controller, then shares the command edge.
The command path
The controller runs on every state sample. It re-clamps the held policy target against the fresh measurement and emits one bounded command. The latency of this path is how long a new measurement takes to become a command. Its cadence is the command rate, which follows the 100 Hz state rate.
The policy path
The policy path follows a state sample through TensorRT inference to the first command that carries the new target. Its latency is about 20 ms, made of two 10 ms waits rather than inference.
❓ Why is the policy path about 20 ms?
- The policy runs on its own 50 Hz clock, which ticks together with the 100 Hz state source. It takes the freshest state sample, which is usually the one from the previous 10 ms tick, so the sample is already 10 ms old when inference starts.
- Inference takes about 0.2 ms. The controller applies the new target only when the next state sample arrives, 10 ms later.
Rarely, the controller runs right after inference on the same tick, and the response takes about 10 ms.
The policy uses every second state sample, so about half of the samples have no response on this path. That is by design, not message loss.
Set the Measurement Window
Decide the window before you capture, so the run is long enough to cover it. The report counts the warmup and the window in state messages from the source port. Both paths share that source, so one window describes both.
Latency budgets: 10 ms for the command path and 40 ms for the policy path.
❓ Why these settings?
- Warmup skips graph startup and the few states before the first policy target.
- The window leaves about 30 command-path and 15 policy-path responses above p99.9.
- The 400-second completion timeout is above the run’s 311 seconds; the default 30 seconds ends long captures with an error.
The command-path report compares each command interval with the expected period of 10 ms,
the default state_source.period. The policy-path report omits the expected period. A
policy target applied on the same tick produces one 10 ms and one 30 ms interval, so the
period error depends on how often that happens, not on jitter.
- The command-path budget is one state period.
- The policy-path budget is the default
motion_control.target_expiry, the age at which the controller discards a policy target.
For how to size a window for another application, see Choose the warmup and window.
For a window of N state messages, expect these counts:
- Command path:
Eligible Sourcesis N, andSelectedis N minus the states without a valid policy target. In mock mode, that is N. - Policy path:
Eligible Sourcesis N, andSelectedis N / 2, one per inference.
Building and Warming Up
Build the application:
Holoscan SDK 5 emits the data-flow markers that the report reads, so the regular build is the one you profile. Then run mock mode once without profiling:
If no TensorRT engine is cached yet, this run builds one beside the policy. Later runs load the cached engine, so captures start without the engine build.
Capturing a Profile
Close other GPU and CPU-heavy work, such as builds or Isaac Sim, first. The run writes to
its own directory under build/benchmarks/so101_reach/:
The capture takes about 311 seconds. The options match Profile an Application.
The mock sink accepts each command before the controller sends the next one. If a busy
host delays the sink by one state period, the run stops early with kBackpressured in
the log. Count such a run as failed and run it again on an idle host.
Exporting the Capture
From the same host shell, open the run directory in the application container:
Stay in the container until Reading the Results. Export the capture to SQLite with message payloads:
The report matches emitted and received messages by their payloads, so the export needs
--include-blobs=true.
Selecting the Measured Paths
List every path the report can reconstruct:
This writes all-paths_data_flow_tracking.csv with one row per path. Mock mode has exactly
two rows. Save each row’s Port Path on a single line:
Measuring Latency and Cadence
Report each path over the window:
responses=first keeps one command per state sample on each path. See
Customize the report
for every option.
Reading the Results
Read the saved warnings first. All three files are empty for a clean capture:
Then print both measured rows. Nsight Systems adds the report options to each output file name, so find the files by prefix:
These fields matter most:
Leave the container with exit. On the host, read the run summary from the log:
The summary covers the whole run, and the report covers only the window. They agree when the counts reconcile:
- Commands for states.
Sent 31100 joint commands for 31103 joint statesmeans three states arrived before the first policy target and produced no command. Expired targets would also leave states without a command. - Inferences.
Ran 15551 reach policy inferences ... skipped 0matches about half of the states. A skipped inference found no state sample newer than one policy period.
Compare With the Example Results
These example results come from one run of this procedure on an x86 workstation. They are not a benchmark record: published application benchmarks target Jetson AGX Thor. Use them to check that your capture has the expected shape: the counts, the 100 Hz command rate and the policy path of about 20 ms.
Example run configuration
- Run date: October 21, 2026, a 300-second window.
- Host: AMD Ryzen Threadripper PRO 3955WX, Linux 6.8.0-146-generic, no CPU isolation, clocks not locked.
- GPU: NVIDIA RTX 5880 Ada Generation, driver 580.178.04.
- Tools: Holoscan SDK 5.0.0, CUDA 13.2.51, TensorRT 10.16.0.72, Nsight Systems 2026.4.1.191.
- Policy: default model, SHA-256
25e08afb6bc6cd62…, TensorRT engine loaded from cache. - Configuration: default
so101_reach_config.yaml. - Other workloads: a desktop session only; no other compute work on the GPU.
- Measurement: warmup 1,000 and window 30,000 state messages,
responses=first.
📄 Example results
The command-path budget is 10 ms and the policy-path budget is 40 ms.
The run ended with this summary:
Repeating the capture shows how deterministic the pipeline is on your host: percentiles should stay close between runs, while maxima vary more. To reduce latency and jitter, see Optimize your host and Optimize an Application.
Recording Conditions and Artifacts
Record the conditions of the run, following Record conditions and artifacts. On the host:
Record the tool versions and the policy hash from inside the container:
The TensorRT version is part of the engine file name in application.log. Note other
workloads on the host yourself, because nothing in the capture records them.
Keep these files from the run directory:
Troubleshooting
⚠️ Capture and report problems
Summary
In this part you learned how to:
- Know what you measure: the command path and the policy path, and what mock mode can establish.
- Capture a run with Nsight Systems and export it with message payloads.
- Report latency and cadence for both paths over a fixed window.
- Check the counts against the run summary.
- Record conditions and artifacts so others can reproduce the result.
Next Steps
- Profile an Application: every report option, including per-response sample export.
- Optimize an Application: reduce latency and jitter after profiling.
- SO-101 sinus motion benchmark: a published benchmark of another SO-101 application.