Optimize your host
CPU and GPU frequency changes, idle states and background work cause latency spikes that make control loops jitter and benchmarks hard to reproduce. This guide reduces them on AGX Thor or Linux x86_64 with an NVIDIA GPU, then checks the result. See platform support for device and software versions.
✨ Agent prompt: Optimize my host
Before tuning
-
On AGX Thor, check your host first with
tools/host_preflight.pyfrom the checkout root, and skip sections that already match the reference configuration:Results describe the current boot. Clock locks and runtime
sysctlchanges reset at reboot unless you persist them, so check again after rebooting. See Check your configuration to interpret the findings. -
Record your starting settings so you can restore them.
-
Stop unrelated GPU workloads and profilers before comparing runs.
-
Check effective settings after each change and measure the same workload again.
-
Record CPU, GPU, driver, CUDA, kernel and tuning settings with your results.
Higher clocks and reduced CPU idle time increase power use and cooling needs. Choose settings for your workload rather than applying every option.
Install the host and kernel
Choose the installation and kernel first; later boot options apply to the selected kernel.
AGX Thor
Linux x86_64
Keep the host installation small
Container-based applications need the Jetson Linux board support package (BSP), drivers and container runtime. They do not need host copies of the full JetPack SDK, samples and developer tools. See Build and run prerequisites for the container runtime.
The desktop is part of the default root filesystem even with Jetson OS alone; follow Run without a desktop after installation if you do not need it.
Use a real-time kernel
PREEMPT_RT allows more kernel work to be preempted, which can reduce scheduling
jitter. Follow the Jetson Linux R39.2.1 real-time kernel instructions
for packages matching your installed release. The RT kernel can be installed
alongside the generic kernel without reflashing.
This real-time kernel is available as a Developer Preview. Evaluate it for your workload before choosing it for deployment.
After installing the RT kernel, set DEFAULT in /boot/extlinux/extlinux.conf
to real-time and reboot. Check the running kernel:
The latter must report 1 for PREEMPT_RT; the kernel name alone is not proof.
Restore the generic kernel
Set DEFAULT in /boot/extlinux/extlinux.conf to primary and reboot.
Reduce scheduling interference
Latency-critical work, such as a control loop, runs best on CPUs that the scheduler, interrupts and kernel threads leave alone. Reserve housekeeping CPUs for system work and isolate the rest with boot options. Applications then place their critical threads on the isolated CPUs; see Isolate CPU execution.
The examples reserve CPUs 0-7 for system work and isolate CPUs 8-13. Replace both CPU lists with the logical CPU IDs for your host, and keep at least one housekeeping CPU. Make sure you can access the host console before editing boot settings. Keep the backup across reboots; if the backup filename already exists, choose a new name instead of overwriting an earlier backup.
AGX Thor
Linux x86_64
The prebuilt JetPack 7.2.1 kernels support these isolation options, which the Thor reference setup uses:
❓ What do these options do?
isolcpus=domainremoves CPUs from scheduler load balancing.managed_irqasks the kernel to avoid them for managed interrupts, on a best-effort basis.irqaffinitysets the default IRQ CPUs;kthread_cpusselects CPUs for eligible kernel threads. Neither guarantees all interrupts or threads move.
Optional: enable tickless and RCU offload with a custom-built kernel
nohz_full suppresses scheduler ticks on selected CPUs when possible;
rcu_nocbs offloads their RCU callbacks. These options require a custom-built
PREEMPT_RT kernel with CONFIG_NO_HZ_FULL and CONFIG_RCU_NOCB_CPU enabled. See
the Jetson Linux real-time kernel build instructions. The prebuilt
kernel does not enable them, so the reference configuration omits them. The
preflight reports missing kernel support or notes that a supported configuration
differs from the reference.
After installing the selected kernel, save a backup of the boot configuration:
Edit the APPEND line for the boot entry you use in /boot/extlinux/extlinux.conf.
Append the options on that same line without removing existing arguments. If you
installed the RT kernel, check which entry DEFAULT selects (real-time or primary).
Reboot.
Restore the boot configuration
Restore the saved file and reboot. Only use this backup while the kernel files referenced by its entries are still installed:
Check cat /proc/cmdline after reboot and inspect /proc/interrupts under load.
If you used a different backup filename, substitute it in the restore command.
Real-time CPU budget
For real-time benchmarks, disable RT bandwidth throttling, as the AGX Thor reference configuration does. A finite budget can suspend continuously runnable real-time workers until the next budget period, adding periodic latency spikes under load.
Record the current value before changing it:
Disabling this limit removes the CPU-time reserve for ordinary tasks. A runaway real-time worker can starve other work on its CPU. Keep benchmark workers on the isolated CPUs and reserve the housekeeping CPUs for system services.
To retain the setting after reboot, add this line to
/etc/sysctl.d/90-isaac-os-rt.conf:
After reboot, verify sysctl kernel.sched_rt_runtime_us reports -1.
To undo the change, remove that line and restore the value you recorded with
sudo sysctl -w kernel.sched_rt_runtime_us=<previous-value>.
Power mode and clocks
Lock clocks after the boot changes above.
AGX Thor
Linux x86_64
For JetPack 7.2.1, check the AGX Thor power guide and confirm which
mode ID selects the power budget you need. The AGX Thor reference uses the qualified
120W mode (mode 1 on T5000).
❓ Why not MAXN?
MAXN is an experimental mode for adjusting clock settings and creating custom power profiles. We do not recommend prolonged heavy workloads in this mode. Higher clock limits do not guarantee higher sustained performance: exceeding the platform power budget can trigger throttling. Use the qualified 120W profile for reference comparisons; treat MAXN measurements as a separate configuration. See the AGX Thor power guide.
Record the current mode and save clock settings:
jetson_clocks locks CPU, GPU and memory clocks to the selected power mode’s maximums.
For the reference setup, select 120W before locking clocks. Follow any reboot
request from nvpmodel before locking clocks:
On the supported Thor installation, jetson_clocks also disables CPU idle states.
The preflight verifies the exposed idle-state controls;
there is no separate PM QoS resume-latency setting to apply.
Keep the settings after reboot
The power mode persists across reboots; clock locks do not. To lock clocks at
every boot, create /etc/systemd/system/isaac-os-jetson-clocks.service:
Enable it, reboot, then check sudo jetson_clocks --show:
To stop locking clocks at boot, run
sudo systemctl disable isaac-os-jetson-clocks.service and reboot.
Restore power and clock settings
If you enabled the boot service, disable it first. To restore a different power mode, run sudo nvpmodel -m <saved-mode-id> and
reboot. The reboot also clears the clock locks.
If you are keeping the current power mode, restore the saved clocks instead:
Run without a desktop
Disabling the desktop frees memory and reduces background CPU/GPU work. On Jetson,
CPU and GPU share memory, so reclaimed desktop RAM is available to either.
Skip this step if your application needs a local display.
Disable optional visualization before running applications without a display.
For SO-101 sinus motion simulation, set sim_so101.enable_camera: false in
applications/so101_sinus_motion/sim/config/so101_sinus_motion_sim_config.yaml.
These commands end graphical sessions. Save your work and run them from SSH or a text console. Check that you can reconnect before changing the boot default.
Record the current state:
Stop the display manager for the current session:
Expect inactive. To restore it, if it was running before:
Optional: boot without a desktop
Change the default for future boots, then reboot:
To restore a previously graphical default:
If your recorded default was different, restore that target instead.
❓ How does this differ from multi-user.target?
stop display-manager.service stops the display manager and its graphical sessions.
systemctl isolate multi-user.target also stops units outside that target’s dependency
set; it can affect more than the desktop. set-default changes the next boot, not the
current session. See systemctl for target behavior.
Check your configuration
AGX Thor
Linux x86_64
Isaac OS reference benchmarks use the AGX Thor reference configuration. Apply it to compare with published results; other settings may suit your workload better.
Inspect the host after your last reboot and just before benchmarking, from the checkout root:
The checker is read-only and compares the host with the reference configuration. If you chose different settings, expect warnings for those differences. Applications still need their own thread placement and scheduling checks.
Warnings (!) flag possible performance effects; information (ℹ) notes reference
differences. Exit codes are 0 for clean or informational results, 1 for incomplete
inspections (✗), and 2 for completed inspections with warnings. Add --verbose
for individual checks and raw readings, or set NO_COLOR=1 to disable terminal colors.
Save a fresh report with each set of results:
The report includes its UTC capture time, checker source hash, and reference revision.
A match means the checked configuration agrees with the reference. Clock checks compare
CPU, GPU, video, and memory readings with the selected mode’s limits in /etc/nvpmodel.conf.
Unlimited (-1) limits use the maximum supported frequency from sysfs. Mismatches warn;
missing limits or readings fail inspection. Actual readings are retained for comparison.
Boot requests such as
kthread_cpus are labeled separately from observed CPU masks.
Clock readings and placement are snapshots; the checker does not monitor throttling,
all interrupt routing, or manually launched desktop processes throughout a run.