Skip to navigation

Optimize your host

View as Markdown

CPU and GPU frequency changes, idle states and background work cause latency spikes that make control loops jitter and benchmarks hard to reproduce. This guide reduces them on AGX Thor or Linux x86_64 with an NVIDIA GPU, then checks the result. See platform support for device and software versions.

✨ Agent prompt: Optimize my host

Before tuning

  • On AGX Thor, check your host first with tools/host_preflight.py from the checkout root, and skip sections that already match the reference configuration:

    sudo -v
    python3 tools/host_preflight.py

    Results describe the current boot. Clock locks and runtime sysctl changes reset at reboot unless you persist them, so check again after rebooting. See Check your configuration to interpret the findings.

  • Record your starting settings so you can restore them.

  • Stop unrelated GPU workloads and profilers before comparing runs.

  • Check effective settings after each change and measure the same workload again.

  • Record CPU, GPU, driver, CUDA, kernel and tuning settings with your results.

Higher clocks and reduced CPU idle time increase power use and cooling needs. Choose settings for your workload rather than applying every option.

Install the host and kernel

Choose the installation and kernel first; later boot options apply to the selected kernel.

Keep the host installation small

Container-based applications need the Jetson Linux board support package (BSP), drivers and container runtime. They do not need host copies of the full JetPack SDK, samples and developer tools. See Build and run prerequisites for the container runtime.

Installation methodMinimal host choice
AGX Thor ISO installerInstall the BSP; the installer includes Docker/container setup without the full JetPack SDK
SDK ManagerFlash Jetson OS; omit optional Jetson SDK components in the second stage

The desktop is part of the default root filesystem even with Jetson OS alone; follow Run without a desktop after installation if you do not need it.

Use a real-time kernel

PREEMPT_RT allows more kernel work to be preempted, which can reduce scheduling jitter. Follow the Jetson Linux R39.2.1 real-time kernel instructions for packages matching your installed release. The RT kernel can be installed alongside the generic kernel without reflashing.

This real-time kernel is available as a Developer Preview. Evaluate it for your workload before choosing it for deployment.

After installing the RT kernel, set DEFAULT in /boot/extlinux/extlinux.conf to real-time and reboot. Check the running kernel:

uname -r
cat /sys/kernel/realtime

The latter must report 1 for PREEMPT_RT; the kernel name alone is not proof.

Set DEFAULT in /boot/extlinux/extlinux.conf to primary and reboot.

Reduce scheduling interference

Latency-critical work, such as a control loop, runs best on CPUs that the scheduler, interrupts and kernel threads leave alone. Reserve housekeeping CPUs for system work and isolate the rest with boot options. Applications then place their critical threads on the isolated CPUs; see Isolate CPU execution.

The examples reserve CPUs 0-7 for system work and isolate CPUs 8-13. Replace both CPU lists with the logical CPU IDs for your host, and keep at least one housekeeping CPU. Make sure you can access the host console before editing boot settings. Keep the backup across reboots; if the backup filename already exists, choose a new name instead of overwriting an earlier backup.

The prebuilt JetPack 7.2.1 kernels support these isolation options, which the Thor reference setup uses:

isolcpus=domain,managed_irq,8-13 irqaffinity=0-7 kthread_cpus=0-7
  • isolcpus=domain removes CPUs from scheduler load balancing. managed_irq asks the kernel to avoid them for managed interrupts, on a best-effort basis.
  • irqaffinity sets the default IRQ CPUs; kthread_cpus selects CPUs for eligible kernel threads. Neither guarantees all interrupts or threads move.

nohz_full suppresses scheduler ticks on selected CPUs when possible; rcu_nocbs offloads their RCU callbacks. These options require a custom-built PREEMPT_RT kernel with CONFIG_NO_HZ_FULL and CONFIG_RCU_NOCB_CPU enabled. See the Jetson Linux real-time kernel build instructions. The prebuilt kernel does not enable them, so the reference configuration omits them. The preflight reports missing kernel support or notes that a supported configuration differs from the reference.

After installing the selected kernel, save a backup of the boot configuration:

sudo cp -a --no-clobber /boot/extlinux/extlinux.conf "$HOME/extlinux.conf.before-isaac-os"

Edit the APPEND line for the boot entry you use in /boot/extlinux/extlinux.conf. Append the options on that same line without removing existing arguments. If you installed the RT kernel, check which entry DEFAULT selects (real-time or primary). Reboot.

Restore the saved file and reboot. Only use this backup while the kernel files referenced by its entries are still installed:

sudo cp -a "$HOME/extlinux.conf.before-isaac-os" /boot/extlinux/extlinux.conf
sudo reboot

Check cat /proc/cmdline after reboot and inspect /proc/interrupts under load. If you used a different backup filename, substitute it in the restore command.

Real-time CPU budget

For real-time benchmarks, disable RT bandwidth throttling, as the AGX Thor reference configuration does. A finite budget can suspend continuously runnable real-time workers until the next budget period, adding periodic latency spikes under load.

Record the current value before changing it:

sysctl kernel.sched_rt_runtime_us
sudo sysctl -w kernel.sched_rt_runtime_us=-1

Disabling this limit removes the CPU-time reserve for ordinary tasks. A runaway real-time worker can starve other work on its CPU. Keep benchmark workers on the isolated CPUs and reserve the housekeeping CPUs for system services.

To retain the setting after reboot, add this line to /etc/sysctl.d/90-isaac-os-rt.conf:

kernel.sched_rt_runtime_us = -1

After reboot, verify sysctl kernel.sched_rt_runtime_us reports -1. To undo the change, remove that line and restore the value you recorded with sudo sysctl -w kernel.sched_rt_runtime_us=<previous-value>.

Power mode and clocks

Lock clocks after the boot changes above.

For JetPack 7.2.1, check the AGX Thor power guide and confirm which mode ID selects the power budget you need. The AGX Thor reference uses the qualified 120W mode (mode 1 on T5000).

MAXN is an experimental mode for adjusting clock settings and creating custom power profiles. We do not recommend prolonged heavy workloads in this mode. Higher clock limits do not guarantee higher sustained performance: exceeding the platform power budget can trigger throttling. Use the qualified 120W profile for reference comparisons; treat MAXN measurements as a separate configuration. See the AGX Thor power guide.

Record the current mode and save clock settings:

sudo nvpmodel -q
sudo jetson_clocks --show
sudo jetson_clocks --store

jetson_clocks locks CPU, GPU and memory clocks to the selected power mode’s maximums. For the reference setup, select 120W before locking clocks. Follow any reboot request from nvpmodel before locking clocks:

sudo nvpmodel -m 1
sudo jetson_clocks
sudo nvpmodel -q
sudo jetson_clocks --show

On the supported Thor installation, jetson_clocks also disables CPU idle states. The preflight verifies the exposed idle-state controls; there is no separate PM QoS resume-latency setting to apply.

Keep the settings after reboot

The power mode persists across reboots; clock locks do not. To lock clocks at every boot, create /etc/systemd/system/isaac-os-jetson-clocks.service:

[Unit]
Description=Lock Jetson clocks for Isaac OS
After=nvpmodel.service
[Service]
Type=oneshot
ExecStart=/usr/bin/jetson_clocks
[Install]
WantedBy=multi-user.target

Enable it, reboot, then check sudo jetson_clocks --show:

sudo systemctl daemon-reload
sudo systemctl enable isaac-os-jetson-clocks.service
sudo reboot

To stop locking clocks at boot, run sudo systemctl disable isaac-os-jetson-clocks.service and reboot.

If you enabled the boot service, disable it first. To restore a different power mode, run sudo nvpmodel -m <saved-mode-id> and reboot. The reboot also clears the clock locks.

If you are keeping the current power mode, restore the saved clocks instead:

sudo jetson_clocks --restore

Run without a desktop

Disabling the desktop frees memory and reduces background CPU/GPU work. On Jetson, CPU and GPU share memory, so reclaimed desktop RAM is available to either. Skip this step if your application needs a local display. Disable optional visualization before running applications without a display. For SO-101 sinus motion simulation, set sim_so101.enable_camera: false in applications/so101_sinus_motion/sim/config/so101_sinus_motion_sim_config.yaml.

These commands end graphical sessions. Save your work and run them from SSH or a text console. Check that you can reconnect before changing the boot default.

Record the current state:

systemctl get-default
systemctl is-active display-manager.service

Stop the display manager for the current session:

sudo systemctl stop display-manager.service
systemctl is-active display-manager.service

Expect inactive. To restore it, if it was running before:

sudo systemctl start display-manager.service

Change the default for future boots, then reboot:

sudo systemctl set-default multi-user.target
sudo reboot

To restore a previously graphical default:

sudo systemctl set-default graphical.target
sudo systemctl start display-manager.service

If your recorded default was different, restore that target instead.

stop display-manager.service stops the display manager and its graphical sessions. systemctl isolate multi-user.target also stops units outside that target’s dependency set; it can affect more than the desktop. set-default changes the next boot, not the current session. See systemctl for target behavior.

Check your configuration

Isaac OS reference benchmarks use the AGX Thor reference configuration. Apply it to compare with published results; other settings may suit your workload better.

Inspect the host after your last reboot and just before benchmarking, from the checkout root:

sudo -v
python3 tools/host_preflight.py

The checker is read-only and compares the host with the reference configuration. If you chose different settings, expect warnings for those differences. Applications still need their own thread placement and scheduling checks.

Warnings (!) flag possible performance effects; information (ℹ) notes reference differences. Exit codes are 0 for clean or informational results, 1 for incomplete inspections (✗), and 2 for completed inspections with warnings. Add --verbose for individual checks and raw readings, or set NO_COLOR=1 to disable terminal colors.

Save a fresh report with each set of results:

python3 tools/host_preflight.py --format json > host-preflight.json

The report includes its UTC capture time, checker source hash, and reference revision. A match means the checked configuration agrees with the reference. Clock checks compare CPU, GPU, video, and memory readings with the selected mode’s limits in /etc/nvpmodel.conf. Unlimited (-1) limits use the maximum supported frequency from sysfs. Mismatches warn; missing limits or readings fail inspection. Actual readings are retained for comparison. Boot requests such as kthread_cpus are labeled separately from observed CPU masks. Clock readings and placement are snapshots; the checker does not monitor throttling, all interrupt routing, or manually launched desktop processes throughout a run.