Operating System Settings#

This chapter describes operating system settings that can affect Vera CPU and Vera Rubin system performance.

Note

Kernel configuration, boot parameters, and operating system defaults vary by distribution and kernel flavor. Verify the effective settings on the target system before you apply these recommendations.

CPU NUMA Affinity#

Important

On Vera-Rubin platforms, it is critical that you use numactl to bind your applications to the local CPU and memory that are in the same NUMA domain. Otherwise the cross-socket performance will be poor.

Determine on which NUMA nodes your applications will run. Refer to Checking the CPUs and Checking the Non-Uniform Memory Access Settings for more information.

To bind to socket 0:

numactl -N 0 -m 0 <your_app>

To bind to specific CPU cores on socket 0:

numactl -C 0-87 -m 0 <your_app>

To bind to socket 1:

numactl -N 1 -m 1 <your_app>

To bind to specific CPU cores on socket 1:

numactl -C 88-175 -m 1 <your_app>

Spatial Multithreading#

The NVIDIA Olympus core in Vera supports Spatial Multithreading (SMT), which exposes two virtual processors for each physical CPU core to the operating system. Each virtual processor is also called a Processing Element (PE), and the two PEs that belong to the same physical core are thread siblings.

Topology#

Each Vera socket has 88 physical cores. With SMT enabled, this becomes 176 PEs per socket. On a single-socket system, the first PE of each core is typically numbered 0-87, and the second PE is typically numbered 88-175. For example, logical CPUs 0 and 88, and logical CPUs 1 and 89, correspond to the same physical core. On multi-socket systems, CPU numbering can differ, so detect the topology at runtime in one of the following ways:

  • lscpu --extended=CPU,CORE shows which logical CPU corresponds to which physical core.

    lscpu --extended=CPU,CORE
    CPU CORE
    0    0
    1    1
    2    2
    3    3
    4    4
    ........
    84   84
    85   85
    86   86
    87   87
    88    0
    89    1
    90    2
    91    3
    ........
    172   84
    173   85
    174   86
    175   87
    
  • The thread siblings are exposed using sysfs:

    $ cat /sys/devices/system/cpu/cpu0/topology/thread_siblings_list
    0,88
    $ cat /sys/devices/system/cpu/cpu1/topology/thread_siblings_list
    1,89
    $ cat /sys/devices/system/cpu/cpu88/topology/thread_siblings_list
    0,88
    

Controlling SMT#

A BIOS option is available to disable SMT at boot time. If SMT is disabled at boot time, the operating system sees only 88 processors per socket. Otherwise, the operating system sees 176 processors per socket.

At runtime, to disable SMT:

echo off | sudo tee /sys/devices/system/cpu/smt/control

When disabled at runtime:

  • On a single-socket system, nproc reports 88 processors. On multi-socket systems, the total scales with the number of sockets.

  • Processors 88-175 are marked as offline in the operating system on a single-socket system.

  • /sys/devices/system/cpu/smt/active reports 0.

To enable it again at runtime:

echo on | sudo tee /sys/devices/system/cpu/smt/control

After enabling again:

  • On a single-socket system, nproc reports 176 processors. On multi-socket systems, the total scales with the number of sockets.

  • Processors 88-175 are marked as online in the operating system on a single-socket system.

  • /sys/devices/system/cpu/smt/active reports 1.

Vera also supports toggling SMT for individual cores at runtime. For example, to disable SMT on physical core 0, take one of its thread siblings offline, such as logical CPU 88 on a single-socket system:

echo 0 | sudo tee /sys/devices/system/cpu/cpu88/online

To bring SMT back online at runtime:

echo 1 | sudo tee /sys/devices/system/cpu/cpu88/online

When SMT is enabled for at least one core on the system, /sys/devices/system/cpu/smt/active reports 1.

Page Size#

Vera supports 4 KB and 64 KB Linux host kernel page sizes. Choose the host kernel page size based on the software stack and workload that you need to run:

  • Use a 4 KB host kernel for Arm Confidential Computing Architecture (CCA) systems. Vera CCA software and hardware checkout flows require a 4 KB host kernel, and CCA is not supported with a 64 KB host kernel. Realm guests can use 4 KB, 16 KB, or 64 KB guest kernel page sizes, but CCA realm internals use 4 KB pages. For a short overview of Vera CCA features, refer to Confidential Compute.

  • Use a 64 KB host kernel for non-CCA systems and general page-size or performance tuning, unless your workload or software stack explicitly requires a 4 KB host kernel.

To configure the Linux host kernel page size, use one of the following Kconfig settings during kernel compilation:

  • 4 KB page size: CONFIG_ARM64_4K_PAGES=y

  • 64 KB page size: CONFIG_ARM64_64K_PAGES=y

The 64 KB page size can benefit applications that allocate large amounts of memory because it can reduce page faults and improve TLB hit rates. The 4 KB page size is required for CCA host kernels and can be useful when compatibility with software that assumes 4 KB pages is more important than the potential performance benefit of a larger base page size.

Note

For non-CCA systems, the recommended default host kernel page size is 64 KB. For CCA systems, use a 4 KB host kernel.

Huge Pages#

Huge pages can benefit applications that allocate large memory regions by reducing TLB misses.

You can use huge pages on Vera systems in the following ways:

  • Transparent Huge Pages (THP)

    • Transparent to the application.

    • Mostly automatic with a few available kernel tuning parameters.

    • When using a 64 KB host kernel page size, the default THP size is 512 MB. Linux 6.9 and later support multi-size THP (mTHP) for smaller THP sizes such as 2 MB (refer to Transparent Huge Pages for more information).

  • Hugetlbfs

    • Does not suffer from fragmentation concerns or from allocation latency because the huge pages are preallocated and indivisible.

    • Requires application modification.

    • Requires sysadmin setup.

Transparent Huge Pages#

Transparent Huge Pages (THP) is transparent to applications. Applications can benefit from huge pages without source code changes.

THP policy can be set to always, madvise, or never. When set to madvise, THP is allocated only for memory regions where the application requests huge pages with madvise(MADV_HUGEPAGE).

Warning

With a 64 KB host kernel page size, the default THP size is 512 MB, which might be too large for many workloads. Linux 6.9 and later support multi-size THP (mTHP), which can use smaller THP sizes such as 2 MB. Prefer 2 MB mTHP unless you have measured a benefit from 512 MB pages.

To check the current THP policy, run the following command. The active policy is marked with brackets:

$ cat /sys/kernel/mm/transparent_hugepage/enabled
always [madvise] never

To change the THP policy, run one of the following commands:

$ echo always | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
$ echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
$ echo never | sudo tee /sys/kernel/mm/transparent_hugepage/enabled

On 64 KB host kernels, 2 MB mTHP requires Linux 6.9 or later. Confirm that the 2 MB sysfs entry exists before changing per-size policy. If this path is absent, 2 MB mTHP is not available on that kernel:

$ ls /sys/kernel/mm/transparent_hugepage/hugepages-2048kB

For most workloads on kernels that support mTHP, disable 512 MB THP and use 2 MB mTHP instead:

$ echo never | sudo tee /sys/kernel/mm/transparent_hugepage/hugepages-524288kB/enabled
$ echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/hugepages-2048kB/enabled

If you have already customized per-size THP policy, record the current values before applying these settings so that you can restore them later.

To restore the documented default per-size policy (512 MB inherits the global THP policy, and 2 MB mTHP is disabled):

$ echo inherit | sudo tee /sys/kernel/mm/transparent_hugepage/hugepages-524288kB/enabled
$ echo never | sudo tee /sys/kernel/mm/transparent_hugepage/hugepages-2048kB/enabled

Linux first-touch NUMA policy places a huge page on the NUMA node of the first CPU that writes to it. The page will not be migrated due to subsequent accesses from other sockets. With 512 MB THP, a large aligned allocation that is initialized with OpenMP (or similar) first-touch can therefore remain in one socket and cause remote memory traffic. The same mechanism exists for every page size, but the chance of sharing a page across sockets scales with page size.

This guide recommends disabling Automatic NUMA Balancing, so the kernel does not migrate pages after first touch. Refer to Automatic NUMA Scheduling and Balancing for more information.

HPC applications that run at least one MPI rank per socket typically allocate per socket and are less exposed to this issue. Workloads that benefit from 512 MB pages can re-enable them.

Starting with glibc 2.43 (for example, Ubuntu 26.04), malloc requests THP by default on AArch64, which is equivalent to the glibc tunable glibc.malloc.hugetlb=1. glibc does not choose the kernel huge-page size. On a 64 KB host kernel, those allocations are typically backed by 512 MB THP if 512 MB pages are enabled. The allocation then behaves like a 512 MB first-touch region: the first CPU that writes to the page places the entire 512 MB page, and threads on other sockets incur remote memory traffic.

To stop malloc from requesting THP for a process, set GLIBC_TUNABLES=glibc.malloc.hugetlb=0.

Refer to Transparent Hugepage Support for more information about THP and mTHP.

Note

The default huge page size is related to the kernel page size. Refer to HugeTLBpage on ARM64 for more information.

Memory Reserved for Huge Page Allocation#

When THP is set to always or madvise, khugepaged raises vm.min_free_kbytes so that the kernel retains enough contiguous free memory for huge-page allocations. With a 64 KB host kernel page size, this reservation can be large:

$ sudo sysctl vm.min_free_kbytes
vm.min_free_kbytes = 17301504

The kernel reserves these pages proportionally across all memory zones, which reduces the memory available to applications on every NUMA node, including the GPU NUMA nodes on Vera Rubin systems. Writing to /sys/kernel/mm/transparent_hugepage/ recalculates the value and overrides any setting that you apply manually.

The reservation scales with the pageblock size, which is set by the CONFIG_PAGE_BLOCK_MAX_ORDER kernel build option, and not with the THP size that you select. To check the value used by the running kernel, run the following command:

$ grep PAGE_BLOCK_MAX_ORDER /boot/config-$(uname -r)
CONFIG_PAGE_BLOCK_MAX_ORDER=10

If your workloads use 2 MB or smaller mTHP, as recommended earlier in this section, a value of 5 optimizes the kernel for 2 MB pages and substantially reduces vm.min_free_kbytes. Changing this option requires a custom kernel build, so validate it against your workload first.

Proactive Compaction#

To reduce fragmentation, proactive compaction performs memory compaction in the background. By making more physically contiguous memory available, it can improve the availability and allocation latency of huge pages and other higher-order allocations.

  • Without proactive compaction, an application can encounter a performance cliff when the kernel must synchronously compact memory to satisfy a huge-page allocation.

  • With proactive compaction, the kernel begins compacting memory in the background when fragmentation reaches a threshold. This can reduce the likelihood and duration of synchronous compaction stalls.

Proactive compaction exposes the /proc/sys/vm/compaction_proactiveness tunable, which accepts values in the 0 to 100 range and has a default value of 20. This tunable determines how aggressively the kernel compacts memory in the background. Aggressive compaction can consume CPU time, migrate application pages, and introduce system-wide latency spikes. The default value is reasonable and should be changed only based on performance data.

To limit the overhead of proactive compaction, you can use on-demand compaction, which is available only when CONFIG_COMPACTION is set. When 1 is written to /proc/sys/vm/compact_memory, all zones are compacted, and free memory is made available in contiguous blocks where possible. This can be useful for huge-page allocation because it directly compacts memory as required. Refer to Documentation for /proc/sys/vm/ for more information.

Hugetlbfs#

With hugetlbfs, the system can preallocate pools of hugetlb pages for applications, but this approach requires application changes. You can specify the minimum number of huge pages reserved by the system, how large the pool can grow, and whether malloc should use hugetlbfs for an application.

Test your application with hugetlbfs before broadly enabling it. If hugetlbfs works for the application, reserving a pool of huge pages at boot time can improve availability because memory is not yet fragmented. As a result, the requested number of huge pages is more likely to be assembled. Refer to HugeTLB Pages for more information.

Configuring Linux Perf#

Refer to Configuring Perf for more information.

Performance Governor#

You can set the CPU governor with the cpupower command. For example, to set the CPU governor to performance, run the following command:

sudo cpupower frequency-set -g performance

Note

On certain distributions, such as Ubuntu, the cpufrequtils package provides a service that might change the CPU governor to ondemand when the system boots. To avoid this behavior, disable the service by running sudo systemctl disable cpufrequtils.

Reducing Periodic Timer Ticks for Streaming Stores#

Some CPU memory write kernels on Vera run most efficiently when the core stays in streaming store mode and stores use the high-bandwidth WriteUnique alloc=0 path. If the same CPU takes frequent timer interrupts, the Arm64 interrupt and exception paths can execute the required DMB instructions. These barriers can reset the store tracker’s streaming state, so the next stores can temporarily use the MakeUnique or the WriteUnique alloc=1 path until streaming behavior is established again. This scenario is most relevant to pinned, long-running, bandwidth-limited CPU store loops that otherwise spend most of their time in WriteUnique alloc=0.

Systems built with CONFIG_HZ=1000 can take a periodic arch_timer interrupt on a busy CPU approximately once per millisecond unless tick suppression is active. For highly tuned RT or HPC-style workloads, consider reducing the periodic OS activity on the CPU cores that run streaming store threads.

To evaluate whether the system is configured for tick suppression, check the kernel configuration, boot parameters, and timer interrupt counts:

grep CONFIG_NO_HZ_FULL /boot/config-$(uname -r)
cat /proc/cmdline
grep arch_timer /proc/interrupts

Consider the following OS settings:

  • Use nohz_full=<cpulist> on CPUs that run one pinned userspace task to suppress periodic ticks while the CPU is in full dynticks mode.

  • Use isolcpus=<cpulist> to keep those CPUs out of the scheduler’s general CPU pool.

  • Use rcu_nocbs=<cpulist> to move RCU callback processing off those CPUs.

  • If kernel rebuilds are practical, use a lower CONFIG_HZ value, such as 250 or 100, to reduce the periodic tick rate.

Leave at least one housekeeping CPU that is outside the nohz_full set. On production systems, reserve enough housekeeping CPUs for timers, RCU, kernel threads, interrupts, and management agents. These parameters do not replace IRQ affinity tuning for device interrupts. For example, on a system where the workload CPU partition is represented by Linux CPU IDs 4-87, keep CPUs 0-3 for housekeeping and isolate the workload CPUs with the following kernel parameters:

nohz_full=4-87 isolcpus=4-87 rcu_nocbs=4-87

Note

These settings are opt-in tuning knobs for specialized workloads. They can reduce the accuracy of CPU accounting from tools such as top and /proc/stat, and transitions into and out of full dynticks mode can add overhead when more than one runnable task is placed on an isolated CPU. Test with the target workload before broadly enabling them.

Init on Alloc#

The CONFIG_INIT_ON_ALLOC_DEFAULT_ON kernel configuration option controls whether the kernel fills newly allocated pages and heap objects with zeroes by default. You can override this setting with the init_on_alloc=[0|1] kernel parameter.

init_on_alloc is a Linux kernel security parameter. When enabled, it automatically fills newly allocated memory pages and heap objects with zeroes. This initialization can affect allocation throughput for high-performance computing workloads.

Note

For maximum throughput in high-performance computing tasks, use init_on_alloc=0. This setting trades off the additional memory initialization provided by init_on_alloc=1.

Distributions and kernel flavors can use different values for CONFIG_INIT_ON_ALLOC_DEFAULT_ON. To check the build-time setting of the running kernel, run the following command:

grep '^CONFIG_INIT_ON_ALLOC_DEFAULT_ON=' /boot/config-$(uname -r)

Boot parameters can override the build-time setting. To check the effective init_on_alloc value after boot, inspect the kernel’s memory auto-initialization message:

sudo dmesg | grep 'mem auto-init: stack:'

Here is example output:

[    0.000000] mem auto-init: stack:all(zero), heap alloc:off, heap free:off

The heap alloc:off field confirms that initialization on allocation is disabled; heap alloc:on means that it is enabled.

Input-Output Memory Management Unit Passthrough#

The Input-Output Memory Management Unit (IOMMU) is a hardware component that translates I/O device virtual addresses, also called I/O virtual addresses (IOVAs), to physical addresses. Different platforms have different IOMMUs, such as DMA Remapping Reporting (DMAR) for Intel IOMMU and the System Memory Management Unit (SMMU) for Arm platforms. Linux provides the iommu.passthrough mode, which controls whether DMA uses IOMMU translation.

Some applications might benefit in bare metal and virtual machine (VM) environments when iommu.passthrough is set to 1. Setting iommu.passthrough=1 on the kernel command line bypasses IOMMU translation for DMA. Setting iommu.passthrough=0 uses IOMMU translation for DMA. Set this value at deployment, or edit the appropriate GRUB configuration files. For the changes to take effect, reboot the system.

The kernel configuration can select passthrough mode by default. Before you add a boot parameter, check the running kernel configuration and boot messages:

grep '^CONFIG_IOMMU_DEFAULT_PASSTHROUGH=' /boot/config-$(uname -r)
sudo dmesg | grep -i iommu

To add kernel parameters, complete the steps for your distribution:

Ubuntu

  1. Create the /etc/default/grub.d/iommu_passthrough.cfg file with the following contents:

    GRUB_CMDLINE_LINUX="$GRUB_CMDLINE_LINUX iommu.passthrough=1"
    
  2. Run the following commands:

    sudo update-grub
    sudo reboot
    

Red Hat

  1. Run the following commands:

    sudo grubby --update-kernel=ALL --args="iommu.passthrough=1"
    sudo reboot
    

SUSE

  1. Edit the /etc/default/grub file.

  2. On the line that contains the GRUB_CMDLINE_LINUX string, append the iommu.passthrough=1 parameter, and run the following commands:

    sudo update-bootloader --refresh
    sudo reboot
    

Automatic NUMA Scheduling and Balancing#

On Vera Rubin systems, NVIDIA recommends that you do not use Automatic NUMA Scheduling and Balancing (AutoNUMA) features of the Linux kernel. AutoNUMA can introduce additional page faults, which can significantly reduce GPU-heavy application performance.

  • To see the status of AutoNUMA, use cat /proc/sys/kernel/numa_balancing.

  • The output is a bitmask. A value of 0 disables AutoNUMA. A value of 1 enables normal NUMA balancing, 2 enables memory-tiering NUMA balancing, and 3 enables both modes.

  • To disable AutoNUMA in a session, run the following command:

    sudo sysctl -w kernel.numa_balancing=0
    
  • To disable AutoNUMA permanently, run the following commands:

    echo "kernel.numa_balancing = 0" | sudo tee \
        /etc/sysctl.d/99-disable-autonuma.conf
    sudo sysctl --system
    

Swap File Size#

This section applies only to Vera Rubin systems.

This guidance applies when GPU memory is onlined to Linux as system memory in NUMA mode. In CDMM mode, GPU memory is managed by the NVIDIA driver and is not onlined to Linux as system memory in a NUMA node.

In NUMA mode, if an application allocates a large enough fraction of CPU memory, the kernel might migrate some pages, possibly from third-party applications, from CPU memory to GPU memory. This behavior has been observed in applications that use deep learning frameworks and might use nearly all CPU memory during initialization. In NUMA mode, if the kernel moves this memory to GPU memory, the memory can be reclaimed only through a swap file. If possible, configure a swap file that is large enough for these scenarios.

Note

On a Vera Rubin system with sufficient disk space, use a swap file that is at least one quarter to one half of the aggregate GPU memory size in the system.

CUDA Driver Module Options#

This section applies only to Vera Rubin systems.

CDMM Mode and NUMA Onlining of GPU Memory#

For Vera Rubin systems, Coherent Driver-Based Memory Management (CDMM) is the default mode. In CDMM mode, the NVIDIA driver manages GPU memory and GPU memory is not onlined to Linux as system memory in a NUMA node. This avoids memory over-reporting and prevents unexpected NUMA-based migrations from CPU memory to GPU memory.

To explicitly enable CDMM mode, add the following NVIDIA driver option:

options nvidia NVreg_CoherentGPUMemoryMode=driver

To explicitly enable NUMA onlining of GPU memory, add the following NVIDIA driver option:

options nvidia NVreg_CoherentGPUMemoryMode=numa

For example, create or update /etc/modprobe.d/nvidia-openrm.conf with the appropriate options nvidia line. If an NVIDIA driver options file already exists, merge this setting into the existing options nvidia line. Reboot the system or reload the NVIDIA driver after changing the option.

Refer to Understanding Memory Management on Hardware-Coherent Platforms for more information about CDMM.

Receive Steering and Connection Tracking for IP Workloads#

This section applies to traffic that passes through the kernel’s IP stack, such as TCP and UDP sockets. It does not apply to RDMA traffic, which bypasses the kernel network stack.

The kernel’s receive and transmit steering mechanisms (RSS, RPS, RFS, accelerated RFS, and XPS) are described in Scaling in the Linux Networking Stack. Without steering, a packet’s receive processing often runs on a different CPU from the CPU where the application reads the socket, so the socket, the poll, and the connection-tracking state move between cores on every packet. On Vera, moving a cache line between cores costs more than keeping it on one core, and the cost increases when the cores are on different sockets. This traffic can become a large share of the kernel time per request for services that are CPU-bound on their IP network path, such as web servers and proxies, RPC services, and load balancers.

The same rule applies to traffic between processes in the same network namespace, which the kernel carries over the loopback interface even when it is addressed to one of the host’s own network addresses. Examples include containers in one Kubernetes pod, an application and its service-mesh sidecar, and a reverse proxy and its application server on one host.

The following operating system settings keep that processing local to the application without changing the application:

  • Receive queue placement. Set the affinity of the adapter’s receive-queue interrupts to CPUs on the adapter’s NUMA node and run the application on the same node. Refer to NUMA Affinity and CPU NUMA Affinity. The irqbalance service can move the interrupts again, so when you set the affinity by hand, stop it or exclude the adapter’s interrupts from it.

  • Receive Flow Steering (RFS). Enable RFS on every receive queue of the interfaces that carry the traffic, so each socket’s receive processing runs on the CPU where the application reads it. Where the adapter supports flow steering in hardware (ethtool -k <interface> lists ntuple-filters), also enable accelerated RFS with ethtool -K <interface> ntuple on. The loopback interface has one receive queue and no interrupt, so RFS is the only steering available for same-host traffic.

  • Connection tracking. Each tracked packet looks up and updates a shared connection entry. If no stateful firewall or NAT rule needs to track a class of traffic, exempt it from connection tracking with a notrack rule in the raw priority that is matched by an interface or a port.

RFS works best when each communicating thread stays on one CPU, so pin the application’s threads with taskset, numactl, or the application’s own affinity settings.

To enable RFS on all receive queues of an interface for an application pinned to CPUs 0-7, size the global flow table, give each queue its share of it, and list the application’s CPUs in each queue’s rps_cpus mask. The mask is a hexadecimal bitmap of Linux CPU IDs; on systems with more than 32 CPUs it is written as comma-separated 32-bit words:

sudo sysctl -w net.core.rps_sock_flow_entries=32768
for q in /sys/class/net/<interface>/queues/rx-*; do
    echo 4096 | sudo tee $q/rps_flow_cnt
    echo ff | sudo tee $q/rps_cpus
done

For a client and a server on the same host, apply the same settings to the single loopback queue and list the CPUs of both. For example, for a server pinned to CPUs 0 and 1 and a client pinned to CPUs 2 and 4, the mask is 17:

echo 32768 | sudo tee /sys/class/net/lo/queues/rx-0/rps_flow_cnt
echo 17 | sudo tee /sys/class/net/lo/queues/rx-0/rps_cpus

To exempt a service’s traffic from connection tracking with nftables, add notrack rules in the raw priority of the prerouting and output hooks. The following example matches a service on TCP port 8080. To exempt all loopback traffic instead, match iif lo and oif lo:

sudo nft add table inet service_notrack
sudo nft add chain inet service_notrack pre \
    '{ type filter hook prerouting priority raw; }'
sudo nft add chain inet service_notrack out \
    '{ type filter hook output priority raw; }'
sudo nft add rule inet service_notrack pre tcp dport 8080 notrack
sudo nft add rule inet service_notrack out tcp sport 8080 notrack

To revert, delete the table and clear the steering settings:

sudo nft delete table inet service_notrack
for q in /sys/class/net/<interface>/queues/rx-*; do
    echo 0 | sudo tee $q/rps_cpus $q/rps_flow_cnt
done
sudo sysctl -w net.core.rps_sock_flow_entries=0
sudo ethtool -K <interface> ntuple off

Note

These settings are opt-in tuning knobs for network services that are CPU-bound on their receive path. RFS adds a flow-table lookup and inter-processor interrupts to the receive path, and it helps only when the application runs on a different CPU from the one that receives its packets. Hardware flow steering support varies by adapter and driver, so check with the adapter vendor. notrack disables stateful firewall and NAT rules for the matched traffic, so check the firewall configuration before adding it. None of these settings persist across a reboot, so use sysctl.d, a udev rule, or a system service to apply them at boot. Test with the target workload before enabling them broadly.

Libvirt Network#

  • For bare metal deployments, if libvirt is installed, it adds iptables rules to allow traffic to and from guests attached to the virbr0 device in the INPUT, FORWARD, OUTPUT, and POSTROUTING chains. If virtualization is not required, removing these rules improves network performance.

  • For bare metal environments where libvirt is not required, disabling the libvirt service and removing the libvirt-installed iptables rules can improve network performance.

To remove the libvirt rules, stop the libvirtd service or manually delete the networking rules.

Warning

Only disable the libvirt service or remove the iptables rules in bare metal environments. These steps will break existing libvirt deployments.

  • Stop the libvirtd service

    If there are no virtual machines and virtualization is not required, disable libvirtd.service and reboot to clear the rules and delete the virbr0 interface.

    sudo systemctl stop libvirtd
    sudo systemctl disable libvirtd
    sudo reboot now
    
  • Delete only the networking rules

    Another way to delete these rules is to run the following commands to deactivate the libvirt network. This action removes the virbr0 bridge, terminates the dnsmasq process, and removes the iptables rules.

    To deactivate the libvirt network named default:

    virsh net-destroy default
    

    Prevent the network from automatically starting on boot:

    virsh net-autostart --network default --disable
    

    To revert the changes, reactivate the default libvirt network:

    virsh net-start default
    

Note

If libvirt is required, avoid NAT and consider using NIC passthrough or SR-IOV for virtual machines to improve performance. Libvirt performance optimization is outside the scope of this guide.

Unified Memory Considerations on Vera Rubin Systems#

Vera Rubin systems enable the CPU and GPU to access a shared address space across LPDDR5X and HBM memory by using NVLink-C2C. This architecture provides flexible, high-bandwidth memory access and introduces considerations for memory placement.

These considerations apply when GPU memory is onlined to Linux as system memory in NUMA mode. In CDMM mode, the NVIDIA driver manages GPU memory, and GPU memory is not onlined to Linux as system memory in a NUMA node. Refer to CUDA Driver Module Options for the driver option syntax.

In NUMA mode, HBM memory attached to the GPU is visible to the operating system and can be used for general memory allocations. As a result, applications that do not explicitly manage NUMA locality might have memory pages allocated from GPU HBM, even when they are not using the GPU.

Although HBM offers significantly higher bandwidth than LPDDR5X, it might have higher latency for CPU access, especially when memory access patterns are irregular or latency sensitive. For performance-critical applications, and especially applications that do not use the GPU, use NUMA-aware memory allocation, such as numactl, libnuma, or mbind, to ensure that memory is allocated from the preferred memory domain. For CPU-only workloads, the preferred memory domain is typically LPDDR5X.

To identify unintended HBM usage, monitor memory allocation using tools such as numastat, nvidia-smi, or hwloc. Pin memory allocations to CPU-local nodes unless HBM’s higher bandwidth is explicitly beneficial for the workload. Being explicit about memory locality helps optimize performance and avoid scenarios where latency-sensitive code suffers from unintended memory placement. In CDMM mode, NUMA tools can still be used to manage system memory, but they do not control GPU memory placement.