Operating System Settings#
This chapter describes operating system settings that can affect Vera CPU and Vera Rubin system performance.
Note
Kernel configuration, boot parameters, and operating system defaults vary by distribution and kernel flavor. Verify the effective settings on the target system before you apply these recommendations.
CPU NUMA Affinity#
Important
On Vera-Rubin platforms, it is critical that you use numactl to bind
your applications to the local CPU and memory that are in the same NUMA domain.
Otherwise the cross-socket performance will be poor.
Determine on which NUMA nodes your applications will run. Refer to Checking the CPUs and Checking the Non-Uniform Memory Access Settings for more information.
To bind to socket 0:
numactl -N 0 -m 0 <your_app>
To bind to specific CPU cores on socket 0:
numactl -C 0-87 -m 0 <your_app>
To bind to socket 1:
numactl -N 1 -m 1 <your_app>
To bind to specific CPU cores on socket 1:
numactl -C 88-175 -m 1 <your_app>
Spatial Multithreading#
The NVIDIA Olympus core in Vera supports Spatial Multithreading (SMT), which exposes two virtual processors for each physical CPU core to the operating system. Each virtual processor is also called a Processing Element (PE), and the two PEs that belong to the same physical core are thread siblings.
Topology#
Each Vera socket has 88 physical cores. With SMT enabled, this becomes 176 PEs per socket. On a single-socket system, the first PE of each core is typically numbered 0-87, and the second PE is typically numbered 88-175. For example, logical CPUs 0 and 88, and logical CPUs 1 and 89, correspond to the same physical core. On multi-socket systems, CPU numbering can differ, so detect the topology at runtime in one of the following ways:
lscpu --extended=CPU,COREshows which logical CPU corresponds to which physical core.lscpu --extended=CPU,CORE CPU CORE 0 0 1 1 2 2 3 3 4 4 ........ 84 84 85 85 86 86 87 87 88 0 89 1 90 2 91 3 ........ 172 84 173 85 174 86 175 87
The thread siblings are exposed using sysfs:
$ cat /sys/devices/system/cpu/cpu0/topology/thread_siblings_list 0,88 $ cat /sys/devices/system/cpu/cpu1/topology/thread_siblings_list 1,89 $ cat /sys/devices/system/cpu/cpu88/topology/thread_siblings_list 0,88
Controlling SMT#
A BIOS option is available to disable SMT at boot time. If SMT is disabled at boot time, the operating system sees only 88 processors per socket. Otherwise, the operating system sees 176 processors per socket.
At runtime, to disable SMT:
echo off | sudo tee /sys/devices/system/cpu/smt/control
When disabled at runtime:
On a single-socket system,
nprocreports 88 processors. On multi-socket systems, the total scales with the number of sockets.Processors 88-175 are marked as
offlinein the operating system on a single-socket system./sys/devices/system/cpu/smt/activereports 0.
To enable it again at runtime:
echo on | sudo tee /sys/devices/system/cpu/smt/control
After enabling again:
On a single-socket system,
nprocreports 176 processors. On multi-socket systems, the total scales with the number of sockets.Processors 88-175 are marked as
onlinein the operating system on a single-socket system./sys/devices/system/cpu/smt/activereports 1.
Vera also supports toggling SMT for individual cores at runtime. For example, to disable SMT on physical core 0, take one of its thread siblings offline, such as logical CPU 88 on a single-socket system:
echo 0 | sudo tee /sys/devices/system/cpu/cpu88/online
To bring SMT back online at runtime:
echo 1 | sudo tee /sys/devices/system/cpu/cpu88/online
When SMT is enabled for at least one core on the system,
/sys/devices/system/cpu/smt/active reports 1.
Page Size#
Vera supports 4 KB and 64 KB Linux host kernel page sizes. Choose the host kernel page size based on the software stack and workload that you need to run:
Use a 4 KB host kernel for Arm Confidential Computing Architecture (CCA) systems. Vera CCA software and hardware checkout flows require a 4 KB host kernel, and CCA is not supported with a 64 KB host kernel. Realm guests can use 4 KB, 16 KB, or 64 KB guest kernel page sizes, but CCA realm internals use 4 KB pages. For a short overview of Vera CCA features, refer to Confidential Compute.
Use a 64 KB host kernel for non-CCA systems and general page-size or performance tuning, unless your workload or software stack explicitly requires a 4 KB host kernel.
To configure the Linux host kernel page size, use one of the following Kconfig settings during kernel compilation:
4 KB page size:
CONFIG_ARM64_4K_PAGES=y64 KB page size:
CONFIG_ARM64_64K_PAGES=y
The 64 KB page size can benefit applications that allocate large amounts of memory because it can reduce page faults and improve TLB hit rates. The 4 KB page size is required for CCA host kernels and can be useful when compatibility with software that assumes 4 KB pages is more important than the potential performance benefit of a larger base page size.
Note
For non-CCA systems, the recommended default host kernel page size is 64 KB. For CCA systems, use a 4 KB host kernel.
Huge Pages#
Huge pages can benefit applications that allocate large memory regions by reducing TLB misses.
You can use huge pages on Vera systems in the following ways:
Transparent Huge Pages (THP)
Transparent to the application.
Mostly automatic with a few available kernel tuning parameters.
When using a 64 KB host kernel page size, the default THP size is 512 MB. Linux 6.9 and later support multi-size THP (mTHP) for smaller THP sizes such as 2 MB (refer to Transparent Huge Pages for more information).
Hugetlbfs
Does not suffer from fragmentation concerns or from allocation latency because the huge pages are preallocated and indivisible.
Requires application modification.
Requires sysadmin setup.
Transparent Huge Pages#
Transparent Huge Pages (THP) is transparent to applications. Applications can benefit from huge pages without source code changes.
THP policy can be set to always, madvise, or never. When set to
madvise, THP is allocated only for memory regions where the application
requests huge pages with madvise(MADV_HUGEPAGE).
Warning
With a 64 KB host kernel page size, the default THP size is 512 MB, which might be too large for many workloads. Linux 6.9 and later support multi-size THP (mTHP), which can use smaller THP sizes such as 2 MB. Prefer 2 MB mTHP unless you have measured a benefit from 512 MB pages.
To check the current THP policy, run the following command. The active policy is marked with brackets:
$ cat /sys/kernel/mm/transparent_hugepage/enabled
always [madvise] never
To change the THP policy, run one of the following commands:
$ echo always | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
$ echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
$ echo never | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
On 64 KB host kernels, 2 MB mTHP requires Linux 6.9 or later. Confirm that the 2 MB sysfs entry exists before changing per-size policy. If this path is absent, 2 MB mTHP is not available on that kernel:
$ ls /sys/kernel/mm/transparent_hugepage/hugepages-2048kB
For most workloads on kernels that support mTHP, disable 512 MB THP and use 2 MB mTHP instead:
$ echo never | sudo tee /sys/kernel/mm/transparent_hugepage/hugepages-524288kB/enabled
$ echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/hugepages-2048kB/enabled
If you have already customized per-size THP policy, record the current values before applying these settings so that you can restore them later.
To restore the documented default per-size policy (512 MB inherits the global THP policy, and 2 MB mTHP is disabled):
$ echo inherit | sudo tee /sys/kernel/mm/transparent_hugepage/hugepages-524288kB/enabled
$ echo never | sudo tee /sys/kernel/mm/transparent_hugepage/hugepages-2048kB/enabled
Linux first-touch NUMA policy places a huge page on the NUMA node of the first CPU that writes to it. The page will not be migrated due to subsequent accesses from other sockets. With 512 MB THP, a large aligned allocation that is initialized with OpenMP (or similar) first-touch can therefore remain in one socket and cause remote memory traffic. The same mechanism exists for every page size, but the chance of sharing a page across sockets scales with page size.
This guide recommends disabling Automatic NUMA Balancing, so the kernel does not migrate pages after first touch. Refer to Automatic NUMA Scheduling and Balancing for more information.
HPC applications that run at least one MPI rank per socket typically allocate per socket and are less exposed to this issue. Workloads that benefit from 512 MB pages can re-enable them.
Starting with glibc 2.43 (for example, Ubuntu 26.04), malloc requests
THP by default on AArch64, which is equivalent to the glibc tunable
glibc.malloc.hugetlb=1. glibc does not choose the kernel huge-page size.
On a 64 KB host kernel, those allocations are typically backed by 512 MB THP
if 512 MB pages are enabled. The allocation then behaves like a 512 MB
first-touch region: the first CPU that writes to the page places the entire
512 MB page, and threads on other sockets incur remote memory traffic.
To stop malloc from requesting THP for a process, set
GLIBC_TUNABLES=glibc.malloc.hugetlb=0.
Refer to Transparent Hugepage Support for more information about THP and mTHP.
Note
The default huge page size is related to the kernel page size. Refer to HugeTLBpage on ARM64 for more information.
Memory Reserved for Huge Page Allocation#
When THP is set to always or madvise, khugepaged raises
vm.min_free_kbytes so that the kernel retains enough contiguous free memory
for huge-page allocations. With a 64 KB host kernel page size, this reservation
can be large:
$ sudo sysctl vm.min_free_kbytes
vm.min_free_kbytes = 17301504
The kernel reserves these pages proportionally across all memory zones, which
reduces the memory available to applications on every NUMA node, including the
GPU NUMA nodes on Vera Rubin systems. Writing to
/sys/kernel/mm/transparent_hugepage/ recalculates the value and overrides
any setting that you apply manually.
The reservation scales with the pageblock size, which is set by the
CONFIG_PAGE_BLOCK_MAX_ORDER kernel build option, and not with the THP size
that you select. To check the value used by the running kernel, run the
following command:
$ grep PAGE_BLOCK_MAX_ORDER /boot/config-$(uname -r)
CONFIG_PAGE_BLOCK_MAX_ORDER=10
If your workloads use 2 MB or smaller mTHP, as recommended earlier in this
section, a value of 5 optimizes the kernel for 2 MB pages and substantially
reduces vm.min_free_kbytes. Changing this option requires a custom kernel
build, so validate it against your workload first.
Proactive Compaction#
To reduce fragmentation, proactive compaction performs memory compaction in the background. By making more physically contiguous memory available, it can improve the availability and allocation latency of huge pages and other higher-order allocations.
Without proactive compaction, an application can encounter a performance cliff when the kernel must synchronously compact memory to satisfy a huge-page allocation.
With proactive compaction, the kernel begins compacting memory in the background when fragmentation reaches a threshold. This can reduce the likelihood and duration of synchronous compaction stalls.
Proactive compaction exposes the
/proc/sys/vm/compaction_proactiveness tunable, which accepts values in the
0 to 100 range and has a default value of 20. This tunable
determines how aggressively the kernel compacts memory in the background.
Aggressive compaction can consume CPU time, migrate application pages, and
introduce system-wide latency spikes. The default value is reasonable and
should be changed only based on performance data.
To limit the overhead of proactive compaction, you can use on-demand compaction,
which is available only when CONFIG_COMPACTION is set. When 1 is
written to /proc/sys/vm/compact_memory, all zones are compacted, and free
memory is made available in contiguous blocks where possible. This can be useful
for huge-page allocation because it directly compacts memory as required. Refer
to Documentation for /proc/sys/vm/
for more information.
Hugetlbfs#
With hugetlbfs, the system can preallocate pools of hugetlb pages for
applications, but this approach requires application changes. You can specify
the minimum number of huge pages reserved by the system, how large the pool can
grow, and whether malloc should use hugetlbfs for an application.
Test your application with hugetlbfs before broadly enabling it. If hugetlbfs works for the application, reserving a pool of huge pages at boot time can improve availability because memory is not yet fragmented. As a result, the requested number of huge pages is more likely to be assembled. Refer to HugeTLB Pages for more information.
Configuring Linux Perf#
Refer to Configuring Perf for more information.
Performance Governor#
You can set the CPU governor with the cpupower command. For example, to set
the CPU governor to performance, run the following command:
sudo cpupower frequency-set -g performance
Note
On certain distributions, such as Ubuntu, the cpufrequtils package
provides a service that might change the CPU governor to ondemand when
the system boots. To avoid this behavior, disable the service by running
sudo systemctl disable cpufrequtils.
Reducing Periodic Timer Ticks for Streaming Stores#
Some CPU memory write kernels on Vera run most efficiently when the core stays
in streaming store mode and stores use the high-bandwidth
WriteUnique alloc=0 path. If the same CPU takes frequent timer
interrupts, the Arm64 interrupt and exception paths can execute the required
DMB instructions. These barriers can reset the store tracker’s streaming
state, so the next stores can temporarily use the MakeUnique or the
WriteUnique alloc=1 path until streaming behavior is established again. This
scenario is most relevant to pinned, long-running, bandwidth-limited CPU store
loops that otherwise spend most of their time in WriteUnique alloc=0.
Systems built with CONFIG_HZ=1000 can take a periodic arch_timer
interrupt on a busy CPU approximately once per millisecond unless tick
suppression is active. For highly tuned RT or HPC-style workloads, consider
reducing the periodic OS activity on the CPU cores that run streaming store
threads.
To evaluate whether the system is configured for tick suppression, check the kernel configuration, boot parameters, and timer interrupt counts:
grep CONFIG_NO_HZ_FULL /boot/config-$(uname -r)
cat /proc/cmdline
grep arch_timer /proc/interrupts
Consider the following OS settings:
Use
nohz_full=<cpulist>on CPUs that run one pinned userspace task to suppress periodic ticks while the CPU is in full dynticks mode.Use
isolcpus=<cpulist>to keep those CPUs out of the scheduler’s general CPU pool.Use
rcu_nocbs=<cpulist>to move RCU callback processing off those CPUs.If kernel rebuilds are practical, use a lower
CONFIG_HZvalue, such as 250 or 100, to reduce the periodic tick rate.
Leave at least one housekeeping CPU that is outside the nohz_full set. On
production systems, reserve enough housekeeping CPUs for timers, RCU, kernel
threads, interrupts, and management agents. These parameters do not replace IRQ
affinity tuning for device interrupts. For example, on a system where the
workload CPU partition is represented by Linux CPU IDs 4-87, keep CPUs 0-3 for
housekeeping and isolate the workload CPUs with the following kernel
parameters:
nohz_full=4-87 isolcpus=4-87 rcu_nocbs=4-87
Note
These settings are opt-in tuning knobs for specialized workloads. They can
reduce the accuracy of CPU accounting from tools such as top and
/proc/stat, and transitions into and out of full dynticks mode can add
overhead when more than one runnable task is placed on an isolated CPU.
Test with the target workload before broadly enabling them.
Init on Alloc#
The CONFIG_INIT_ON_ALLOC_DEFAULT_ON kernel configuration option controls
whether the kernel fills newly allocated pages and heap objects with zeroes by
default. You can override this setting with the init_on_alloc=[0|1] kernel
parameter.
init_on_alloc is a Linux kernel security parameter. When enabled, it
automatically fills newly allocated memory pages and heap objects with zeroes.
This initialization can affect allocation throughput for high-performance
computing workloads.
Note
For maximum throughput in high-performance computing tasks, use
init_on_alloc=0. This setting trades off the additional memory
initialization provided by init_on_alloc=1.
Distributions and kernel flavors can use different values for
CONFIG_INIT_ON_ALLOC_DEFAULT_ON. To check the build-time setting of the
running kernel, run the following command:
grep '^CONFIG_INIT_ON_ALLOC_DEFAULT_ON=' /boot/config-$(uname -r)
Boot parameters can override the build-time setting. To check the effective
init_on_alloc value after boot, inspect the kernel’s memory
auto-initialization message:
sudo dmesg | grep 'mem auto-init: stack:'
Here is example output:
[ 0.000000] mem auto-init: stack:all(zero), heap alloc:off, heap free:off
The heap alloc:off field confirms that initialization on allocation is
disabled; heap alloc:on means that it is enabled.
Input-Output Memory Management Unit Passthrough#
The Input-Output Memory Management Unit (IOMMU) is a hardware component that
translates I/O device virtual addresses, also called I/O virtual addresses
(IOVAs), to physical addresses. Different platforms have different IOMMUs, such
as DMA Remapping Reporting (DMAR) for Intel IOMMU and the System Memory
Management Unit (SMMU) for Arm platforms. Linux provides the
iommu.passthrough mode, which controls whether DMA uses IOMMU translation.
Some applications might benefit in bare metal and virtual machine (VM)
environments when iommu.passthrough is set to 1. Setting
iommu.passthrough=1 on the kernel command line bypasses IOMMU translation
for DMA. Setting iommu.passthrough=0 uses IOMMU translation for DMA. Set
this value at deployment, or edit the appropriate GRUB configuration files. For
the changes to take effect, reboot the system.
The kernel configuration can select passthrough mode by default. Before you add a boot parameter, check the running kernel configuration and boot messages:
grep '^CONFIG_IOMMU_DEFAULT_PASSTHROUGH=' /boot/config-$(uname -r)
sudo dmesg | grep -i iommu
To add kernel parameters, complete the steps for your distribution:
Ubuntu
Create the
/etc/default/grub.d/iommu_passthrough.cfgfile with the following contents:GRUB_CMDLINE_LINUX="$GRUB_CMDLINE_LINUX iommu.passthrough=1"
Run the following commands:
sudo update-grub sudo reboot
Red Hat
Run the following commands:
sudo grubby --update-kernel=ALL --args="iommu.passthrough=1" sudo reboot
SUSE
Edit the
/etc/default/grubfile.On the line that contains the
GRUB_CMDLINE_LINUXstring, append theiommu.passthrough=1parameter, and run the following commands:sudo update-bootloader --refresh sudo reboot
Automatic NUMA Scheduling and Balancing#
On Vera Rubin systems, NVIDIA recommends that you do not use Automatic NUMA Scheduling and Balancing (AutoNUMA) features of the Linux kernel. AutoNUMA can introduce additional page faults, which can significantly reduce GPU-heavy application performance.
To see the status of AutoNUMA, use
cat /proc/sys/kernel/numa_balancing.The output is a bitmask. A value of
0disables AutoNUMA. A value of1enables normal NUMA balancing,2enables memory-tiering NUMA balancing, and3enables both modes.To disable AutoNUMA in a session, run the following command:
sudo sysctl -w kernel.numa_balancing=0
To disable AutoNUMA permanently, run the following commands:
echo "kernel.numa_balancing = 0" | sudo tee \ /etc/sysctl.d/99-disable-autonuma.conf sudo sysctl --system
Swap File Size#
This section applies only to Vera Rubin systems.
This guidance applies when GPU memory is onlined to Linux as system memory in NUMA mode. In CDMM mode, GPU memory is managed by the NVIDIA driver and is not onlined to Linux as system memory in a NUMA node.
In NUMA mode, if an application allocates a large enough fraction of CPU memory, the kernel might migrate some pages, possibly from third-party applications, from CPU memory to GPU memory. This behavior has been observed in applications that use deep learning frameworks and might use nearly all CPU memory during initialization. In NUMA mode, if the kernel moves this memory to GPU memory, the memory can be reclaimed only through a swap file. If possible, configure a swap file that is large enough for these scenarios.
Note
On a Vera Rubin system with sufficient disk space, use a swap file that is at least one quarter to one half of the aggregate GPU memory size in the system.
CUDA Driver Module Options#
This section applies only to Vera Rubin systems.
CDMM Mode and NUMA Onlining of GPU Memory#
For Vera Rubin systems, Coherent Driver-Based Memory Management (CDMM) is the default mode. In CDMM mode, the NVIDIA driver manages GPU memory and GPU memory is not onlined to Linux as system memory in a NUMA node. This avoids memory over-reporting and prevents unexpected NUMA-based migrations from CPU memory to GPU memory.
To explicitly enable CDMM mode, add the following NVIDIA driver option:
options nvidia NVreg_CoherentGPUMemoryMode=driver
To explicitly enable NUMA onlining of GPU memory, add the following NVIDIA driver option:
options nvidia NVreg_CoherentGPUMemoryMode=numa
For example, create or update /etc/modprobe.d/nvidia-openrm.conf with the
appropriate options nvidia line. If an NVIDIA driver options file already
exists, merge this setting into the existing options nvidia line. Reboot the
system or reload the NVIDIA driver after changing the option.
Refer to Understanding Memory Management on Hardware-Coherent Platforms for more information about CDMM.
Receive Steering and Connection Tracking for IP Workloads#
This section applies to traffic that passes through the kernel’s IP stack, such as TCP and UDP sockets. It does not apply to RDMA traffic, which bypasses the kernel network stack.
The kernel’s receive and transmit steering mechanisms (RSS, RPS, RFS, accelerated RFS, and XPS) are described in Scaling in the Linux Networking Stack. Without steering, a packet’s receive processing often runs on a different CPU from the CPU where the application reads the socket, so the socket, the poll, and the connection-tracking state move between cores on every packet. On Vera, moving a cache line between cores costs more than keeping it on one core, and the cost increases when the cores are on different sockets. This traffic can become a large share of the kernel time per request for services that are CPU-bound on their IP network path, such as web servers and proxies, RPC services, and load balancers.
The same rule applies to traffic between processes in the same network namespace, which the kernel carries over the loopback interface even when it is addressed to one of the host’s own network addresses. Examples include containers in one Kubernetes pod, an application and its service-mesh sidecar, and a reverse proxy and its application server on one host.
The following operating system settings keep that processing local to the application without changing the application:
Receive queue placement. Set the affinity of the adapter’s receive-queue interrupts to CPUs on the adapter’s NUMA node and run the application on the same node. Refer to NUMA Affinity and CPU NUMA Affinity. The
irqbalanceservice can move the interrupts again, so when you set the affinity by hand, stop it or exclude the adapter’s interrupts from it.Receive Flow Steering (RFS). Enable RFS on every receive queue of the interfaces that carry the traffic, so each socket’s receive processing runs on the CPU where the application reads it. Where the adapter supports flow steering in hardware (
ethtool -k <interface>listsntuple-filters), also enable accelerated RFS withethtool -K <interface> ntuple on. The loopback interface has one receive queue and no interrupt, so RFS is the only steering available for same-host traffic.Connection tracking. Each tracked packet looks up and updates a shared connection entry. If no stateful firewall or NAT rule needs to track a class of traffic, exempt it from connection tracking with a
notrackrule in therawpriority that is matched by an interface or a port.
RFS works best when each communicating thread stays on one CPU, so pin the
application’s threads with taskset, numactl, or the application’s
own affinity settings.
To enable RFS on all receive queues of an interface for an application
pinned to CPUs 0-7, size the global flow table, give each queue its share
of it, and list the application’s CPUs in each queue’s rps_cpus mask.
The mask is a hexadecimal bitmap of Linux CPU IDs; on systems with more
than 32 CPUs it is written as comma-separated 32-bit words:
sudo sysctl -w net.core.rps_sock_flow_entries=32768
for q in /sys/class/net/<interface>/queues/rx-*; do
echo 4096 | sudo tee $q/rps_flow_cnt
echo ff | sudo tee $q/rps_cpus
done
For a client and a server on the same host, apply the same settings to the
single loopback queue and list the CPUs of both. For example, for a server
pinned to CPUs 0 and 1 and a client pinned to CPUs 2 and 4, the mask is
17:
echo 32768 | sudo tee /sys/class/net/lo/queues/rx-0/rps_flow_cnt
echo 17 | sudo tee /sys/class/net/lo/queues/rx-0/rps_cpus
To exempt a service’s traffic from connection tracking with nftables,
add notrack rules in the raw priority of the prerouting and output
hooks. The following example matches a service on TCP port 8080. To
exempt all loopback traffic instead, match iif lo and oif lo:
sudo nft add table inet service_notrack
sudo nft add chain inet service_notrack pre \
'{ type filter hook prerouting priority raw; }'
sudo nft add chain inet service_notrack out \
'{ type filter hook output priority raw; }'
sudo nft add rule inet service_notrack pre tcp dport 8080 notrack
sudo nft add rule inet service_notrack out tcp sport 8080 notrack
To revert, delete the table and clear the steering settings:
sudo nft delete table inet service_notrack
for q in /sys/class/net/<interface>/queues/rx-*; do
echo 0 | sudo tee $q/rps_cpus $q/rps_flow_cnt
done
sudo sysctl -w net.core.rps_sock_flow_entries=0
sudo ethtool -K <interface> ntuple off
Note
These settings are opt-in tuning knobs for network services that are
CPU-bound on their receive path. RFS adds a flow-table lookup and
inter-processor interrupts to the receive path, and it helps only when
the application runs on a different CPU from the one that receives its
packets. Hardware flow steering support varies by adapter and driver,
so check with the adapter vendor. notrack disables stateful firewall
and NAT rules for the matched traffic, so check the firewall
configuration before adding it. None of these settings persist across
a reboot, so use sysctl.d, a udev rule, or a system service to
apply them at boot. Test with the target workload before enabling them
broadly.
Libvirt Network#
For bare metal deployments, if libvirt is installed, it adds iptables rules to allow traffic to and from guests attached to the
virbr0device in the INPUT, FORWARD, OUTPUT, and POSTROUTING chains. If virtualization is not required, removing these rules improves network performance.For bare metal environments where libvirt is not required, disabling the libvirt service and removing the libvirt-installed iptables rules can improve network performance.
To remove the libvirt rules, stop the libvirtd service or manually delete
the networking rules.
Warning
Only disable the libvirt service or remove the iptables rules in bare metal environments. These steps will break existing libvirt deployments.
Stop the libvirtd service
If there are no virtual machines and virtualization is not required, disable
libvirtd.serviceand reboot to clear the rules and delete thevirbr0interface.sudo systemctl stop libvirtd sudo systemctl disable libvirtd sudo reboot now
Delete only the networking rules
Another way to delete these rules is to run the following commands to deactivate the libvirt network. This action removes the
virbr0bridge, terminates thednsmasqprocess, and removes the iptables rules.To deactivate the libvirt network named default:
virsh net-destroy default
Prevent the network from automatically starting on boot:
virsh net-autostart --network default --disable
To revert the changes, reactivate the default libvirt network:
virsh net-start default
Note
If libvirt is required, avoid NAT and consider using NIC passthrough or SR-IOV for virtual machines to improve performance. Libvirt performance optimization is outside the scope of this guide.
Unified Memory Considerations on Vera Rubin Systems#
Vera Rubin systems enable the CPU and GPU to access a shared address space across LPDDR5X and HBM memory by using NVLink-C2C. This architecture provides flexible, high-bandwidth memory access and introduces considerations for memory placement.
These considerations apply when GPU memory is onlined to Linux as system memory in NUMA mode. In CDMM mode, the NVIDIA driver manages GPU memory, and GPU memory is not onlined to Linux as system memory in a NUMA node. Refer to CUDA Driver Module Options for the driver option syntax.
In NUMA mode, HBM memory attached to the GPU is visible to the operating system and can be used for general memory allocations. As a result, applications that do not explicitly manage NUMA locality might have memory pages allocated from GPU HBM, even when they are not using the GPU.
Although HBM offers significantly higher bandwidth than LPDDR5X, it might have
higher latency for CPU access, especially when memory access patterns are
irregular or latency sensitive. For performance-critical applications, and
especially applications that do not use the GPU, use NUMA-aware memory
allocation, such as numactl, libnuma, or mbind, to ensure that
memory is allocated from the preferred memory domain. For CPU-only workloads,
the preferred memory domain is typically LPDDR5X.
To identify unintended HBM usage, monitor memory allocation using tools such as
numastat, nvidia-smi, or hwloc. Pin memory allocations to CPU-local
nodes unless HBM’s higher bandwidth is explicitly beneficial for the workload.
Being explicit about memory locality helps optimize performance and avoid
scenarios where latency-sensitive code suffers from unintended memory placement.
In CDMM mode, NUMA tools can still be used to manage system memory, but they do
not control GPU memory placement.