Advanced Performance Tuning

The Performance Tuning Guide can be used in the XLIO case for detailed instructions on how to optimally tune your machines for XLIO performance.

1. Check which NUMA is related to your interface.

   Copy

   Copied!

               
   
               
   cat /sys/class/net/<interface_name>/device/numa_node
           

   Example:

   Copy

   Copied!

               
   
               
   [root@r-host142 ~]# cat /sys/class/net/ens5/device/numa_node
   1
           

   The output above shows that your device is installed next to NUMA

2. Bind application to NUMA node:

   Copy

   Copied!

               
   
               
   #<host> numactl --cpunodebind=<NUMA_NODE> <your_application>
           

3. Check which CPU is related to the specific NUMA.

   The output above shows that:

   • CPUs 0-13 & 28-41 are related to NUMA 0• CPUs 14-27 & 42-55 are related to NUMA 1

   Since we want to use NUMA 1, one of the following CPUs should be used: 14-27 & 42-55

4. Use the "taskset" command to run the XLIO process on a specific CPU.

   • Server-Side:

     Copy

     Copied!

                 
     
                 
     LD_PRELOAD=libxlio.so taskset -c 15 sockperf sr -i < IP of FIRST machine MLX interface >
             

   • Client-Side:

     Copy

     Copied!

                 
     
                 
     LD_PRELOAD=libxlio.so taskset -c 15 sockperf pp -i < IP of FIRST machine MLX interface >
             

   In this example, we use CPU 15 that belongs to NUMA 1. You can also use "numactl - -hardware".

XLIO can leverage Huge Pages to reduce TLB misses and improve memory allocation efficiency.

Check supported Huge Page sizes:

            

            
#(host) ls /sys/kernel/mm/hugepages/
        

Allocate Huge Pages:

            

            
#(host) echo <number_of_hugepages> | sudo tee /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
OR
#(host) echo <number_of_hugepages> | sudo tee /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages
        

NUMA-aware Huge Pages allocation (recommended):

We recommend that Huge Pages be allocated on the same NUMA node as the application to reduce memory access latency and optimize performance.

            

            
#(host) echo <number_of_hugepages> | sudo tee /sys/devices/system/node/<NUMA_NODE>/hugepages/hugepages-2048kB/nr_hugepages
OR
#(host) echo <number_of_hugepages> | sudo tee /sys/devices/system/node/<NUMA_NODE>/hugepages/hugepages-1048576kB/nr_hugepages
        

When configuring the BIOS, please pay attention to the following:

1. Turbo mode - Increases CPU core frequency when only portion of the cores is active.

2. Performance Mode

   1. Max performance mode - A preset for maximum system performance - May increase system power consumption.

3. Power modes

   1. C-states and P-states - Prevent/Allow CPU sleeping on idle - Disabling C/P states may reduce latency but increases system power consumption when the system is idle.

4. Hyperthreading – there is no right answer if you should have it ON or OFF.

   • ON means more CPU to handle other application and Kernel tasks, so the amortized cost will be smaller for each CPU

   • OFF means do not share cache with other tasks, so cache utilization may improve

5. SMI interrupts.

   Look for "Processor Power and Utilization Monitoring" and "Memory Pre-Failure Notification" SMIs.

   The OS is not aware of these interrupts, so the only way you might be able to notice them is by reading the CPU msr register.

Please make sure to carefully read your vendor BIOS tuning guide as the configuration options differ per vendor.

• Disable cppc_cpufreq

  Copy

  Copied!

              
  
              
  rmmod cppc_cpufreq
          

The following table provides guidance for tuning XLIO parameters to reduce excessive CPU utilization. Please follow the recommendations below to adjust the XLIO behavior.