Basic Performance Tuning
The Performance Tuning Guide can be used in the XLIO case for detailed instructions on how to optimally tune your machines for XLIO performance.
Check which NUMA is related to your interface.
cat /sys/
class
/net/<interface_name>/device/numa_nodeExample:
[root
@r
-host142 ~]# cat /sys/class
/net/ens5/device/numa_node1
The output above shows that your device is installed next to NUMA
Check which CPU is related to the specific NUMA.
The output above shows that:
• CPUs 0-13 & 28-41 are related to NUMA 0• CPUs 14-27 & 42-55 are related to NUMA 1
Since we want to use NUMA 1, one of the following CPUs should be used: 14-27 & 42-55
Use the "taskset" command to run the XLIO process on a specific CPU.
Server-Side:
LD_PRELOAD=libxlio.so taskset -c
15
sockperf sr -i < IP of FIRST machine MLXinterface
>Client-Side:
LD_PRELOAD=libxlio.so taskset -c
15
sockperf pp -i < IP of FIRST machine MLXinterface
>
In this example, we use CPU 15 that belongs to NUMA 1. You can also use "numactl - -hardware".
Each machine has its own BIOS parameters. It is important to implement any server manufacturer and Linux distribution tuning recommendations for lowest latency.
When configuring the BIOS, please pay attention to the following:
Enable Max performance mode.
Enable Turbo mode.
Power modes – disable C-states and P-states, do not let the CPU sleep on idle.
Hyperthreading – there is no right answer if you should have it ON or OFF.
• ON means more CPU to handle kernel tasks, so the amortized cost will be smaller for each CPU
• OFF means do not share cache with other CPUs, so cache utilization is better
If all of your system jitter is under control, it is recommended to turn is OFF, if not keep it ON.
Disable SMI interrupts.
Look for "Processor Power and Utilization Monitoring" and "Memory Pre-Failure Notification" SMIs.
The OS is not aware of these interrupts, so the only way you might be able to notice them is by reading the CPU msr register.
Please make sure to carefully read your vendor BIOS tuning guide as the configuration options differ per vendor.