HBN-Only Performance Optimizations
This document specifies some recommended Performance Optimization steps for the HBN-Only DPF Deployment:
The following kernel parameters can be added to worker nodes to make sure that the the cores that belong to the same NUMA Node as the DPU are isolated and reserved for workload pods. Please substitute the values for isolcpus, nohz_full, and rcu_nocbs with the CPU cores from the NUMA node which the BlueField-3 is connected to:
intel_iommu=on iommu=pt numa_balancing=disable processor.max_cstate=0 isolcpus=28-55,84-111 nohz_full=28-55,84-111 rcu_nocbs=28-55,84-111
The following parameters can be set in kubelet configuration file (typically at: /var/lib/kubelet/config.yaml) for using the "static" CPU Manager Policy and reserving the CPU cores that belong to NUMA Nodes that the DPU is not connected to for system usage. Kubernetes will not use these cores for workload pods.
Add the following lines to the file, please substitute the values with the CPU cores from the NUMA nodes which the BlueField-3 is not connected to:
cpuManagerPolicy: static
reservedSystemCPUs: 0-27,56-83
Add the controlPlaneMTU and the highSpeedMTU parameters to the OperatorConfig YAML, for example:
---
apiVersion: operator.dpu.nvidia.com/v1alpha1
kind: DPFOperatorConfig
metadata:
name: dpfoperatorconfig
namespace: dpf-operator-system
spec:
provisioningController:
bfbPVCName: "bfb-pvc"
dmsTimeout: 900
kamajiClusterManager:
disable: false
networking:
controlPlaneMTU: 9000
highSpeedMTU: 9000
Please add the following lines to the OVS commands script section in the DPUFlavor YAML:
_ovs-vsctl set Interface p0 mtu_request=9216
_ovs-vsctl set Interface p1 mtu_request=9216
For example:
_ovs-vsctl set Open_vSwitch . other_config:doca-init=true
_ovs-vsctl set Open_vSwitch . other_config:dpdk-max-memzones=50000
_ovs-vsctl set Open_vSwitch . other_config:hw-offload=true
_ovs-vsctl set Open_vSwitch . other_config:pmd-quiet-idle=true
_ovs-vsctl set Open_vSwitch . other_config:max-idle=20000
_ovs-vsctl set Open_vSwitch . other_config:max-revalidator=5000
_ovs-vsctl --if-exists del-br ovsbr1
_ovs-vsctl --if-exists del-br ovsbr2
_ovs-vsctl --may-exist add-br br-sfc
_ovs-vsctl set bridge br-sfc datapath_type=netdev
_ovs-vsctl set bridge br-sfc fail_mode=secure
_ovs-vsctl --may-exist add-port br-sfc p0
_ovs-vsctl set Interface p0 type=dpdk
_ovs-vsctl set Interface p0 mtu_request=9216
_ovs-vsctl set Port p0 external_ids:dpf-type=physical
_ovs-vsctl --may-exist add-port br-sfc p1
_ovs-vsctl set Interface p1 type=dpdk
_ovs-vsctl set Interface p1 mtu_request=9216
_ovs-vsctl set Port p1 external_ids:dpf-type=physical
Please increase the hugepages allocation in the DPUFlavor YAML:
- hugepages=8072