Secondary CNI support for HBN-OVNK use case
This section covers an advanced configuration of the secondary CNI feature of the Host Based Networking and OVN Kubernetes use case. Enabling and configuring this feature allows for the creation of pods with secondary networks which are also accelerated by OVN Kubernetes as the secondary CNI.
Before proceeding with this advanced configuration, please ensure you have reviewed the Host Based Networking and OVN Kubernetes configuration guide first and completed that first. This advanced configuration builds upon that setup and provides additional steps to enable secondary cni support.
Upgrade the OVN Kubernetes CNI components from the helm chart. Ensure environment variables are set before running this command.
envsubst < manifests/01-cni-installation/ovn-kubernetes_secondary_network.yml | helm upgrade --install -n ovn-kubernetes ovn-kubernetes ${OVN_KUBERNETES_REPO_URL}/ovn-kubernetes-chart --version $TAG --values -
OVN-Kubernetes Helm values
commonManifests:
enabled: true
nodeWithoutDPUManifests:
enabled: true
controlPlaneManifests:
enabled: true
ovnMultiNetworkEnable: true # enables secondary CNI/network support
nodeWithDPUManifests:
enabled: true
nodeMgmtPortNetdev: $DPU_P0_VF1
dpuServiceAccountNamespace: dpf-operator-system
gatewayOpts: --gateway-interface=$DPU_P0
## Note this CIDR is followed by a trailing /24 which informs OVN Kubernetes on how to split the CIDR per node.
podNetwork: $POD_CIDR/24
serviceNetwork: $SERVICE_CIDR
k8sAPIServer: https://$TARGETCLUSTER_API_SERVER_HOST:$TARGETCLUSTER_API_SERVER_PORT
Verification
These verification commands may need to be run multiple times to ensure the condition is met.
Verify the CNI installation with:
## Ensure all nodes in the cluster are ready.
kubectl wait --for=condition=ready nodes --all
## Ensure all pods in the ovn-kubernetes namespace are ready.
kubectl wait --for=condition=ready --namespace ovn-kubernetes pods --all --timeout=300s
Ensure environment variables are set before running this command.
cat manifests/02-dpudeployment-modifications/*.yaml | envsubst | kubectl apply -f -
OVN DPUServiceTemplate to deploy OVN workloads to the DPUs
---
apiVersion: svc.dpu.nvidia.com/v1alpha1
kind: DPUServiceTemplate
metadata:
name: ovn
namespace: dpf-operator-system
spec:
deploymentServiceName: "ovn"
helmChart:
source:
repoURL: $OVN_KUBERNETES_REPO_URL
chart: ovn-kubernetes-chart
version: $TAG
values:
commonManifests:
enabled: true
dpuManifests:
enabled: true
ovnMultiNetworkEnable: true # enables secondary CNI/network support
nodeMgmtPortNetdev: $DPU_P0_VF1
leaseNamespace: "ovn-kubernetes"
gatewayOpts: "--gateway-interface=br-ovn"
Verification
These verification commands may need to be run multiple times to ensure the condition is met.
Note that the DPUService name will have a random suffix. For example, ovn-hbn-doca-hbn-l2xsl. Use the correct name for the verification.
Verify the DPU and Service installation with:
## Ensure the DPUServices are created and have been reconciled.
kubectl wait --for=condition=ApplicationsReconciled --namespace dpf-operator-system dpuservices -l svc.dpu.nvidia.com/owned-by-dpudeployment=dpf-operator-system_ovn-hbn
## Ensure the DPUServiceIPAMs have been reconciled
kubectl wait --for=condition=DPUIPAMObjectReconciled --namespace dpf-operator-system dpuserviceipam --all
## Ensure the DPUServiceInterfaces have been reconciled
kubectl wait --for=condition=ServiceInterfaceSetReconciled --namespace dpf-operator-system dpuserviceinterface --all
## Ensure the DPUServiceChains have been reconciled
kubectl wait --for=condition=ServiceChainSetReconciled --namespace dpf-operator-system dpuservicechain --all
If you want to create pods with secondary networks, first create a secondary network NetworkAttachmentDefinition.
kubectl apply -f manifests/03-test-traffic/nad_bf3_p0_vfs.yaml
Now you can create pods with secondary network interfaces using the following command:
kubectl apply -f manifests/03-test-traffic/pods-secondary-network.yaml
Once the pods are running, you can check the network interfaces inside the pods to verify that the secondary network interfaces have been created. They should be created with the interface name net1 and should have an IP address from the 192.168.100.0/24 as defined in the NetworkAttachmentDefinition.
You can ping or run iperf traffic between the pods using the secondary network interfaces to test connectivity and performance.