DPF Book Template - RDG for DPF with OVN-Kubernetes and HBN Services Demo
NVIDIA Docs Hub  NVIDIA Networking  Networking Solutions  DPF Book Template - RDG for DPF with OVN-Kubernetes and HBN Services Demo  Host Configuration

On This Page

Host Configuration

Warning

Make sure that the BIOS settings on the worker node servers have SR-IOV enabled and that the servers are tuned for maximum performance.

Warning

All worker nodes must have the same PCIe placement for the BlueField-3 NIC and must show the same interface name.

Hypervisor Installation and Configuration

The hypervisor used in this Reference Deployment Guide (RDG) is based on Ubuntu 24.04 with KVM.

While this document does not detail the KVM installation process, it is important to note that the setup requires the following ISOs to deploy the Firewall, Jump, and MaaS virtual machines (VMs):

  • Ubuntu 24.04

  • pfSense-CE-2.7.2

To implement the solution, three Linux bridges must be created on the hypervisor:

Note

Ensure a DHCP record is configured for the lab-br bridge interface in your trusted LAN to assign it an IP address.

  • lab-br – connects the Firewall VM to the trusted LAN.

  • mgmt-br – Connects the various VMs to the host management network.

  • hs-br – Connects the Firewall VM to the high-speed network.

Additionally, an MTU of 9000 must be configured on the management and high-speed bridges (mgmt-br and hs-br) as well as their uplink interfaces to ensure optimal performance.

Hypervisor netplan configuration

Copy
Copied!
            

            
network:
    ethernets:
        eno1:
            dhcp4: false
        eno2:
            dhcp4: false
            mtu: 9000
        ens2f0np0:
            dhcp4: false
            mtu: 9000
    bridges:
      lab-br:
         interfaces: [eno1]
         dhcp4: true
      mgmt-br:
         interfaces: [eno2]
         dhcp4: false
         mtu: 9000
      hs-br:
         interfaces: [ens2f0np0]
         dhcp4: false
         mtu: 9000
    version: 2

Apply the configuration:

Hypervisor Console

Copy
Copied!
            

            
$ sudo netplan apply

Prepare Infrastructure Servers

Firewall VM - pfSense Installation and Interface Configuration

Download the pfSense CE (Community Edition) ISO to your hypervisor and proceed with the software installation.

Suggested spec:

  • vCPU: 2

  • RAM: 2GB

  • Storage: 10GB

  • Network interfaces

    • Bridge device connected to lab-br

    • Bridge device connected to mgmt-br

    • Bridge device connected to hs-br

The Firewall VM must be connected to all three Linux bridges on the hypervisor. Before beginning the installation, ensure that three virtual network interfaces of type "Bridge device" are configured. Each interface should be connected to a different bridge (lab-br, mgmt-br, and hs-br) as illustrated in the diagram below.

FW_VM_NIC-version-1-modificationdate-1751545013377-api-v2.png

After completing the installation, the setup wizard displays a menu with several options, such as "Assign Interfaces" and "Reboot System." During this phase, you must configure the network interfaces for the Firewall VM.

  1. Select Option 2: "Set interface(s) IP address" and configure the interfaces as follows:

    • WAN (lab-br) – Trusted LAN IP (Static/DHCP)

    • LAN (mgmt-br) – Static IP 10.0.110.254/24

    • OPT1 (hs-br) – Static IP 172.169.50.1/30

  2. Once the interface configuration is complete, use a web browser within the host management network to access the Firewall web interface and finalize the configuration.

Next, proceed with the installation of the Jump VM. This VM will serve as a platform for running a browser to access the Firewall’s web interface for post-installation configuration.

Jump VM

Suggested specifications:

  • vCPU: 4

  • RAM: 8GB

  • Storage: 25GB

  • Network interface: Bridge device, connected to mgmt-br

Procedure:

  1. Proceed with a standard Ubuntu 24.04 installation. Use the following login credentials across all hosts in this setup:

    Username

    Password

    depuser

    user

  2. Enable internet connectivity and DNS resolution by creating the following Netplan configuration:

    Note

    Use 10.0.110.254 as a temporary DNS nameserver until the MaaS VM is installed and configured. After completing the MaaS installation, update the Netplan file to replace this address with the MaaS IP: 10.0.110.252.

    Jump Node netplan

    Copy
    Copied!
                
    
            
    network:
    ethernets:
        enp1s0:
            dhcp4: false
            addresses: [10.0.110.253/24]
            nameservers:
              search: [dpf.rdg.local.domain]
              addresses: [10.0.110.254]
            routes:
              - to: default
                via: 10.0.110.254
    version: 2

  3. Apply the configuration:

    Jump Node Console

    Copy
    Copied!
                
    
            
    depuser@jump:~$ sudo netplan apply

  4. Update and upgrade the system:

    Jump Node Console

    Copy
    Copied!
                
    
            
    depuser@jump:~$ sudo apt update -y
depuser@jump:~$ sudo apt upgrade -y

  5. Install and configure the Xfce desktop environment and XRDP (complementary packages for RDP):

    Jump Node Console

    Copy
    Copied!
                
    
            
    depuser@jump:~$ sudo apt install -y xfce4 xfce4-goodies
depuser@jump:~$ sudo apt install -y lightdm-gtk-greeter
depuser@jump:~$ sudo apt install -y xrdp
depuser@jump:~$ echo "xfce4-session" | tee .xsession
depuser@jump:~$ sudo systemctl restart xrdp

  6. Install Firefox for accessing the Firewall web interface:

    Jump Node Console

    Copy
    Copied!
                
    
            
    $ sudo apt install -y firefox

  7. Install and configure an NFS server with the /mnt/dpf_share directory:

    Jump Node Console

    Copy
    Copied!
                
    
            
    $ sudo apt install -y nfs-server
$ sudo mkdir -m 777 /mnt/dpf_share
$ sudo vi /etc/exports

  8. Add the following line to /etc/exports:

    Jump Node Console

    Copy
    Copied!
                
    
            
    /mnt/dpf_share 10.0.110.0/24(rw,sync,no_subtree_check)

  9. Restart the NFS server:

    Jump Node Console

    Copy
    Copied!
                
    
            
    $ sudo systemctl restart nfs-server

  10. Create the directory bfb under /mnt/dpf_share with the same permissions as the parent directory:

    Jump Node Console

    Copy
    Copied!
                
    
            
    $ sudo mkdir -m 777 /mnt/dpf_share/bfb

  11. Generate an SSH key pair for depuser in the jump node (later on will be imported to the admin user in MaaS to enable password-less login to the provisioned servers):

    Jump Node Console

    Copy
    Copied!
                
    
            
    depuser@jump:~$ ssh-keygen -t rsa

  12. Reboot the jump node to display the graphical user interface:

    Jump Node Console

    Copy
    Copied!
                
    
            
    depuser@jump:~$ sudo reboot

    Note

    After setting up port-forwarding rules on the firewall (next steps), remote login to the graphical interface of the Jump node will be available.

    Concurrent login to the local graphical console and using RDP isn't possible, make sure to first log out from the local console when switching to RDP connection.

Firewall VM – Web Configuration

From your Jump node, open Firefox web browser and go to the pfSense web UI (http://10.0.110.254, default credentials are admin/pfsense). You should see a page similar to the following:

Note

The IP addresses from the trusted LAN network under "DNS servers" and "Interfaces - WAN" are blurred.

firewall_main_page_blur-version-1-modificationdate-1751545012863-api-v2.png

Proceed with the following configurations:

Note

The following screenshots display only a part of the configuration view. Make sure to not miss any of the steps mentioned below!

  • Interfaces

    • WAN – mark “Enable interface”, unmark “Block private networks and loopback addresses”

    • LAN – mark “Enable interface”, “IPv4 configuration type”: Static IPv4 ("IPv4 Address": 10.0.110.254/24, "IPv4 Upstream Gateway": None), “MTU”: 9000

    • OPT1 – mark “Enable interface”, “IPv4 configuration type”: Static IPv4 ("IPv4 Address": 172.169.50.1/30, "IPv4 Upstream Gateway": None), “MTU”: 9000

      Firewall_LAN_Interface-version-1-modificationdate-1751545012407-api-v2.png

  • Firewall:

    • NAT -> Port Forward -> Add rule -> “Interface”: WAN, “Address Family”: IPv4, “Protocol”: TCP, “Destination”: WAN address, “Destination port range”: (“From port”: SSH, “To port”: SSH), “Redirect target IP”: (“Type”: Address or Alias, “Address”: 10.0.110.253), “Redirect target port”: SSH, “Description”: NAT SSH

    • NAT -> Port Forward -> Add rule -> “Interface”: WAN, “Address Family”: IPv4, “Protocol”: TCP, “Destination”: WAN address, “Destination port range”: (“From port”: MS RDP, “To port”: MS RDP), “Redirect target IP”: (“Type”: Address or Alias, “Address”: 10.0.110.253), “Redirect target port”: MS RDP, “Description”: NAT RDP

      pfsense_nat_forward_ssh-version-1-modificationdate-1751545012043-api-v2.png

      Firewall_NAT_rules-version-1-modificationdate-1751545011570-api-v2.png

    • Rules -> OPT1 -> Add rule -> “Action”: Pass, “Interface”: OPT1, “Address Family”: IPv4+IPv6, “Protocol”: Any, “Source”: Any, “Destination”: Any

      Firewall_OPT1_Rules-version-1-modificationdate-1751545011253-api-v2.png

  • System:

    • Routing → Gateways → Add → “Interface”: OPT1, “Address Family”: IPv4, “Name”: switch, “Gateway”: 172.169.50.2 → Click "Save"→ Under "Default Gateway" - "Default gateway IPv4" choose WAN_DHCP → Click "Save"

      pfsense_add_gateway-version-1-modificationdate-1751545010797-api-v2.png

      Note

      Note that the IP addresses from the Trusted LAN network under "Gateway" and "Monitor IP" are blurred.

      pfsense_default_gw_blur-version-1-modificationdate-1751545010387-api-v2.png

    • Routing → Static Routes → Add → “Destination network”: 10.0.120.0/22, “Gateway”: switch – 172.169.50.2, “Description”: To HS network → Click "Save"

      pfsense_add_static_route-version-1-modificationdate-1751545009950-api-v2.png

      Firewall_System_StaticRoute-version-1-modificationdate-1751545009573-api-v2.png

MaaS VM

Suggested specifications:

  • vCPU: 4

  • RAM: 4GB

  • Storage: 50GB

  • Network interface: Bridge device, connected to mgmt-br

Procedure:

  1. Perform a regular Ubuntu installation on the MaaS VM.

  2. Create the following Netplan configuration to enable internet connectivity and DNS resolution:

    Note

    Use 10.0.110.254 as a temporary DNS nameserver. After the MaaS installation, replace this with the MaaS IP address (10.0.110.252) in both the Jump and MaaS VM Netplan files.

    MaaS netplan

    Copy
    Copied!
                
    
            
    network:
    ethernets:
        enp1s0:
            dhcp4: false
            addresses: [10.0.110.252/24]
            nameservers:
              search: [dpf.rdg.local.domain]
              addresses: [10.0.110.254]
            routes:
              - to: default
                via: 10.0.110.254
    version: 2

  3. Apply the netplan configuration:

    MaaS Console

    Copy
    Copied!
                
    
            
    depuser@maas:~$ sudo netplan apply

  4. Update and upgrade the system:

    MaaS Console

    Copy
    Copied!
                
    
            
    depuser@maas:~$ sudo apt update -y
depuser@maas:~$ sudo apt upgrade -y

  5. Install PostgreSQL and configure the database for MaaS:

    MaaS Console

    Copy
    Copied!
                
    
            
    $ sudo -i
# apt install -y postgresql
# systemctl enable --now postgresql
# systemctl disable --now systemd-timesyncd
# export MAAS_DBUSER=maasuser
# export MAAS_DBPASS=maaspass
# export MAAS_DBNAME=maas
# sudo -i -u postgres psql -c "CREATE USER \"$MAAS_DBUSER\" WITH ENCRYPTED PASSWORD '$MAAS_DBPASS'"
# sudo -i -u postgres createdb -O "$MAAS_DBUSER" "$MAAS_DBNAME"

  6. Install MaaS:

    MaaS Console

    Copy
    Copied!
                
    
            
    # snap install maas

  7. Initialize MaaS:

    MaaS Console

    Copy
    Copied!
                
    
            
    # maas init region+rack --maas-url http://10.0.110.252:5240/MAAS --database-uri "postgres://$MAAS_DBUSER:$MAAS_DBPASS@localhost/$MAAS_DBNAME"

  8. Create an admin account:

    MaaS Console

    Copy
    Copied!
                
    
            
    # maas createadmin --username admin --password admin --email admin@example.com

  9. Save the admin API key:

    MaaS Console

    Copy
    Copied!
                
    
            
    # maas apikey --username admin > admin-apikey

  10. Log in to the MaaS server:

    MaaS Console

    Copy
    Copied!
                
    
            
    # maas login admin http://localhost:5240/MAAS "$(cat admin-apikey)"

  11. Configure MaaS (Substitute <Trusted_LAN_NTP_IP> and <Trusted_LAN_DNS_IP> with the IP addresses in your environment):

    MaaS Console

    Copy
    Copied!
                
    
            
    # maas admin domain update maas name="dpf.rdg.local.domain"
# maas admin maas set-config name=ntp_servers value="<Trusted_LAN_NTP_IP>"
# maas admin maas set-config name=network_discovery value="disabled"
# maas admin maas set-config name=upstream_dns value="<Trusted_LAN_DNS_IP>"
# maas admin maas set-config name=dnssec_validation value="no"
# maas admin maas set-config name=default_osystem value="ubuntu"

  12. Define and configure IP ranges and subnets:

    MaaS Console

    Copy
    Copied!
                
    
            
    # maas admin ipranges create type=dynamic start_ip="10.0.110.51" end_ip="10.0.110.120"
# maas admin ipranges create type=dynamic start_ip="10.0.110.21" end_ip="10.0.110.30"
# maas admin ipranges create type=reserved start_ip="10.0.110.10" end_ip="10.0.110.10" comment="c-plane VIP"
# maas admin ipranges create type=reserved start_ip="10.0.110.200" end_ip="10.0.110.200" comment="kamaji VIP"
# maas admin ipranges create type=reserved start_ip="10.0.110.251" end_ip="10.0.110.254" comment="dpfmgmt"
# maas admin vlan update 0 untagged dhcp_on=True primary_rack=maas mtu=9000
# maas admin dnsresources create fqdn=kube-vip.dpf.rdg.local.domain ip_addresses=10.0.110.10
# maas admin dnsresources create fqdn=jump.dpf.rdg.local.domain ip_addresses=10.0.110.253
# maas admin dnsresources create fqdn=fw.dpf.rdg.local.domain ip_addresses=10.0.110.254
# maas admin fabrics create
Success.
Machine-readable output follows:
{
"class_type": null,
"name": "fabric-1",
"id": 1,
...
# maas admin subnets create name="fake-dpf" cidr="20.20.20.0/24" fabric=1

  13. Complete MaaS setup:

    1. Connect to the Jump node GUI and access the MaaS UI at http://10.0.110.252:5240/MAAS.

    2. On the first page, verify the "Region Name" and "DNS Forwarder," then continue.

    3. On the image selection page, select Ubuntu 24.04 LTS (amd64) and sync the image.

      maas_OS_Image_Mix_Good-version-1-modificationdate-1751545009250-api-v2.png

    4. Import the previously generated SSH key (id_rsa.pub) for the depuser into the MaaS admin user profile and finalize the setup.

      import_sshkey-version-1-modificationdate-1751545008823-api-v2.png

  14. Configure DHCP snippets:

    1. Navigate to Settings → DHCP Snippets → Add Snippet.

    2. Fill in the following fields:

      1. Name: dpf-mgmt

      2. Toggle on "Enabled"

      3. Type: IP Range

      4. Applies to: 10.0.110.21-10.0.110.30

    3. Fill in the content of the DHCP snippet field with the following (replace MAC address as appropriate with your workers MGMT interface MAC):

      DHCP snippet

      Copy
      Copied!
                  
      
            
      # worker1
host worker1 {
   # 
   # Node DHCP snippets
   # 
 
   hardware ethernet 04:32:01:60:0d:da;
   fixed-address 10.0.110.21;
}
# worker2
host worker2 {
   # 
   # Node DHCP snippets
   # 
 
   hardware ethernet 04:32:01:5f:cb:e0;
   fixed-address 10.0.110.22;
}

  15. Go to Settings → Deploy, set "Default OS release" to Ubuntu 24.04 LTS Noble Numbat, and save.

    maas_os-version_deployment-version-1-modificationdate-1751545008400-api-v2.png

  16. Update the DNS nameserver IP address in both Jump and MaaS VM Netplan files from 10.0.110.254 to 10.0.110.252 and reapply the configuration.

K8s Master VMs

Suggested specifications:

  • vCPU: 8

  • RAM: 16GB

  • Storage: 100GB

  • Network interface: Bridge device, connected to mgmt-br

  1. Before provisioning the Kubernetes (K8s) Master VMs with MaaS, create the required virtual disks with empty storage. Use the following one-liner to create three 100 GB QCOW2 virtual disks:

    Hypervisor Console

    Copy
    Copied!
                
    
            
    $ for i in $(seq 1 3); do qemu-img create -f qcow2 /var/lib/libvirt/images/master$i.qcow2 100G; done

    This command generates the following disks in the /var/lib/libvirt/images/ directory:

    • master1.qcow2

    • master2.qcow2

    • master3.qcow2

  2. Configure VMs in virt-manager:

    1. Open virt-manager and create three virtual machines:

      • Assign the corresponding virtual disk (master1.qcow2, master2.qcow2, or master3.qcow2) to each VM.

      • Configure each VM with the suggested specifications (vCPU, RAM, storage, and network interface).

    2. During the VM setup, ensure the NIC is selected under the Boot Options tab. This ensures the VMs can PXE boot for MaaS provisioning.

    3. Once the configuration is complete, shut down all the VMs.

  3. After the VMs are created and configured, proceed to provision them via the MaaS interface. MaaS will handle the OS installation and further setup as part of the deployment process.

Provision Master VMs and Worker Nodes Using MaaS

Master VMs

Install virsh and Set Up SSH Access

  1. SSH to the MaaS VM from the Jump node:

    MaaS Console

    Copy
    Copied!
                
    
            
    depuser@jump:~$ ssh maas
depuser@maas:~$ sudo -i

  2. Install the virsh client to communicate with the hypervisor:

    MaaS Console

    Copy
    Copied!
                
    
            
    # apt install -y libvirt-clients

  3. Generate an SSH key for the root user and copy it to the hypervisor user in the libvirtd group:

    MaaS Console

    Copy
    Copied!
                
    
            
    # ssh-keygen -t rsa
# ssh-copy-id ubuntu@<hypervisor_MGMT_IP>

  4. Verify SSH access and virsh communication with the hypervisor:

    MaaS Console

    Copy
    Copied!
                
    
            
    # virsh -c qemu+ssh://ubuntu@<hypervisor_MGMT_IP>/system list --all

    Expected output:

    MaaS Console

    Copy
    Copied!
                
    
            
     Id   Name          State
------------------------------
 1    fw     running
 2    jump   running
 3    maas   running
 -    master1       shut off
 -    master2       shut off
 -    master3       shut off

  5. Copy the SSH key to the required MaaS directory (for snap-based installations):

    MaaS Console

    Copy
    Copied!
                
    
            
    # mkdir -p /var/snap/maas/current/root/.ssh
# cp .ssh/id_rsa* /var/snap/maas/current/root/.ssh/

Get MAC Addresses of the Master VMs

Retrieve the MAC addresses of the Master VMs:

MaaS Console

Copy
Copied!
            

            
# for i in $(seq 1 3); do virsh -c qemu+ssh://ubuntu@<hypervisor_MGMT_IP>/system dumpxml master$i | grep 'mac address'; done

Example output:

MaaS Console

Copy
Copied!
            

            
<mac address='52:54:00:a9:9c:ef'/>
<mac address='52:54:00:19:6b:4d'/>
<mac address='52:54:00:68:39:7f'/>


Add Master VMs to MaaS

  1. Add the Master VMs to MaaS:

    Info

    Once added, MaaS will automatically start the newly added VMs commissioning (discovery and introspection).

    MaaS Console

    Copy
    Copied!
                
    
            
    # maas admin machines create hostname=master1 architecture=amd64/generic mac_addresses='52:54:00:a9:9c:ef' power_type=virsh power_parameters_power_address=qemu+ssh://ubuntu@<hypervisor_MGMT_IP>/system power_parameters_power_id=master1 skip_bmc_config=1 testing_scripts=none
Success.
Machine-readable output follows:
{
    "description": "",
    "status_name": "Commissioning",
...
    "status": 1, 
...
    "system_id": "c3seyq",
...
    "fqdn": "master1.dpf.rdg.local.domain",
    "power_type": "virsh",
...
    "status_message": "Commissioning",
    "resource_uri": "/MAAS/api/2.0/machines/c3seyq/"
}
 
# maas admin machines create hostname=master2 architecture=amd64/generic mac_addresses='52:54:00:19:6b:4d' power_type=virsh power_parameters_power_address=qemu+ssh://ubuntu@<hypervisor_MGMT_IP>/system power_parameters_power_id=master2 skip_bmc_config=1 testing_scripts=none
 
# maas admin machines create hostname=master3 architecture=amd64/generic mac_addresses='52:54:00:68:39:7f' power_type=virsh power_parameters_power_address=qemu+ssh://ubuntu@<hypervisor_MGMT_IP>/system power_parameters_power_id=master3 skip_bmc_config=1 testing_scripts=none

    Repeat the command for master2 and master3 with their respective MAC addresses.

  2. Verify commissioning by waiting for the status to change to "Ready" in MaaS.

    maas_masters_commission_virsh_updated-version-1-modificationdate-1751545007960-api-v2.png

    After commissioning, the next phase is the deployment (OS provisioning).

Configure OVS Bridges on Master VMs

To be able to have persistency across reboots, create an OVS-bridge from each management interface of the master nodes and assign it a static IP address.

For each Master VM:

  1. Create an OVS bridge in the MaaS Network tab:

    1. Navigate to Network → Management Interface → Create Bridge.

    2. Configure as follows:

      1. Name: brenp1s0 (prefix br added to the interface name)

      2. Bridge Type: Open vSwitch (ovs)

      3. Subnet: 10.0.110.0/24

      4. IP Mode: Static Assign

      5. Address: Assign 10.0.110.1 for master1, 10.0.110.2 for master2, and 10.0.110.3 for master3.

        maas_master1_ovs_bridge_updated-version-1-modificationdate-1751545007603-api-v2.png

  2. Save the interface settings for each VM.

Deploy Master VMs Using Cloud-Init

  1. Use the following cloud-init script to configure the necessary software and ensure OVS bridge persistency:

    Note

    Replace enp1s0 and brenp1s0 in the following cloud-init with your interface names as displayed in MaaS network tab.

    Master nodes cloud-init

    Copy
    Copied!
                
    
            
    #cloud-config
system_info:
  default_user:
    name: depuser
    passwd: "$6$jOKPZPHD9XbG72lJ$evCabLvy1GEZ5OR1Rrece3NhWpZ2CnS0E3fu5P1VcZgcRO37e4es9gmriyh14b8Jx8gmGwHAJxs3ZEjB0s0kn/"
    lock_passwd: false
    groups: [adm, audio, cdrom, dialout, dip, floppy, lxd, netdev, plugdev, sudo, video]
    sudo: ["ALL=(ALL) NOPASSWD:ALL"]
    shell: /bin/bash
ssh_pwauth: True
package_upgrade: true
runcmd:
    - apt-get update
    - apt-get -y install openvswitch-switch nfs-common
    - |
      UPLINK_MAC=$(cat /sys/class/net/enp1s0/address)
      ovs-vsctl set Bridge brenp1s0 other-config:hwaddr=$UPLINK_MAC
      ovs-vsctl br-set-external-id brenp1s0 bridge-id brenp1s0 -- br-set-external-id brenp1s0 bridge-uplink enp1s0

  2. Deploy the master VMs:

    1. Select all three Master VMs → Actions → Deploy.

    2. Toggle Cloud-init user-data and paste the cloud-init script.

    3. Start the deployment and wait for the status to change to "Ubuntu 24.04 LTS".

      maas_master_vms_deployment_before-version-1-modificationdate-1751545007193-api-v2.png

      maas_master_vms_deployment_complete_updated-version-1-modificationdate-1751545006827-api-v2.png

Verify Deployment

  • SSH into the Master VMs from the Jump node:

    Jump Node Console

    Copy
    Copied!
                
    
            
    depuser@jump:~$ ssh master1
depuser@master1:~$

  • Run sudo without password:

    Master1 Console

    Copy
    Copied!
                
    
            
    depuser@master1:~$ sudo -i
root@master1:~#

  • Verify installed packages:

    Master1 Console

    Copy
    Copied!
                
    
            
    root@master1:~# apt list --installed | egrep 'openvswitch-switch|nfs-common'
nfs-common/noble-updates,now 1:2.6.4-3ubuntu5.1 amd64 [installed]
openvswitch-switch/noble-updates,noble-security,noble-security,now 3.3.0-1ubuntu3.2 amd64 [installed]

  • Check OVS bridge attributes:

    Master1 Console

    Copy
    Copied!
                
    
            
    root@master1:~# ovs-vsctl list bridge brenp1s0

    Output example:

    Master1 Console

    Copy
    Copied!
                
    
            
    ...
external_ids        : {bridge-id=brenp1s0, bridge-uplink=enp1s0, netplan="true", "netplan/global/set-fail-mode"=standalone, "netplan/mcast_snooping_enable"="false", "netplan/rstp_enable"="false"}
...
other_config        : {hwaddr="52:54:00:a9:9c:ef"}
...

  • Verify that enp1s0 and brenp1s0 are configured with 9000 MTU (replace enp1s0 and brenp1s0 with your interface names):

    Master1 Console

    Copy
    Copied!
                
    
            
    root@master1:~# ip a show enp1s0; ip a show brenp1s0
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast master ovs-system state UP group default qlen 1000
    link/ether 52:54:00:a9:9c:ef brd ff:ff:ff:ff:ff:ff
4: brenp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 52:54:00:a9:9c:ef brd ff:ff:ff:ff:ff:ff
    inet 10.0.110.1/24 brd 10.0.110.255 scope global brenp1s0
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fea9:9cef/64 scope link
       valid_lft forever preferred_lft forever

Finalize Setup

Reboot the Master VMs to complete the provisioning.

Master1 Console

Copy
Copied!
            

            
root@master1:~# reboot

Worker Nodes

Create Worker Machines in MaaS

  1. Add the worker nodes to MaaS using ipmi as the power type. Replace placeholders with your specific IPMI credentials and IP addresses:

    MaaS Console

    Copy
    Copied!
                
    
            
    # maas admin machines create hostname=worker1 architecture=amd64 power_type=ipmi power_parameters_power_driver=LAN_2_0 power_parameters_power_user=<IPMI_username_worker1> power_parameters_power_pass=<IPMI_password_worker1> power_parameters_power_address=<IPMI_address_worker1>

    Output example:

    MaaS Console

    Copy
    Copied!
                
    
            
    ...
Success.
Machine-readable output follows:
{
    "description": "",
    "status_name": "Commissioning",
...
    "status": 1,
...
    "system_id": "pbskd3",
...
    "fqdn": "worker1.dpf.rdg.local.domain",
...
    "power_type": "ipmi",
...
    "resource_uri": "/MAAS/api/2.0/machines/pbskd3/"
}

  2. Repeat the command for worker2 with its respective credentials:

    MaaS Console

    Copy
    Copied!
                
    
            
    # maas admin machines create hostname=worker2 architecture=amd64 power_type=ipmi power_parameters_power_driver=LAN_2_0 power_parameters_power_user=<IPMI_username_worker2> power_parameters_power_pass=<IPMI_password_worker2> power_parameters_power_address=<IPMI_address_worker2>

Once added, MaaS will automatically start commissioning the worker nodes (discovery and introspection).

Create a Tag for Kernel Parameters

Create an entity called "Tag" to configure kernel parameters for the worker nodes.

  1. In the MaaS UI sidebar, go to Organization → Tags → Create New Tag and define

    • "Tag name": compute_performance

    • "Kernel options":

  2. Substitute the values for isolcpus, nohz_full, and rcu_nocbs to the CPU cores in the NUMA node which the BlueField-3 is connected to:

    Note

    If you are not sure in which NUMA node BlueField is connected to, you can later perform this step after the worker node is deployed (although redeployment would be necessary).

    Kernel options for worker nodes

    Copy
    Copied!
                
    
            
    intel_iommu=on iommu=pt numa_balancing=disable processor.max_cstate=0 isolcpus=28-55,84-111 nohz_full=28-55,84-111 rcu_nocbs=28-55,84-111

  3. Apply the tag:

    1. Go to Machines → Select a worker node → Configuration → Edit Tag → Select compute_performance → Save.

    2. Repeat for the other worker node.

Adjust Network Settings

For each worker node, configure the network interfaces:

  • Management Adapter:

    • Go to Network → Select the host management adapter (e.g., ens15f0) → Create Bridge

    • Name: br-dpu

    • Bridge Type: Standard

    • Subnet: 10.0.110.0/24

    • IP Mode: DHCP

    • Save the interface

  • BlueField Adapter:

    • Select P0 on the BlueField adapter (e.g., ens5f0np0) → Actions → Edit Physical

    • Fabric: Fabric-1

    • Subnet: 20.20.20.0/24 (fake-dpf)

    • IP Mode: DHCP

    • Save the interface

Repeat these steps for the second worker node.

maas_edit_physical_interface_v25.4-version-1-modificationdate-1751545006460-api-v2.png

Deploy Worker Nodes Using Cloud-Init

  1. Use the following cloud-init script for deployment. Replace ens5f0np0 with your actual interface name:

    Worker node cloud-init

    Copy
    Copied!
                
    
            
    #cloud-config
system_info:
  default_user:
    name: depuser
    passwd: "$6$jOKPZPHD9XbG72lJ$evCabLvy1GEZ5OR1Rrece3NhWpZ2CnS0E3fu5P1VcZgcRO37e4es9gmriyh14b8Jx8gmGwHAJxs3ZEjB0s0kn/"
    lock_passwd: false
    groups: [adm, audio, cdrom, dialout, dip, floppy, lxd, netdev, plugdev, sudo, video]
    sudo: ["ALL=(ALL) NOPASSWD:ALL"]
    shell: /bin/bash
ssh_pwauth: True
package_upgrade: true
write_files:
  - path: /etc/sysctl.d/99-custom-netfilter.conf
    owner: root:root
    permissions: '0644'
    content: |
      net.bridge.bridge-nf-call-iptables=0
 
runcmd:
  - apt-get update
  - apt-get -y install nfs-common
  - sysctl --system 
  - sed -i '/^\s*ens5f0np0:/,/^\s*mtu:/ { /^\s*mtu:/d }' /etc/netplan/*.yaml
  - netplan apply

  2. Deploy the worker nodes by selecting the worker nodes in MaaS → Actions → Deploy → Customize options → Enable Cloud-init user-data → Paste the cloud-init script → Deploy.

Verify Deployment

After the deployment is complete verify that the worker nodes have been deployed successfully with the following commands:

  • SSH without password from the jump node:

    Jump Node Console

    Copy
    Copied!
                
    
            
    depuser@jump:~$ ssh worker1
depuser@worker1:~$

  • Run sudo without password:

    Worker1 Console

    Copy
    Copied!
                
    
            
    depuser@worker1:~$ sudo -i
root@worker1:~#

  • Validate that nfs-common package was installed:

    Worker1 Console

    Copy
    Copied!
                
    
            
    root@worker1:~# apt list --installed | grep 'nfs-common'
nfs-common/noble-updates,now 1:2.6.4-3ubuntu5.1 amd64 [installed]

  • /proc/cmdline is configured with the correct parameters and that IOMMU is indeed in passthrough mode:

    Worker1 Console

    Copy
    Copied!
                
    
            
    root@worker1:~# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-6.8.0-60-generic root=UUID=a4aed791-5a14-459e-a754-a80d524f5fff ro intel_iommu=on iommu=pt numa_balancing=disable processor.max_cstate=0 isolcpus=28-55,84-111 nohz_full=28-55,84-111 rcu_nocbs=28-55,84-111
 
root@worker1:~# dmesg | grep 'type: Passthrough'
[    5.033173] iommu: Default domain type: Passthrough (set via kernel command line)

  • br_netfilter module is not loaded:

    Worker1 Console

    Copy
    Copied!
                
    
            
    root@worker1:~# lsmod | grep br_netfilter
root@worker1:~#

  • P0 interface has dhcp4 set to true and does not have mtu line in netplan configuration file.

    Worker1 Console

    Copy
    Copied!
                
    
            
    root@worker1:~# cat /etc/netplan/50-cloud-init.yaml
network:
...
		ens5f0np0:
            dhcp4: true
            match:
                macaddress: "58:a2:e1:95:54:ca"
            set-name: "ens5f0np0"
...

  • ens15f0 and br-dpu are with 9000 MTU (replace ens15f0 with your interface name):

    Worker1 Console

    Copy
    Copied!
                
    
            
    root@worker1:~# ip a show ens15f0; ip a show br-dpu
2: ens15f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq master br-dpu state UP group default qlen 1000
    link/ether 04:32:01:60:0d:da brd ff:ff:ff:ff:ff:ff
    altname enp53s0f0
8: br-dpu: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
    link/ether 04:32:01:60:0d:da brd ff:ff:ff:ff:ff:ff
    inet 10.0.110.21/24 metric 100 brd 10.0.110.255 scope global dynamic br-dpu
       valid_lft 403sec preferred_lft 403sec
    inet6 fe80::632:1ff:fe60:dda/64 scope link
       valid_lft forever preferred_lft forever

Finalize Deployment

Reboot the worker nodes:

Jump Node Console

Copy
Copied!
            

            
root@worker1:~# reboot

The infrastructure is now ready for the K8s deployment.

maas_worker_nodes_after_deployment_updated_2-version-1-modificationdate-1751545006047-api-v2.png

© Copyright 2025, NVIDIA. Last updated on Jul 10, 2025.
content here