NVIDIA Support for TripleO Ussuri Application Notes
NVIDIA Docs Hub  NVIDIA Networking  Networking Software  NVIDIA Support for TripleO Ussuri Application Notes  BareMetal Provision with Bluefield

On This Page

BareMetal Provision with Bluefield

BlueField® SmartNIC adapters accelerate a wide range of applications through flexible data and control-plane offloading. Enabling a more efficient use of compute resources, BlueField adapters empower the CPU to focus on running applications rather than on networking or security processing. Additionally, as software-defined adapters, BlueField SmartNICs ensure the ultimate flexibility by adapting to future protocols and features through simple software updates.

Supported Platforms

Network Interface Cards

Firmware Version

BFB

BlueField-2

24.28.2006

CentOS 7.6 (3.1)

Mellanox BlueField SmartNIC Supported Features

Mellanox BlueField SmartNIC supports the following Features:

  • Mellanox ASAP2 Accelerated Switching and Packet Processing® for Open vSwitch (OVS) delivers flexible, highly efficient virtual switching and routing capabilities. OVS accelerations can be further enhanced using BlueField processing and memory. For example, the scale of OVS actions can be increased by utilizing BlueField internal memory, and more OVS actions and vSwitch/vRouter implementations can be supported.

  • Network overlay technology (VXLAN) offload, including encapsulation and decapsulation, allows the traditional offloads to operate on the tunneled protocols, and offload Network Address Translation (NAT) routing capabilities.

Deploying Overcloud with Ironic

Starting from a fresh RHEL Bare Metal server, install and configure the undercloud according to the official TripleOinstallation documentation.
Follow this link for TripleO instructions to prepare bare metal overcloud.

  1. Add provisioning network to network_data.yaml.

    Copy
    Copied!
                
    
            
    - name: OcProvisioning
  vip: true
  name_lower: oc_provisioning
  vlan: 215
  ip_subnet: '172.16.4.0/24'
  allocation_pools: [{'start': '172.16.4.1', 'end': '172.16.4.2'}]
  ipv6_subnet: 'fd00:fd00:fd00:7000::/64'
  ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:7000::10', 'end': 'fd00:fd00:fd00:700 addresses 0:ffff:ffff:ffff:fffe'}]
  mtu: 1500

Add provisioning network to controller.yaml (under ovs_bridge mlnx-br config).

Copy
Copied!
            

            
  vlan_id:
   get_param: OcProvisioningNetworkVlanID
   ip_netmask:
   get_param: OcProvisioningIpSubnet

Preparing BlueField

  1. Install the latest operating system on BlueField according to the BlueField Installation Guide. Please use BFB RHEL 7.6 and above.

  2. Enable PXE boot and set PXE retries on the BlueField FlexBoot.

    image2020-11-9_23-5-11.png

    image2020-11-9_23-5-24.png

  3. Bluefield has several operation modes. For proper operation, Bluefield must be configured as an embedded function, where the embedded Arm system controls the NIC resources and datapath.

    Make sure the following is configured in the Bluefield host:

    Copy
    Copied!
                
    
            
    $ mlxconfig -d /dev/mst/mt41682_pciconf0 q | grep -E "ECPF|CPU"
         INTERNAL_CPU_MODEL                  EMBEDDED_CPU(1)
         ECPF_ESWITCH_MANAGER                ECPF(1)
         ECPF_PAGE_SUPPLIER                  ECPF(1)

    In case of a different configuration, run the following command, and power the host:

    Copy
    Copied!
                
    
            
    $ mst start
$ mlxconfig -d /dev/mst/mt41682_pciconf0 s INTERNAL_CPU_MODEL=1

    For security and isolation purposes, it is possible to restrict the host from performing operations that can compromise the SmartNIC. See Restricting-Smart-NIC-Host.

Creating Neutron Agent Container on BlueField

To run OpenStack service as a container, the image and a starting script will be required.

  1. Pull the image to the docker repository.

  2. If the host has access to https://hub.docker.com/, the container can be pulled directly. Pull the image from the docker repository:

    Copy
    Copied!
                
    
            
    BlueField$ docker pull mellanox/centos-binary-neutron-openvswitch-agent-ussuri-aarch64:latest

  3. If the host has no access to https://hub.docker.com/, the container should be pulled on another server, saved and pushed to the host. Pull the image on another server:

    Copy
    Copied!
                
    
            
    host_with_internet$ docker pull mellanox/centos-binary-neutron-openvswitch-agent-ussuri-aarch64:latest
 
host_with_internet$ docker save mellanox/centos-binary-neutron-
 
openvswitch-agent-ussuri-aarch64:latest > /tmp/centos-binary-neutron-openvswitch-agent-ussuri-aarch64.tar
 
host_with_internet$ scp /tmp/centos-binary-neutron-openvswitch-agent-ussuri-aarch64.tar BlueField:/tmp
 
BlueField$ docker load -i /tmp/centos-binary-neutron-openvswitch-agent-ussuri-aarch64.tar

  4. Download neutron_ovs_agent_launcher.sh.

    Copy
    Copied!
                
    
            
    BlueField$ curl https://raw.githubusercontent.com/Mellanox/neutron-openvswitch-agent-builder/ussuri/neutron_ovs_agent_launcher.sh -o /root/neutron_ovs_agent_launcher.sh
 
BlueField$ chmod +x /root/neutron_ovs_agent_launcher.sh

  5. Create the container. Use the following script to create and start the container:

    Copy
    Copied!
                
    
            
    BlueField$ curl https://raw.githubusercontent.com/Mellanox/neutron-openvswitch-agent-builder/ussuri/neutron_ovs_agent_start.sh -o /root/neutron_ovs_agent_start.sh
BlueField$ chmod +x /root/neutron_ovs_agent_start.sh
 
BlueField$ /root/neutron_ovs_agent_start.sh

Bluefield Network Configuration

  1. Create VLAN interface with InternalApi VLAN in BlueField. Inside BlueField, create a network script in /etc/sysconfig/network-scripts/.

    Copy
    Copied!
                
    
            
    vi /etc/sysconfig/network-scripts/ifcfg-p0.<InternalApi vlan>
DEVICE=p0.<InternalApi vlan>
BOOTPROTO=none
ONBOOT=yes
IPADDR=<InternalApi free IP>
PREFIX=<InternalApi Subnet>
VLAN=yes

  2. Add the script to add an Uplink representor to the br-phys (bridge for VLAN).

    Copy
    Copied!
                
    
            
    DEVICE=p0
TYPE=OVSPort
DEVICETYPE=ovs
OVS_BRIDGE=br-phys
ONBOOT=yes

  3. Add the script to configure the Bluefield OOB port:

    Copy
    Copied!
                
    
            
    vi /etc/sysconfig/network-scripts/ifcfg-oob_net0 
NAME="oob_net0"
DEVICE="oob_net0"
NM_CONTROLLED="yes"
PEERDNS="yes"
ONBOOT="yes"
BOOTPROTO="dhcp"
TYPE=Ethernet

  4. Copy the /etc/hosts from the controller node to each of the BlueField hosts:

    Copy
    Copied!
                
    
            
    $ rsync -av /etc/hosts co-bm-2-arm:/etc/hosts

  5. Update the /etc/hosts on the controller and on BlueField. The IP address and the hostname of all Bluefield hosts in the cluster should look like the following:

    Copy
    Copied!
                
    
            
    172.16.2.251 co-bm-1-arm.localdomain co-bm-1-arm
172.16.1.251 co-bm-1-arm.storage.localdomain co-bm-1-arm.storage
172.16.2.251 co-bm-1-arm.internalapi.localdomain co-bm-1-arm.internalapi
172.16.0.251 co-bm-1-arm.tenant.localdomain co-bm-1-arm.tenant
192.168.24.251 co-bm-1-arm.ctlplane.localdomain co-bm-1-arm.ctlplane
172.16.2.252 co-bm-2-arm.localdomain co-bm-2-arm
172.16.1.252 co-bm-2-arm.storage.localdomain co-bm-2-arm.storage
172.16.2.252 co-bm-2-arm.internalapi.localdomain co-bm-2-arm.internalapi
172.16.0.252 co-bm-2-arm.tenant.local

    It is required for the provision network, which is usually a flat or a VLAN network.

  6. Restart the network service:

    Copy
    Copied!
                
    
            
    $ systemctl restart network

  7. Get Neutron configuration to the BlueField:

    1. Copy the Neutron configuration from the controller to the BlueField in the /var/lib/config-data/puppet-generated/neutron/etc/neutron/ directory.

    2. Copy the following files to BlueField:

      Copy
      Copied!
                  
      
            
      etc/neutron/neutron.conf
etc/neutron/plugins/ml2/ml2_conf.ini
etc/neutron/plugins/ml2/openvswitch_agent.ini

  8. Update the Neutron configuration. On the BlueField, changes must be made to set the correct values for the BlueField host:

    1. In the /etc/neutron/neutron.conf file, change the following:

      Copy
      Copied!
                  
      
            
      bind_host=<Bluefield IP>
host=<Bluefield hostname>

    2. In the /etc/neutron/plugins/ml2/ openvswitch_agent.ini file, change the following:

      Copy
      Copied!
                  
      
            
      ovs]baremetal_smartnic=True
[securitygroup]
firewall_driver=openvswitch

  9. Update the openvsiwtch with hw-offload enable:

    Copy
    Copied!
                
    
            
    $ systemctl enable openvswitch.service
$ ovs-vsctl set Open_vSwitch . other_config:hw-offload=true
$ systemctl restart openvswitch.service
$ docker restart neutron_ovs_agent

VXLAN Tenant Network Configuration in BlueField

  1. Create a VLAN interface with a tenant VLAN on BlueField by creating a network script in /etc/sysconfig/network-scripts/:

    Copy
    Copied!
                
    
            
    vi /etc/sysconfig/network-scripts/ifcfg-p0.<Tenant vlan>
DEVICE=p0.<Tenant vlan>
BOOTPROTO=none
ONBOOT=yes
IPADDR=<Tenant free IP>
PREFIX=<Tenant Subnet>
VLAN=yes

  2. Restart the network service:

    Copy
    Copied!
                
    
            
    $ systemctl restart network

  3. In the BlueField, edit the /etc/neutron/plugins/ml2/openvswitch_agent.ini file, as follows:

    Copy
    Copied!
                
    
            
    [ovs]	
local_ip=[BlueField vxlan tunnel ip]
[agent]
l2_population=True
arp_responder=True

  4. In the BlueField, restart the neutron OVS agent:

    Copy
    Copied!
                
    
            
    $ docker restart neutron_ovs_agent

  5. In the controller, update the /var/lib/config-data/puppet-generated/neutron/etc/neutron/plugins/ml2/ml2_conf.ini file, as follows:

    Copy
    Copied!
                
    
            
    mechanism_drivers=openvswitch,l2population

  6. In the controller, restart the neutron API:

    Copy
    Copied!
                
    
            
    $ podman restart neutron_api

    Lag Configuration

    LAG configuration allows the bare metal host to output traffic through a single network interface. Traffic arbitration is done in the internal Bluefield host.

    The steps consist of setting up LAG over Uplink representors in BlueField, and connecting the bond device to the the physical OVS bridge (br-phys) utilizing linux bonding module. These configurations are persisted using network scripts, but any other persistent network configuration mechanism is acceptable.

    Similarly to the instructions in Network Configuration in BlueField, over the bond it is required to:

    1. Configure the VLAN interface to be used for OpenStack API traffic

    2. Connect the bond to OVS

    3. Enable hardware offload in OVS

    4. Restart neutron_ovs_agent_container

    Bond device network script:

    Copy
    Copied!
                
    
            
    # network script for bond device
cat << EOF > /etc/sysconfig/network-scripts/ifcfg-bond0 
DEVICE=bond0
NAME=bond0 TYPE=Bond
BONDING_MASTER=yes
ONBOOT=yes
BOOTPROTO=none
BONDING_OPTS="mode=active-backup miimon=100"
NM_CONTROLLED="no"
EOF
 
# network script for vlan device over bond for API traffic
cat << EOF > /etc/sysconfig/network-scripts/ifcfg-bond0.<InternalApi vlan> DEVICE=bond0.<InternalApi vlan>
BOOTPROTO=none
ONBOOT=yes
IPADDR=<InternalApi free IP>
PREFIX=<InternalApi Subnet>
VLAN=yes

    Uplink representors network scripts:

    Copy
    Copied!
                
    
            
    # network script for first uplink representor
$ cat << EOF > /etc/sysconfig/network-scripts/ifcfg-p0
DEVICE=p0
NAME=p0
TYPE=Ethernet 
BOOTPROTO=none
ONBOOT=yes
MASTER=bond0
SLAVE=yes
NM_CONTROLLED="no"
EOF
 
# network script for second uplink representor
 
$ cat << EOF > /etc/sysconfig/network-scripts/ifcfg-p1
DEVICE=p1
NAME=p1
TYPE=Ethernet
BOOTPROTO=none
ONBOOT=yes
MASTER=bond0
SLAVE=yes
NM_CONTROLLED="no"
EOF

    To apply configuration invoke without reboot:

    Copy
    Copied!
                
    
            
    $ systemctl restart network

    Connect bond0 to the bridge:

    Copy
    Copied!
                
    
            
    $ ovs-vsctl add-port br-phys bond0

Creating Overcloud Networks

A bare metal environment needs at least two networks: provisioning network and tenant network.

  1. Create a provisioning network.

    Copy
    Copied!
                
    
            
    openstack network create --share --provider-network-type vlan --provider-physical-network datacentre --provider-segment 215 provisioning
openstack subnet create --network provisioning --subnet-range 172.16.4.0/24 --gateway 172.16.4.6 --allocation-pool start=172.16.4.201,end=172.16.4.250 provisioning-subnet

  2. Open the following port in the default security group of the provisioning network:

    Copy
    Copied!
                
    
            
    ADMIN_SECURITY_GROUP_ID=$(openstack security group list --project admin -f value -c ID)
openstack security group rule create ${ADMIN_SECURITY_GROUP_ID} --protocol udp --dst-port 67 --egress --description DHCP67
openstack security group rule create ${ADMIN_SECURITY_GROUP_ID} --protocol udp --dst-port 68 --ingress --description DHCP68
openstack security group rule create ${ADMIN_SECURITY_GROUP_ID} --protocol udp --dst-port 69 --egress --description TFTP69
openstack security group rule create ${ADMIN_SECURITY_GROUP_ID} --protocol tcp --dst-port 6385 --egress --description Ironic6385
openstack security group rule create ${ADMIN_SECURITY_GROUP_ID} --protocol tcp --dst-port 9999 --ingress --description Ironic9999
openstack security group rule create ${ADMIN_SECURITY_GROUP_ID} --protocol tcp --dst-port 9999 --egress --description Ironic9999
openstack security group rule create ${ADMIN_SECURITY_GROUP_ID} --protocol tcp --dst-port 3260 --ingress --description IscsiDeploy3260
openstack security group rule create ${ADMIN_SECURITY_GROUP_ID} --protocol tcp --dst-port 80 --egress --description DirectDeploy80
openstack security group rule create ${ADMIN_SECURITY_GROUP_ID} --protocol tcp --dst-port 8088 --egress --description DirectDeploy8088
openstack security group rule create ${ADMIN_SECURITY_GROUP_ID} --protocol tcp --dst-port 443 --egress --description DirectDeploy443 
 
SERVICE_SECURITY_GROUP_ID=$(openstack security group list --project service -f value -c ID)
openstack security group rule create ${SERVICE_SECURITY_GROUP_ID} --protocol udp --dst-port 67 --egress --description DHCP67
openstack security group rule create ${SERVICE_SECURITY_GROUP_ID} --protocol udp --dst-port 68 --ingress --description DHCP68
openstack security group rule create ${SERVICE_SECURITY_GROUP_ID} --protocol udp --dst-port 69 --egress --description TFTP69
openstack security group rule create ${SERVICE_SECURITY_GROUP_ID} --protocol tcp --dst-port 6385 --egress --description Ironic6385
openstack security group rule create ${SERVICE_SECURITY_GROUP_ID} --protocol tcp --dst-port 9999 --ingress --description Ironic9999
openstack security group rule create ${SERVICE_SECURITY_GROUP_ID} --protocol tcp --dst-port 9999 --egress --description Ironic9999
openstack security group rule create ${SERVICE_SECURITY_GROUP_ID} --protocol tcp --dst-port 3260 --ingress --description IscsiDeploy3260
openstack security group rule create ${SERVICE_SECURITY_GROUP_ID} --protocol tcp --dst-port 80 --egress --description DirectDeploy80
openstack security group rule create ${SERVICE_SECURITY_GROUP_ID} --protocol tcp --dst-port 8088 --egress --description DirectDeploy8088
openstack security group rule create ${SERVICE_SECURITY_GROUP_ID} --protocol tcp --dst-port 443 --egress --description DirectDeploy443

  3. Create a tenant network.

    Copy
    Copied!
                
    
            
    openstack network create tenant-net
openstack subnet create --network tenant-net --subnet-range 192.0.3.0/24 --allocation-pool start=192.0.3.10,end=192.0.3.20 tenant-subnet

Bare-Metal Flavor

To create a bare metal flavor, run the following command:

Copy
Copied!
            

            
openstack flavor create --ram 1024 --disk 20 --vcpus 1 baremetal
openstack flavor set baremetal --property resources:CUSTOM_BAREMETAL=1
openstack flavor set baremetal --property resources:VCPU=0
openstack flavor set baremetal --property resources:MEMORY_MB=0
openstack flavor set baremetal --property resources:DISK_GB=0

  1. Create the baremetal.yaml file:

    Copy
    Copied!
                
    
            
    nodes:
  -
    name: node-1
    driver: ipmi
    driver_info: null
    ipmi_address: host-ilo
    ipmi_username: admin
    ipmi_password: admin
    resource_class: baremetal
    network_interface: neutron
    properties: null
    cpu_arch: x86_64
    local_gb: 419
    cpus: 24
    memory_mb: 131072
    capabilities: 'boot_option:local'
    ports:
      -
        address: '50:6b:4b:2f:0b:f4'
        pxe_enabled: true
        is_smartnic: true
        physical_network: datacentre
        local_link_connection:
          hostname: BF-BM
          port_id: pf0hpf

  2. Create a baremetal node:

    Copy
    Copied!
                
    
            
    $ openstack baremetal node create -f baremetal.yaml

  3. Specify the deployment kernel and deployment ramdisk on each node:

    Copy
    Copied!
                
    
            
    KERNEL_UUID=`openstack image show bm-deploy-kernel -f value -c id`
INITRAMFS_UUID=`openstack image show bm-deploy-ramdisk -f value -c id`
openstack baremetal node set NODE_UUID --driver-info deploy_kernel=KERNEL_UUID --driver-info deploy_ramdisk=INITRAMFS_UUID

  4. Set the node to available state:

    Copy
    Copied!
                
    
            
    $ ironic node-set-provision-state node-1 manage
$ ironic node-set-provision-state node-1 provide

    Boot the Bare Metal Instance

    To boot the bare metal instance, run the following command:

    Copy
    Copied!
                
    
            
    openstack server create --flavor baremetal --image bm_centos --nic net-id=tenant-net node-1

© Copyright 2023, NVIDIA. Last updated on May 23, 2023.