Mediated Devices

The Mellanox mediated devices deliver flexibility in allowing to create accelerated devices without SR-IOV on the Bluefield® system. These mediated devices support NIC and RDMA, and offer the same level of ASAP² offloads as SR-IOV VFs. Mediates devices are supported using mlx5 sub-function acceleration technology.

1. To support sub-functions, PCIe BAR2 must be enabled. Run:

   Copy

   Copied!

               
   
               
   $ mlxconfig -d /dev/mst/mst41682_pciconf0 s PF_BAR2_SIZE=4 PF_BAR2_ENABLE=True
           

   Cold reboot the BlueField host system so that the above settings can be applied on subsequent reboot sequences.

2. By default, the firmware allows for a large number of maximum mdev devices. You must set the maximum number of mediated devices to 2 or 4 devices. Run:

   Copy

   Copied!

               
   
               
   $ echo 4 > /sys/bus/pci/devices/0000:05:00.0/mdev_supported_types/mlx5_core-local/max_mdevs
           

3. Mediated devices are uniquely identified using UUID. To create one, run:

   Copy

   Copied!

               
   
               
   $ uuidgen
   $ echo 49d0e9ac-61b8-4c91-957e-6f6dbc42557d > /sys/bus/pci/devices/0000:05:00.0/mdev_supported_types/mlx5_core-local/create
           

4. By default, if the driver vfio_mdev is loaded, newly created mdev devices are bound to it. To make use of this newly created mdev device in order to create a netdevice and RDMA device, you must first unbind it from that driver. Run:

   Copy

   Copied!

               
   
               
   $ echo 49d0e9ac-61b8-4c91-957e-6f6dbc42557d > /sys/bus/mdev/drivers/vfio_mdev/unbind
           

5. Configure a MAC address for the mdev device. Run:

   Copy

   Copied!

               
   
               
   $ echo 00:11:22:33:44:55 > /sys/bus/mdev/devices/49d0e9ac-61b8-4c91-957e-6f6dbc42557d/devlink-compat-config/mac_addr
           

6. Query the representor netdevice of the mdev device. Run:

   Copy

   Copied!

               
   
               
   $ cat /sys/bus/mdev/devices/49d0e9ac-61b8-4c91-957e-6f6dbc42557d/devlink-compat-config/netdev
           

7. Bind the mediated device to mlx5_core driver. Run:

   Copy

   Copied!

               
   
               
   $ echo 49d0e9ac-61b8-4c91-957e-6f6dbc42557d > /sys/bus/mdev/drivers/mlx5_core/bind
           

   When an mdev device is bound to the mlx5_core driver, its respective netdevice and/or RDMA device is also created.

8. To inspect the netdevice and RDMA device for the mdev, run:

   Copy

   Copied!

               
   
               
   $ ls /sys/bus/mdev/devices/49d0e9ac-61b8-4c91-957e-6f6dbc42557d/net/
   $ ls /sys/bus/mdev/devices/49d0e9ac-61b8-4c91-957e-6f6dbc42557d/infiniband/