The Mellanox mediated devices deliver flexibility in allowing to create accelerated devices without SR-IOV on the Bluefield® system. These mediated devices support NIC and RDMA, and offer the same level of ASAP2 offloads as SR-IOV VFs. Mediates devices are supported using mlx5 sub-function acceleration technology.
To support sub-functions, PCIe BAR2 must be enabled. Run:
$ mlxconfig -d /dev/mst/mst41682_pciconf0 s PF_BAR2_SIZE=4 PF_BAR2_ENABLE=True
Cold reboot the BlueField host system so that the above settings can be applied on subsequent reboot sequences.
By default, the firmware allows for a large number of maximum mdev devices. You must set the maximum number of mediated devices to 2 or 4 devices. Run:
$ echo 4 > /sys/bus/pci/devices/0000:05:00.0/mdev_supported_types/mlx5_core-local/max_mdevs
Mediated devices are uniquely identified using UUID. To create one, run:
$ uuidgen $ echo 49d0e9ac-61b8-4c91-957e-6f6dbc42557d > /sys/bus/pci/devices/0000:05:00.0/mdev_supported_types/mlx5_core-local/create
By default, if the driver vfio_mdev is loaded, newly created mdev devices are bound to it. To make use of this newly created mdev device in order to create a netdevice and RDMA device, you must first unbind it from that driver. Run:
$ echo 49d0e9ac-61b8-4c91-957e-6f6dbc42557d > /sys/bus/mdev/drivers/vfio_mdev/unbind
Configure a MAC address for the mdev device. Run:
$ echo 00:11:22:33:44:55 > /sys/bus/mdev/devices/49d0e9ac-61b8-4c91-957e-6f6dbc42557d/devlink-compat-config/mac_addr
Query the representor netdevice of the mdev device. Run:
$ cat /sys/bus/mdev/devices/49d0e9ac-61b8-4c91-957e-6f6dbc42557d/devlink-compat-config/netdev
Bind the mediated device to mlx5_core driver. Run:
$ echo 49d0e9ac-61b8-4c91-957e-6f6dbc42557d > /sys/bus/mdev/drivers/mlx5_core/bind
When an mdev device is bound to the mlx5_core driver, its respective netdevice and/or RDMA device is also created.
To inspect the netdevice and RDMA device for the mdev, run:
$ ls /sys/bus/mdev/devices/49d0e9ac-61b8-4c91-957e-6f6dbc42557d/net/ $ ls /sys/bus/mdev/devices/49d0e9ac-61b8-4c91-957e-6f6dbc42557d/infiniband/