A scalable function (SF) is a lightweight function that has a parent PCIe function on which it is deployed. An mlx5 SF has its own function capabilities and its own resources. This means that an SF has its own dedicated queues (txq, rxq, cq, eq) which are neither shared nor stolen from the parent PCIe function.
No special support is needed from system BIOS to use SFs. SFs co-exist with PCIe SR-IOV virtual functions. SFs do not require enabling PCIe SR-IOV.
Scalable Function Configuration
The following procedure offers a guide on using scalable functions with upstream Linux kernel.
- Make sure your firmware version supports SFs (20.30.1004 and above).
Enable SF support in device. Run:
- Cold reboot the system for the configuration to take effect.
Mandatory Kernel Configuration
Support for Linux kernel mlx5 SFs must be enabled as it is disabled by default.
The following two Kconfig flags must be enabled.
Software Control and Commands
SFs use a 4-step process as follows:
SFs are managed using mlxdevm tool. It is located under directory
Display the physical (i.e. uplink) port of the PF. Run:
Add an SF. Run:
An added SF is still not usable for the end-user application. It can only be used after configuration and activation.
SF number ≥1000 is reserved for the virtio-net controller.
When an SF is added on the external controller (e.g. DPU) users must supply the controller number. In a single host DPU case, there is only one controller starting with controller number 1.
Example of adding an SF for PF0 of external controller 1:
Show the newly added devlink port by its port index or its representor device.
Set the MAC address of the SF. Run:
Configure OVS. Run:
Activate the SF. Run:
Activating the SF results in creating an auxiliary device and initiating driver load sequence for netdevice, RDMA, and VDPA devices. Once the operational state is marked as attached, a driver is attached to this SF and device loading begins.
An application interested in using the SF netdevice and rdma device must monitor the RDMA and netdevices either through udev monitor or poll the sysfs hierarchy of the SF's auxiliary device.
By default, SF is attached to the configuration driver mlx5_core.sf_cfg. Users must unbind an SF from the configuration and bind it to the mlx5_core.sf driver to make use of it. Run:
View the new state of the SF. Run:
View the auxiliary device of the SF. Run:
There can be hundreds of auxiliary SF devices on the auxiliary bus. Each SF's auxiliary device contains a unique sfnum and PCI information.
View the parent PCI device of the SF. Run:
View the devlink instance of the SF device. Run:
View the port and netdevice associated with the SF. Run:
View the RDMA device for the SF. Run:
Deactivate SF. Run:
Deactivating the SF triggers driver unload in the host system. Once SF is deactivated, its operational state changes to "detached". An orchestration system should poll for the operational state to be changed to "detached" before deleting the SF. This ensures a graceful hot-unplug.
Delete SF. Run:
Finally, once the state is "inactive" and the operational state is "detached" the user can safely delete the SF. For faster provisioning, a user can reconfigure and active the SF again without deletion.