Shared RQ Mode

When creating 1 send queue (SQ) and 1 receive queue (RQ), each representor consumes ~3MB memory per single channel. Scaling this to the desired 1024 representors (SFs and/or VFs) would require ~3GB worth of memory for single channel. A major chunk of the 3MB is contributed by RQ allocation (receive buffers and SKBs). Therefore, to make efficient use of memory, shared RQ mode is implemented so PF/VF/SF representors share receive queues owned by the uplink representor.

The feature is enabled by default. To disable it:

  1. Edit the field ALLOW_SHARED_RQ in /etc/mellanox/mlnx-bf.conf as follows:

    Copy
    Copied!
                

    ALLOW_SHARED_RQ="no"

  2. Restart the driver. Run:

    Copy
    Copied!
                

    /etc/init.d/openibd restart

To connect from the host to BlueField in shared RQ mode, please refer to section Verifying Connection from Host to BlueField.

Warning

PF/VF representor to PF/VF communication on the host is not possible.

The following behavior is observed in shared RQ mode:

  • It is expected to see a 0 in the rx_bytes and rx_packets and valid vport_rx_packets and vport_rx_bytes after running traffic. Example output:

    Copy
    Copied!
                

    # ethtool -S pf0hpf NIC statistics: rx_packets: 0 rx_bytes: 0 tx_packets: 66946 tx_bytes: 8786869 vport_rx_packets: 546093 vport_rx_bytes: 321100036 vport_tx_packets: 549449 vport_tx_bytes: 321679548

  • Ethtool usage – in this mode, it is not possible to change/set the ring or coalesce parameters for the RX side using ethtool. Changing channels also only affects the TX side.

© Copyright 2023, NVIDIA. Last updated on Feb 8, 2024.