image image image image image

On This Page


Port Type Management/VPI Cards Configuration

Ports of ConnectX-4 adapter cards and above can be individually configured to work as InfiniBand or Ethernet ports. By default, both VPI ports are initialized as InfiniBand ports. If you wish to change the port type, use the mlxconfig script after the driver is loaded.
For further information on how to set the port type, please refer to the MFT User Manual (nvidia.com/en-us/networking/ → Products → Software → InfiniBand/VPI Software → MFT—Firmware Tools)



Counters

Counters are used to provide information about how well an operating system, an application, a service, or a driver is performing. The counter data help determine system bottlenecks and fine-tune the system and application performance. The operating system, network, and devices provide counter data that an application can consume to provide users with a graphical view of how well the system is performing.
The counter index is a Queue Pair (QP) attribute given in the QP context. Multiple QPs may be associated with the same counter set. If multiple QPs share the same counter, the counter value will represent the cumulative total.

RoCE Counters

  • RoCE counters are available only through sysfs located under:
    • # /sys/class/infiniband/<device>/ports/*/counters/
    • # /sys/class/infiniband/<device>/ports/*/hw_counters/

For mlx5 port and RoCE counters, refer to the Understanding mlx5 Linux Counters Community post.

SR-IOV Counters

Physical Function can also read Virtual Functions' port counters through sysfs located under # /sys/class/net/<interface_name>/device/sriov/<index>/stats/

ethtool Counters

The ethtool counters are counted in different places, according to which they are divided into groups. Each counters group may also have different counter types.


For the full list of supported ethtool counters, refer to the Understanding mlx5 ethtool Counters community post.

Persistent Naming 


To avoid network interface renaming after boot or driver restart, create a file names  "/etc/udev/rules.d/85-net-persistent-names.rules" with rules to set explicit interface names (the number, 85, must be more than 83).

  • Example for Ethernet interfaces: 

    #PCI device 15b3:1019 (mlx5_core)
    #NAME:="some name" , := is used to make sure that device name will be persistent.
    SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:02:c9:fa:c3:50", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME:="eth1"
    SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:02:c9:fa:c3:51", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME:="eth2"
  • Example for IPoIB interfaces: 

    1. ATTR{type}=="32" is IPoIB interfaces
    #2. Find address value for each IB interfaces:
    #cat /sys/class/net/ib0/address | head -1 | cut -d":" -f"13-"
    #24:8a:07:03:00:a5:29:38
    #Use above output to place into ATTR{address}.
    #Address match must start with ?* and only reference the last 8 bytes of the address
    SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{type}=="32", ATTR{address}=="?*24:8a:07:03:00:a5:29:38", NAME:="ib0"
    SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{type}=="32", ATTR{address}=="?*24:8a:07:03:00:a5:29:39", NAME:="ib1"

Interrupt Request (IRQ) Naming

Once IRQs are allocated by the driver, they are named mlx5_comp<x>@pci:<pci_addr>. The IRQs corresponding to the channels in use are renamed to <interface>-<x>, while the rest maintain their default name.
The mlx5_core driver allocates all IRQs during loading time to support the maximum possible number of channels. Once the driver is up, no further IRQs are freed or allocated. Changing the number of working channels does not re-allocate or free the IRQs.

The following example demonstrates how reducing the number of channels affects the IRQs names.

$ ethtool -l ens1
Channel parameters for ens1:
Pre-set maximums:
RX:             0
TX:             0
Other:          0
Combined:       12

Current hardware settings:
RX:             0
TX:             0
Other:          0
Combined:       12

$ cat /proc/interrupts

98:          0          0	      0          0         0          0      7935          0         0          0          0         0  IR-PCI-MSI-edge      mlx5_async@pci:0000:81:00.0
99:          0         0          0          0         0          0         1          0         0          0         0          0  IR-PCI-MSI-edge     ens1-0
100:         0         0          0          0         0          0         1          0         0          0         0          0  IR-PCI-MSI-edge     ens1-1
101:         0         0          0          0         0          0         1          0         0          0         0          0  IR-PCI-MSI-edge     ens1-2
 102:        0         0          0          0         0          0         1          0         0          0         0          0  IR-PCI-MSI-edge     ens1-3
103:         0         0          0          0         0          0         1          0         0          0         0          0  IR-PCI-MSI-edge     ens1-4
104:         0         0          0          0         0          0         1          0         0          0         0          0  IR-PCI-MSI-edge     ens1-5
105:         0         0          0          0         0          0         1          0         0          0         0          0  IR-PCI-MSI-edge     ens1-6
106:         0         0          0          0         0          0         1          0         0          0         0          0  IR-PCI-MSI-edge     ens1-7
107:         0         0          0          0         0          0         1          0         0          0         0          0  IR-PCI-MSI-edge     ens1-8
108:         0         0          0          0         0          0         1          0         0          0         0          0  IR-PCI-MSI-edge     ens1-9
109:         0         0          0          0         0          0         1          0         0          0         0          0  IR-PCI-MSI-edge     ens1-10
110:         0         0          0          0         0          0         1          0         0          0         0          0  IR-PCI-MSI-edge     ens1-11



$ ethtool -L ens1 combined 4 
$ ethtool -l ens1
Channel parameters for ens1:
…
Current hardware settings:
RX:             0
TX:             0
Other:          0
Combined:       4

$ cat /proc/interrupts
98:          0         0          0          0         0          0      8455          0         0          0         0          0  IR-PCI-MSI-edge     mlx5_async@pci:0000:81:00.0
99:          0         0          0          0         0          0         1          2         0          0         0          0  IR-PCI-MSI-edge     ens1-0
100:         0         0          0          0         0          0         1          0         2          0         0          0  IR-PCI-MSI-edge     ens1-1
101:         0         0          0          0         0          0         1          0         0          2         0          0  IR-PCI-MSI-edge     ens1-2
102:         0         0          0          0         0          0         1          0         0          0         2          0  IR-PCI-MSI-edge     ens1-3
103:         0         0          0          0         0          0         1          0         0          0         0          1  IR-PCI-MSI-edge     mlx5_comp4@pci:0000:81:00.0
104:         0         0          0          0         0          0         2          0         0          0         0          0  IR-PCI-MSI-edge     mlx5_comp5@pci:0000:81:00.0
105:         0         0          0          0         0          0         1          1         0          0         0          0  IR-PCI-MSI-edge     mlx5_comp6@pci:0000:81:00.0
106:         0         0          0          0         0          0         1          0         1          0         0          0  IR-PCI-MSI-edge     mlx5_comp7@pci:0000:81:00.0
107:         0         0          0          0         0          0         1          0         0          1         0          0  IR-PCI-MSI-edge     mlx5_comp8@pci:0000:81:00.0
108:         0         0          0          0         0          0         1          0         0          0         1          0  IR-PCI-MSI-edge     mlx5_comp9@pci:0000:81:00.0
109:         0         0          0          0         0          0         1          0         0          0         0          1  IR-PCI-MSI-edge     mlx5_comp10@pci:0000:81:00.0
110:         0         0          0          0         0          0         2          0         0          0         0          0  IR-PCI-MSI-edge     mlx5_comp11@pci:0000:81:00.0