NVIDIA MLNX_EN Documentation v4.9-7.1.0.0 LTS

Interface

ConnectX®-3/ConnectX®-3 Pro ports can be individually configured to work as InfiniBand or Ethernet ports. By default both ConnectX ports are initialized as InfiniBand ports. If you wish to change the port type use the connectx_port_config script after the driver is loaded.
Running "/sbin/connectx_port_config -s" will show current port configuration for all ConnectX devices.
Port configuration is saved in the file: /etc/infiniband/connectx.conf. This saved configuration is restored at driver restart only if restarting via "/etc/init.d/openibd restart".
Possible port types are:

  • eth – Ethernet

  • ib – InfiniBand

  • auto - link sensing mode - detect port type based on the attached network type. If no link is detected, the driver retries link sensing every few seconds.

The port link type can be configured for each device in the system at a run time using the "/sbin/ connectx_port_config" script. This utility will prompt for the PCI device to be modified (if there is only one it will be selected automatically).
In the next stage, the user will be prompted for the desired mode for each port. The desired port configuration will then be set for the selected device.
This utility also has a non-interactive mode:

Copy
Copied!
            

/sbin/connectx_port_config \[\[-d|--device <PCI device ID>\] -c|--conf <port1,port2>\]"

Auto-Sensing enables the NIC to automatically sense the link type (InfiniBand or Ethernet) based on the link partner and load the appropriate driver stack (InfiniBand or Ethernet).

For example, if the first port is connected to an InfiniBand switch and the second to Ethernet switch, the NIC will automatically load the first switch as InfiniBand and the second as Ethernet.

Upon driver startup:

  1. Sense the adapter card's port type:
    If a valid cable or module is connected (QSFP, SFP+, or SFP with EEPROM in the cable/module):

  • Set the port type to the sensed link type (IB/Ethernet) Otherwise:

  • Set the port type as default (Ethernet)

During driver run time:

  • Sense a link every 3 seconds if no link is sensed/detected

  • If sensed, set the port type as sensed

Ports of ConnectX-4 adapter cards and above can be individually configured to work as InfiniBand or Ethernet ports. By default, both VPI ports are initialized as InfiniBand ports. If you wish to change the port type, use the mlxconfig script after the driver is loaded.
For further information on how to set the port type, please refer to the MFT User Manual.

Counters are used to provide information about how well an operating system, an application, a service, or a driver is performing. The counter data help determine system bottlenecks and fine-tune the system and application performance. The operating system, network, and devices provide counter data that an application can consume to provide users with a graphical view of how well the system is performing.
The counter index is a Queue Pair (QP) attribute given in the QP context. Multiple QPs may be associated with the same counter set. If multiple QPs share the same counter, the counter value will represent the cumulative total.

  • ConnectX®-3 supports 127 different counters which are allocated as follows:

    • 4 counters reserved for PF - 2 counters for each port

    • 2 counters reserved for VF - 1 counter for each port

    • All other counters if exist are allocated by demand

RoCE Counters

  • RoCE counters are available only through sysfs located under:

    • # /sys/class/infiniband/<device>/ports/*/counters/

    • # /sys/class/infiniband/<device>/ports/*/hw_counters/

For mlx4 port and RoCE counters, refer to the Understanding mlx4 Linux Counters Community post.
For mlx5 port and RoCE counters, refer to the Understanding mlx5 Linux Counters Community post.

SR-IOV Counters

  • Physical Function can also read Virtual Functions' port counters through sysfs located under:

    • For mlx4 drivers - # /sys/class/net/eth*/vf*/statistics/

    • For mlx5 drivers - # /sys/class/net/<interface_name>/device/sriov/<index>/stats/

ethtool Counters

The ethtool counters are counted in different places, according to which they are divided into groups. Each counters group may also have different counter types.

worddave635fed9c99097774044df72a47e9130.png

For the full list of supported ethtool counters, refer to the Understanding mlx5 ethtool Counters community post.

To avoid network interface renaming after boot or driver restart, set the desired constant interface name in the "/etc/udev/rules.d/70-persistent-net.rules" file.

  • Example for Ethernet interfaces:

    Copy
    Copied!
                

    PCI device 0x15b3:0x1003 (mlx4_core) SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?", ATTR{address}=="00:02:c9:fa:c3:50", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth", NAME="eth1" SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?", ATTR{address}=="00:02:c9:fa:c3:51", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth", NAME="eth2" SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?", ATTR{address}=="00:02:c9:e9:56:a1", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth", NAME="eth3" SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?", ATTR{address}=="00:02:c9:e9:56:a2", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth", NAME="eth4" 

  • Example for IPoIB interfaces:

    Copy
    Copied!
                

    SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{dev_id}=="0x0", ATTR{type}=="32", NAME="ib0" SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{dev_id}=="0x1", ATTR{type}=="32", NAME="ib1"

Once IRQs are allocated by the driver, they are named mlx5_comp<x>@pci:<pci_addr>. The IRQs corresponding to the channels in use are renamed to <interface>-<x>, while the rest maintain their default name.
The mlx5_core driver allocates all IRQs during loading time to support the maximum possible number of channels. Once the driver is up, no further IRQs are freed or allocated. Changing the number of working channels does not re-allocate or free the IRQs.

The following example demonstrates how reducing the number of channels affects the IRQs names.

Copy
Copied!
            

$ ethtool -l ens1 Channel parameters for ens1: Pre-set maximums: RX: 0 TX: 0 Other: 0 Combined: 12   Current hardware settings: RX: 0 TX: 0 Other: 0 Combined: 12   $ cat /proc/interrupts   98: 0 0 0 0 0 0 7935 0 0 0 0 0 IR-PCI-MSI-edge mlx5_async@pci:0000:81:00.0 99: 0 0 0 0 0 0 1 0 0 0 0 0 IR-PCI-MSI-edge ens1-0 100: 0 0 0 0 0 0 1 0 0 0 0 0 IR-PCI-MSI-edge ens1-1 101: 0 0 0 0 0 0 1 0 0 0 0 0 IR-PCI-MSI-edge ens1-2  102: 0 0 0 0 0 0 1 0 0 0 0 0 IR-PCI-MSI-edge ens1-3 103: 0 0 0 0 0 0 1 0 0 0 0 0 IR-PCI-MSI-edge ens1-4 104: 0 0 0 0 0 0 1 0 0 0 0 0 IR-PCI-MSI-edge ens1-5 105: 0 0 0 0 0 0 1 0 0 0 0 0 IR-PCI-MSI-edge ens1-6 106: 0 0 0 0 0 0 1 0 0 0 0 0 IR-PCI-MSI-edge ens1-7 107: 0 0 0 0 0 0 1 0 0 0 0 0 IR-PCI-MSI-edge ens1-8 108: 0 0 0 0 0 0 1 0 0 0 0 0 IR-PCI-MSI-edge ens1-9 109: 0 0 0 0 0 0 1 0 0 0 0 0 IR-PCI-MSI-edge ens1-10 110: 0 0 0 0 0 0 1 0 0 0 0 0 IR-PCI-MSI-edge ens1-11       $ ethtool -L ens1 combined 4  $ ethtool -l ens1 Channel parameters for ens1: … Current hardware settings: RX: 0 TX: 0 Other: 0 Combined: 4   $ cat /proc/interrupts 98: 0 0 0 0 0 0 8455 0 0 0 0 0 IR-PCI-MSI-edge mlx5_async@pci:0000:81:00.0 99: 0 0 0 0 0 0 1 2 0 0 0 0 IR-PCI-MSI-edge ens1-0 100: 0 0 0 0 0 0 1 0 2 0 0 0 IR-PCI-MSI-edge ens1-1 101: 0 0 0 0 0 0 1 0 0 2 0 0 IR-PCI-MSI-edge ens1-2 102: 0 0 0 0 0 0 1 0 0 0 2 0 IR-PCI-MSI-edge ens1-3 103: 0 0 0 0 0 0 1 0 0 0 0 1 IR-PCI-MSI-edge mlx5_comp4@pci:0000:81:00.0 104: 0 0 0 0 0 0 2 0 0 0 0 0 IR-PCI-MSI-edge mlx5_comp5@pci:0000:81:00.0 105: 0 0 0 0 0 0 1 1 0 0 0 0 IR-PCI-MSI-edge mlx5_comp6@pci:0000:81:00.0 106: 0 0 0 0 0 0 1 0 1 0 0 0 IR-PCI-MSI-edge mlx5_comp7@pci:0000:81:00.0 107: 0 0 0 0 0 0 1 0 0 1 0 0 IR-PCI-MSI-edge mlx5_comp8@pci:0000:81:00.0 108: 0 0 0 0 0 0 1 0 0 0 1 0 IR-PCI-MSI-edge mlx5_comp9@pci:0000:81:00.0 109: 0 0 0 0 0 0 1 0 0 0 0 1 IR-PCI-MSI-edge mlx5_comp10@pci:0000:81:00.0 110: 0 0 0 0 0 0 2 0 0 0 0 0 IR-PCI-MSI-edge mlx5_comp11@pci:0000:81:00.0

© Copyright 2023, NVIDIA. Last updated on Sep 5, 2023.