InfiniBand Interface

MLNX-OS offers the option of viewing the transceiver information of a module or cable connected to a specific interface. The information is a set of read-only parameters burned onto the EEPROM of the transceiver by the manufacture. The parameters include identifier (connector type), cable type, speed and additional inventory attributes.

To display transceiver information of a specific interface, run:

Copy
Copied!
            

switch (config) # show interfaces ib 1/36 transceiver Slot 1 port 36 state identifier : QSFP+ cable/module type : Passive copper, unequalized infiniband speeds : SDR , DDR , QDR , FDR, HDR, NDR vendor : Mellanox cable length : 2m part number : MC2207130-0A1 revision : A3 serial number : MT1324VS02215

Warning

The indicated cable length is rounded up to the nearest natural number.

NVIDIA switch systems offer high power transceiver (e.g. LR4) support on all ports of the Switch-IB ® family switch systems.

If a high power transceiver (e.g. LR4) is inserted to a port that does not support it, the link does not go up, and the following warning message is displayed: “Warning: High power transceiver is not supported” when the command “show interfaces ib” is run.

Forward Error Correction (FEC) mechanism adds extra data to the transmitted information. The receiving device uses this additional data to verify that the received data contains no errors. If the receiving side discovers errors within the received data it is able to correct some of these errors. The number or errors that can be corrected depends on the FEC algorithm.

Switch-IB® EDR (100Gb/s) NVIDIA-to-NVIDIA InfiniBand connections enable standard low-latency Reed Solomon (LL RS) FEC on active optical cables longer than 30 meters and passive copper cables longer than 2m.

Warning

This feature is available only for Quantum and Quantum-2 based switch systems (including modular systems).

Warning

To split a port in an externally managed 1U switch system, please refer to section “Using mlxconfig to Split a Port in a Remotely Managed Switch” in the MFT documentation.

In-band management is blocked by default on MLNX-OS switches, but can be enabled via the CLI command "system manage inband-ib". Once enabled, interaction between the different sources of commands is possible. That is, you may split a port via MAD and display the results on CLI, or split a port via CLI and query the result via MAD. Logically, it is similar to having two CLI users.

The break-out cable is a unique NVIDIA capability, where a single physical quad-lane QSFP port is divided into 2 dual-lane ports. It maximizes the flexibility of the end user to use the NVIDIA switch with a combination of dual-lane and quad-lane interfaces according to the specific requirements of its network. All system ports may be split into 2-lane ports and, on modular systems, only external ports may be split. Splitting a port changes the notation of that port from x/y to x/y/z with “x/y” indicating the previous notation of the port prior to the split and “z” indicating the number of the resulting single-lane port (1,2). Each sub-physical port is then handled as an individual port. For example, splitting port 5 into 2 lanes gives the following new ports: 1/5/1 & 1/5/2 and on modular systems, splitting port 5 of device 1 on leaf 1 will give the following ports: 1/1/5/1 and 1/1/5/2.

For Quantum-2 based systems, there will be an additional hierarchy—the cage level, which, for Quantum-2 systems, contains 2 physical ports.

As such, a representation of an interface for Quantum-2 systems will be <Asic/Cage/Port> (e.g., “interface ib 1/3/1 and interface ib 1/3/2” represents 2 ports 1 & 2 that are located within cage 3 of the system).
For split interfaces on Quantum-2 systems, the representation will be <Asic/Cage/Port/split > (e.g., “interface ib 1/3/1/1 and interface ib 1/3/1/2” represents split ports 1 & 2 that are split of port 1 located within cage 3 of the system).
For example, QM9700 system has 32 cages, but 64 ports are represented from “interface ib 1/1/1-1/32/2”.

Warning

Splitting the interface deletes all configuration on that interface.

Warning

In order to use this feature, the system’s profile must be configured to “ib split-ready” as described in section “Changing System Profile to Allow for Split-Ready Configuration” using the command “system profile”.

On modular systems, both managements should be configured with split-ready profile.

Changing System Profile to Allow for Split-Ready Configuration

If system does not have split-ready configuration, change its profile to allow for it:

  1. Change the system’s profile to “ib split-ready”. Run:

    Copy
    Copied!
                

    switch (config) # system profile ib split-ready Warning - confirming will cause system reboot and all configuration will be deleted Type 'yes' to confirm profile change: yes

    Warning

    On modular systems, system’s profile need to be changed to “ib split-ready” for both managements simultaneously.

  2. Verify system profile configuration. Run:

    Copy
    Copied!
                

    switch (config) # show system profile Profile: ib Number of SWIDs: 1 Adaptive Routing: yes Adaptive Routing Groups: 1792 Split Ready: yes IB Routing: no

Changing the Module Type to a Split Mode

To split an interface:

  1. Shut down the interface.

    Examples:

    Copy
    Copied!
                

    switch (config)# interface ib 1/4 shutdown     or     switch (config) # interface ib 1/4 switch (config interface ib 1/4) # shutdown

  2. Split the ports as desired. Run:

    Copy
    Copied!
                

    switch (config interface ib 1/4) # module-type qsfp-split-2

  3. New ports can be shown by the interfaces IB status command:

    Copy
    Copied!
                

    switch (config) # show interfaces ib status Interface Description IB Subnet Speed Current line rate Logical port state Physical port state --------- ----------- --------- ------- ----------------- ------------------ ------------------- IB1/1/1 infiniband-default edr 25.0 Gbps Active LinkUp IB1/1/2 infiniband-default - - Down Polling IB1/2 infiniband-default edr 100.0 Gbps Active LinkUp IB1/3 infiniband-default - - Down Polling IB1/4/1 infiniband-default - - Down Polling IB1/4/2 infiniband-default - - Down Polling IB1/5 infiniband-default - - Down Polling IB1/6 infiniband-default - - Down Polling IB1/7 infiniband-default - - Down Polling IB1/8 infiniband-default - - Down Polling

Warning

The above examples were executed on 1U systems, but are available also for modular systems.

Unsplitting a Split Port

To unsplit a split port:

  1. Shut down all of the split ports. Run:

    Copy
    Copied!
                

    switch (config) # interface ib 1/4/1 switch (config interface ib 1/4/1) # shutdown switch (config interface ib 1/4/1) # exit switch (config) # interface ib 1/4/2 switch (config interface ib 1/4/2) # shutdown switch (config interface ib 1/4/2) # exit

  2. From the first member of the split (1/4/1), change the module-type back to QSFP. Run:

    Copy
    Copied!
                

    switch (config interface ib 1/4/1) # module-type qsfp

Warning

For both split and unsplit operations, a “force” option may be utilized to allow the operation without previously shutting down the ports.

Warning

The above examples were executed on 1U systems, but are available also for modular systems.


© Copyright 2023, NVIDIA. Last updated on Nov 9, 2023.