InfiniBand Interface
MLNX-OS offers the option of viewing the transceiver information of a module or cable connected to a specific interface. The information is a set of read-only parameters burned onto the EEPROM of the transceiver by the manufacture. The parameters include identifier (connector type), cable type, speed and additional inventory attributes.
To display transceiver information of a specific interface, run:
switch (config) # show interfaces ib 1/36 transceiver
Slot 1 port 36 state
identifier : QSFP+
cable/module type : Passive copper, unequalized
infiniband speeds : SDR , DDR , QDR , FDR, HDR, NDR
vendor : Mellanox
cable length : 2m
part number : MC2207130-0A1
revision : A3
serial number : MT1324VS02215
The indicated cable length is rounded up to the nearest natural number.
NVIDIA switch systems offer high power transceiver (e.g. LR4) support on all ports of the Switch-IB ® family switch systems.
If a high power transceiver (e.g. LR4) is inserted to a port that does not support it, the link does not go up, and the following warning message is displayed: “Warning: High power transceiver is not supported” when the command “show interfaces ib” is run.
Forward Error Correction (FEC) mechanism adds extra data to the transmitted information. The receiving device uses this additional data to verify that the received data contains no errors. If the receiving side discovers errors within the received data it is able to correct some of these errors. The number or errors that can be corrected depends on the FEC algorithm.
Switch-IB® EDR (100Gb/s) NVIDIA-to-NVIDIA InfiniBand connections enable standard low-latency Reed Solomon (LL RS) FEC on active optical cables longer than 30 meters and passive copper cables longer than 2m.
This feature is available only for Quantum and Quantum-2 based switch systems (including modular systems).
To split a port in an externally managed 1U switch system, please refer to section “Using mlxconfig to Split a Port in a Remotely Managed Switch” in the MFT documentation.
In-band management is blocked by default on MLNX-OS switches, but can be enabled via the CLI command "system manage inband-ib". Once enabled, interaction between the different sources of commands is possible. That is, you may split a port via MAD and display the results on CLI, or split a port via CLI and query the result via MAD. Logically, it is similar to having two CLI users.
The break-out cable is a unique NVIDIA capability, where a single physical quad-lane QSFP port is divided into 2 dual-lane ports. It maximizes the flexibility of the end user to use the NVIDIA switch with a combination of dual-lane and quad-lane interfaces according to the specific requirements of its network. All system ports may be split into 2-lane ports and, on modular systems, only external ports may be split. Splitting a port changes the notation of that port from x/y to x/y/z with “x/y” indicating the previous notation of the port prior to the split and “z” indicating the number of the resulting single-lane port (1,2). Each sub-physical port is then handled as an individual port. For example, splitting port 5 into 2 lanes gives the following new ports: 1/5/1 & 1/5/2 and on modular systems, splitting port 5 of device 1 on leaf 1 will give the following ports: 1/1/5/1 and 1/1/5/2.
For Quantum-2 based systems, there will be an additional hierarchy—the cage level, which, for Quantum-2 systems, contains 2 physical ports.
As such, a representation of an interface for Quantum-2 systems will be <Asic/Cage/Port> (e.g., “interface ib 1/3/1 and interface ib 1/3/2” represents 2 ports 1 & 2 that are located within cage 3 of the system).
For split interfaces on Quantum-2 systems, the representation will be <Asic/Cage/Port/split > (e.g., “interface ib 1/3/1/1 and interface ib 1/3/1/2” represents split ports 1 & 2 that are split of port 1 located within cage 3 of the system).
For example, QM9700 system has 32 cages, but 64 ports are represented from “interface ib 1/1/1-1/32/2”.
Splitting the interface deletes all configuration on that interface.
In order to use this feature, the system’s profile must be configured to “ib split-ready” as described in section “Changing System Profile to Allow for Split-Ready Configuration” using the command “system profile”.
On modular systems, both managements should be configured with split-ready profile.
Changing System Profile to Allow for Split-Ready Configuration
If system does not have split-ready configuration, change its profile to allow for it:
Change the system’s profile to “ib split-ready”. Run:
switch(config) # system profile ib split-ready Warning - confirming will cause system reboot and all configuration will be deleted Type'yes'to confirm profile change: yesNoteOn modular systems, system’s profile need to be changed to “ib split-ready” for both managements simultaneously.
Verify system profile configuration. Run:
switch(config) # show system profile Profile: ib Number of SWIDs:1Adaptive Routing: yes Adaptive Routing Groups:1792Split Ready: yes IB Routing: no
Changing the Module Type to a Split Mode
To split an interface:
Shut down the interface.
Examples:
switch(config)#interfaceib1/4shutdown orswitch(config) #interfaceib1/4switch(configinterfaceib1/4) # shutdownSplit the ports as desired. Run:
switch(configinterfaceib1/4) # module-type qsfp-split-2New ports can be shown by the interfaces IB status command:
switch(config) # show interfaces ib status Interface Description IB Subnet Speed Current line rate Logical port state Physical port state --------- ----------- --------- ------- ----------------- ------------------ ------------------- IB1/1/1infiniband-defaultedr25.0Gbps Active LinkUp IB1/1/2infiniband-default- - Down Polling IB1/2infiniband-defaultedr100.0Gbps Active LinkUp IB1/3infiniband-default- - Down Polling IB1/4/1infiniband-default- - Down Polling IB1/4/2infiniband-default- - Down Polling IB1/5infiniband-default- - Down Polling IB1/6infiniband-default- - Down Polling IB1/7infiniband-default- - Down Polling IB1/8infiniband-default- - Down Polling
The above examples were executed on 1U systems, but are available also for modular systems.
Unsplitting a Split Port
To unsplit a split port:
Shut down all of the split ports. Run:
switch(config) #interfaceib1/4/1switch(configinterfaceib1/4/1) # shutdownswitch(configinterfaceib1/4/1) # exitswitch(config) #interfaceib1/4/2switch(configinterfaceib1/4/2) # shutdownswitch(configinterfaceib1/4/2) # exitFrom the first member of the split (1/4/1), change the module-type back to QSFP. Run:
switch(configinterfaceib1/4/1) # module-type qsfp
For both split and unsplit operations, a “force” option may be utilized to allow the operation without previously shutting down the ports.
The above examples were executed on 1U systems, but are available also for modular systems.