Ethernet Interfaces
Ethernet interfaces have the following physical set of configurable parameters:
Admin state – enabling or disabling the interface
Flow control – admin state per direction (send or receive)
MTU (Maximum Transmission Unit) – 1500-9216 bytes
Speed – 1/10/40/56/100GbE (depending interface type and system)
Description – user defined string
Module-type – the type of the module plugged in the interface
To use 100GbE QSFP interfaces as 25/10GbE (via QSA adapter), the speed must be manually set with the command “speed 25000” or “speed 10000” respectively under the interface configuration mode.
The breakout cable is a unique NVIDIA capability, where a single physical quad-lane QSFPor double density QSFP (QSFP-DD) port can be divided into various types. QSFP can be divided into 2 dual-lane ports or 4 single-lane ports. QSFP-DD can be divided in 2 quard-lane ports or 4 dual-lane ports.It maximizes the flexibility of the end user to use NVIDIA switch with a combination of different interfaces according to the specific requirements of its network. Certain ports cannot be split at all, and there are ports that can be split into 2 ports only (for more information please refer to your Switch Hardware User Manual). Splitting a port changes the notation of that port from x/y to x/y/z with “x/y” indicating the previous notation of the port prior to the split and “z” indicating the number of the resulting sub-physical port (1,2 or 1,2,3,4). Each sub-physical port is then handled as an individual port. For example, splitting port 10 into 4 lanes gives the following new ports: 1/10/1, 1/10/2, 1/10/3, 1/10/4.
qsfp-split-4 operations result in blocking ports that are additional to the one being split. A set of hardware restrictions determine which of the ports can be split.
Specific ports can be split by using a QSFP 1X4 breakout cable to split one single-lane port into 4 lanes (4 SFP+ connectors). These 4 lanes then go one lane to each of the 4 SFP+ connectors.
Splitting the interface deletes all configuration on that interface.
When splitting an interface’s traffic into 4 data streams (split qsfp into four lanes), one of the other ports on the switch is disabled (unmapped). To see the exact splitting options available per system, refer to the cabling chapter in each specific system’s hardware user manual.
To see the exact splitting options available per system, refer to each specific system’s hardware user manual (Cabling chapter) located on the company website.
Breakout Cables Behavior on SN3800 Switch Systems
SN3800 switch systems currently supports only splitting a port to 2.
Changing the Module Type to a Split Mode
To split an interface:
Shut down all the ports related to the interface. Run:
• In case of qsfp-split-2, shut down the current interface only• In case of qsfp-split-4, shut down the current interface and the other interface according switch system’s specifications.switch
(config) #interface
ethernet1
/3
switch
(configinterface
ethernet1
/3
) # shutdownswitch
(configinterface
ethernet1
/3
) # exitswitch
(config) #interface
ethernet1
/4
switch
(configinterface
ethernet1
/4
) # shutdownSplit
Split the ports specifying the desired module type to be used (QSFP or QSFP-DD).Run:
switch
(configinterface
ethernet1
/3
) # module-type qsfp-split-4
The following warning will be displayed:
The following interfaces will be unmapped:
1
/3
1
/4
. Type “YES” when asked to confirm the split.The <ports> field in the warning refers to the affected ports from splitting port <inf> in the applied command.
WarningPlease beware that in some products splitting a port into a specific type prevents you from accessing the splittable port and an additional one. For example, splitting a port 3 into qsfp-4 on SN2700, makes ports 3 and 4 inaccessible.
This affects the following systems:
• SN2700 – makes one port inaccessible if split port into qsfp-4
Unsplitting a Split Port
Shut down all of the split ports. Run:
switch
(configinterface
ethernet1
/4
/4
) # shutdownswitch
(configinterface
ethernet1
/4
/4
) # exitswitch
(config) #interface
ethernet1
/4
/3
switch
(configinterface
ethernet1
/4
/3
) # shutdownswitch
(configinterface
ethernet1
/4
/3
) # exitswitch
(config) #interface
ethernet1
/4
/2
switch
(configinterface
ethernet1
/4
/2
) # shutdownswitch
(configinterface
ethernet1
/4
/2
) # exitswitch
(config) #interface
ethernet1
/4
/1
switch
(configinterface
ethernet1
/4
/1
) # shutdownFrom the first member of the split (1/4/1), change the module-type back to QSFP. Run:
switch
(configinterface
ethernet1
/4
/1
) # no module-typeWarningThe module-type can be changed only from the first member of the split and not from the interface which has been split.
The following warning will be displayed:
The following interfaces will be unmapped:
1
/4
/1
1
/4
/2
1
/4
/3
1
/4
/4
.Type “YES” when prompted with “Type 'YES' to confirm unsplit.”
NVIDIA offers proprietary speed of 56Gb/s per Ethernet interface.
To achieve 56GbE link speed, run the following on the desired interface:
switch
(config) # interface
ethernet 1
/1
switch
(config interface
ethernet 1
/1
) # speed 56G
For more information about this feature and its potential applications, please refer to the following community post:
NVIDIA Onyx offers the option of viewing the transceiver information of a module or cable connected to a specific interface. The information is a set of read-only parameters burned onto the EEPROM of the transceiver by the manufacture. The parameters include identifier (connector type), cable type, speed and additional inventory attributes.
To display transceiver information of a specific interface, run:
switch
(config) # show interfaces ethernet 1
/20
transceiver
Port 1
/20
state
identifier : QSFP+
cable/module type : Passive copper, unequalized
ethernet speed and type: 56GigE
vendor : Mellanox
cable length : 1m
part number : MC2207130-001
revision : A3
serial number : MT1238VS04936
The indicated cable length is rounded up to the nearest natural number.
For more information about this feature and its potential applications, please refer to the following community post:
NVIDIA switch systems offer high power transceiver (LR4) support in the following ports:
Transceiver |
Switch OPN |
Supported Ports |
||
Speed |
Protocol |
Power Consumption [W] |
||
40GbE |
LR4/ER4 |
3.5 |
SN2100/SN2410/SN2700 |
All ports |
100GbE |
3.5 |
SN2100/SN2410/SN2700 |
All ports |
|
100GbE |
4.5 |
SN2100 |
1, 2, 15, 16 |
|
SN2410 |
49, 50, 55, 56 |
|||
SN2700 |
1, 2, 31, 32 |
If a high power transceiver (e.g. LR4) is inserted to a port that does not support it, the link does not go up, and the following warning message is displayed: “Warning: High power transceiver is not supported” when the command “show interfaces ethernet” is run.
Forward Error Correction (FEC) mechanism adds extra data to the transmitted information. The receiving device uses this additional data to verify that the received data contains no errors. If the receiving side discovers errors within the received data it is able to correct some of these errors. The number or errors that can be corrected depends on the FEC algorithm and the amount of redundant data.
100GbE NVIDIA-to-NVIDIA Ethernet connections always enable standard Reed Solomon (RS) FEC on all cables.
If a NVIDIAsystem is connected to a 3rd party system, then FEC is only activated if the 3rd party requests it also.
FEC Modes on All Speeds
Speed |
FEC Mode |
200GbE |
KP4 (enhanced RS FEC) |
100/50/25GbE |
RS FEC |
40/10/1GbE |
No FEC |
The Port Recirculation feature allows the user to configure one of the ports as a recirculation port. When a user configures a physical Ethernet port as a recirculation port, the control of the port will move from the user to the operating system. The interface will no longer be available to the user, but rather be allocated by the operating system for other applications. For instance, on Spectrum-based systems, enabling What-Just-Happened buffer telemetry requires configuring one of the ports as a recirculation port. In this case, the operating system will use this port to get buffer dropped packets from all the other ports and present them to the user.