Ethernet Interfaces

NVIDIA Onyx User Manual v3.10.4006

Ethernet interfaces have the following physical set of configurable parameters:

  • Admin state – enabling or disabling the interface

  • Flow control – admin state per direction (send or receive)

  • MTU (Maximum Transmission Unit) – 1500-9216 bytes

  • Speed – 1/10/40/56/100GbE (depending interface type and system)

  • Description – user defined string

  • Module-type – the type of the module plugged in the interface

Warning

To use 100GbE QSFP interfaces as 25/10GbE (via QSA adapter), the speed must be manually set with the command “speed 25000” or “speed 10000” respectively under the interface configuration mode.

The breakout cable is a unique NVIDIA capability, where a single physical quad-lane QSFPor double density QSFP (QSFP-DD) port can be divided into various types. QSFP can be divided into 2 dual-lane ports or 4 single-lane ports. QSFP-DD can be divided in 2 quard-lane ports or 4 dual-lane ports.It maximizes the flexibility of the end user to use NVIDIA switch with a combination of different interfaces according to the specific requirements of its network. Certain ports cannot be split at all, and there are ports that can be split into 2 ports only (for more information please refer to your Switch Hardware User Manual). Splitting a port changes the notation of that port from x/y to x/y/z with “x/y” indicating the previous notation of the port prior to the split and “z” indicating the number of the resulting sub-physical port (1,2 or 1,2,3,4). Each sub-physical port is then handled as an individual port. For example, splitting port 10 into 4 lanes gives the following new ports: 1/10/1, 1/10/2, 1/10/3, 1/10/4.

Break-Out_Cable-version-1-modificationdate-1709536831900-api-v2.PNG

qsfp-split-4 operations result in blocking ports that are additional to the one being split. A set of hardware restrictions determine which of the ports can be split.

Specific ports can be split by using a QSFP 1X4 breakout cable to split one single-lane port into 4 lanes (4 SFP+ connectors). These 4 lanes then go one lane to each of the 4 SFP+ connectors.

Warning

Splitting the interface deletes all configuration on that interface.

When splitting an interface’s traffic into 4 data streams (split qsfp into four lanes), one of the other ports on the switch is disabled (unmapped). To see the exact splitting options available per system, refer to the cabling chapter in each specific system’s hardware user manual.
To see the exact splitting options available per system, refer to each specific system’s hardware user manual (Cabling chapter) located on the company website.

Breakout Cables Behavior on SN3800 Switch Systems

SN3800 switch systems currently supports only splitting a port to 2.

Changing the Module Type to a Split Mode

To split an interface:

  1. Shut down all the ports related to the interface. Run:
    • In case of qsfp-split-2, shut down the current interface only• In case of qsfp-split-4, shut down the current interface and the other interface according switch system’s specifications.

    Copy
    Copied!
                

    switch (config) # interface ethernet 1/3 switch (config interface ethernet 1/3) # shutdown switch (config interface ethernet 1/3) # exit switch (config) # interface ethernet 1/4 switch (config interface ethernet 1/4) # shutdown

  2. Split
    Split the ports specifying the desired module type to be used (QSFP or QSFP-DD).

    Run:

    Copy
    Copied!
                

    switch (config interface ethernet 1/3) # module-type qsfp-split-4

  3. The following warning will be displayed:

    Copy
    Copied!
                

    The following interfaces will be unmapped: 1/3 1/4. Type “YES” when asked to confirm the split.

    The <ports> field in the warning refers to the affected ports from splitting port <inf> in the applied command.

    Warning

    Please beware that in some products splitting a port into a specific type prevents you from accessing the splittable port and an additional one. For example, splitting a port 3 into qsfp-4 on SN2700, makes ports 3 and 4 inaccessible.

    This affects the following systems:

    • SN2700 – makes one port inaccessible if split port into qsfp-4

Unsplitting a Split Port

  1. Shut down all of the split ports. Run:

    Copy
    Copied!
                

    switch (config interface ethernet 1/4/4) # shutdown switch (config interface ethernet 1/4/4) # exit switch (config) # interface ethernet 1/4/3 switch (config interface ethernet 1/4/3) # shutdown switch (config interface ethernet 1/4/3) # exit switch (config) # interface ethernet 1/4/2 switch (config interface ethernet 1/4/2) # shutdown switch (config interface ethernet 1/4/2) # exit switch (config) # interface ethernet 1/4/1 switch (config interface ethernet 1/4/1) # shutdown

  2. From the first member of the split (1/4/1), change the module-type back to QSFP. Run:

    Copy
    Copied!
                

    switch (config interface ethernet 1/4/1) # no module-type

    Warning

    The module-type can be changed only from the first member of the split and not from the interface which has been split.

    The following warning will be displayed:

    Copy
    Copied!
                

    The following interfaces will be unmapped: 1/4/1 1/4/2 1/4/3 1/4/4.

  3. Type “YES” when prompted with “Type 'YES' to confirm unsplit.”

NVIDIA offers proprietary speed of 56Gb/s per Ethernet interface.

To achieve 56GbE link speed, run the following on the desired interface:

Copy
Copied!
            

switch (config) # interface ethernet 1/1 switch (config interface ethernet 1/1) # speed 56G

For more information about this feature and its potential applications, please refer to the following community post:

NVIDIA Onyx offers the option of viewing the transceiver information of a module or cable connected to a specific interface. The information is a set of read-only parameters burned onto the EEPROM of the transceiver by the manufacture. The parameters include identifier (connector type), cable type, speed and additional inventory attributes.

To display transceiver information of a specific interface, run:

Copy
Copied!
            

switch (config) # show interfaces ethernet 1/20 transceiver Port 1/20 state identifier : QSFP+ cable/module type : Passive copper, unequalized ethernet speed and type: 56GigE vendor : Mellanox cable length : 1m part number : MC2207130-001 revision : A3 serial number : MT1238VS04936

Warning

The indicated cable length is rounded up to the nearest natural number.

For more information about this feature and its potential applications, please refer to the following community post:

NVIDIA switch systems offer high power transceiver (LR4) support in the following ports:

Transceiver

Switch OPN

Supported Ports

Speed

Protocol

Power Consumption [W]

40GbE

LR4/ER4

3.5

SN2100/SN2410/SN2700

All ports

100GbE

3.5

SN2100/SN2410/SN2700

All ports

100GbE

4.5

SN2100

1, 2, 15, 16

SN2410

49, 50, 55, 56

SN2700

1, 2, 31, 32

If a high power transceiver (e.g. LR4) is inserted to a port that does not support it, the link does not go up, and the following warning message is displayed: “Warning: High power transceiver is not supported” when the command “show interfaces ethernet” is run.

Forward Error Correction (FEC) mechanism adds extra data to the transmitted information. The receiving device uses this additional data to verify that the received data contains no errors. If the receiving side discovers errors within the received data it is able to correct some of these errors. The number or errors that can be corrected depends on the FEC algorithm and the amount of redundant data.

100GbE NVIDIA-to-NVIDIA Ethernet connections always enable standard Reed Solomon (RS) FEC on all cables.

If a NVIDIAsystem is connected to a 3rd party system, then FEC is only activated if the 3rd party requests it also.

FEC Modes on All Speeds

Speed

FEC Mode

200GbE

KP4 (enhanced RS FEC)

100/50/25GbE

RS FEC

40/10/1GbE

No FEC

The Port Recirculation feature allows the user to configure one of the ports as a recirculation port. When a user configures a physical Ethernet port as a recirculation port, the control of the port will move from the user to the operating system. The interface will no longer be available to the user, but rather be allocated by the operating system for other applications. For instance, on Spectrum-based systems, enabling What-Just-Happened buffer telemetry requires configuring one of the ports as a recirculation port. In this case, the operating system will use this port to get buffer dropped packets from all the other ports and present them to the user.

© Copyright 2023, NVIDIA. Last updated on Mar 5, 2024.