ESF Maintenance, Monitoring and Troubleshooting

NVIDIA Onyx User Manual v3.10.4006

To upgrade the MLAG cluster, the standby switch should be upgraded first, then (after reboot with the upgraded software) the slave will rejoin the MLAG cluster.

After that, the master can be upgraded.

When the master reboots with the upgraded software, the other standby node (which is running) becomes the master. After the old master reboots, it joins the cluster and then the configuration is set.

For a more detailed description of NVIDIA Onyx upgrade procedure, please refer to the following posts:

This section provides information and tools to monitor and debug the deployed fabric.

It is recommended to ensure that the below conditions are followed:

  1. Both switches are part of the same management subnet (connected to the same switch or more but on the same subnet).

  2. The management network is connected on mgmt0 port.

  3. The mlag-port-channel number is identical in both switches (recommended but not obligatory).

  4. The same switch version is installed on both switches.

  5. The IPL link is in UP state. try to ping the other switch via the IPL ping.

  6. Align the MLAG interface mode on both the server and the switch.
    For example, if you select LACP mode on the MLAG interface (active), mode 4 should be configured on the bond interface.

Below are failure scenarios followed by monitoring and debug instructions.

The following scenarios are discussed:

  • IPL link Down

  • 'Inactive Ports' and 'Active-Partial' Status on the “show mlag” command

  • Management Port is Down but IPL port is UP

  • MLAG Cluster issues

  • IPL issues

  • MLAG port issues

IPL link Down

The IPL link should be configured as port-channel with 2 or more ports, but in some scenarios both ports may be in “Down” state. In this case only the master switch will pass traffic.

If we run “show mlag” command when only one “mlag-port-channel” port is configured, we will get the following:

Master:

Copy
Copied!
            

mti-mar-sx04 [my-new-domain: master] (config) # show mlag Admin status: Enabled Operational status: Up Reload-delay: 30 sec Keepalive-interval: 1 sec Upgrade-timeout: 60 min System-mac: 00:00:5e:00:01:5d MLAG Ports Configuration Summary: Configured: 1 Disabled: 0 Enabled: 1 MLAG Ports Status Summary: Inactive: 0 Active-partial: 0 Active-full: 1 MLAG IPLs Summary: ID Group Vlan Operational Local Peer Port-Channel Interface State IP address IP address -------------------------------------------------------------------------- 1 Po1 4000 Up 10.10.10.2 10.10.10.1 MLAG Members Summary: System-id State Hostname ------------------------------------- E4:1D:2D:37:50:88 Up <mti-mar-sx04> E4:1D:2D:37:54:88 Up mti-mar-sx03 mti-mar-sx04 [my-new-domain: master] (config) #

Standby:

Copy
Copied!
            

mti-mar-sx03 [my-new-domain: standby] (config) # show mlag Admin status: Enabled Operational status: Up Reload-delay: 30 sec Keepalive-interval: 1 sec Upgrade-timeout: 60 min System-mac: 00:00:5e:00:01:5d MLAG Ports Configuration Summary: Configured: 1 Disabled: 0 Enabled: 1 MLAG Ports Status Summary: Inactive: 0 Active-partial: 0 Active-full: 1 MLAG IPLs Summary: ID Group Vlan Operational Local Peer Port-Channel Interface State IP address IP address -------------------------------------------------------------------------- 1 Po1 4000 Up 10.10.10.1 10.10.10.2 MLAG Members Summary: System-id State Hostname ------------------------------------- E4:1D:2D:37:54:88 Up <mti-mar-sx03> E4:1D:2D:37:50:88 Up mti-mar-sx04 mti-mar-sx03 [my-new-domain: standby] (config) #

When shutting down the IPL port on the master switch:

Copy
Copied!
            

mti-mar-sx04 [my-new-domain: master] (config) # interface port-channel 1 shutdown mti-mar-sx04 [my-new-domain: master] (config) # show mlag Admin status: Enabled Operational status: Up Reload-delay: 30 sec Keepalive-interval: 1 sec Upgrade-timeout: 60 min System-mac: 00:00:5e:00:01:5d MLAG Ports Configuration Summary: Configured: 1 Disabled: 0 Enabled: 1 MLAG Ports Status Summary: Inactive: 0 Active-partial: 0 Active-full: 1 MLAG IPLs Summary: ID Group Vlan Operational Local Peer Port-Channel Interface State IP address IP address -------------------------------------------------------------------------- 1 Po1 4000 Down 10.10.10.2 10.10.10.1 MLAG Members Summary: System-id State Hostname ------------------------------------- E4:1D:2D:37:50:88 Up <mti-mar-sx04> E4:1D:2D:37:54:88 Down mti-mar-sx03 mti-mar-sx04 [my-new-domain: master] (config) #

Standby switch:

Copy
Copied!
            

mti-mar-sx03 [my-new-domain: standby] (config) # show mlag Admin status: Enabled Operational status: Down Reload-delay: 30 sec Keepalive-interval: 1 sec Upgrade-timeout: 60 min System-mac: 00:00:5e:00:01:5d MLAG Ports Configuration Summary: Configured: 1 Disabled: 1 Enabled: 0 MLAG Ports Status Summary: Inactive: 0 Active-partial: 0 Active-full: 1 MLAG IPLs Summary: ID Group Vlan Operational Local Peer Port-Channel Interface State IP address IP address -------------------------------------------------------------------------- 1 Po1 4000 Down 10.10.10.1 10.10.10.2 MLAG Members Summary: System-id State Hostname ------------------------------------- E4:1D:2D:37:54:88 Peering <mti-mar-sx03> E4:1D:2D:37:50:88 Down mti-mar-sx04 mti-mar-sx03 [my-new-domain: standby] (config) #


'Inactive Ports' and 'Active-Partial' Status on the “show mlag” command

By default, all ethernet ports are admin UP, while the mlag-port-channels are down, as in most cases the full network configuration is done first and then the mlag-port-channel is enabled. Make sure to enable the ports when creating mlag-port-channel and adding ethernet interface to it (either static or LACP).

Note: When one port is down, it doesn't mean that the whole mlag-port-channel is down.

MLAG Ports Status Summary:

  • Inactive - all ports in the mlag-port-channel are down (on both switches).

  • Active-partial - some ports are down (example below, on one switch)

  • Active-full - normal condition, all is good.

When one mlag-port-channel is down, we will see the following output:

Copy
Copied!
            

mti-mar-sx03 [my-new-domain: master] (config) # interface mlag-port-channel 10 shutdown mti-mar-sx03 [my-new-domain: master] (config) # show mlag Admin status: Enabled Operational status: Up Reload-delay: 30 sec Keepalive-interval: 1 sec Upgrade-timeout: 60 min System-mac: 00:00:5e:00:01:5d MLAG Ports Configuration Summary:Configured: 1 Disabled: 0 Enabled: 1 MLAG Ports Status Summary:Inactive: 0 Active-partial: 1 Active-full: 0 MLAG IPLs Summary: ID Group Vlan Operational Local Peer Port-Channel Interface State IP address IP address -------------------------------------------------------------------------- 1 Po1 4000 Up 10.10.10.1 10.10.10.2 MLAG Members Summary: System-id State Hostname ------------------------------------- E4:1D:2D:37:54:88 Up <mti-mar-sx03>E4:1D:2D:37:50:88 Up mti-mar-sx04 mti-mar-sx03 [my-new-domain: master] (config) #

To enable it:

Copy
Copied!
            

mti-mar-sx03 [my-new-domain: master] (config) # interface mlag-port-channel 10 no shutdown mti-mar-sx03 [my-new-domain: master] (config) # show mlag Admin status: Enabled Operational status: Up Reload-delay: 30 sec Keepalive-interval: 1 sec Upgrade-timeout: 60 min System-mac: 00:00:5e:00:01:5d MLAG Ports Configuration Summary: Configured: 1 Disabled: 0 Enabled: 1 MLAG Ports Status Summary: Inactive: 0 Active-partial: 0 Active-full: 1 MLAG IPLs Summary: ID Group Vlan Operational Local Peer Port-Channel Interface State IP address IP address -------------------------------------------------------------------------- 1 Po1 4000 Up 10.10.10.1 10.10.10.2 MLAG Members Summary: System-id State Hostname ------------------------------------- E4:1D:2D:37:54:88 Up <mti-mar-sx03> E4:1D:2D:37:50:88 Up mti-mar-sx04 mti-mar-sx03 [my-new-domain: master] (config) #

Management Port is Down but IPL port is UP

When there is no ping between the two servers on mgmt0 (e.g. mgmt0 port is Down, or any management switch problem that blocks traffic between the switches on mgmt0) - both switches will pass traffic.

There is no mentioning of the second switch in the cluster.

The “ show mlag” and “ show mlag-vip” output will look like this:

Copy
Copied!
            

mti-mar-sx04 [my-new-domain: master] (config) # show mlag Admin status: Enabled Operational status: Up Reload-delay: 30 sec Keepalive-interval: 1 sec Upgrade-timeout: 60 min System-mac: 00:00:5e:00:01:5d MLAG Ports Configuration Summary: Configured: 1 Disabled: 0 Enabled: 1 MLAG Ports Status Summary: Inactive: 0 Active-partial: 0 Active-full: 1 MLAG IPLs Summary: ID Group Vlan Operational Local Peer Port-Channel Interface State IP address IP address -------------------------------------------------------------------------- 1 Po1 4000 Up 10.10.10.2 10.10.10.1 MLAG Members Summary: System-id State Hostname ------------------------------------- E4:1D:2D:37:50:88 Up <mti-mar-sx04> E4:1D:2D:37:54:88 Up - mti-mar-sx04 [my-new-domain: master] (config) # mti-mar-sx04 [my-new-domain: master] (config) # show mlag-vip MLAG VIP ======== MLAG group name: my-new-domain MLAG VIP address: 10.20.2.205/24 Active nodes: 1 Hostname VIP-State IP Address ---------------------------------------------------- mti-mar-sx04 master 10.20.2.54 mti-mar-sx04 [my-new-domain: master] (config) #


MLAG Cluster Issues

After adding the two switches to the cluster, wait for a few seconds. One switch will become Master, while the other one will become the slave. When performing remove/add/cluster change operations, always wait for the switch to go to “standalone master” before continuing.

Run "show mlag-vip"

Copy
Copied!
            

mti-mar-sx03 [my-mlag-vip-domain: master] (config) # show mlag-vip MLAG VIP ======== MLAG group name: my-mlag-vip-domain MLAG VIP address: 10.20.2.205/24 Active nodes: 2 Hostname VIP-State IP Address ---------------------------------------------------- mti-mar-sx03 master 10.20.2.53 mti-mar-sx04 standby 10.20.2.54 mti-mar-sx03 [my-new-domain: master] (config) #

Verify that the two switches are in the cluster. The other MLAG switch must reflect the same information.

If one switch does not see this MLAG-Domain do the following:

Run "show ip route":

Copy
Copied!
            

mti-mar-sx03 [my-mlag-vip-domain: master] (config) # show ip route VRF Name: default ----------------------------- Destination Mask Gateway Interface Source Distance/Metric default 0.0.0.0 10.20.0.251 mgmt0 DHCP 0/0 10.20.0.0 255.255.0.0 0.0.0.0 mgmt0 direct 0/0 10.10.10.0 255.255.255.0 0.0.0.0

The management subnet must only point out of the MGMT port. inband management is acceptable. If there is a conflict, the MGMT Keep alive is sent out on the wrong port and not advertised to another switch.

In case the switch still does not see the cluster: The MGMT keep alive is broadcast to a well known multicast DNS group – 224.0.0.251. Check to see if both switches are advertising to this group. It is likely that the mgmt. port will see a lot of traffic. This output will need to be captured and analyzed.

Copy
Copied!
            

mti-mar-sx03 [my-mlag-vip-domain: master] (config) # tcpdump -i mgmt0 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on mgmt0, link-type EN10MB (Ethernet), capture size 96 bytes 06:42:15.330780 IP mti-mar-sx03.mti.labs.mlnx.mdns > 224.0.0.251.mdns: 0 [2a] PTR (Cache flush)? _tcn_MLAG-DOMAIN._tcp.local. (117)

This is a transmission from master to the multicast group. Before we have a master, both switches will see this frame, and both will transmit it. After the cluster is formed, only the master will transmit this. If this frame is not seen, the cluster will not form.

IPL issues

IPL Link needs to be up for MLAG peer ports and sync data to be available. The IPL VLAN is local to the MLAG switches and can be any number. VLAN 4000 or higher is typically used for control vlans and is recommended.

The “show mlag” command shows IPL link state and other valuable information.

The IPL link needs to be Up. Both switches must be in Up State in the “Member” summary. Peering or down are not a good state. Peering could be a transient state but should move to UP eventually.

Copy
Copied!
            

mti-mar-sx03 [my-mlag-vip-domain: master] (config) # show mlag Admin status: Enabled Operational status: Up Reload-delay: 30 sec Keepalive-interval: 1 sec Upgrade-timeout: 60 min System-mac: 00:00:5e:00:01:5d << Both switches should show the same System MAC Address MLAG Ports Configuration Summary: Configured: 1 Disabled: 0 Enabled: 1 MLAG Ports Status Summary: Inactive: 0 Active-partial: 0 Active-full: 1 MLAG IPLs Summary: ID Group Vlan Operational Local Peer Port-Channel Interface State IP address IP address -------------------------------------------------------------------------- 1 Po1 4000 Up 10.10.10.1 10.10.10.2 MLAG Members Summary: System-id State Hostname ------------------------------------- E4:1D:2D:37:54:88 Up <mti-mar-sx03> E4:1D:2D:37:50:88 Up mti-mar-sx04

In case IPL is up and still member ports are not visible, try ping the remote IPL interface. Ping the local switch and then the MLAG Peer switch IPL IP address. If ping doesn’t go through use tcpdump to debug this case. In case link is up and ping is lossy, check for traffic on the IPL interface. During normal operation, IPL traffic is a few frames per second at the most. If you see a lot of traffic, it is likely an indication of a loop in the setup.

Copy
Copied!
            

switch (config) # tcpdump -i vlan4000

The other usual suspects are checking if both sides are set to static, or LACP. Check interface transceiver for matching serial numbers to identify cabling issues.

MLAG Port Issues

A healthy MLAG should show all ports as UP (P) and MLAG must be (U).

Copy
Copied!
            

mti-mar-sx03 [my-mlag-vip-domain: master] (config) # show interface mlag-port-channel summary MLAG Port-Channel Flags: D-Down, U-Up P-Partial UP, S - suspended by MLAG Port Flags: D - Down, P - Up in port-channel (members) S - Suspend in port-channel (members), I - Individual Group Port-Channel Type Local Ports Peer Ports (D/U/P/S) (D/P/S/I) (D/P/S/I) -------------------------------------------------------------------------------- 1 Mpo1(U) LACP Eth1/10(P) Eth1/10(P) mti-mar-sx03 [my-mlag-vip-domain: master] (config) #

“Partial” means that all ports are down on the MLAG-peer switch side. This could be a result of interface MLAG being shut on the remote side or mlag protocol shut on remote side.

Copy
Copied!
            

mti-mar-sx03 [my-mlag-vip-domain: master] (config) # show interface mlag-port-channel summary MLAG Port-Channel Flags: D-Down, U-Up P-Partial UP, S - suspended by MLAG Port Flags: D - Down, P - Up in port-channel (members) S - Suspend in port-channel (members), I - Individual Group Port-Channel Type Local Ports Peer Ports (D/U/P/S) (D/P/S/I) (D/P/S/I) -------------------------------------------------------------------------------- 1 Mpo1(P) LACP Eth1/10(P) Eth1/10(D)

Peer ports not being visible means that ports in the MLAG-Peer switch are either not added in the MLAG or there are cluster issues .

Copy
Copied!
            

mti-mar-sx03 [my-mlag-vip-domain: master] (config) # show interface mlag-port-channel summary MLAG Port-Channel Flags: D-Down, U-Up P-Partial UP, S - suspended by MLAG Port Flags: D - Down, P - Up in port-channel (members) S - Suspend in port-channel (members), I - Individual Group Port-Channel Type Local Ports Peer Ports (D/U/P/S) (D/P/S/I) (D/P/S/I) -------------------------------------------------------------------------------- 1 Mpo1(P) LACP Eth1/10(P) SX1012-B [MLAG-DOMAIN: master] (config) #

If the physical port shows (S) that could result from either receiving no PDUs from the remote side or by receiving a PDU that doesn’t match what is being received on other members of the MLAG port-channel

Check the LACP counters to see continuous increment of counters, both sent and receive must increment. One every second for fast retransmit and one every 30 seconds for slow retransmit.

Copy
Copied!
            

mti-mar-sx03 [my-mlag-vip-domain: master] (config) # show lacp counters LACPDUs Marker Marker Response LACPDUs Port Sent Recv Sent Recv Sent Recv Illegal Unknown ----------------------------------------------------------------------------- ... Mlag-port-channel: 1 ------------------ 1/10 0 0 0 0 35 27 0 0

In case the lacp counters are incrementing and port is still down, then check the SID received on different port of the MLAG. They should match across all MLAG ports.

Copy
Copied!
            

mti-mar-sx03 [my-mlag-vip-domain: master] (config) #show lacp interfaces neighbors Flags: A - Device is in Active mode P - Device is in Passive mode MLAG channel group 1 neighbors Port 1/10 ---------- Partner System ID : e4:1d:2d:37:48:80 (This is the System-ID received on this port from the remote switch. It must match for all ports connected to the same switch) Partner System priority : 32768 Flags : A LACP Partner Port Priority : 32768 LACP Partner Oper Key : 13845 (LACP OPER KEY must match across all ports in the same MLAG port-channel) LACP Partner Port State : 0xbc Port State Flags Decode ------------------------ Activity : Active Aggregation State : Aggregation, Sync, Collecting, Distributing,

To check the SID used by the NVIDIA switch use this command:

Copy
Copied!
            

mti-mar-sx03 [my-mlag-vip-domain: master] (config) # show lacp interfaces mlag-port-channel 1 system-identifier Priority: 32768 MAC: 00:00:5e:00:01:06

Check the lacp property across all ports in an MLAG:

Copy
Copied!
            

mti-mar-sx03 [my-mlag-vip-domain: master] (config) # show lacp interfaces eth 1/10 Port : 1/10 ------------- Port State = Bundle MLAG Channel Group : 1 Pseudo mlag-port-channel = Mpo1 LACP port-priority = 32768 LACP Rate = Slow LACP Activity : Active LACP Timeout : Short Aggregation State : Aggregation, Sync, Collecting, Distributing, LACP Port Admin Oper Port Port Port State Priority Key Key Number State ------------------------------------------------------------------- 1/7 Bundle 32768 29001 29001 0x7 0x0 (This is what we advertise to the remote switch- the Admin and Oper keys must match across all ports in a port-channel)


© Copyright 2023, NVIDIA. Last updated on Mar 5, 2024.