ESF Maintenance, Monitoring and Troubleshooting
To upgrade the MLAG cluster, the standby switch should be upgraded first, then (after reboot with the upgraded software) the slave will rejoin the MLAG cluster.
After that, the master can be upgraded.
When the master reboots with the upgraded software, the other standby node (which is running) becomes the master. After the old master reboots, it joins the cluster and then the configuration is set.
For a more detailed description of NVIDIA Onyx upgrade procedure, please refer to the following posts:
This section provides information and tools to monitor and debug the deployed fabric.
It is recommended to ensure that the below conditions are followed:
Both switches are part of the same management subnet (connected to the same switch or more but on the same subnet).
The management network is connected on mgmt0 port.
The mlag-port-channel number is identical in both switches (recommended but not obligatory).
The same switch version is installed on both switches.
The IPL link is in UP state. try to ping the other switch via the IPL ping.
Align the MLAG interface mode on both the server and the switch.
For example, if you select LACP mode on the MLAG interface (active), mode 4 should be configured on the bond interface.
Below are failure scenarios followed by monitoring and debug instructions.
The following scenarios are discussed:
IPL link Down
'Inactive Ports' and 'Active-Partial' Status on the “show mlag” command
Management Port is Down but IPL port is UP
MLAG Cluster issues
IPL issues
MLAG port issues
IPL link Down
The IPL link should be configured as port-channel with 2 or more ports, but in some scenarios both ports may be in “Down” state. In this case only the master switch will pass traffic.
If we run “show mlag” command when only one “mlag-port-channel” port is configured, we will get the following:
Master:
mti-mar-sx04 [my-new
-domain: master] (config) # show mlag
Admin status: Enabled
Operational status: Up
Reload-delay: 30
sec
Keepalive-interval: 1
sec
Upgrade-timeout: 60
min
System-mac: 00
:00
:5e:00
:01
:5d
MLAG Ports Configuration Summary:
Configured: 1
Disabled: 0
Enabled: 1
MLAG Ports Status Summary:
Inactive: 0
Active-partial: 0
Active-full: 1
MLAG IPLs Summary:
ID Group Vlan Operational Local Peer
Port-Channel Interface State IP address IP address
--------------------------------------------------------------------------
1
Po1 4000
Up 10.10
.10.2
10.10
.10.1
MLAG Members Summary:
System-id State Hostname
-------------------------------------
E4:1D:2D:37
:50
:88
Up <mti-mar-sx04>
E4:1D:2D:37
:54
:88
Up mti-mar-sx03
mti-mar-sx04 [my-new
-domain: master] (config) #
Standby:
mti-mar-sx03 [my-new
-domain: standby] (config) # show mlag
Admin status: Enabled
Operational status: Up
Reload-delay: 30
sec
Keepalive-interval: 1
sec
Upgrade-timeout: 60
min
System-mac: 00
:00
:5e:00
:01
:5d
MLAG Ports Configuration Summary:
Configured: 1
Disabled: 0
Enabled: 1
MLAG Ports Status Summary:
Inactive: 0
Active-partial: 0
Active-full: 1
MLAG IPLs Summary:
ID Group Vlan Operational Local Peer
Port-Channel Interface State IP address IP address
--------------------------------------------------------------------------
1
Po1 4000
Up 10.10
.10.1
10.10
.10.2
MLAG Members Summary:
System-id State Hostname
-------------------------------------
E4:1D:2D:37
:54
:88
Up <mti-mar-sx03>
E4:1D:2D:37
:50
:88
Up mti-mar-sx04
mti-mar-sx03 [my-new
-domain: standby] (config) #
When shutting down the IPL port on the master switch:
mti-mar-sx04 [my-new
-domain: master] (config) # interface
port-channel 1
shutdown
mti-mar-sx04 [my-new
-domain: master] (config) # show mlag
Admin status: Enabled
Operational status: Up
Reload-delay: 30
sec
Keepalive-interval: 1
sec
Upgrade-timeout: 60
min
System-mac: 00
:00
:5e:00
:01
:5d
MLAG Ports Configuration Summary:
Configured: 1
Disabled: 0
Enabled: 1
MLAG Ports Status Summary:
Inactive: 0
Active-partial: 0
Active-full: 1
MLAG IPLs Summary:
ID Group Vlan Operational Local Peer
Port-Channel Interface State IP address IP address
--------------------------------------------------------------------------
1
Po1 4000
Down 10.10
.10.2
10.10
.10.1
MLAG Members Summary:
System-id State Hostname
-------------------------------------
E4:1D:2D:37
:50
:88
Up <mti-mar-sx04>
E4:1D:2D:37
:54
:88
Down mti-mar-sx03
mti-mar-sx04 [my-new
-domain: master] (config) #
Standby switch:
mti-mar-sx03 [my-new
-domain: standby] (config) # show mlag
Admin status: Enabled
Operational status: Down
Reload-delay: 30
sec
Keepalive-interval: 1
sec
Upgrade-timeout: 60
min
System-mac: 00
:00
:5e:00
:01
:5d
MLAG Ports Configuration Summary:
Configured: 1
Disabled: 1
Enabled: 0
MLAG Ports Status Summary:
Inactive: 0
Active-partial: 0
Active-full: 1
MLAG IPLs Summary:
ID Group Vlan Operational Local Peer
Port-Channel Interface State IP address IP address
--------------------------------------------------------------------------
1
Po1 4000
Down 10.10
.10.1
10.10
.10.2
MLAG Members Summary:
System-id State Hostname
-------------------------------------
E4:1D:2D:37
:54
:88
Peering <mti-mar-sx03>
E4:1D:2D:37
:50
:88
Down mti-mar-sx04
mti-mar-sx03 [my-new
-domain: standby] (config) #
'Inactive Ports' and 'Active-Partial' Status on the “show mlag” command
By default, all ethernet ports are admin UP, while the mlag-port-channels are down, as in most cases the full network configuration is done first and then the mlag-port-channel is enabled. Make sure to enable the ports when creating mlag-port-channel and adding ethernet interface to it (either static or LACP).
Note: When one port is down, it doesn't mean that the whole mlag-port-channel is down.
MLAG Ports Status Summary:
Inactive - all ports in the mlag-port-channel are down (on both switches).
Active-partial - some ports are down (example below, on one switch)
Active-full - normal condition, all is good.
When one mlag-port-channel is down, we will see the following output:
mti-mar-sx03 [my-new
-domain: master] (config) # interface
mlag-port-channel 10
shutdown
mti-mar-sx03 [my-new
-domain: master] (config) # show mlag
Admin status: Enabled
Operational status: Up
Reload-delay: 30
sec
Keepalive-interval: 1
sec
Upgrade-timeout: 60
min
System-mac: 00
:00
:5e:00
:01
:5d
MLAG Ports Configuration Summary:Configured: 1
Disabled: 0
Enabled: 1
MLAG Ports Status Summary:Inactive: 0
Active-partial: 1
Active-full: 0
MLAG IPLs Summary:
ID Group Vlan Operational Local Peer
Port-Channel Interface State IP address IP address
--------------------------------------------------------------------------
1
Po1 4000
Up 10.10
.10.1
10.10
.10.2
MLAG Members Summary:
System-id State Hostname
-------------------------------------
E4:1D:2D:37
:54
:88
Up <mti-mar-sx03>E4:1D:2D:37
:50
:88
Up mti-mar-sx04
mti-mar-sx03 [my-new
-domain: master] (config) #
To enable it:
mti-mar-sx03 [my-new
-domain: master] (config) # interface
mlag-port-channel 10
no shutdown
mti-mar-sx03 [my-new
-domain: master] (config) # show mlag
Admin status: Enabled
Operational status: Up
Reload-delay: 30
sec
Keepalive-interval: 1
sec
Upgrade-timeout: 60
min
System-mac: 00
:00
:5e:00
:01
:5d
MLAG Ports Configuration Summary:
Configured: 1
Disabled: 0
Enabled: 1
MLAG Ports Status Summary:
Inactive: 0
Active-partial: 0
Active-full: 1
MLAG IPLs Summary:
ID Group Vlan Operational Local Peer
Port-Channel Interface State IP address IP address
--------------------------------------------------------------------------
1
Po1 4000
Up 10.10
.10.1
10.10
.10.2
MLAG Members Summary:
System-id State Hostname
-------------------------------------
E4:1D:2D:37
:54
:88
Up <mti-mar-sx03>
E4:1D:2D:37
:50
:88
Up mti-mar-sx04
mti-mar-sx03 [my-new
-domain: master] (config) #
Management Port is Down but IPL port is UP
When there is no ping between the two servers on mgmt0 (e.g. mgmt0 port is Down, or any management switch problem that blocks traffic between the switches on mgmt0) - both switches will pass traffic.
There is no mentioning of the second switch in the cluster.
The “ show mlag” and “ show mlag-vip” output will look like this:
mti-mar-sx04 [my-new
-domain: master] (config) # show mlag
Admin status: Enabled
Operational status: Up
Reload-delay: 30
sec
Keepalive-interval: 1
sec
Upgrade-timeout: 60
min
System-mac: 00
:00
:5e:00
:01
:5d
MLAG Ports Configuration Summary:
Configured: 1
Disabled: 0
Enabled: 1
MLAG Ports Status Summary:
Inactive: 0
Active-partial: 0
Active-full: 1
MLAG IPLs Summary:
ID Group Vlan Operational Local Peer
Port-Channel Interface State IP address IP address
--------------------------------------------------------------------------
1
Po1 4000
Up 10.10
.10.2
10.10
.10.1
MLAG Members Summary:
System-id State Hostname
-------------------------------------
E4:1D:2D:37
:50
:88
Up <mti-mar-sx04>
E4:1D:2D:37
:54
:88
Up -
mti-mar-sx04 [my-new
-domain: master] (config) #
mti-mar-sx04 [my-new
-domain: master] (config) # show mlag-vip
MLAG VIP
========
MLAG group name: my-new
-domain
MLAG VIP address: 10.20
.2.205
/24
Active nodes: 1
Hostname VIP-State IP Address
----------------------------------------------------
mti-mar-sx04 master 10.20
.2.54
mti-mar-sx04 [my-new
-domain: master] (config) #
MLAG Cluster Issues
After adding the two switches to the cluster, wait for a few seconds. One switch will become Master, while the other one will become the slave. When performing remove/add/cluster change operations, always wait for the switch to go to “standalone master” before continuing.
Run "show mlag-vip"
mti-mar-sx03 [my-mlag-vip-domain: master] (config) # show mlag-vip
MLAG VIP
========
MLAG group name: my-mlag-vip-domain
MLAG VIP address: 10.20
.2.205
/24
Active nodes: 2
Hostname VIP-State IP Address
----------------------------------------------------
mti-mar-sx03 master 10.20
.2.53
mti-mar-sx04 standby 10.20
.2.54
mti-mar-sx03 [my-new
-domain: master] (config) #
Verify that the two switches are in the cluster. The other MLAG switch must reflect the same information.
If one switch does not see this MLAG-Domain do the following:
Run "show ip route":
mti-mar-sx03 [my-mlag-vip-domain: master] (config) # show ip route
VRF Name: default
-----------------------------
Destination Mask Gateway Interface Source Distance/Metric
default
0.0
.0.0
10.20
.0.251
mgmt0 DHCP 0
/0
10.20
.0.0
255.255
.0.0
0.0
.0.0
mgmt0 direct 0
/0
10.10
.10.0
255.255
.255.0
0.0
.0.0
The management subnet must only point out of the MGMT port. inband management is acceptable. If there is a conflict, the MGMT Keep alive is sent out on the wrong port and not advertised to another switch.
In case the switch still does not see the cluster: The MGMT keep alive is broadcast to a well known multicast DNS group – 224.0.0.251. Check to see if both switches are advertising to this group. It is likely that the mgmt. port will see a lot of traffic. This output will need to be captured and analyzed.
mti-mar-sx03 [my-mlag-vip-domain: master] (config) # tcpdump -i mgmt0
tcpdump: verbose output suppressed, use -v or -vv for
full protocol decode
listening on mgmt0, link-type EN10MB (Ethernet), capture size 96
bytes
06
:42
:15.330780
IP mti-mar-sx03.mti.labs.mlnx.mdns > 224.0
.0.251
.mdns: 0
[2a] PTR (Cache flush)? _tcn_MLAG-DOMAIN._tcp.local. (117
)
This is a transmission from master to the multicast group. Before we have a master, both switches will see this frame, and both will transmit it. After the cluster is formed, only the master will transmit this. If this frame is not seen, the cluster will not form.
IPL issues
IPL Link needs to be up for MLAG peer ports and sync data to be available. The IPL VLAN is local to the MLAG switches and can be any number. VLAN 4000 or higher is typically used for control vlans and is recommended.
The “show mlag” command shows IPL link state and other valuable information.
The IPL link needs to be Up. Both switches must be in Up State in the “Member” summary. Peering or down are not a good state. Peering could be a transient state but should move to UP eventually.
mti-mar-sx03 [my-mlag-vip-domain: master] (config) # show mlag
Admin status: Enabled
Operational status: Up
Reload-delay: 30
sec
Keepalive-interval: 1
sec
Upgrade-timeout: 60
min
System-mac: 00
:00
:5e:00
:01
:5d << Both switches should show the same System MAC Address
MLAG Ports Configuration Summary:
Configured: 1
Disabled: 0
Enabled: 1
MLAG Ports Status Summary:
Inactive: 0
Active-partial: 0
Active-full: 1
MLAG IPLs Summary:
ID Group Vlan Operational Local Peer
Port-Channel Interface State IP address IP address
--------------------------------------------------------------------------
1
Po1 4000
Up 10.10
.10.1
10.10
.10.2
MLAG Members Summary:
System-id State Hostname
-------------------------------------
E4:1D:2D:37
:54
:88
Up <mti-mar-sx03>
E4:1D:2D:37
:50
:88
Up mti-mar-sx04
In case IPL is up and still member ports are not visible, try ping the remote IPL interface. Ping the local switch and then the MLAG Peer switch IPL IP address. If ping doesn’t go through use tcpdump to debug this case. In case link is up and ping is lossy, check for traffic on the IPL interface. During normal operation, IPL traffic is a few frames per second at the most. If you see a lot of traffic, it is likely an indication of a loop in the setup.
switch
(config) # tcpdump -i vlan4000
The other usual suspects are checking if both sides are set to static, or LACP. Check interface transceiver for matching serial numbers to identify cabling issues.
MLAG Port Issues
A healthy MLAG should show all ports as UP (P) and MLAG must be (U).
mti-mar-sx03 [my-mlag-vip-domain: master] (config) # show interface
mlag-port-channel summary
MLAG Port-Channel Flags: D-Down, U-Up
P-Partial UP, S - suspended by MLAG
Port Flags: D - Down, P - Up in port-channel (members)
S - Suspend in port-channel (members), I - Individual
Group
Port-Channel Type Local Ports Peer Ports
(D/U/P/S) (D/P/S/I) (D/P/S/I)
--------------------------------------------------------------------------------
1
Mpo1(U) LACP Eth1/10
(P) Eth1/10
(P)
mti-mar-sx03 [my-mlag-vip-domain: master] (config) #
“Partial” means that all ports are down on the MLAG-peer switch side. This could be a result of interface MLAG being shut on the remote side or mlag protocol shut on remote side.
mti-mar-sx03 [my-mlag-vip-domain: master] (config) # show interface
mlag-port-channel summary
MLAG Port-Channel Flags: D-Down, U-Up
P-Partial UP, S - suspended by MLAG
Port Flags: D - Down, P - Up in port-channel (members)
S - Suspend in port-channel (members), I - Individual
Group
Port-Channel Type Local Ports Peer Ports
(D/U/P/S) (D/P/S/I) (D/P/S/I)
--------------------------------------------------------------------------------
1
Mpo1(P) LACP Eth1/10
(P) Eth1/10
(D)
Peer ports not being visible means that ports in the MLAG-Peer switch are either not added in the MLAG or there are cluster issues .
mti-mar-sx03 [my-mlag-vip-domain: master] (config) # show interface
mlag-port-channel summary
MLAG Port-Channel Flags: D-Down, U-Up
P-Partial UP, S - suspended by MLAG
Port Flags: D - Down, P - Up in port-channel (members)
S - Suspend in port-channel (members), I - Individual
Group
Port-Channel Type Local Ports Peer Ports
(D/U/P/S) (D/P/S/I) (D/P/S/I)
--------------------------------------------------------------------------------
1
Mpo1(P) LACP Eth1/10
(P)
SX1012-B [MLAG-DOMAIN: master] (config) #
If the physical port shows (S) that could result from either receiving no PDUs from the remote side or by receiving a PDU that doesn’t match what is being received on other members of the MLAG port-channel
Check the LACP counters to see continuous increment of counters, both sent and receive must increment. One every second for fast retransmit and one every 30 seconds for slow retransmit.
mti-mar-sx03 [my-mlag-vip-domain: master] (config) # show lacp counters
LACPDUs Marker Marker Response LACPDUs
Port Sent Recv Sent Recv Sent Recv Illegal Unknown
-----------------------------------------------------------------------------
...
Mlag-port-channel: 1
------------------
1
/10
0
0
0
0
35
27
0
0
In case the lacp counters are incrementing and port is still down, then check the SID received on different port of the MLAG. They should match across all MLAG ports.
mti-mar-sx03 [my-mlag-vip-domain: master] (config) #show lacp interfaces neighbors
Flags:
A - Device is in Active mode
P - Device is in Passive mode
MLAG channel group 1
neighbors
Port 1
/10
----------
Partner System ID : e4:1d:2d:37
:48
:80
(This is the System-ID received on this
port from the remote switch
. It must match for
all ports connected to the same switch
)
Partner System priority : 32768
Flags : A
LACP Partner Port Priority : 32768
LACP Partner Oper Key : 13845
(LACP OPER KEY must match across all ports in the same MLAG port-channel)
LACP Partner Port State : 0xbc
Port State Flags Decode
------------------------
Activity : Active
Aggregation State : Aggregation, Sync, Collecting, Distributing,
To check the SID used by the NVIDIA switch use this command:
mti-mar-sx03 [my-mlag-vip-domain: master] (config) # show lacp interfaces mlag-port-channel 1
system-identifier
Priority: 32768
MAC: 00
:00
:5e:00
:01
:06
Check the lacp property across all ports in an MLAG:
mti-mar-sx03 [my-mlag-vip-domain: master] (config) # show lacp interfaces eth 1
/10
Port : 1
/10
-------------
Port State = Bundle
MLAG Channel Group : 1
Pseudo mlag-port-channel = Mpo1
LACP port-priority = 32768
LACP Rate = Slow
LACP Activity : Active
LACP Timeout : Short
Aggregation State : Aggregation, Sync, Collecting, Distributing,
LACP Port Admin Oper Port Port
Port State Priority Key Key Number State
-------------------------------------------------------------------
1
/7
Bundle 32768
29001
29001
0x7
0x0
(This is what we advertise to the remote switch
- the Admin and Oper keys must match across all ports in a port-channel)