NVIDIA MLNX-OS User Manual v3.10.4006
NVIDIA MLNX-GW User Manual for NVIDIA Skyway Appliance v8.2.2200

Chassis Management

The chassis manager provides the user access to the following information:

Accessible Parameters

Description

switch temperatures

Displays system’s temperature

power supply voltages

Displays power supplies’ voltage levels

fan unit

Displays system fans’ status

power unit

Displays system power consumers

Flash memory

Displays information about system memory utilization.

Additionally, it monitors:

  • AC power to the PSUs

  • DC power out from the PSUs

  • Chassis failures

The system health monitor scans the system to decide whether or not the system is healthy. When the monitor discovers that one of the system's modules (leaf, spine, fan, or power supply) is in an unhealthy state or returned from an unhealthy state, it notifies the users through the following methods:

  • System logs—accessible to the user at any time as they are saved permanently on the system

  • Status LEDs—changed by the system health monitor when an error is found in the system and is resolved

  • Email/SNMP traps—notification on any error found in the system and resolved

Re-Notification on Errors

When the system is in an unhealthy state, the system health monitor notifies the user about the current unresolved issue every X seconds. The user can configure the re-notification gap by running the “health notif-cntr <counter>” command.

System Health Monitor Alerts Scenarios

System Health Monitor sends notification alerts in the following cases:

Alert Message

Scenario

Notification Indicator

Recovery Action

Recovery Message

<fan_name> speed is below minimal range

A chassis fan speed is below minimal threshold:

15% of maximum speed

Email, fan LED and system status LED set red, log alert, SNMP.

Check the fan and replace it if required

“<fan_name> has been restored to its normal state”

Fan <fan_number> speed in spine number <spine_number> is below minimal range

A spine fan speed is below minimal threshold:

30% of maximum speed

Email, fan LED and system status LED set red, log alert, SNMP

Check the fan and replace it if required

“Fan speed <fan_number> in spine number <spine_number> has been restored to its normal state”

<fan_name> is unresponsive

A chassis fan is not responsive on the switch system

Email, fan LED and system status LED set red, log alert, SNMP

Check fan connectivity and replace it if required

“<fan_name> has been restored to its normal state”

Fan <fan_number> in spine number <spine_number> is unresponsive

A spine fan is not responsive on the switch system

Email, fan LED and system status LED set red, log alert, SNMP

Check fan connectivity and replace it if required

“Fan <fan_number> in spine number <spine_number> has been restored to its normal state”

<fan_name> is not present

A chassis fan is missing

Email, fan LED and system status LED set red, log alert, SNMP

Insert a fan unit

“<fan_name> has been restored to its normal state”

Fan <fan_number> in spine number <spine_number> is not present.

A spine fan is missing

Email, fan LED and system status LED set red, log alert, SNMP

Insert a fan unit

“Fan <fan_number> in spine number <spine_number> has been restored to its normal state”

Insufficient number of working fans in the system

Insufficient number of working fans in the system

Email, fan LED and system status LED set red, log alert, SNMP

Plug in additional fans or change faulty fans

“The system currently has sufficient number of working fans”

Power Supply <ps_number> voltage is out of range

The power supply voltage is out of range.

Email, power supply LED and system status LED set red, log alert, SNMP

Check the power connection of the PS

“Power Supply <ps_number> voltage is in range”

Power supply <ps_number> temperature is too hot

A power supply unit temperature is higher than the maximum threshold of 70 Celsius on the switch system

Email, power supply LED and system status LED set red, log alert, SNMP

Check chassis fans connections. On switch systems, check system fan connections.

“Power supply <ps_number> temperature is back to normal”

Power Supply <number> is unresponsive

A power supply is malfunctioning or disconnected

Email, system status and power supply LED set red, log alert, SNMP

Connect power cable or replace malfunctioning PS

“Power supply has been removed” or “PS has been restored to its normal state”

Unit/leaf/spine <leaf/spine number> is unresponsive

A leaf/spine is not responsive

Email, system status LED set red, log alert, SNMP

Check leaf/spine connectivity and replace it if required

“Leaf/spine number <leaf/spine number> has been restored to its normal state”

Unit/leaf/spine voltage is out of range

One of the voltages on the switch system is below minimal threshold or higher than the maximum threshold - both thresholds are 15% of the expected voltage

Email, system status LED set red, log alert, SNMP

Check leaf connectivity

“Unit voltage is in range”

ASIC temperature is too hot

An ASIC unit temperature is higher than the maximum threshold of 105 Celsius on switch systems

Email, system status LED set red, log alert, SNMP

Check the fan’s system

“ASIC temperature is back to normal”

Power Supply Options

MLNX-OS offers power redundancy configurations and monitoring for modular switch systems. Modular switch systems have the following redundancy configuration modes:

  • “combined”—no power supply is reserved. The redundancy is not enabled.

  • “ps-redundant”—one power supply unit is redundant to the rest. The system can work with one less power supply unit.

  • “grid-redundant”—the power supplies are split into two logical power supply grids, first half of the PSUs belongs to grid A and the second half to grid B. The systems can work with only one grid. When using grid-redundancy mode the power budget is calculated according to the minimum power budget between the grids. This mode is available on CS75xx chassis systems. During switch initialization, or hot-plugging of switch components, MLNX-OS enables and/or disables switch components according to the available power budget.

MLNX-OS may send power alarms (via SNMP or email) as follow:

  • If the available budget is insufficient for all the system components an “insufficientPower” event is generated. In this mode several switch components may be disabled.

  • If the total power of the system is insufficient for redundancy, a “lowPower” event is generated

  • If a connected power supply provides below 1.6K Watts or grid-redundancy mode is configured and a power supply is connected to a 110V grid, then a “powerRedundancyMismatch event is generated, where grid redundancy can not be achieved in such configuration.

In case of an insufficient-power mode, the order in which the FRUs are turned ON is first spines (1,2,3...max) and then the leafs (1,2,3...max), while the order of the FRUs in case of turning them OFF is first the spines (max...3) and then the leafs (max...1). The management modules are not affected.

For the trap OID, please refer to the Mellanox-MIB file.

Warning

Power cycle is needed after changing power redundancy mode on a modular switch system.

Width Reduction Power Saving

Link width reduction (LWR) is a
NVIDIA

proprietary power saving feature to be utilized to economize the power usage of the fabric. LWR may be used to manually or automatically configure a certain connection between
NVIDIA switch

systems to lower the width of a link from 4X operation to 1X based on the traffic flow.

LWR is relevant only for
InfiniBand FDR

speeds in which the links are operational at a 4X width.

Warning

When “show interfaces” is used, a port’s speed appears unchanged even when only one lane is active.

LWR has three operating modes per interface:

  • Disabled—LWR does not operate and the link remains in 4X under all circumstances.

  • Automatic—the link automatically alternates between 4X and 1X based on traffic flow.

  • Force—a port is forced to operate in 1X mode lowering the throughput capability of the port. This mode should be chosen in cases where constant low throughput is expected on the port for a certain time period—after which the port should be configured to one of the other two modes, to allow higher throughput to pass through the port.

The following table describes LWR configuration behavior:

Switch-A Configuration

Switch-B Configuration

Behavior

Disable

Disable

LWR is disabled

Disable

Force

Transmission from Switch-B to Switch-A operates at 1X. On the opposite direction, LWR is disabled.

Disable

Auto

Depending on traffic flow, transmission from Switch-B to Switch-A may operate at 1X. On the opposite direction, LWR is disabled.

Auto

Force

Transmission from Switch-B to Switch-A operates at 1 lane. Transmission from Switch-A to Switch-B may operate at 1X depending on the traffic.

Auto

Auto

Width of the connection depends on the traffic flow

Force

Force

Connection between the switches operates at 1x

Managing Chassis Power

It is possible to power down or power up modules in a chassis by using the commands “power enable” and “no power enable”.

  1. Run the command “show power” to get a list of modules that are available to power up or down.
  2. To power down a desired module, run:
    Copy
    Copied!
                

    switch (config) # no power enable <module>

  3. To power up a desired module, run:
    Copy
    Copied!
                

    switch (config) # power enable <module>

  4. Using the ”show power” command it is possible to see the power consumption of the system and also the power consumption by power supply unit.
  1. Display module’s temperature. Run:

    Copy
    Copied!
                

    switch (config) # show temperature --------------------------------------------------------- Module Component Reg CurTemp Status (Celsius) --------------------------------------------------------- MGMT SIB T1 33.00 OK MGMT Board AMB temp T1 24.50 OK MGMT Ports AMB temp T1 27.00 OK MGMT CPU package Sensor T1 29.00 OK MGMT CPU Core Sensor T1 28.00 OK MGMT CPU Core Sensor T2 24.00 OK PS1 power-mon T1 22.00 OK PS2 power-mon T1 23.00 OK

  2. Display measured voltage levels of power supplies. Run:

    Copy
    Copied!
                

    switch (config) # show voltage -------------------------------------------------------------------------------------- Module Power Meter Reg Expected Actual Status High Low Voltage Voltage Range Range -------------------------------------------------------------------------------------- MGMT acdc-monitor1 DDR3 0.675V 0.68 0.67 OK 0.78 0.57 MGMT acdc-monitor1 CPU 0.9V 0.90 0.86 OK 1.03 0.77 MGMT acdc-monitor1 SYS 3.3V 3.30 3.36 OK 3.79 2.80 MGMT acdc-monitor1 CPU 1.8V 1.80 1.82 OK 2.07 1.53 MGMT acdc-monitor1 CPU/PCH 1.05V 1.05 1.06 OK 1.21 0.89 MGMT acdc-monitor1 CPU 1.05V 1.05 1.06 OK 1.21 0.89 MGMT acdc-monitor1 DDR3 1.35V 1.35 1.35 OK 1.55 1.15 MGMT acdc-monitor1 USB 5V 5.00 5.04 OK 5.75 4.25 MGMT acdc-monitor1 1.05V LAN 1.50 1.51 OK 1.72 1.27 MGMT ASICVoltMonitor1 Asic 1.2V 1.20 1.21 OK 1.38 1.02 MGMT ASICVoltMonitor1 Asic 3.3V 3.30 3.31 OK 3.79 2.80 MGMT ASICVoltMonitor2 Vcore SX 0.95 0.96 OK 1.09 0.81 MGMT ASICVoltMonitor2 Asic 1.8V 1.80 1.81 OK 2.07 1.53 MGMT acdc-monitor2 3.3V Switch IB 3.30 3.36 OK 3.79 2.80 PS1 power-mon vout 12V 12.00 12.07 OK 13.80 10.20

  3. Display the fan speed and status. Run:

    Copy
    Copied!
                

    switch (config) # show fan ----------------------------------------------------- Module Device Fan Speed Status (RPM) ----------------------------------------------------- FAN1 FAN F1 6297.00 OK FAN1 FAN F2 5421.00 OK FAN2 FAN F1 6355.00 OK FAN2 FAN F2 5378.00 OK FAN3 FAN F1 6183.00 OK FAN3 FAN F2 5421.00 OK FAN4 FAN F1 6268.00 OK FAN4 FAN F2 5399.00 OK PS1 FAN F1 10336.00 OK PS2 FAN - - NOT PRESENT

  4. Display the voltage current and status of each module in the system. Run:

    Copy
    Copied!
                

    switch (config) # show power consumers ------------------------------------------------------------------ Module Device Sensor Power Voltage Current Status [Watts] [Volts] [Amp] ------------------------------------------------------------------ PS1 power-mon input 39.94 12.07 3.31 OK MGMT acdc-monitor1 input 2.11 12.00 0.18 OK   Total power used : 42.05 Watts

The OS can access USB devices attached to switch systems. USB devices are automatically recognized and mounted upon insertion. To access a USB device for reading or writing a file, you need to provide the path to the file on the mounted USB device in the following format:

Copy
Copied!
            

scp://username:password@hostname/var/mnt/usb1/<file name>

While username and password are the admin username and password and hostname is the IP of the switch.

Examples:

  • To fetch an image from a USB device, run the command:

    Copy
    Copied!
                

    switch (config) # image fetch scp://username:password@hostname/var/mnt/usb1/<image filename>

  • To save log file (my-logfile) to a USB device under the name “test_logfile” using the command “logging files”, run:

    Copy
    Copied!
                

    switch (config) # logging files upload my-logfile scp://username:password@hostname/var/mnt/usb1/test_logfile

  • To safely remove the USB and to flush the cache, after writing (log files, for example) to a USB, use the “usb eject” command:

    Copy
    Copied!
                

    switch (config) # usb eject

The unit identification (UID) LED is a hardware feature used as a means of locating a specific switch system in a server room.

To activate the UID LED on a switch system, run:

Copy
Copied!
            

switch (config) # led MGMT uid on

To verify the LED status, run:

Copy
Copied!
            

switch (config) # show leds Module LED Status -------------------------------------------------------------------------- MGMT STATUS Green MGMT FAN1 Green MGMT FAN2 Green MGMT FAN3 Green MGMT FAN4 Green MGMT PS_STATUS Green MGMT PS1 Green MGMT PS2 Green MGMT UID Blue

To deactivate the UID LED on a switch system, run:

Copy
Copied!
            

switch (config) # led MGMT uid off

NVIDIA high end management modular switch systems support redundant management modules. Chassis HA reduces downtime as it assures continuity of the work even when a management module dies. Chassis HA management allows the systems administrator to associate a single IP address with the appliance. Connecting to that IP address allows the user to change and review the system’s chassis parameters regardless of the active management module.

Chassis High Availability Nodes Roles

Every node in the Chassis HA has one of the following roles/modes:

  • Master—the node that manages chassis configurations and services the chassis IP addresses

  • Slave—the node that replaces the Master node and takes over its responsibilities once the Master node is down

Warning

The master node is the only node that has access to chassis components such as temperature, inventory and firmware.

The CPU role of the current management node can be recognized by following one these methods:

  • Running the command “show chassis ha”

    Copy
    Copied!
                

    switch (config) # show chassis ha 2-node HA state: Box management IPv4: 10.7.146.44/24 Box management IPv6: fdfd:fdfd:7:145::1033:47fd/64 interface : mgmt0 local role : master local slot : 1 other state : not-present reset count : 0

  • Check the LEDs in the management modules as displayed in the figure below

  • Go to the WebUI →System →Modules page and see the information on the LEDs

Malfunctioned CPU Behavior

When a CPU in not responding to an internal communication with the other CPU, the non responding CPU will be reset by the other CPU. Each time a CPU resets, a counter is incremented. After 5 resets a CPU is considered malfunctioned and will be shut down.

To verify how many times a CPU is reset, run:

Copy
Copied!
            

switch [default: master] (config) # show chassis ha 2-node HA state: Box management IPv4: 10.7.146.44/24 Box management IPv6: fdfd:fdfd:7:145::1033:47fd/64 interface : mgmt0 local role : master local slot : 1 other state : not-present reset count : 0

To verify if a CPU has been shut down, either run:

Copy
Copied!
            

switch [default: master] (config) # show chassis ha 2-node HA state: Box management IPv4: 10.7.146.44/24 Box management IPv6: fdfd:fdfd:7:145::1033:47fd/64 interface : mgmt0 local role : master local slot : 1 other state : not-present reset count : 0

Or check the system page in the WebUI, the management figure will be grayed out.
To enable the malfunctioned CPU, first replace it and run “chassis ha reset other”.

Box IP Centralized Location

Box IP (BIP) centralized management infrastructure enables you to configure and monitor the system. The BIP continues to function even if one of the management blades dies. Box IP is defined by running the command “chassis ha bip <board IP address>”. The created BIP is used as the master IP’s alias. For example:

Copy
Copied!
            

switch [standalone: master] (config) # chassis ha bip 192.168.10.100 255.255.255.0

System Configuration

System configuration changes should be performed by the master using the BIP otherwise they are overridden by the master configuration.

Chassis HA is based on database replication enabling the entire master configuration to be replicated to the slave. Data such as chassis configuration is replicated. However, run time information such as time, logs, active user lists, is not copied. Additionally, node specific configuration information such as host name and IP address is not copied.

Warning

Chassis HA requires connectivity of both management modules (mgmt0, mgmt1) in the same broadcast domain.

The SM commands are only visible to the SM HA master in a modular system. This is node would display "master" in its CLI prompt.

Copy
Copied!
            

switch [standalone: master] (config) #

If the node shows "slave" or "unknown", the node is not the "master" and thus would not be able to use the IB SM commands.

"unknown" indicates that mgmt0 is not LinkUp and is not assigned a valid IPv4 address. On modular systems, the mgmt0 interface on all installed management modules must be:

  • LinkUp

  • With a valid IPv4 address

  • In the same L2 broadcast domain

Even if only one module is installed, it must have a mgmt0 interface that is LinkUp and with a valid IPv4 address.

Takeover Functionally

Management CPU functional takeover takes up to 20-30 seconds. However, when plugging in a module, you need to wait for approximately 3 minutes before making any other hardware change. During the takeover process, the Master LED status is differentiated by a color scheme. To verify the system’s status, run the “show chassis ha” command on both managements.

If the CPU malfunctions, the system resets it 5 times in an attempt to solve the issue. If the CPU is not activated after the reset, the system powers it off as well as its attached spine. Once the CPU is powered off, the user should replace the malfunctioned CPU module. To power on the CPU and the attached spine, plug the module in, log into the Master CPU and run the “chassis ha power enable other” command.

Warning

Although the LEDs are functional during the takeover, wait for approximately 3 minutes before making any other hardware change.

Master example:

Copy
Copied!
            

switch [default: master] (config) # show chassis ha 2-node HA state: Box management IPv4: 10.7.146.44/24 Box management IPv6: fdfd:fdfd:7:145::1033:47fd/64  interface : mgmt0 local role : master local slot : 1 other state : not-present reset count : 0

Slave example:

Copy
Copied!
            

switch [default: master] (config) # show chassis ha 2-node HA state: Box management IPv4: 10.7.146.44/24 Box management IPv6: fdfd:fdfd:7:145::1033:47fd/64 interface : mgmt0 local role : master local slot : 1 other state : not-present reset count : 0

Warning

Not following these instructions may result in some errors in the log. These errors may be safely ignored.

Rebooting 1U Switches

To reboot a 1U switch system, run:

Copy
Copied!
            

switch (config) # reload

Rebooting Modular Switches

NVIDIA high end management modular switch systems support redundant management modules. Chassis HA reduces downtime as it assures continuity of the work even when a management module dies. Chassis HA management allows the systems administrator to associate a single IP address with the appliance. Connecting to that IP address allows the user to change and review the system’s chassis parameters regardless of the active management module.

To reboot modular switches:

  1. Connect to the box IP (BIP). Please refer to “Box IP Centralized Location” for more information.

  2. Reboot the slave management. Run:

    Copy
    Copied!
                

    switch [default: master] (config) # chassis ha reset other

  3. Reboot the master management. Run:

    Copy
    Copied!
                

    switch [default: master] (config) # reload

Onyx supports viewing all active events on the system. The following events may be observed with the command “show system hardware events”.

Event Name

Description

Ethernet Family

Invalid Mac (SMAC=MC)

Source MAC is a multicast address

Invalid Mac (SMAC=DMAC)

Source MAC is same as destination mac address

Invalid Ethertype

Packet has an unknown Ethertype (0x05DC < ethertype < 0x600)

IP Routing Family

Ingress Router interface is disabled

Ingress packet has been dropped because incoming L3 interface is admin down

Mismatched IP (UC DIP over MC/BC Mac)

Packet MAC is multicast/broadcast but destination IP is unicast

Invalid IP (DIP=loopback)

Destination IP is loopback IP

(For IPv6: DIP==::1/128 or DIP==0:0:0:0:0:ffff:7f00:0/104

For IPv4: DIP==127.0.0.0/8)

Invalid IP (SIP=MC)

Source IP is multicast address

(For IPv6: SIP == FF00::/8

For IPv4: SIP == 224.0.0.0: 239.255.255.255 aka 224.0.0.0/4)

Invalid IP (SIP=unspecified)

Source IP is unspecified

Invalid IP (SIP=DIP)

Source IP is identical to destination IP

Mismatched MC Mac

Packet’s multicast MAC does not correspond to packet’s MC IP address

IPv6 neighbor not resolved

IPv6 neighbor not resolved

Invalid IPv6 (SIP=Link Local)

Source IP is link local (IPv6)

MC RPF check failure

Multicast RPF check failure

TTL expired

TTL value is zero

Egress Router interface is disabled

Egress packet has been dropped because outgoing L3 interface is admin/oper is down

IPv4 neighbor not resolved

Entry not found for destination

Tunnel Family

NVE Decap fragmentation error

Fragmentation error during decapsulation

© Copyright 2023, NVIDIA. Last updated on May 23, 2023.