Chassis Management

The chassis manager provides the user access to the following information:

Additionally, it monitors:

• AC power to the PSUs

• DC power out from the PSUs

• Chassis failures

The system health monitor scans the system to decide whether or not the system is healthy. When the monitor discovers that one of the system's modules (leaf, spine, fan, or power supply) is in an unhealthy state or returned from an unhealthy state, it notifies the users through the following methods:

• System logs—accessible to the user at any time as they are saved permanently on the system

• Status LEDs—changed by the system health monitor when an error is found in the system and is resolved

• Email/SNMP traps—notification on any error found in the system and resolved

Re-Notification on Errors

When the system is in an unhealthy state, the system health monitor notifies the user about the current unresolved issue every X seconds. The user can configure the re-notification gap by running the “health notif-cntr <counter>” command.

System Health Monitor Alerts Scenarios

System Health Monitor sends notification alerts in the following cases:

Power Supply Options

MLNX-OS offers power redundancy configurations and monitoring for modular switch systems. Modular switch systems have the following redundancy configuration modes:

• “combined”—no power supply is reserved. The redundancy is not enabled.

• “ps-redundant”—one power supply unit is redundant to the rest. The system can work with one less power supply unit.

• “grid-redundant”—the power supplies are split into two logical power supply grids, first half of the PSUs belongs to grid A and the second half to grid B. The systems can work with only one grid. When using grid-redundancy mode the power budget is calculated according to the minimum power budget between the grids. This mode is available on CS75xx chassis systems. During switch initialization, or hot-plugging of switch components, MLNX-OS enables and/or disables switch components according to the available power budget.

MLNX-OS may send power alarms (via SNMP or email) as follow:

• If the available budget is insufficient for all the system components an “insufficientPower” event is generated. In this mode several switch components may be disabled.

• If the total power of the system is insufficient for redundancy, a “lowPower” event is generated

• If a connected power supply provides below 1.6K Watts or grid-redundancy mode is configured and a power supply is connected to a 110V grid, then a “powerRedundancyMismatch” event is generated, where grid redundancy can not be achieved in such configuration.

In case of an insufficient-power mode, the order in which the FRUs are turned ON is first spines (1,2,3...max) and then the leafs (1,2,3...max), while the order of the FRUs in case of turning them OFF is first the spines (max...3) and then the leafs (max...1). The management modules are not affected.

For the trap OID, please refer to the Mellanox-MIB file.

Width Reduction Power Saving

Link width reduction (LWR) is a
NVIDIA

proprietary power saving feature to be utilized to economize the power usage of the fabric. LWR may be used to manually or automatically configure a certain connection between
NVIDIA switch

systems to lower the width of a link from 4X operation to 1X based on the traffic flow.

LWR is relevant only for
InfiniBand FDR

speeds in which the links are operational at a 4X width.

LWR has three operating modes per interface:

• Disabled—LWR does not operate and the link remains in 4X under all circumstances.

• Automatic—the link automatically alternates between 4X and 1X based on traffic flow.

• Force—a port is forced to operate in 1X mode lowering the throughput capability of the port. This mode should be chosen in cases where constant low throughput is expected on the port for a certain time period—after which the port should be configured to one of the other two modes, to allow higher throughput to pass through the port.

The following table describes LWR configuration behavior:

Managing Chassis Power

It is possible to power down or power up modules in a chassis by using the commands “power enable” and “no power enable”.

1. Run the command “show power” to get a list of modules that are available to power up or down.

2. To power down a desired module, run:

   Copy

   Copied!

               
   
               
   switch (config) # no power enable <module>
           

3. To power up a desired module, run:

   Copy

   Copied!

               
   
               
   switch (config) # power enable <module>
           

4. Using the ”show power” command it is possible to see the power consumption of the system and also the power consumption by power supply unit.

1. Display module’s temperature. Run:

   Copy

   Copied!

               
   
               
   switch (config) # show temperature 
   ---------------------------------------------------------
   Module      Component              Reg  CurTemp    Status
                                           (Celsius) 
   ---------------------------------------------------------
   MGMT        SIB                    T1   33.00      OK    
   MGMT        Board AMB temp         T1   24.50      OK    
   MGMT        Ports AMB temp         T1   27.00      OK    
   MGMT        CPU package Sensor     T1   29.00      OK    
   MGMT        CPU Core Sensor        T1   28.00      OK    
   MGMT        CPU Core Sensor        T2   24.00      OK    
   PS1         power-mon              T1   22.00      OK    
   PS2         power-mon              T1   23.00      OK 
           

2. Display measured voltage levels of power supplies. Run:

   Copy

   Copied!

               
   
               
   switch (config) # show voltage
   --------------------------------------------------------------------------------------
   Module   Power Meter          Reg              Expected  Actual   Status  High   Low
                                                  Voltage   Voltage          Range  Range
   --------------------------------------------------------------------------------------
   MGMT     acdc-monitor1        DDR3 0.675V      0.68      0.67     OK      0.78   0.57
   MGMT     acdc-monitor1        CPU 0.9V         0.90      0.86     OK      1.03   0.77
   MGMT     acdc-monitor1        SYS 3.3V         3.30      3.36     OK      3.79   2.80
   MGMT     acdc-monitor1        CPU 1.8V         1.80      1.82     OK      2.07   1.53
   MGMT     acdc-monitor1        CPU/PCH 1.05V    1.05      1.06     OK      1.21   0.89
   MGMT     acdc-monitor1        CPU 1.05V        1.05      1.06     OK      1.21   0.89
   MGMT     acdc-monitor1        DDR3 1.35V       1.35      1.35     OK      1.55   1.15
   MGMT     acdc-monitor1        USB 5V           5.00      5.04     OK      5.75   4.25
   MGMT     acdc-monitor1        1.05V LAN        1.50      1.51     OK      1.72   1.27
   MGMT     ASICVoltMonitor1     Asic 1.2V        1.20      1.21     OK      1.38   1.02
   MGMT     ASICVoltMonitor1     Asic 3.3V        3.30      3.31     OK      3.79   2.80
   MGMT     ASICVoltMonitor2     Vcore SX         0.95      0.96     OK      1.09   0.81
   MGMT     ASICVoltMonitor2     Asic 1.8V        1.80      1.81     OK      2.07   1.53
   MGMT     acdc-monitor2        3.3V Switch IB   3.30      3.36     OK      3.79   2.80
   PS1      power-mon            vout 12V         12.00     12.07    OK      13.80  10.20
           

3. Display the fan speed and status. Run:

   Copy

   Copied!

               
   
               
   switch (config) # show fan
   -----------------------------------------------------
   Module          Device          Fan  Speed     Status
                                        (RPM)
   -----------------------------------------------------
   FAN1            FAN             F1   6297.00   OK
   FAN1            FAN             F2   5421.00   OK
   FAN2            FAN             F1   6355.00   OK
   FAN2            FAN             F2   5378.00   OK
   FAN3            FAN             F1   6183.00   OK
   FAN3            FAN             F2   5421.00   OK
   FAN4            FAN             F1   6268.00   OK
   FAN4            FAN             F2   5399.00   OK
   PS1             FAN             F1   10336.00  OK
   PS2             FAN             -    -         NOT PRESENT
           

4. Display the voltage current and status of each module in the system. Run:

   Copy

   Copied!

               
   
               
   switch (config) # show power consumers
   ------------------------------------------------------------------
   Module  Device            Sensor  Power   Voltage  Current  Status
                                     [Watts] [Volts]  [Amp]
   ------------------------------------------------------------------
   PS1     power-mon         input   39.94   12.07    3.31     OK
   MGMT    acdc-monitor1     input   2.11    12.00    0.18     OK
    
   Total power used : 42.05 Watts
           

The OS can access USB devices attached to switch systems. USB devices are automatically recognized and mounted upon insertion. To access a USB device for reading or writing a file, you need to provide the path to the file on the mounted USB device in the following format:

            

            
scp://username:password@hostname/var/mnt/usb1/<file name>
        

While username and password are the admin username and password and hostname is the IP of the switch.

Examples:

• To fetch an image from a USB device, run the command:

  Copy

  Copied!

              
  
              
  switch (config) # image fetch scp://username:password@hostname/var/mnt/usb1/<image filename>
          

• To save log file (my-logfile) to a USB device under the name “test_logfile” using the command “logging files”, run:

  Copy

  Copied!

              
  
              
  switch (config) # logging files upload my-logfile scp://username:password@hostname/var/mnt/usb1/test_logfile
          

• To safely remove the USB and to flush the cache, after writing (log files, for example) to a USB, use the “usb eject” command:

  Copy

  Copied!

              
  
              
  switch (config) # usb eject
          

The unit identification (UID) LED is a hardware feature used as a means of locating a specific switch system in a server room.

To activate the UID LED on a switch system, run:

            

            
switch (config) # led MGMT uid on
        

To verify the LED status, run:

            

            
switch (config) # show leds
Module   LED            Status
--------------------------------------------------------------------------
MGMT     STATUS         Green
MGMT     FAN1           Green
MGMT     FAN2           Green
MGMT     FAN3           Green
MGMT     FAN4           Green
MGMT     PS_STATUS      Green
MGMT     PS1            Green
MGMT     PS2            Green
MGMT     UID            Blue
        

To deactivate the UID LED on a switch system, run:

            

            
switch (config) # led MGMT uid off
        

NVIDIA high end management modular switch systems support redundant management modules. Chassis HA reduces downtime as it assures continuity of the work even when a management module dies. Chassis HA management allows the systems administrator to associate a single IP address with the appliance. Connecting to that IP address allows the user to change and review the system’s chassis parameters regardless of the active management module.

Chassis High Availability Nodes Roles

Every node in the Chassis HA has one of the following roles/modes:

• Master—the node that manages chassis configurations and services the chassis IP addresses

• Slave—the node that replaces the Master node and takes over its responsibilities once the Master node is down

The CPU role of the current management node can be recognized by following one these methods:

• Running the command “show chassis ha”

  Copy

  Copied!

              
  
              
  switch (config) # show chassis ha
  2-node HA state:
    Box management IPv4: 10.7.146.44/24
    Box management IPv6: fdfd:fdfd:7:145::1033:47fd/64
    interface          : mgmt0
    local role         : master
    local slot         : 1
    other state        : not-present
    reset count        : 0
          

• Check the LEDs in the management modules as displayed in the figure below

• Go to the WebUI → System → Modules page and see the information on the LEDs

Malfunctioned CPU Behavior

When a CPU in not responding to an internal communication with the other CPU, the non responding CPU will be reset by the other CPU. Each time a CPU resets, a counter is incremented. After 5 resets a CPU is considered malfunctioned and will be shut down.

To verify how many times a CPU is reset, run:

            

            
switch [default: master] (config) # show chassis ha
2-node HA state:
  Box management IPv4: 10.7.146.44/24
  Box management IPv6: fdfd:fdfd:7:145::1033:47fd/64
  interface          : mgmt0
  local role         : master
  local slot         : 1
  other state        : not-present
  reset count        : 0
        

To verify if a CPU has been shut down, either run:

            

            
switch [default: master] (config) # show chassis ha
2-node HA state:
  Box management IPv4: 10.7.146.44/24
  Box management IPv6: fdfd:fdfd:7:145::1033:47fd/64
  interface          : mgmt0
  local role         : master
  local slot         : 1
  other state        : not-present
  reset count        : 0
        

Or check the system page in the WebUI, the management figure will be grayed out.
To enable the malfunctioned CPU, first replace it and run “chassis ha reset other”.

Box IP Centralized Location

Box IP (BIP) centralized management infrastructure enables you to configure and monitor the system. The BIP continues to function even if one of the management blades dies. Box IP is defined by running the command “chassis ha bip <board IP address>”. The created BIP is used as the master IP’s alias. For example:

            

            
switch [standalone: master] (config) # chassis ha bip 192.168.10.100 255.255.255.0
        

System Configuration

System configuration changes should be performed by the master using the BIP otherwise they are overridden by the master configuration.

Chassis HA is based on database replication enabling the entire master configuration to be replicated to the slave. Data such as chassis configuration is replicated. However, run time information such as time, logs, active user lists, is not copied. Additionally, node specific configuration information such as host name and IP address is not copied.

            

            
switch [standalone: master] (config) #
        

Takeover Functionally

Management CPU functional takeover takes up to 20-30 seconds. However, when plugging in a module, you need to wait for approximately 3 minutes before making any other hardware change. During the takeover process, the Master LED status is differentiated by a color scheme. To verify the system’s status, run the “show chassis ha” command on both managements.

If the CPU malfunctions, the system resets it 5 times in an attempt to solve the issue. If the CPU is not activated after the reset, the system powers it off as well as its attached spine. Once the CPU is powered off, the user should replace the malfunctioned CPU module. To power on the CPU and the attached spine, plug the module in, log into the Master CPU and run the “chassis ha power enable other” command.

Master example:

            

            
switch [default: master] (config) # show chassis ha
2-node HA state:
 Box management IPv4: 10.7.146.44/24
 Box management IPv6: fdfd:fdfd:7:145::1033:47fd/64
 interface      	: mgmt0
 local role   	    : master
 local slot   	    : 1
 other state  	    : not-present
 reset count  	    : 0
        

Slave example:

            

            
switch [default: master] (config) # show chassis ha
2-node HA state:
  Box management IPv4: 10.7.146.44/24
  Box management IPv6: fdfd:fdfd:7:145::1033:47fd/64
  interface          : mgmt0
  local role         : master
  local slot         : 1
  other state        : not-present
  reset count        : 0
        

Rebooting 1U Switches

To reboot a 1U switch system, run:

            

            
switch (config) # reload
        

Rebooting Modular Switches

NVIDIA high end management modular switch systems support redundant management modules. Chassis HA reduces downtime as it assures continuity of the work even when a management module dies. Chassis HA management allows the systems administrator to associate a single IP address with the appliance. Connecting to that IP address allows the user to change and review the system’s chassis parameters regardless of the active management module.

To reboot modular switches:

1. Connect to the box IP (BIP). Please refer to “Box IP Centralized Location” section for more information.

2. Reboot the slave management. Run:

   Copy

   Copied!

               
   
               
   switch [default: master] (config) # chassis ha reset other
           

3. Reboot the master management. Run:

   Copy

   Copied!

               
   
               
   switch [default: master] (config) # reload
           

Onyx supports viewing all active events on the system. The following events may be observed with the command show system hardware events.