Q3200-RA/Q3400-RA XDR 800Gb/s InfiniBand Switch Systems User Manual

System Bring-Up of Managed Systems

This quick start guide provides an end-to-end setup process for installing and running NVOS.

Note

Before running NVUE commands, please refer to the NVIDIA User Experience (NVUE) section in the NVIDIA NVOS User Manual for InfiniBand Switches.

Note

NVOS supports configurations through NVUE only. Running Linux commands or directly modifying underlying Linux files might result in undefined behavior.

This guide requires an intermediate-level Linux knowledge. An understanding of text editing, Unix file permissions, and process monitoring is necessary. A variety of text editors are pre-installed, including vi and nano.

Access to a Linux or UNIX shell in needed. For Windows users, employing a Linux environment like Cygwin as your command line tool is recommended for interacting with NVOS.

Note

If moving from MLNX-OS to NVOS operating system, please refer to the following appendix: Moving from MLNX-OS to NVOS NVIDIA NVOS User Manual for InfiniBand Switches.

NVOS provides two methods for initial provisioning:

When starting NVOS for the first time, the management port send a DHCP request. To determine the IP address of the switch, you can cross reference the MAC address of the switch with your DHCP server. The MAC address is typically located on the side of the switch or on the box in which the unit ships.

Login Credentials

The default installation includes two accounts:

  • The user account (admin) has sudo privileges. The admin account uses the default password admin.

  • The user account (monitor) has read only privileges. The monitor account uses the default password monitor.

Upon first login, it will be required to change the default passwords for the admin and monitor accounts. The new password must comply with the default password hardening rules (see Password Hardening section in the NVIDIA NVOS User Manual for InfiniBand Switches).

The new configured password is treated like any other configuration. If the configuration is not saved, the user will need to reconfigure it upon the system's next boot.

Copy
Copied!
            

You are required to change your password immediately (administrator enforced).   ███╗ ██╗██╗ ██╗ ██████╗ ███████╗ ████╗ ██║██║ ██║██╔═══██╗██╔════╝ ██╔██╗ ██║██║ ██║██║ ██║███████╗ ██║╚██╗██║╚██╗ ██╔╝██║ ██║╚════██║ ██║ ╚████║ ╚████╔╝ ╚██████╔╝███████║ ╚═╝ ╚═══╝ ╚═══╝ ╚═════╝ ╚══════╝   WARNING: Your password has expired. You must change your password now! New password: Retype new password:

In this quick start guide, use the admin account to configure NVOS.

All accounts except root can use remote SSH login. Note that the default SSH auto-logout is 15 minutes.

NVOS supports multiple services to authenticate users and authorize switch tasks, both local and remote authentication/authorization methods.

For more information, please refer to Authentication Authorization and Accounting section in the NVIDIA NVOS User Manual for InfiniBand Switches.

Serial Console Management

It is recommended to perform management and configuration over the network, either in-band or out-of-band. A serial console is fully supported.

Typically, switches ship from the manufacturer with a mating DB9 serial cable. Switches with ONIE are always set to a 115200 baud rate.

Recommended Serial Connection Settings

Parameter

Setting

Baud Rate

115200

Data bits

8

Stop bits

1

Parity

None

Flow Control

None


Wired Ethernet Management

An NVOS switch always provides two dedicated Ethernet management port called eth0 and eth1. This interface is specifically for out-of-band management use. The management interface uses DHCP for addressing by default.

Set Static IP Address

To set a static IP address, run the following (ipv6 supported as well):

Copy
Copied!
            

admin@nvos:~$ nv set interface eth0 ip address 192.0.2.42/24  admin@nvos:~$ nv set interface eth0 ip gateway 192.0.2.1  admin@nvos:~$ nv config apply

Note

Configuring static IP address will trigger unsolicited announcement messages to the gateway in order to reveal the devices's MAC address.

Configure the Hostname

The hostname identifies the switch; as such, make sure the hostname is configured in a unique and descriptive way. For more information, see Hostname section in the NVIDIA NVOS User Manual for InfiniBand Switches.

Configure the System Clock

NVOS relies on the system clock for displaying the time and timestamping messages. User can control the system time zone and clock settings.

Manual Settings

Time Zone

Default time zone is set to Coordinated Universal Time (UTC). User may change the time zone configuration by executing the following:

Copy
Copied!
            

admin@nvos:~$ nv set system timezone Etc/UTC admin@nvos:~$ nv config apply


Clock

The system date and time can be manually changed. NTP servers configured on the switch will supersede any manually entered date-time settings.

Copy
Copied!
            

admin@nvos:~$ nv action change system date-time 2024-12-24 10:25:13

NTP

NVOS supports Network Time Protocol (NTP) for synchronizing the time of the system. For more information see Network Time Protocol section in the NVIDIA NVOS User Manual for InfiniBand Switches

NTP Servers from DHCP

By default, NTP will obtain the NTP servers from DHCP. It is possible to disable it by executing the following:

Copy
Copied!
            

admin@nvos:~$ nv set system ntp dhcp disabled admin@nvos:~$ nv config apply


Configure NTP Servers

A new NTP server can be added by executing the following:

Copy
Copied!
            

admin@nvos:~$ nv set system ntp server 10.11.100.5 admin@nvos:~$ nv config apply

NTP can be configured to accept packets coming from an authenticated source only.

NTP authentication is disabled by default.

An authentication key may be created and used to authenticate incoming NTP packets. To ensure the key is utilized, the following conditions must be met:

  • Enable NTP authentication

    Copy
    Copied!
                

    admin@nvos:~$ nv set system ntp authentication enabled admin@nvos:~$ nv config apply

  • Add a new NTP key, and associate it with the server:

    Copy
    Copied!
                

    admin@nvos:~$ nv set system ntp key 15 value SECRET admin@nvos:~$ nv set system ntp key 15 trusted yes admin@nvos:~$ nv set system ntp server 10.11.100.5 key 15 admin@nvos:~$ nv config apply

  • NTP server must use the same key.

NVOS support various attributes for NTP servers and keys, for more infomation please see NTP Commands section in the NVIDIA NVOS User Manual for InfiniBand Switches.

Configure Syslog Servers

User can configure the system to send syslog logs to a centralized logging server ( one or more):

Copy
Copied!
            

admin@nvos:~$ nv set system syslog server 10.20.30.40 admin@nvos:~$ nv config apply

For more syslog configurations, please see Remote Logging section in the NVIDIA NVOS User Manual for InfiniBand Switches .

By default, NVOS enables all data plane ports (every InfiniBand port).

To view link status, run the nv show interface command or nv show platform transceiver.

Warning

When running NVUE commands to configure the switch, run the nv config save command before you reboot. The command saves the applied configuration to the startup configuration so that the changes persist after the reboot.

    
admin@nvos:~$ nv config save    

The connection-mode attribute indicates to which peer this port is connected. This feature is dedicated to Q3200-RA system which can be connected to NDR systems (or older) and to XDR systems.

In case the port is connected to XDR systems, the connection-mode should be XDR. If it connected to NDR system, the connection-mode should be NDR. By default, all the ports are configured to connection-mode XDR.

When connecting port to NDR system, the port connection-mode needs to be changed with the set command (nv set interface link connection-mode) to 'ndr' in order to have link up with full bandwidth. Changing port connection-mode will move the port to down and up again when configuration ends.

© Copyright 2025, NVIDIA. Last updated on Mar 20, 2025.