The following subsections illustrate how to install a Network OS using different methods.
From DHCP Server
Perform the following in your DHCP server.
Define a DHCP filename by configuring it in /etc/dhcpd.conf. Run:
filename "tftpboot_location/onie-installer-x86_64-mlnx_x86-r5.2.0004.bin";
Enable auto-discovery mode (if disabled). Run:
ONIE:/ # onie-start Starting: discover... done. ONIE:/ #
The switch system will fetch and install the OS image from the IP address discovered in the previous step (e.g. 20.20.20.20—this address will be used throughout this manual).
Info: Fetching http://20.20.20.20/onie-installer-x86_64-mlnx_x86-r5.2.0004 ... ONIE: Executing installer: http://20.20.20.20/onie-installer-x86_64-mlnx_x86-r5.2.0004 Verifying image checksum ... OK. Preparing image archive ... OK. MLNX OS Installer: platform: x86_64-mlnx_x86-r5.0 ====== Invoking MLNX OS installer via kexec ====== Network will disconnect, further logs may be reviewed via serial console [1018004.495413] sd 0:0:0:0: [sda] Synchronizing SCSI cache [1018004.501473] Starting new kernel [ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu ...
From TFTP Server
Perform the following in your TFTP server.
Move the image file under the TFTP boot location.
Enable auto-discovery mode (if disabled). Run:
ONIE:/ # onie-start Starting: discover... done.
The switch system will fetch and install the OS image.
Info: Fetching tftp://20.20.20.20/onie-installer-x86_64-mlnx_x86-r5.2.0004 ... ONIE: Executing installer: tftp://20.20.20.20/onie-installer-x86_64-mlnx_x86-r5.2.0004 Verifying image checksum ... OK. Preparing image archive ... OK. MLNX OS Installer: platform: x86_64-mlnx_x86-r5.0 ====== Invoking MLNX OS installer via kexec ====== Network will disconnect, further logs may be reviewed via serial console [1018004.495413] sd 0:0:0:0: [sda] Synchronizing SCSI cache [1018004.501473] Starting new kernel [ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu ...
From IPv6 Neighbors
To install a Network OS from IPv6 neighbors using an installer test script:
Boot ONIE in install mode by selecting "Install OS" in the ONIE menu.
GNU GRUB version 2.02 +---------------------------------------------+ |*ONIE: Install OS | | ONIE: Rescue | | ONIE: Uninstall OS | | ONIE: Update ONIE | | ONIE: Embed ONIE | | | | | +---------------------------------------------+
Prepare a small installer test script at the deployment server (using HTTP server).
# cat /var/www/html/onie-isntaller-mlnx #/bin/sh echo "Mellanox ONIE discovery over eth0 ipv6 neighbors test" exit 1
Let the automatic installer detect the IPv6 neighbor you prepared.
ONIE: Using DHCPv6 addr: eth0: fe80::3652:b5fe:fecd:897d ONIE: Using DHCPv6 addr: eth1: fe80::202a:c9df:fe6e:68ab discover: installer mode detected. Running installer. Please press Enter to activate this console. Info: Attempting http://[fe80::3640:b5ff:febf:897d%eth0]/onie-installer ... Info: Attempting http://[fe80::3640:b5ff:febf:897d%eth0]/onie-installer.bin ... Info: Attempting http://[fe80::202:c9ff:fe63:68a0%eth0]/onie-installer-x86_64-mlnx_msn2700-r0 ... Info: Attempting http://[fe80::202:c9ff:fe63:68a0%eth0]/onie-installer-x86_64-mlnx_msn2700-r0.bin ... Info: Attempting http://[fe80::202:c9ff:fe63:68a0%eth0]/onie-installer-x86_64-mlnx_msn2700 ... Info: Attempting http://[fe80::202:c9ff:fe63:68a0%eth0]/onie-installer-x86_64-mlnx_msn2700.bin ... Info: Attempting http://[fe80::202:c9ff:fe63:68a0%eth0]/onie-installer-mlnx_msn2700 ... Info: Attempting http://[fe80::202:c9ff:fe63:68a0%eth0]/onie-installer-mlnx_msn2700.bin ... Info: Attempting http://[fe80::202:c9ff:fe63:68a0%eth0]/onie-installer-x86_64-mlnx ... Info: Attempting http://[fe80::202:c9ff:fe63:68a0%eth0]/onie-installer-x86_64-mlnx.bin ... Info: Attempting http://[fe80::202:c9ff:fe63:68a0%eth0]/onie-installer-x86_64 ... Info: Attempting http://[fe80::202:c9ff:fe63:68a0%eth0]/onie-installer-x86_64.bin ...
From Local USB Drive
ONIE discovery and installation from USB does not support EXFAT (NTFS). Use VFAT formatting instead.
To install a Network OS from a USB drive:
Put the NOS image inside the USB root directory.
Mount and install the NOS version. Run:
ONIE:/ # mkdir /mnt/tmpusb ONIE:/ # mount /dev/sdb1 /mnt/tmpusb ONIE:/ # cd /mnt/tmpusb ONIE:/ # mnt/tmpusb # sync ONIE:/ # onie-nos-install /mnt/tmpusb/onie-installer-x86_64-mlnx_x86-r5.2.0004.bin Rebooting into ONIE (re)install mode... ONIE:/ # umount: can't remount rootfs read-only The system is going down NOW! Sent SIGTERM to all processes Sent SIGKILL to all processes Restarting system. ...
Via SCP Local File
To install a Network OS from a via SCP local file:
Select "Install OS" in the ONIE GRUB menu.
Disable DHCP auto-discovery (for non-DHCP configurations only). Run:
ONIE:/ # onie-stop discover: installer mode detected. Stopping: discover...start-stop-daemon: warning: killing process done.
Assign static IP and netmask for eth0 (for non-DHCP configurations only):
ONIE:/ # ifconfig eth0 192.168.1.200 netmask 255.255.255.0
Ping SCP server to verify connection:
ONIE:/ # ping 192.168.1.21
Create directory to which to copy NOS image. Run:
ONIE:/ # mkdir /tmp/nos
Copy NOS image to /tmp/nos:
ONIE:/ # scp root@192.168.1.21:/tftpboot/cumulus-linux-3.5.3-mlx-amd64.bin /tmp/nos ... Do you want to continue connecting? (y/n) yes root@192.168.1.21’s password: ... cumulus-linux-3.5.3-mlx-amd64.bin 100% 212MB 19.3MB/s 00:11
cd to directory and install NOS image:
ONIE:/ # cd /tmp/nos ONIE:/tmp/nos # onie-nos-install cumulus-linux-3.5.3-mlx-amd64.bin discover: installer mode detected. Stopping: discover... done. ONIE: Executing installer: cumulus-linux-3.5.3-mlx-amd64.bin Verifying image checksum ...OK. Preparing image archive ... OK. Verifying image architecture ...OK. Verifying system ram ...OK. Setting up installer ...
Secured Installation and Update
Starting version 5.3.0008, ONIE supports a secured installation and update procedure. It is only enabled when Secured Boot is enabled in the BIOS. In this case, ONIE validates CMS signature of the installed NOS.
In addition, when using ONIE in Secured Mode, the ONIE itself also must have a CMS signature in the update procedure.
When loading an ONIE system, the system loads the following GRUB menu:
GNU GRUB version 2.02
+---------------------------------------------+
|*ONIE: Install OS |
| ONIE: Rescue |
| ONIE: Uninstall OS |
| ONIE: Update ONIE |
| ONIE: Embed ONIE |
| |
| |
+---------------------------------------------+
For a full description of the different boot mode options and their function, please refer to the official ONIE documentation referenced in the “Related Documentation” section.
Rescue Mode
To boot in Rescue mode:
Select "Rescue" in the ONIE GRUB menu:
GNU GRUB version 2.02 +---------------------------------------------+ | ONIE: Install OS | |*ONIE: Rescue | | ONIE: Uninstall OS | | ONIE: Update ONIE | | ONIE: Embed ONIE | | | | | +---------------------------------------------+
Run "rescue":
# rescue ONIE: Rescue Mode ... Version : master-201604201145 Build Date: 2016-04-20T13:08+0300 [ 5.798037] i8042: No controller found Info: Mounting kernel filesystems... done. Info: Mounting LABEL=ONIE-BOOT on /mnt/onie-boot ... ... discover: Rescue mode detected. Installer disabled. Please press Enter to activate this console. To check the install status inspect /var/log/onie.log. Try this: tail -f /var/log/onie.log ** Rescue Mode Enabled **
ONIE Update
To update the ONIE:
Select "Update ONIE" in the ONIE GRUB menu:
GNU GRUB version 2.02 +---------------------------------------------+ | ONIE: Install OS | | ONIE: Rescue | | ONIE: Uninstall OS | |*ONIE: Update ONIE | | ONIE: Embed ONIE | | | | | +---------------------------------------------+
Run the command "onie-self-update":
ONIE:/ # onie-self-update http://20.20.20.20/onie-updater-x86_64-mlnx_x86-r5.2.0004 topping: discover... done. Info: Fetching http://20.20.20.20/onie-updater-x86_64-mlnx_x86-r5.2.0004 ... Connecting to 20.20.20.20 (20.20.20.20:80) installer 100% |*******************************| 24601k 0:00:00 ETA ONIE: Executing installer: http://<ip address>/onie-updater-x86_64-mlnx_x86-r5.2.0004 Verifying image checksum ... OK. Preparing image archive ... OK. ONIE: Version : master-201604201145 ONIE: Architecture : x86_64 ONIE: Machine : mlnx_x86 ONIE: Machine Rev : 5.2.0004 ONIE: Config Version: 1 Installing ONIE on: /dev/sda Rebooting... + killall -TERM dropbear + killall -TERM syslogd + [ -x /lib/onie/shutdown-machine ] + /lib/onie/s[1205783.482229] LPCI2C DBG: LPC Generic Decode Range 3 deconfigured: 0xfc2501 [1205783.490022] LPCI2C DBG: LPC Generic Decode Range 2 deconfigured: 0xfc2001 lpci2c 15706 0 [1205783.498523] LPCI2C DBG: lpci2c removed - Live 0xffffffffa0000000 (O) ...
Uninstall
To uninstall the OS from a system:
Select "Uninstall OS" in the ONIE GRUB menu:
GNU GRUB version 2.02 +---------------------------------------------+ | ONIE: Install OS | | ONIE: Rescue | |*ONIE: Uninstall OS | | ONIE: Update ONIE | | ONIE: Embed ONIE | | | | | +---------------------------------------------+
Run the command "uninstall":
ONIE:/ # uninstall ONIE: OS Uninstall Mode ... Version : master-201604201145 Build Date: 2016-04-20T13:08+0300 [ 5.796828] i8042: No controller found Info: Mounting kernel filesystems... done. Info: Mounting LABEL=ONIE-BOOT on /mnt/onie-boot ... [ 10.270675] LPCI2C DBG: LPC Generic Decode Range 2 configured: 0xfc2001 [ 10.277368] LPCI2C DBG: LPC Generic Decode Range 3 configured: 0xfc2501 ... ONIE:/ # + /bin/umount -a -r The system is going down NOW! ...
Embed
To boot in Embed mode:
Select rescue mode in the ONIE GRUB menu:
GNU GRUB version 2.02 +---------------------------------------------+ | ONIE: Install OS | | ONIE: Rescue | | ONIE: Uninstall OS | | ONIE: Update ONIE | |*ONIE: Embed ONIE | | | | | +---------------------------------------------+
Run the "onie-self-update" command:
ONIE:/ # onie-self-update -e http://20.20.20.20/sx_mlnx_os/onie/x86.r5.2.0004/on ie-updater-x86_64-mlnx_x86-r5.2.0004.bin Stopping: discover... done. Info: Fetching http://20.20.20.20/sx_mlnx_os/onie/x86.r5.2.0004/onie-updater-x86_64-mlnx_x86-r5.2.0004 ... ... Installing ONIE on: /dev/sda Rebooting... + killall -TERM dropbear + killall -TERM syslogd + [ -x /lib/onie/shutdown-machine ] ... ONIE:/ # + /bin/umount -a -r The system is going down NOW! ... [ 2274.038521] reboot: machine restart ...
Reinstall
There are two possible methods to reinstall the network OS on a x86 system:
This procedure is relevant to ONIE version 5.1.00.xx and higher.
In order to update BIOS through ONIE:
Boot to ONIE update mode.During a system boot in grub menu choose ONIE and after that select "ONIE: Update ONIE" option.
In booted ONIE, start the stop discover process:
# onie-stop
BIOS release directory contains BIOS ROM file and its MD5sum.
Copy these files to any directory on the target system. For example:
# scp /auto/sw_system_release/sx_mlnx_bios/IvyBridge/0ABZS017_02.02.x01/ Release/0ABZS017.rom* root@10.209.48.47:/tmp
Perform BIOS update. For example:
# bios_update.sh /tmp/0ABZS017.rom
Do not interrupt BIOS update process.
When calling the bios_update.sh script, the absolute path to BIOS image must be provided.Power cycle your system.
Check new BIOS version from ONIE update mode. For example:
# bios_version.sh BIOS Version: 4.6.5 BIOS Release Date: 15/04/2018 BIOS SubVersion: 0ABZS017_02.02.001
ONIE allows collecting extended debug information (mostly HW related) about a system. This ability is more relevant for non-NVIDIA-proprietary NOSs or to ONIE bare-metal systems.
In order to gather this information:
Select rescue mode in the ONIE GRUB menu:
GNU GRUB version 2.02 +---------------------------------------------+ | ONIE: Install OS | |*ONIE: Rescue | | ONIE: Uninstall OS | | ONIE: Update ONIE | | ONIE: Embed ONIE | | | | | +---------------------------------------------+
WarningDo not go to "Install OS" or "Embed ONIE" as it can destroy already installed NOS.
Run "onie-support <directory_path>". For example:
onie-support /tmp
Command execution may take 16-30 seconds depending on your system.
Once done, the tarball onie-support-mlnx-<system-type>.tar.bz2 is created at the specified directory. For example:Success: Support tarball created: /tmp/onie-support-mlnx_msn3700.tar.bz2
Copy this file to the server, open it, and check the gathered information or send it for further analysis.
scp /tmp/onie-support-mlnx_msn3700.tar.bz2 <user>@<server>:/tmp
On the server:
tar xjf onie-support-mlnx_msn3700.tar.bz2
The information is compiled in the following files:
Filename
Description
bios_version.txt
Exact BIOS version, subversion, and release date
blkid.txt
Disk partitions, their labels, and UUID
fdisk.txt
Additional information about disk partitions
chassis_inventory.txt
System inventory (S/N, P/N of system modules)
chassis_fru.txt
Decoded chassis/system FRU (EEPROM VPD)
system_vpd.bin
Full raw dump of system VPD (EEPROM FRU)
chassis_devs.txt
HW devices and their info
chassis_modules.txt
Information about modules and their presence
chassis_power.txt
System power information
chassis_temp.txt
System temperature information
chassis_volt.txt
System voltage information
cpld_dump.txt
Full raw CPLD registers dump
cpuinfo.txt
CPU information
dmidecode_memory.txt
Information about installed DIMMs
dmidecode_system.txt
System important SMBIOS info (type 1, 2 and 3)
dmesg.txt
Last boot kernel logs
kernel_cmdline.txt
Kernel command line parameters
dmidecode_bios_logs.txt
BIOS logs reported to OS
etho_info.txt
Information about eth0 interface
i2cscan.txt
Scanning information of all available I2C buses (including all virtual I2C buses)
ismart.txt
SSD information for Innodisk vendor (if their SSDF is installed)
smartctl.txt
SSD information for all vendors
lspci.txt
Scan of all PCIe devices, and verbose PCIe information about NVIDIA ASIC device
onie-syseeprom.txt
Decoded ONIE system EEPROM information
onie-sysinfo.txt
System information provided by ONIE
machine-build.conf
ONIE build information
machine-live.conf
ONIE run-time information
machine.conf
Common information of previous 2 files
runtime-export-env.txt
ONIE exported environment information
runtime-process.txt
Running processes
runtime-set-env.txt
ONIE environment variables
log/
Contains ONIE boot and run logs
ONIE also contains many debug tools that can be run manually, with different options aside from collecting debug info with onie-support.
One tool is "chassismgr":
chassismgr <command> [parameters...]
You may run "chassismgr" without parameters for help menu.
Another tool is "vpddata" which allows reading and writing raw VPD data. For example, it can allow full VPD recovery.