NVIDIA® Cumulus Linux is the first full-featured Debian Buster-based, Linux operating system for the networking industry.

This user guide provides in-depth documentation on the Cumulus Linux installation process, system configuration and management, network solutions, and monitoring and troubleshooting recommendations. In addition, the quick start guide provides an end-to-end setup process to get you started.

Cumulus Linux 5.6 includes the NVIDIA NetQ agent and CLI. You can use NetQ to monitor and manage your data center network infrastructure and operational health. Refer to the NVIDIA NetQ documentation for details.

For a list of the new features in this release, see What's New. For bug fixes and known issues present in this release, refer to the Cumulus Linux 5.6 Release Notes.

Try It Pre-built Demos

The Cumulus Linux documentation includes pre-built Try It demos for certain Cumulus Linux features. The Try It demos run a simulation in NVIDIA Air; a cloud hosted platform that works exactly like a real world production deployment. Use the Try It demos to examine switch configuration for a feature. For more information, see Try It Pre-built Demos.

Open Source Contributions

To implement various Cumulus Linux features, NVIDIA has forked various software projects, like CFEngine Netdev and some Puppet Labs packages. Some of the forked code resides in the NVIDIA Networking GitHub repository and some is available as part of the Cumulus Linux repository as Debian source packages.

NVIDIA has also developed and released new applications as open source. The list of open source projects is on the Cumulus Linux packages page.

Download the User Guide

Use one of the following methods to download the Cumulus Linux user guide and view it offline:

• Host the documentation on a local host using hugo.
• For a fully functional copy of the user guide, download a zip file of an HTML documentation build for offline use. Download the desired version, extract it locally, then open cumulus-linux-56.html in your web browser.
• To view this user guide as a single page to print to a PDF with limited functionality, click here. Click on the link one time and use the web browser print-to-PDF option to save the PDF locally.

This document supports the Cumulus Linux 5.6 release, and lists new platforms, features, and enhancements.

• For a list of open and fixed issues in Cumulus Linux 5.6, see the Cumulus Linux 5.6 Release Notes.
• To upgrade to Cumulus Linux 5.6, follow the steps in Upgrading Cumulus Linux.

What’s New in Cumulus Linux 5.6.0

Cumulus Linux 5.6.0 supports new platforms, contains several new features and improvements, and provides bug fixes.

Platforms

• NVIDIA SN5600 (800G Spectrum-4) - (supports 1G on the bonus port)
• NVIDIA SN3750-SX (200G Spectrum-2) is now generally available

New Features and Enhancements

• FRR upgrade to version 8.4.3
• PTP stack upgrade to linuxptp 4.0
• Adaptive routing is generally available for a switch with the Spectrum-4 ASIC and provides enhancements that include support for VXLAN-encapsulated RoCE traffic and BGP W-ECMP
• PVRST with VLAN-aware bridges
• QOS PFC watchdog for lossless queues
• PTP one-step clock timestamp mode for Spectrum-2 and Spectrum-3
• PAM4 encoding support for the NVIDIA SN4410 switch
• Fan airflow direction in NVUE nv show platform environment fan and Linux smonctl -v command output; a fan direction mismatch triggers a log message
• NVUE enhancements include:
  • SSH commands
  • Enable and disable external API access commands (Cumulus Linux 5.6 and later enables the NVUE REST API by default; be sure to disable or secure the API if needed)
  • ARP neighbor base reachable timer and ND neighbor base reachable timer commands
  • IGMP fast leave processing and last member query count commands
  • Clear BGP route commands to clear all BGP sessions and to refresh routes for all neighbors
  • Clear PIM state and statistics commands
  • Clear EVPN duplicate address commands
  • Show IGMP group commands
  • Show commands to see the ports mapped to a bridge and the VLANs mapped to bridge ports
  • VRRP show commands include operational data
  • Changes to the nv show platform command outputs to improve readability:
    • Shows a concise summary of all sensors
    • Improved memory format
    • Modified disk size output from KB to GB
    • Output fields display n/a for VX models where needed
    • Removed the applied column from the nv show platform hardware component device command
    • Removed the applied and pending columns from the nv show platform software installed python3-nvue command
  • The nv show commands provide a --filter option to filter output data
  • EVPN multihoming configuration with NVUE no longer supports a 10-byte ESI value starting with a non 00 hex value

nv show router pbr nexthop-group
nv show router pbr nexthop-group <nexthop-group-id>
nv show bridge domain <domain-id> port
nv show bridge domain <domain-id> stp counters
nv show bridge domain <domain-id> stp port
nv show bridge domain <domain-id> stp port <interface-id>
nv show bridge domain <domain-id> stp vlan
nv show bridge domain <domain-id> stp vlan <vid>
nv show evpn vni <vni-id> remote-vtep
nv show qos pfc-watchdog
nv show interface <interface-id> ip igmp group
nv show interface <interface-id> ip igmp group <static-group-id>
nv show interface <interface-id> bridge domain <domain-id> stp vlan
nv show interface <interface-id> bridge domain <domain-id> stp vlan <vid>
nv show interface <interface-id> qos pfc-watchdog
nv show interface <interface-id> qos pfc-watchdog status
nv show interface <interface-id> qos pfc-watchdog status <qos-tc-id>
nv show service ptp <instance-id> counters
nv show system api
nv show system api listening-address
nv show system api listening-address <listening-address-id>
nv show system api connections
nv show system global arp
nv show system global arp garbage-collection-threshold
nv show system global nd
nv show system global nd garbage-collection-threshold
nv show system ssh-server
nv show system ssh-server max-unauthenticated
nv show system ssh-server vrf
nv show system ssh-server vrf <vrf-id>
nv show system ssh-server allow-users
nv show system ssh-server allow-users <user-id>
nv show system ssh-server deny-users
nv show system ssh-server deny-users <user-id>
nv show system ssh-server port
nv show system ssh-server port <port-id>
nv show system ssh-server active-sessions

nv set router adaptive-routing link-utilization-threshold (on|off)
nv set router password-obfuscation (enabled|disabled)
nv set bridge domain <domain-id> stp vlan <vid>
nv set bridge domain <domain-id> stp vlan <vid> bridge-priority 4096-61440
nv set bridge domain <domain-id> stp vlan <vid> hello-time 1-10
nv set bridge domain <domain-id> stp vlan <vid> forward-delay 4-30
nv set bridge domain <domain-id> stp vlan <vid> max-age 6-40
nv set bridge domain <domain-id> stp mode (rstp|pvrst)
nv set qos pfc-watchdog polling-interval 100-5000
nv set qos pfc-watchdog robustness 1-1000
nv set interface <interface-id> ip igmp fast-leave
nv set interface <interface-id> ip igmp last-member-query-count
nv set interface <interface-id> router adaptive-routing link-utilization-threshold 1-100
nv set interface <interface-id> bridge domain <domain-id> stp vlan <vid>
nv set interface <interface-id> bridge domain <domain-id> stp vlan <vid> priority 0-240
nv set interface <interface-id> bridge domain <domain-id> stp vlan <vid> path-cost
nv set interface <interface-id> bridge domain <domain-id> stp path-cost 1-200000000 
nv set interface <interface-id> qos pfc-watchdog state (enable|disable)
nv set service ptp <instance-id> profile <profile-id> two-step (on|off)
nv set service ptp <instance-id> two-step (on|off)
nv set system api listening-address <listening-address-id>
nv set system api state (enabled|disabled)
nv set system api port 1-65535
nv set system global arp base-reachable-time (30-2147483|auto)
nv set system global nd base-reachable-time (30-2147483|auto)
nv set system ssh-server max-unauthenticated session-count 1-10000
nv set system ssh-server max-unauthenticated throttle-percent 1-100
nv set system ssh-server max-unauthenticated throttle-start 1-10000
nv set system ssh-server vrf <vrf-id>
nv set system ssh-server allow-users <user-id>
nv set system ssh-server deny-users <user-id>
nv set system ssh-server port <port-id>
nv set system ssh-server authentication-retries 3-100
nv set system ssh-server login-timeout 1-600
nv set system ssh-server inactive-timeout <value>
nv set system ssh-server permit-root-login (disabled|prohibit-password|forced-commands-only|enabled)
nv set system ssh-server max-sessions-per-connection 1-100
nv set system ssh-server state (enabled|disabled)

nv unset router adaptive-routing link-utilization-threshold
nv unset router password-obfuscation
nv unset bridge domain <domain-id> stp vlan
nv unset bridge domain <domain-id> stp vlan <vid>
nv unset bridge domain <domain-id> stp vlan <vid> bridge-priority
nv unset bridge domain <domain-id> stp vlan <vid> hello-time
nv unset bridge domain <domain-id> stp vlan <vid> forward-delay
nv unset bridge domain <domain-id> stp vlan <vid> max-age
nv unset bridge domain <domain-id> stp mode
nv unset qos pfc-watchdog
nv unset qos pfc-watchdog polling-interval
nv unset qos pfc-watchdog robustness
nv unset interface <interface-id> ip igmp fast-leave
nv unset interface <interface-id> ip igmp last-member-query-count
nv unset interface <interface-id> bridge domain <domain-id> stp vlan
nv unset interface <interface-id> bridge domain <domain-id> stp vlan <vid>
nv unset interface <interface-id> bridge domain <domain-id> stp vlan <vid> priority
nv unset interface <interface-id> bridge domain <domain-id> stp vlan <vid> path-cost
nv unset interface <interface-id> bridge domain <domain-id> stp path-cost
nv unset interface <interface-id> qos pfc-watchdog
nv unset interface <interface-id> qos pfc-watchdog state
nv unset service ptp <instance-id> profile <profile-id> two-step
nv unset service ptp <instance-id> two-step
nv unset system api
nv unset system api listening-address
nv unset system api listening-address <listening-address-id>
nv unset system api state
nv unset system api port
nv unset system api port 1-65535
nv unset system global arp base-reachable-time
nv unset system global nd base-reachable-time
nv unset system ssh-server
nv unset system ssh-server max-unauthenticated
nv unset system ssh-server max-unauthenticated session-count
nv unset system ssh-server max-unauthenticated throttle-percent
nv unset system ssh-server max-unauthenticated throttle-start
nv unset system ssh-server vrf
nv unset system ssh-server vrf <vrf-id>
nv unset system ssh-server allow-users
nv unset system ssh-server allow-users <user-id>
nv unset system ssh-server deny-users
nv unset system ssh-server deny-users <user-id>
nv unset system ssh-server port
nv unset system ssh-server port <port-id>
nv unset system ssh-server authentication-retries
nv unset system ssh-server login-timeout
nv unset system ssh-server inactive-timeout
nv unset system ssh-server permit-root-login
nv unset system ssh-server max-sessions-per-connection
nv unset system ssh-server state

nv action clear router policy prefix-list
nv action clear router policy prefix-list <prefix-list-id>
nv action clear router policy prefix-list <prefix-list-id> rule <rule-id> match <match-id>
nv action clear router bgp
nv action clear router bgp in prefix-filter
nv action clear router bgp out
nv action clear router bgp soft
nv action clear router bgp soft in
nv action clear router bgp soft out
nv action clear router igmp interfaces
nv action clear evpn vni
nv action clear evpn vni <vni-id>
nv action clear evpn vni <vni-id> mac <mac-address-id>
nv action clear evpn vni <vni-id> host <ip-address-id>
nv action clear interface <interface-id> qos pfc-watchdog deadlock-count
nv action clear vrf <vrf-id> router bgp in prefix-filter
nv action clear vrf <vrf-id> router bgp out
nv action clear vrf <vrf-id> router bgp soft
nv action clear vrf <vrf-id> router bgp soft in
nv action clear vrf <vrf-id> router bgp soft out
nv action clear vrf <vrf-id> router pim interfaces
nv action clear vrf <vrf-id> router pim interface-traffic

This quick start guide provides an end-to-end setup process for installing and running Cumulus Linux.

Prerequisites

This guide assumes you have intermediate-level Linux knowledge. You need to be familiar with basic text editing, Unix file permissions, and process monitoring. A variety of text editors are pre-installed, including vi and nano.

You must have access to a Linux or UNIX shell. If you are running Windows, use a Linux environment like Cygwin as your command line tool for interacting with Cumulus Linux.

Get Started

Cumulus Linux is on the switch by default. To upgrade to a different Cumulus Linux release or re-install Cumulus Linux, refer to Installation Management. To show the current Cumulus Linux release on the switch, run the NVUE nv show system command.

When starting Cumulus Linux for the first time, the management port makes a DHCPv4 request. To determine the IP address of the switch, you can cross reference the MAC address of the switch with your DHCP server. The MAC address is typically located on the side of the switch or on the box in which the unit ships.

To get started:

• Log in to Cumulus Linux on the switch and change the default credentials.
• Configure Cumulus Linux. This quick start guide provides instructions on changing the hostname of the switch, setting the date and time, and configuring switch ports and a loopback interface.

Login Credentials

The default installation includes two accounts:

• The system account (root) has full system privileges. Cumulus Linux locks the root account password by default (which prohibits login).

• The user account (cumulus) has sudo privileges. The cumulus account uses the default password cumulus.

  When you log in for the first time with the cumulus account, Cumulus Linux prompts you to change the default password. After you provide a new password, the SSH session disconnects and you have to reconnect with the new password.

In this quick start guide, you use the cumulus account to configure Cumulus Linux.

All accounts except root can use remote SSH login; you can use sudo to grant a non-root account root-level access. Commands that change the system configuration require this elevated level of access.

For more information about sudo, see Using sudo to Delegate Privileges.

Serial Console Management

NVIDIA recommends you perform management and configuration over the network, either in band or out of band. A serial console is fully supported.

Typically, switches ship from the manufacturer with a mating DB9 serial cable. Switches with ONIE are always set to a 115200 baud rate.

Wired Ethernet Management

A Cumulus Linux switch always provides at least one dedicated Ethernet management port called eth0. This interface is specifically for out-of-band management use. The management interface uses DHCPv4 for addressing by default.

To set a static IP address:

cumulus@switch:~$ nv set interface eth0 ip address 192.0.2.42/24
cumulus@switch:~$ nv set interface eth0 ip gateway 192.0.2.1
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo nano /etc/network/interfaces
# Management interface
auto eth0
iface eth0
    address 192.0.2.42/24
    gateway 192.0.2.1

Configure the Hostname

The hostname identifies the switch; make sure you configure the hostname to be unique and descriptive.

To change the hostname:

cumulus@switch:~$ nv set system hostname leaf01
cumulus@switch:~$ nv config apply

Configure the Time Zone

The default time zone on the switch is UTC (Coordinated Universal Time). Change the time zone on your switch to be the time zone for your location.

To update the time zone:

cumulus@switch:~$ nv set system timezone US/Eastern
cumulus@switch:~$ nv config apply

Verify the System Time

Verify that the date and time on the switch are correct with the Linux date command:

cumulus@switch:~$ date
Mon 21 Nov 2022 06:30:37 PM UTC

If the date and time are incorrect, the switch does not synchronize with automation tools, such as Puppet, and returns errors after you restart switchd.

To set the software clock according to the configured time zone, run the Linux sudo date -s command; for example:

cumulus@switch:~$ sudo date -s "Tue Jan 26 00:37:13 2021"

For more information about setting the system time, see Setting the Date and Time.

NTP and PTP

• NTP starts at boot by default on the switch and the NTP configuration includes default servers. To customize NTP, see NTP.
• PTP is off by default on the switch. To configure PTP, see PTP.

Configure Breakout Ports with Splitter Cables

If you are using 4x10G DAC or AOC cables, or you want to break out (split) switch ports, configure the breakout ports; see Switch Port Attributes.

Test Cable Connectivity

By default, Cumulus Linux disables all data plane ports (every Ethernet port except the management interface, eth0). To test cable connectivity, administratively enable physical ports.

cumulus@switch:~$ nv set interface swp1
cumulus@switch:~$ nv config apply

cumulus@switch:~$ nv set interface swp1-52
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo ip link set swp1 up

cumulus@switch:~$ sudo su -
cumulus@switch:~$ for i in /sys/class/net/*; do iface=`basename $i`; if [[ $iface == swp* ]]; then ip link set $iface up fi done

Configure Layer 2 Ports

Cumulus Linux does not put all ports into a bridge by default. To create a bridge and configure one or more front panel ports as members of the bridge:

cumulus@switch:~$ nv set interface swp1 bridge domain br_default
cumulus@switch:~$ nv config apply

cumulus@switch:~$ nv set interface swp1-3,swp6,swp14-20 bridge domain br_default
cumulus@switch:~$ nv config apply

...
auto br_default
iface br_default
    bridge-ports swp1
...

...
auto br_default
iface br_default
    bridge-ports glob swp1-10 swp12 glob swp14-20
...

cumulus@switch:~$ sudo ifquery -a

cumulus@switch:~$ sudo ifup -a

For more information about Ethernet bridges, see Ethernet Bridging - VLANs.

Configure Layer 3 Ports

You can configure a front panel port or bridge interface as a layer 3 port.

cumulus@switch:~$ nv set interface swp1 ip address 10.0.0.0/31
cumulus@switch:~$ nv config apply

cumulus@switch:~$ nv set interface swp1-2 bridge domain br_default
cumulus@switch:~$ nv set bridge domain br_default vlan 10
cumulus@switch:~$ nv set interface vlan10 ip address 10.1.10.2/24
cumulus@switch:~$ nv set bridge domain br_default untagged 1
cumulus@switch:~$ nv config apply

auto swp1
iface swp1
  address 10.0.0.0/31

auto br_default
iface br_default
    address 10.1.10.2/24
    bridge-ports swp1 swp2
    bridge-pvid 1

cumulus@switch:~$ sudo ifquery -a

cumulus@switch:~$ sudo ifup -a

Configure a Loopback Interface

Cumulus Linux has a preconfigured loopback interface. When the switch boots up, the loopback interface, called lo, is up and assigned an IP address of 127.0.0.1.

To add an IP address to a loopback interface, configure the lo interface:

cumulus@switch:~$ nv set interface lo ip address 10.10.10.1/32
cumulus@switch:~$ nv config apply

auto lo
iface lo inet loopback
    address 10.10.10.1

You can add multiple loopback addresses. For more information, see Interface Configuration and Management.

cumulus@switch:~$ nv config save

Show Platform and System Settings

• To show the hostname of the switch, the time zone, and the version of Cumulus Linux running on the switch, run the NVUE nv show system command.
• To show switch platform information, such as the ASIC model, CPU, hard disk drive size, RAM size, and port layout, run the NVUE nv show platform hardware command.

Next Steps

You are now ready to configure the switch according to your needs. This guide provides separate sections that describe how to configure system, layer 1, layer 2, layer 3, and network virtualization settings. Each section includes example configurations and pre-built demos.

For a deep dive into the NVUE object model that provides a CLI to simplify configuration, see NVUE.

This section describes how to manage, install, and upgrade Cumulus Linux on your switch.

The Cumulus Linux operating system resides on a switch as a disk image. This section discusses how to manage the image.

To install a new Cumulus Linux image, refer to Installing a New Cumulus Linux Image. To upgrade Cumulus Linux, refer to Upgrading Cumulus Linux.

Reprovision the System (Restart the Installer)

Reprovisioning the system deletes all system data from the switch.

To stage an ONIE installer from the network (where ONIE automatically locates the installer), run the onie-select -i command. You must reboot the switch to start the install process.

cumulus@switch:~$ sudo onie-select -i
WARNING:
WARNING: Operating System install requested.
WARNING: This will wipe out all system data.
WARNING:
Are you sure (y/N)? y
Enabling install at next reboot...done.
Reboot required to take effect.

To cancel a pending reinstall operation, run the onie-select -c command:

cumulus@switch:~$ sudo onie-select -c
Cancelling pending install at next reboot...done.

To stage an installer located in a specific location, run the onie-install -i <location> command. You can specify a local, absolute or relative path, an HTTP or HTTPS server, SCP or FTP server. You can also stage a Zero Touch Provisioning (ZTP) script along with the installer. You typically use the onie-install command with the -a option to activate installation. If you do not specify the -a option, you must reboot the switch to start the installation process.

The following example stages the installer located at http://203.0.113.10/image-installer together with the ZTP script located at http://203.0.113.10/ztp-script and activates installation and ZTP:

cumulus@switch:~$ sudo onie-install -i http://203.0.113.10/image-installer
cumulus@switch:~$ sudo onie-install -z http://203.0.113.10/ztp-script
cumulus@switch:~$ sudo onie-install -a

You can also specify these options together in the same command. For example:

cumulus@switch:~$ sudo onie-install -i http://203.0.113.10/image-installer -z http://203.0.113.10/ztp-script -a

To see more onie-install options, run man onie-install.

Migrate from Cumulus Linux to ONIE (Uninstall All Images and Remove the Configuration)

To remove all installed images and configurations, and return the switch to its factory defaults, run the onie-select -k command.

cumulus@switch:~$ sudo onie-select -k
WARNING:
WARNING: Operating System uninstall requested.
WARNING: This will wipe out all system data.
WARNING:
Are you sure (y/N)? y
Enabling uninstall at next reboot...done.
Reboot required to take effect.

You must reboot the switch to start the uninstallation process.

cumulus@switch:~$ sudo onie-select -c
Cancelling pending uninstall at next reboot...done.

Boot Into Rescue Mode

If your system becomes unresponsive, you can correct certain issues by booting into ONIE rescue mode, which uses unmounted file systems. You can use various Cumulus Linux utilities to try and resolve a problem.

To reboot the system into ONIE rescue mode, run the onie-select -r command:

cumulus@switch:~$ sudo onie-select -r
WARNING:
WARNING: Rescue boot requested.
WARNING:
Are you sure (y/N)? y
Enabling rescue at next reboot...done.
Reboot required to take effect.

cumulus@switch:~$ sudo onie-select -c
Cancelling pending rescue at next reboot...done.

Inspect the Image File

The Cumulus Linux image file is executable. From a running switch, you can display, extract, and verify the contents of the image file.

To display the contents of the Cumulus Linux image file, pass the info option to the image file. For example, to display the contents of an image file called onie-installer located in the /var/lib/cumulus/installer directory:

cumulus@switch:~$ sudo /var/lib/cumulus/installer/onie-installer info
Verifying image checksum ...OK.
Preparing image archive ... OK.
Control File Contents
=====================
Description: Cumulus Linux 4.1.0
Release: 4.1.0
Architecture: amd64
Switch-Architecture: bcm-amd64
Build-Id: dirtyz224615f
Build-Date: 2019-05-17T16:34:22+00:00
Build-User: clbuilder
Homepage: http://www.cumulusnetworks.com/
Min-Disk-Size: 1073741824
Min-Ram-Size: 536870912
mkimage-version: 0.11.111_gbcf0

To extract the contents of the image file, use with the extract <path> option. For example, to extract an image file called onie-installer located in the /var/lib/cumulus/installer directory to the mypath directory:

cumulus@switch:~$ sudo /var/lib/cumulus/installer/onie-installer extract mypath
total 181860
-rw-r--r-- 1 4000 4000       308 May 16 19:04 control
drwxr-xr-x 5 4000 4000      4096 Apr 26 21:28 embedded-installer
-rw-r--r-- 1 4000 4000  13273936 May 16 19:04 initrd
-rw-r--r-- 1 4000 4000   4239088 May 16 19:04 kernel
-rw-r--r-- 1 4000 4000 168701528 May 16 19:04 sysroot.tar

To verify the contents of the image file, use with the verify option. For example, to verify the contents of an image file called onie-installer located in the /var/lib/cumulus/installer directory:

cumulus@switch:~$ sudo /var/lib/cumulus/installer/onie-installer verify
Verifying image checksum ...OK.
Preparing image archive ... OK.
./cumulus-linux-bcm-amd64.bin.1: 161: ./cumulus-linux-bcm-amd64.bin.1: onie-sysinfo: not found
Verifying image compatibility ...OK.
Verifying system ram ...OK.

Related Information

You can install a new Cumulus Linux image using ONIE, an open source project (equivalent to PXE on servers) that enables the installation of network operating systems (NOS) on bare metal switches.

Before you install Cumulus Linux, the switch can be in two different states:

• The switch does not contain an image (the switch is only running ONIE).
• Cumulus Linux is already on the switch but you want to use ONIE to reinstall Cumulus Linux or upgrade to a newer version.

The sections below describe some of the different ways you can install the Cumulus Linux image. Steps show how to install directly from ONIE (if no image is on the switch) and from Cumulus Linux (if the image is already on the switch). For additional methods to find and install the Cumulus Linux image, see the ONIE Design Specification.

You can download a Cumulus Linux image from the NVIDIA Enterprise support portal.

In the following procedures:

• You can name your Cumulus Linux image using any of the ONIE naming schemes mentioned here.
• Run the sudo onie-install -h command to show the ONIE installer options.

Install Using a DHCP/Web Server With DHCP Options

To install Cumulus Linux using a DHCP or web server with DHCP options, set up a DHCP/web server on your laptop and connect the eth0 management port of the switch to your laptop. After you connect the cable, the installation proceeds as follows:

1. The switch boots up and requests an IP address (DHCP request).

2. The DHCP server acknowledges and responds with DHCP option 114 and the location of the installation image.

3. ONIE downloads the Cumulus Linux image, installs, and reboots.

   You are now running Cumulus Linux.

Here is an example DHCP configuration with an ISC DHCP server:

subnet 172.0.24.0 netmask 255.255.255.0 {
  range 172.0.24.20 172.0.24.200;
  option default-url = "http://172.0.24.14/onie-installer-x86_64";
}

Here is an example DHCP configuration with dnsmasq (static address assignment):

dhcp-host=sw4,192.168.100.14,6c:64:1a:00:03:ba,set:sw4
dhcp-option=tag:sw4,114,"http://roz.rtplab.test/onie-installer-x86_64"

If you do not have a web server, you can use this free Apache example.

Install Using a DHCP/Web Server without DHCP Options

Follow the steps below if you can log into the switch on a serial console (ONIE), or log in on the console or with ssh (Install from Cumulus Linux).

Install Using a Web Server With no DHCP

Follow the steps below if you can log into the switch on a serial console (ONIE), or you can log in on the console or with ssh (Install from Cumulus Linux) but no DHCP server is available.

Install Using FTP Without a Web Server

Follow the steps below if your laptop is on the same network as the switch eth0 interface but no DHCP server is available.

Install Using a Local File

Follow the steps below to install the Cumulus Linux image referencing a local file.

Install Using a USB Drive

Follow the steps below to install the Cumulus Linux image using a USB drive.

Prepare for USB Installation

1. From the NVIDIA Enterprise support portal, download the appropriate Cumulus Linux image for your platform.

2. From a computer, prepare your USB drive by formatting it using one of the supported formats: FAT32, vFAT or EXT2.

   ▼ Optional: Prepare a USB Drive inside Cumulus Linux

   a. Insert your USB drive into the USB port on the switch running Cumulus Linux and log in to the switch. Examine output from cat /proc/partitions and sudo fdisk -l [device] to determine the location of your USB drive. For example, sudo fdisk -l /dev/sdb.

   These instructions assume your USB drive is the /dev/sdb device, which is typical if you insert the USB drive after the machine is already booted. However, if you insert the USB drive during the boot process, it is possible that your USB drive is the /dev/sda device. Make sure to modify the commands below to use the proper device for your USB drive.

   b. Create a new partition table on the USB drive. If the parted utility is not on the system, install it with sudo -E apt-get install parted.

   sudo parted /dev/sdb mklabel msdos
   

   c. Create a new partition on the USB drive:

   sudo parted /dev/sdb -a optimal mkpart primary 0% 100%
   

   d. Format the partition to your filesystem of choice using one of the examples below:

   sudo mkfs.ext2 /dev/sdb1
   sudo mkfs.msdos -F 32 /dev/sdb1
   sudo mkfs.vfat /dev/sdb1
   

   To use mkfs.msdos or mkfs.vfat, you need to install the dosfstools package from the Debian software repositories, as they are not included by default.

   e. To continue installing Cumulus Linux, mount the USB drive to move files:

   sudo mkdir /mnt/usb
   sudo mount /dev/sdb1 /mnt/usb
   

3. Copy the Cumulus Linux image to the USB drive, then rename the image file to onie-installer-x86_64.

   You can also use any of the ONIE naming schemes mentioned here.

   When using a MAC or Windows computer to rename the installation file, the file extension can still be present. Make sure you remove the file extension so that ONIE can detect the file.

4. Insert the USB drive into the switch, then prepare the switch for installation:

   • If the switch is offline, connect to the console and power on the switch.
   • If the switch is already online in ONIE, use the reboot command.

   SSH sessions to the switch get dropped after this step. To complete the remaining instructions, connect to the console of the switch. Cumulus Linux switches display their boot process to the console; you need to monitor the console specifically to complete the next step.

5. Monitor the console and select the ONIE option from the first GRUB screen shown below.

   

6. Cumulus Linux on x86 uses GRUB chainloading to present a second GRUB menu specific to the ONIE partition. No action is necessary in this menu to select the default option ONIE: Install OS.

   

7. The switch recognizes the USB drive and mounts it automatically. Cumulus Linux installation begins.

8. After installation completes, the switch automatically reboots into the newly installed instance of Cumulus Linux.

ONIE Installation Options

You can run several installer command line options from ONIE to perform basic switch configuration automatically after installation completes and Cumulus Linux boots for the first time. These options enable you to:

• Set a unique password for the cumulus user
• Provide an initial network configuration
• Execute a ZTP script to perform necessary configuration

ONIE:/ # wget http://myserver.datacenter.com/cumulus-linux-4.4.0-mlx-amd64.bin
ONIE:/ # chmod 755 cumulus-linux-4.4.0-mlx-amd64.bin
ONIE:/ # ./cumulus-linux-4.4.0-mlx-amd64.bin --password 'MyP4$$word'

You can run more than one option in the same command.

Set the cumulus User Password

The default cumulus user account password is cumulus. When you log into Cumulus Linux for the first time, you must provide a new password for the cumulus account, then log back into the system.

To automate this process, you can specify a new password from the command line of the installer with the --password '<clear text-password>' option. For example, to change the default cumulus user password to MyP4$$word:

ONIE:/ # ./cumulus-linux-4.4.0-mlx-amd64.bin --password 'MyP4$$word'

To provide a hashed password instead of a clear text password, use the --hashed-password '<hash>' option. An encrypted hash maintains a secure management network.

1. Generate a sha-512 password hash with the following openssl command. The example command generates a sha-512 password hash for the password MyP4$$word.

   user@host:~$ openssl passwd -6 'MyP4$$word'
   6$LXOrvmOkqidBGqu7$dy0dpYYllekNKOY/9LLrobWA4iGwL4zHsgG97qFQWAMZ3ZzMeyz11JcqtgwKDEgYR6RtjfDtdPCeuj8eNzLnS.
   

2. Specify the new password from the command line of the installer with the --hashed-password '<hash>' command:

   ONIE:/ # ./cumulus-linux-4.4.0-mlx-amd64.bin  --hashed-password '6$LXOrvmOkqidBGqu7$dy0dpYYllekNKOY/9LLrobWA4iGwL4zHsgG97qFQWAMZ3ZzMeyz11JcqtgwKDEgYR6RtjfDtdPCeuj8eNzLnS.'
   

Provide Initial Network Configuration

To provide initial network configuration automatically when Cumulus Linux boots for the first time after installation, use the --interfaces-file <filename> option. For example, to copy the contents of a file called network.intf into the /etc/network/interfaces file and run the ifreload -a command:

ONIE:/ # ./cumulus-linux-4.4.0-mlx-amd64.bin  --interfaces-file network.intf

Execute a ZTP Script

To run a ZTP script that contains commands to execute after Cumulus Linux boots for the first time after installation, use the --ztp <filename> option. For example, to run a ZTP script called initial-conf.ztp:

ONIE:/ # ./cumulus-linux-4.4.0-mlx-amd64.bin --ztp initial-conf.ztp

The ZTP script must contain the CUMULUS-AUTOPROVISIONING string near the beginning of the file and must reside on the ONIE filesystem. Refer to Zero Touch Provisioning - ZTP.

If you use the --ztp option together with any of the other command line options, the ZTP script takes precedence and the switch ignores other command line options.

Change the Default BIOS Password

To provide a layer of security and to prevent unauthorized access to the switch, NVIDIA recommends you change the default BIOS password. The default BIOS password is admin.

To change the default BIOS password:

1. During system boot, press Ctrl+B through the serial console while the BIOS version prints.

   

2. From the Security menu, select Administrator Password.

3. Follow the prompts.

Edit the Cumulus Linux Image (Advanced)

The Cumulus Linux disk image file contains a BASH script that includes a set of variables. You can set these variables to be able to install a fully configured system with a single image file.

...
CL_INSTALLER_PASSWORD=''
CL_INSTALLER_HASHED_PASSWORD=''
CL_INSTALLER_LICENSE=''
CL_INSTALLER_INTERFACES_FILENAME=''
CL_INSTALLER_INTERFACES_CONTENT=''
CL_INSTALLER_ZTP_FILENAME=''
CL_INSTALLER_QUIET=""
CL_INSTALLER_FORCEINST=""
CL_INSTALLER_INTERACTIVE=""
CL_INSTALLER_EXTRACTDIR=""
CL_INSTALLER_PAYLOAD_SHA256="72a8c3da28cda7a610e272b67fa1b3a54c50248bf6abf720f73ff3d10e79ae76"

head -20 cumulus-linux-4.4.0-mlx-amd64.bin > cumulus-linux-4.4.0-mlx-amd64.bin.1

sed -e '1,20d' cumulus-linux-4.4.0-mlx-amd64.bin > cumulus-linux-4.4.0-mlx-amd64.bin.2

cat cumulus-linux-4.4.0-mlx-amd64.bin.1 cumulus-linux-4.4.0-mlx-amd64.bin.2 > cumulus-linux-4.4.0-mlx-amd64.bin.final

...
CL_INSTALLER_PAYLOAD_SHA256='d14a028c2a3a2bc9476102bb288234c415a2b01f828ea62ac332e42f'
CL_INSTALLER_PASSWORD='MyP4$$word'
CL_INSTALLER_HASHED_PASSWORD=''
CL_INSTALLER_LICENSE='customer@datacenter.com|4C3YMCACDiK0D/EnrxlXpj71FBBNAg4Yrq+brza4ZtJFCInvalid'
CL_INSTALLER_INTERFACES_FILENAME=''
CL_INSTALLER_INTERFACES_CONTENT='# This file describes the network interfaces available on your system and how to activate them.

source /etc/network/interfaces.d/*.intf

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet dhcp
	vrf mgmt

auto bridge
iface bridge
    bridge-ports swp1 swp2
    bridge-pvid 1
    bridge-vids 10 11
    bridge-vlan-aware yes

auto mgmt
iface mgmt
	address 127.0.0.1/8
	address ::1/128
	vrf-table auto
'
CL_INSTALLER_ZTP_FILENAME=''
...

Secure Boot

Secure Boot validates each binary image loaded during system boot with key signatures that correspond to a stored trusted key in firmware.

Secure Boot settings are in the BIOS Security menu. To access BIOS, press Ctrl+B through the serial console during system boot while the BIOS version prints:

To access the BIOS menu, use admin which is the default BIOS password:

NVIDIA recommends changing the default BIOS password; navigate to Security and select Administrator Password.

To validate or change the Secure Boot mode, navigate to Security and select Secure Boot:

In the Secure Boot menu, you can enable and disable Secure Boot mode. To install an unsigned version of Cumulus Linux or access ONIE without a prompt for a username and password, set Secure Boot to disabled:

To access ONIE when Secure Boot is enabled, authentication is necessary. The default username and password are both root:

​ONIE: Rescue Mode ...
Platform  : x86_64-mlnx_x86-r0
Version   : 2021.02-5.3.0006-rc3-115200
Build Date: 2021-05-20T14:27+03:00
Info: Mounting kernel filesystems... done.

Info: Mounting ONIE-BOOT on /mnt/onie-boot ...
[   17.011057] ext4 filesystem being mounted at /mnt/onie-boot supports timestamps until 2038 (0x7fffffff)
Info: Mounting EFI System on /boot/efi ...
Info: BIOS mode: UEFI
Info: Using eth0 MAC address: b8:ce:f6:3c:62:06
Info: eth0:  Checking link... up.
Info: Trying DHCPv4 on interface: eth0
ONIE: Using DHCPv4 addr: eth0: 10.20.84.226 / 255.255.255.0
Starting: klogd... done.
Starting: dropbear ssh daemon... done.
Starting: telnetd... done.
discover: Rescue mode detected.  Installer disabled.

Please press Enter to activate this console. To check the install status inspect /var/log/onie.log.
Try this:  tail -f /var/log/onie.log

** Rescue Mode Enabled **
login: root
Password: root
ONIE:~ #

To validate the Secure Boot status of a system from Cumulus Linux, run the mokutil --sb-state command.

cumulus@leaf01:mgmt:~$ mokutil --sb-state
SecureBoot enabled

Related Information

• ONIE Design Specification
• NVIDIA Enterprise support portal
• Managing Cumulus Linux Disk Images

This topic describes how to upgrade Cumulus Linux on your switch.

Consider deploying, provisioning, configuring, and upgrading switches using automation, even with small networks or test labs. During the upgrade process, you can upgrade dozens of devices in a repeatable manner. Using tools like Ansible, Chef, or Puppet for configuration management greatly increases the speed and accuracy of the next major upgrade; these tools also enable you to quickly swap failed switch hardware.

Before You Upgrade

Back up Configuration Files

Understanding the location of configuration data is important for successful upgrades, migrations, and backup. As with other Linux distributions, the /etc directory is the primary location for all configuration data in Cumulus Linux. The following list contains the files you need to back up and migrate to a new release. Make sure you examine any changed files. Make the following files and directories part of a backup strategy.

The following commands verify which files have changed compared to the previous Cumulus Linux install. Be sure to back up any changed files.

• Run the sudo dpkg --verify command to show a list of changed files.
• Run the egrep -v '^$|^#|=""$' /etc/default/isc-dhcp-* command to see if any of the generated /etc/default/isc-* files have changed.

Back Up and Restore Configuration with NVUE

You can back up and restore the configuration file with NVUE only if you used NVUE commands to configure the switch you want to upgrade.

To back up and restore the configuration file:

1. Save the configuration to the /etc/nvue.d/startup.yaml file with the nv config save command:

   cumulus@switch:~$ nv config save
   saved
   

2. Copy the /etc/nvue.d/startup.yaml file off the switch to a different location.

3. After upgrade is complete, restore the configuration. Copy the /etc/nvue.d/startup.yaml file to the switch, then run the nv config apply startup command:

   cumulus@switch:~$ nv config apply startup
   applied
   

For information about the NVUE object model and commands, see NVIDIA User Experience - NVUE.

Create a cl-support File

Before and after you upgrade the switch, run the cl-support script to create a cl-support archive file. The file is a compressed archive of useful information for troubleshooting. If you experience any issues during upgrade, you can send this archive file to the Cumulus Linux support team to investigate.

1. Create the cl-support archive file with the cl-support command:

cumulus@switch:~$ sudo cl-support

2. Copy the cl-support file off the switch to a different location.

3. After upgrade is complete, run the cl-support command again to create a new archive file:

cumulus@switch:~$ sudo cl-support

Upgrade Cumulus Linux

You can upgrade Cumulus Linux in one of two ways:

• Install a Cumulus Linux image of the new release, using ONIE.
• Upgrade only the changed packages using the sudo -E apt-get update and sudo -E apt-get upgrade command.

Cumulus Linux also provides ISSU to upgrade an active switch with minimal disruption to the network. See In-Service-System-Upgrade-ISSU.

Install a Cumulus Linux Image or Upgrade Packages?

The decision to upgrade Cumulus Linux by either installing a Cumulus Linux image or upgrading packages depends on your environment and your preferences. Here are some recommendations for each upgrade method.

Install a Cumulus Linux image if you are performing a rolling upgrade in a production environment and if you are using up-to-date and comprehensive automation scripts. This upgrade method enables you to choose the exact release to which you want to upgrade and is the only method available to upgrade your switch to a new release train (for example, from 4.4.3 to 5.6.0).

Be aware of the following when installing the Cumulus Linux image:

• Installing a Cumulus Linux image is destructive; any configuration files on the switch are not saved; copy them to a different server before you start the Cumulus Linux image install.
• You must move configuration data to the new OS using ZTP or automation while the OS is first booted, or soon afterwards using out-of-band management.
• Moving a configuration file can cause issues.
• Identifying all the locations of configuration data is not always an easy task. See Before You Upgrade Cumulus Linux above.
• Merge conflicts with configuration file changes in the new release sometimes go undetected.
• If configuration files do not restore correctly, you cannot ssh to the switch from in-band management. Use out-of-band connectivity (eth0 or console).
• You must reinstall and reconfigure third-party applications after upgrade.

Run package upgrade if you are upgrading from Cumulus Linux 5.0.0 to a later 5.x release, or if you use third-party applications (package upgrade does not replace or remove third-party applications, unlike the Cumulus Linux image install).

Be aware of the following when upgrading packages:

• You cannot upgrade the switch to a new release train. For example, you cannot upgrade the switch from 4.x to 5.x.
• You can only use package upgrade to upgrade a switch with an image install to a maximum of two releases; for example, you can package upgrade a switch running the Cumulus Linux 5.4 image to 5.5 or 5.6 (5.4 plus two releases).
• The sudo -E apt-get upgrade command might restart or stop services as part of the upgrade process.
• The sudo -E apt-get upgrade command might disrupt core services by changing core service dependency packages.
• After you upgrade, account UIDs and GIDs created by packages might be different on different switches, depending on the configuration and package installation history.
• Cumulus Linux does not support the sudo -E apt-get dist-upgrade command. Be sure to use sudo -E apt-get upgrade when upgrading packages.
• You can check the base image with the grep RELEASE /etc/image-release syntax.
• Occasionally, a release contains a base OS upgrade and does not support package upgrade; release notes indicate when a release does not support package upgrade.

Cumulus Linux Image Install (ONIE)

ONIE is an open source project (equivalent to PXE on servers) that enables the installation of network operating systems (NOS) on a bare metal switch.

To upgrade the switch:

1. Back up the configurations off the switch.

2. Download the Cumulus Linux image.

3. Install the Cumulus Linux image with the onie-install -a -i <image-location> command, which boots the switch into ONIE. The following example command installs the image from a web server, then reboots the switch. There are additional ways to install the Cumulus Linux image, such as using FTP, a local file, or a USB drive. For more information, see Installing a New Cumulus Linux Image.

   cumulus@switch:~$ sudo onie-install -a -i http://10.0.1.251/cumulus-linux-5.6.0-mlx-amd64.bin && sudo reboot
   

4. Restore the configuration files to the new release (NVIDIA does not recommend restoring files with automation).

5. Verify correct operation with the old configurations on the new release.

6. Reinstall third party applications and associated configurations.

Package Upgrade

Cumulus Linux completely embraces the Linux and Debian upgrade workflow, where you use an installer to install a base image, then perform any upgrades within that release train with sudo -E apt-get update and sudo -E apt-get upgrade commands. Any packages that have changed after the base install get upgraded in place from the repository. All switch configuration files remain untouched, or in rare cases merged (using the Debian merge function) during the package upgrade.

When you use package upgrade to upgrade your switch, configuration data stays in place during the upgrade. If the new release updates a previously changed configuration file, the upgrade process prompts you to either specify the version you want to use or evaluate the differences.

Disk Space Requirements

Make sure you have enough disk space to perform a package upgrade. Cumulus Linux 5.6.0 requires:

• 0.6GB of free disk space to upgrade from 5.5
• 1.5GB of free disk space to upgrade from 5.4

Before you upgrade, run the sudo df -h command to show how much disk space you are currently using on the switch.

cumulus@switch:~$ sudo df -h
Filesystem      Size   Used   Avail   Use%    Mounted on
udev            7.7G      0    7.7G     0%    /dev
tmpfs           1.6G    18M    1.6G     2%    /run
/dev/sda4        28G   7.9G     18G    31%    /
tmpfs           7.7G   277M    7.4G     4%    /dev/shm
tmpfs           5.0M      0    5.0M     0%    /run/lock
tmpfs           7.7G      0    7.7G     0%    /sys/fs/cgroup
tmpfs           7.7G    16K    7.7G     1%    /tmp
overlay          28G   7.9G     18G    31%   

Upgrade the Switch

To upgrade the switch using package upgrade:

1. Back up the configurations from the switch.

2. Fetch the latest update metadata from the repository.

   cumulus@switch:~$ sudo -E apt-get update
   

3. Review potential upgrade issues (in some cases, upgrading new packages might also upgrade additional existing packages due to dependencies).

   cumulus@switch:~$ sudo -E apt-get upgrade --dry-run
   

4. Upgrade all the packages to the latest distribution.

   cumulus@switch:~$ sudo -E apt-get upgrade
   

   If you do not need to reboot the switch after the upgrade completes, the upgrade ends, restarts all upgraded services, and logs messages in the /var/log/syslog file similar to the ones shown below. In the examples below, the process only upgrades the frr package.

   Policy: Service frr.service action stop postponed
   Policy: Service frr.service action start postponed
   Policy: Restarting services: frr.service
   Policy: Finished restarting services
   Policy: Removed /usr/sbin/policy-rc.d
   Policy: Upgrade is finished
   

   If the upgrade process encounters changed configuration files that have new versions in the release to which you are upgrading, you see a message similar to this:

   Configuration file '/etc/frr/daemons'
   ==> Modified (by you or by a script) since installation.
   ==> Package distributor has shipped an updated version.
   What would you like to do about it ? Your options are:
   Y or I : install the package maintainer's version
   N or O : keep your currently-installed version
   D : show the differences between the versions
   Z : start a shell to examine the situation
   The default action is to keep your current version.
   *** daemons (Y/I/N/O/D/Z) [default=N] ?
   

   • To see the differences between the currently installed version and the new version, type D.
   • To keep the currently installed version, type N. The new package version installs with the suffix .dpkg-dist (for example, /etc/frr/daemons.dpkg-dist). When the upgrade completes and before you reboot, merge your changes with the changes from the newly installed file.
   • To install the new version, type I. Your currently installed version has the suffix .dpkg-old.
   • Cumulus Linux includes /etc/apt/sources.list in the cumulus-archive-keyring package. During upgrade, you must select if you want the new version from the package or the existing file.

   When the upgrade is complete, you can search for the files with the sudo find / -mount -type f -name '*.dpkg-*' command.

   If you see errors for expired GPG keys that prevent you from upgrading packages, follow the steps in Upgrading Expired GPG Keys.

5. Reboot the switch if the upgrade messages indicate that you need to perform a system restart.

   cumulus@switch:~$ sudo -E apt-get upgrade
   ... upgrade messages here ...
   
   *** Caution: Service restart prior to reboot could cause unpredictable behavior
   *** System reboot required ***
   cumulus@switch:~$ sudo reboot
   

6. Verify correct operation with the old configurations on the new version.

Upgrade Notes

Package upgrade always updates to the latest available release in the Cumulus Linux repository. For example, if you are currently running Cumulus Linux 5.0.0 and run the sudo -E apt-get upgrade command on that switch, the packages upgrade to the latest releases in the latest 5.x release.

Because Cumulus Linux is a collection of different Debian Linux packages, be aware of the following:

• The /etc/os-release and /etc/lsb-release files update to the currently installed Cumulus Linux release when you upgrade the switch using either package upgrade or Cumulus Linux image install. For example, if you run sudo -E apt-get upgrade and the latest Cumulus Linux release on the repository is 5.6.0, these two files display the release as 5.6.0 after the upgrade.
• The /etc/image-release file updates only when you run a Cumulus Linux image install. Therefore, if you run a Cumulus Linux image install of Cumulus Linux 5.5.0, followed by a package upgrade to 5.5.1 using sudo -E apt-get upgrade, the /etc/image-release file continues to display Cumulus Linux 5.5.0, which is the originally installed base image.

Upgrade Switches in an MLAG Pair

If you are using MLAG to dual connect two switches in your environment, follow the steps below to upgrade the switches.

Roll Back a Cumulus Linux Installation

Even the most well planned and tested upgrades can result in unforeseen problems and sometimes the best solution is to roll back to the previous state. These main strategies require detailed planning and execution:

• Flatten and rebuild. If the OS becomes unusable, you can use orchestration tools to reinstall the previous OS release from scratch and then rebuild the configuration automatically.
• Restore to a previous state using a backup configuration captured before the upgrade.

The method you employ is specific to your deployment strategy. Providing detailed steps for each scenario is outside the scope of this document.

Third Party Packages

If you install any third party applications on a Cumulus Linux switch, configuration data is typically installed in the /etc directory, but it is not guaranteed. It is your responsibility to understand the behavior and configuration file information of any third party packages installed on the switch.

After you upgrade using a full Cumulus Linux image install, you need to reinstall any third party packages or any Cumulus Linux add-on packages.

Related Information

• Upgrades: Network Device Worldview and Linux Host Worldview Comparison
• Automation Solutions
• ONIE Design Specification
• Multi-Chassis Link Aggregation - MLAG
• Zero Touch Provisioning - ZTP

To manage additional applications in the form of packages and to install the latest updates, use the Advanced Packaging Tool (apt).

Update the Package Cache

To work correctly, apt relies on a local cache listing of the available packages. You must populate the cache initially, then periodically update it with sudo -E apt-get update:

cumulus@switch:~$ sudo -E apt-get update
Ign:1 copy:/var/lib/cumulus/cumulus-local-apt-archive cumulus-local-apt-archive InRelease
Get:2 copy:/var/lib/cumulus/cumulus-local-apt-archive cumulus-local-apt-archive Release [1,115 B]
Ign:3 copy:/var/lib/cumulus/cumulus-local-apt-archive cumulus-local-apt-archive Release.gpg
Get:4 http://security.debian.org buster/updates InRelease [65.4 kB]                 
Hit:5 http://deb.debian.org/debian buster InRelease                                 
Get:6 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
Get:7 http://deb.debian.org/debian buster-backports InRelease [46.7 kB]
Get:8 http://deb.debian.org/debian buster-updates/main Sources.diff/Index [8,608 B] 
Get:9 http://deb.debian.org/debian buster-updates/main amd64 Packages.diff/Index [8,608 B]
Get:10 http://deb.debian.org/debian buster-updates/main Sources 2021-09-28-1420.03.pdiff [185 B]
Get:10 http://deb.debian.org/debian buster-updates/main Sources 2021-09-28-1420.03.pdiff [185 B]
Get:11 http://deb.debian.org/debian buster-updates/main amd64 Packages 2021-09-28-1420.03.pdiff [184 B]               
Get:11 http://deb.debian.org/debian buster-updates/main amd64 Packages 2021-09-28-1420.03.pdiff [184 B]               
Get:12 http://deb.debian.org/debian buster-backports/main Sources.diff/Index [27.8 kB]                     
Get:13 http://deb.debian.org/debian buster-backports/main amd64 Packages.diff/Index [27.8 kB]                         
Hit:14 http://apps3.cumulusnetworks.com/repos/deb CumulusLinux-4 InRelease                                            
Get:15 http://security.debian.org buster/updates/main Sources [200 kB]                             
Get:16 http://security.debian.org buster/updates/main amd64 Packages [305 kB]              
Hit:17 http://apt.cumulusnetworks.com/repo CumulusLinux-4-latest InRelease                       
Get:18 http://deb.debian.org/debian buster-backports/main Sources 2021-10-02-0801.17.pdiff [681 B]
Get:19 http://deb.debian.org/debian buster-backports/main Sources 2021-10-02-1405.24.pdiff [31 B]
Get:19 http://deb.debian.org/debian buster-backports/main Sources 2021-10-02-1405.24.pdiff [31 B]
Get:20 http://deb.debian.org/debian buster-backports/main amd64 Packages 2021-10-02-1405.24.pdiff [178 B]
Get:20 http://deb.debian.org/debian buster-backports/main amd64 Packages 2021-10-02-1405.24.pdiff [178 B]
Fetched 744 kB in 1s (982 kB/s)
Reading package lists... Done

List Available Packages

After the cache populates, use the apt-cache command to search the cache and find the packages of interest or to get information about an available package.

Here are examples of the search and show sub-commands:

cumulus@switch:~$ apt-cache search tcp
collectd-core - statistics collection and monitoring daemon (core system)
fakeroot - tool for simulating superuser privileges
iperf - Internet Protocol bandwidth measuring tool
iptraf-ng - Next Generation Interactive Colorful IP LAN Monitor
libfakeroot - tool for simulating superuser privileges - shared libraries
libfstrm0 - Frame Streams (fstrm) library
libibverbs1 - Library for direct userspace use of RDMA (InfiniBand/iWARP)
libnginx-mod-stream - Stream module for Nginx
libqt4-network - Qt 4 network module
librtr-dev - Small extensible RPKI-RTR-Client C library - development files
librtr0 - Small extensible RPKI-RTR-Client C library
libwiretap8 - network packet capture library -- shared library
libwrap0 - Wietse Venema's TCP wrappers library
libwrap0-dev - Wietse Venema's TCP wrappers library, development files
netbase - Basic TCP/IP networking system
nmap-common - Architecture independent files for nmap
nuttcp - network performance measurement tool
openssh-client - secure shell (SSH) client, for secure access to remote machines
openssh-server - secure shell (SSH) server, for secure access from remote machines
openssh-sftp-server - secure shell (SSH) sftp server module, for SFTP access from remote machines
python-dpkt - Python 2 packet creation / parsing module for basic TCP/IP protocols
rsyslog - reliable system and kernel logging daemon
socat - multipurpose relay for bidirectional data transfer
tcpdump - command-line network traffic analyzer

cumulus@switch:~$ apt-cache show tcpdump
Package: tcpdump
Version: 4.9.3-1~deb10u1
Installed-Size: 1109
Maintainer: Romain Francoise <rfrancoise@debian.org>
Architecture: amd64
Replaces: apparmor-profiles-extra (<< 1.12~)
Depends: libc6 (>= 2.14), libpcap0.8 (>= 1.5.1), libssl1.1 (>= 1.1.0)
Suggests: apparmor (>= 2.3)
Breaks: apparmor-profiles-extra (<< 1.12~)
Size: 400060
SHA256: 3a63be16f96004bdf8848056f2621fbd863fadc0baf44bdcbc5d75dd98331fd3
SHA1: 2ab9f0d2673f49da466f5164ecec8836350aed42
MD5sum: 603baaf914de63f62a9f8055709257f3
Description: command-line network traffic analyzer
 This program allows you to dump the traffic on a network. tcpdump
 is able to examine IPv4, ICMPv4, IPv6, ICMPv6, UDP, TCP, SNMP, AFS
 BGP, RIP, PIM, DVMRP, IGMP, SMB, OSPF, NFS and many other packet
 types.
 .
 It can be used to print out the headers of packets on a network
 interface, filter packets that match a certain expression. You can
 use this tool to track down network problems, to detect attacks
 or to monitor network activities.
Description-md5: f01841bfda357d116d7ff7b7a47e8782
Homepage: http://www.tcpdump.org/
Multi-Arch: foreign
Section: net
Priority: optional
Filename: pool/upstream/t/tcpdump/tcpdump_4.9.3-1~deb10u1_amd64.deb

List Packages Installed on the System

The apt-cache command shows information about all the packages available in the repository. To see which packages are actually installed on your system with the version, run the following command.

cumulus@switch:~$ nv show platform software installed
Installed Package   description                                                      package                  version
-----------------   -------------------                                              ----------               --------------------
acpi                displays information on ACPI devices                             acpi                     1.7-1.1
acpi-support-base   scripts for handling base ACPI events such as the power button   acpi-support-base        0.142-8
acpid               Advanced Configuration and Power Interface event daemon          acpid                    1:2.0.31-1
...

cumulus@switch:~$ dpkg -l
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                Version                   Architecture Description
+++-===================-=========================-============-=================================
ii  acpi                1.7-1.1                   amd64        displays information on ACPI devices
ii  acpi-support-base   0.142-8                   all          scripts for handling base ACPI events such as th
ii  acpid               1:2.0.31-1                amd64        Advanced Configuration and Power Interface event
ii  adduser             3.118                     all          add and remove users and groups
ii  apt                 1.8.2                     amd64        commandline package manager
ii  arping              2.19-6                    amd64        sends IP and/or ARP pings (to the MAC address)
ii  arptables           0.0.4+snapshot20181021-4  amd64        ARP table administration
...

Show the Version of a Package

To show the version of a specific package installed on the system:

cumulus@switch:~$ nv show platform software installed vrf
             running              applied  pending  description
-----------  -------------------  -------  -------  -----------
description  Linux tools for VRF                    Description
package      vrf                                    Package
version      1.0-cl5.6.0u9                         Version

cumulus@switch:~$ dpkg -l vrf
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name       Version      Architecture Description
+++-==========-============-============-=================================
ii  vrf        1.0-cl5.6.0u9    amd64        Linux tools for VRF

Upgrade Packages

To upgrade all the packages installed on the system to their latest versions, run the following commands:

cumulus@switch:~$ sudo -E apt-get update
cumulus@switch:~$ sudo -E apt-get upgrade

The system lists the packages for upgrade and prompts you to continue.

The above commands upgrade all installed versions with their latest versions but do not install any new packages.

Add New Packages

To add a new package, first ensure the package is not already on the system:

cumulus@switch:~$ dpkg -l | grep <name of package>

• If the package is already on the system, you can update the package from the Cumulus Linux repository as part of the package upgrade process, which upgrades all packages on the system. See Upgrade Packages above.
• If the package is not already on the system, add it by running sudo -E apt-get install <name of package>. This retrieves the package from the Cumulus Linux repository and installs it on your system together with any other dependent packages. The following example adds the tcpreplay package to the system:

cumulus@switch:~$ sudo -E apt-get update
cumulus@switch:~$ sudo -E apt-get install tcpreplay
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
tcpreplay
0 upgraded, 1 newly installed, 0 to remove and 1 not upgraded.
Need to get 436 kB of archives.
After this operation, 1008 kB of additional disk space will be used
...

You can install several packages at the same time:

cumulus@switch:~$ sudo -E apt-get install <package1> <package2> <package3>

Add Packages From Another Repository

As shipped, Cumulus Linux searches the Cumulus Linux repository for available packages. You can add additional repositories to search by adding them to the list of sources that apt-get consults. See man sources.list for more information.

NVIDIA adds features or makes bug fixes to certain packages; do not replace these packages with versions from other repositories.

If you want to install packages that are not in the Cumulus Linux repository, the procedure is the same as above, but with one additional step.

Installing packages outside of the Cumulus Linux repository requires the use of sudo -E apt-get; however, depending on the package, you can use easy-install and other commands.

To install a new package, complete the following steps:

1. Run the dpkg command to ensure that the package is not already installed on the system:

   cumulus@switch:~$ dpkg -l | grep <name of package>
   

2. If the package is already on the system, ensure it is the version you need. If it is an older version, update the package from the Cumulus Linux repository:

   cumulus@switch:~$ sudo -E apt-get update
   cumulus@switch:~$ sudo -E apt-get install <name of package>
   cumulus@switch:~$ sudo -E apt-get upgrade
   

3. If the package is not on the system, the package source location is not in the /etc/apt/sources.list file. Edit and add the appropriate source to the file. For example, add the following if you want a package from the Debian repository that is not in the Cumulus Linux repository:

   deb http://http.us.debian.org/debian buster main
   deb http://security.debian.org/ buster/updates main
   

   Otherwise, /etc/apt/sources.list lists the repository but comments it out. To uncomment the repository, remove the # at the start of the line, then save the file.

4. Run sudo -E apt-get update, then install the package and upgrade:

   cumulus@switch:~$ sudo -E apt-get update
   cumulus@switch:~$ sudo -E apt-get install <name of package>
   cumulus@switch:~$ sudo -E apt-get upgrade
   

Add Packages from the Cumulus Linux Local Archive

Cumulus Linux contains a local archive embedded in the Cumulus Linux image. This archive, cumulus-local-apt-archive, contains the packages you need to install ifplugd, LDAP, RADIUS or TACACS+ without a network connection.

The archive contains the following packages:

• audisp-tacplus
• ifplugd
• libdaemon0
• libnss-ldapd
• libnss-mapuser
• libnss-tacplus
• libpam-ldapd
• libpam-radius-auth
• libpam-tacplus
• libtac2
• libtacplus-map1
• nslcd

Add these packages with apt-get update && apt-get install, as described above.

Related Information

• Debian GNU/Linux FAQ, Ch 8 Package management tools
• man pages for apt-get, dpkg, sources.list, apt_preferences

Use ZTP to deploy network devices in large-scale environments. On first boot, Cumulus Linux runs ZTP, which executes the provisioning automation that deploys the device for its intended role in the network.

The provisioning framework allows you to execute a one-time, user-provided script. You can develop this script using a variety of automation tools and scripting languages. You can also use it to add the switch to a configuration management (CM) platform such as Puppet, Chef, CFEngine or a custom, proprietary tool.

While developing and testing the provisioning logic, you can use the ztp command in Cumulus Linux to run your provisioning script manually on a device.

ZTP in Cumulus Linux can run automatically in one of the following ways, in this order:

1. Through a local file
2. Using a USB drive inserted into the switch (ZTP-USB)
3. Through DHCP

Use a Local File

ZTP only looks one time for a ZTP script on the local file system when the switch boots. ZTP searches for an install script that matches an ONIE-style waterfall in /var/lib/cumulus/ztp, looking for the most specific name first, and ending at the most generic:

• 'cumulus-ztp-' + architecture + '-' + vendor + '_' + model + '-r' + revision
• 'cumulus-ztp-' + architecture + '-' + vendor + '_' + model
• 'cumulus-ztp-' + vendor + '_' + model
• 'cumulus-ztp-' + architecture
• 'cumulus-ztp'

You can also trigger the ZTP process manually by running the ztp --run <URL> command, where the URL is the path to the ZTP script.

Use a USB Drive

At minimum, the script must:

• Install the Cumulus Linux operating system.
• Copy over a basic configuration to the switch.
• Restart the switch or the relevant services to get switchd up and running with that configuration.

Follow these steps to perform ZTP using a USB drive:

1. Copy the installation image to the USB drive.
2. The ztp process searches the root filesystem of the newly mounted drive for filenames matching an ONIE-style waterfall (see the patterns and examples above), looking for the most specific name first, and ending at the most generic.
3. ZTP parses the contents of the script to ensure it contains the CUMULUS-AUTOPROVISIONING flag (see example scripts).

ZTP Over DHCP

If the ztp process does not discover a local ONIE script or applicable USB drive, it checks DHCP every ten seconds for up to five minutes for the presence of a ZTP URL specified in /var/run/ztp.dhcp. The URL can be any of HTTP, HTTPS, FTP, or TFTP.

For ZTP using DHCP, provisioning initially takes place over the management network and initiates through a DHCP hook. A DHCP option specifies a configuration script. The ZTP process requests this script from the Web server and the script executes locally.

The ZTP process over DHCP follows these steps:

1. The first time you boot Cumulus Linux, eth0 makes a DHCP request. By default, Cumulus Linux sends DHCP option 60 (the vendor class identifier) with the value cumulus-linux x86_64 to identify itself to the DHCP server.
2. The DHCP server offers a lease to the switch.
3. If option 239 is in the response, the ZTP process starts.
4. The ZTP process requests the contents of the script from the URL, sending additional HTTP headers containing details about the switch.
5. ZTP parses the contents of the script to ensure it contains the CUMULUS-AUTOPROVISIONING flag (see example scripts).
6. If provisioning is necessary, the script executes locally on the switch with root privileges.
7. ZTP examines the return code of the script. If the return code is 0, ZTP marks the provisioning state as complete in the autoprovisioning configuration file.

Trigger ZTP Over DHCP

If you have not yet provisioned the switch, you can trigger the ZTP process over DHCP when eth0 uses DHCP and one of the following events occur:

• The switch boots.
• You plug a cable into or unplug a cable from the eth0 port.
• You disconnect, then reconnect the switch power cord.

You can also run the ztp --run <URL> command, where the URL is the path to the ZTP script.

Configure the DHCP Server

During the DHCP process over eth0, Cumulus Linux requests DHCP option 239. This option specifies the custom provisioning script.

For example, the /etc/dhcp/dhcpd.conf file for an ISC DHCP server looks like:

option cumulus-provision-url code 239 = text;

  subnet 192.0.2.0 netmask 255.255.255.0 {
  range 192.0.2.100 192.168.0.200;
  option cumulus-provision-url "http://192.0.2.1/demo.sh";
}

In addition, you can specify the hostname of the switch with the host-name option:

subnet 192.168.0.0 netmask 255.255.255.0 {
  range 192.168.0.100 192.168.0.200;
  option cumulus-provision-url "http://192.0.2.1/demo.sh";
  host dc1-tor-sw1 { hardware ethernet 44:38:39:00:1a:6b; fixed-address 192.168.0.101; option host-name "dc1-tor-sw1"; }
}

DHCP on Front Panel Ports

ZTP runs DHCP on all the front panel switch ports and on any active interface. ZTP assesses the list of active ports on every retry cycle. When it receives the DHCP lease and option 239 is present in the response, ZTP starts to execute the script.

Inspect HTTP Headers

The following HTTP headers in the request to the web server retrieve the provisioning script:

Header                        Value                 Example
------                        -----                 -------
User-Agent                                          CumulusLinux-AutoProvision/0.4
CUMULUS-ARCH                  CPU architecture      x86_64
CUMULUS-BUILD                                       5.1.0
CUMULUS-MANUFACTURER                                odm
CUMULUS-PRODUCTNAME                                 switch_model
CUMULUS-SERIAL                                      XYZ123004
CUMULUS-BASE-MAC                                    44:38:39:FF:40:94
CUMULUS-MGMT-MAC                                    44:38:39:FF:00:00
CUMULUS-VERSION                                     5.1.0
CUMULUS-PROV-COUNT                                  0
CUMULUS-PROV-MAX                                    32

Write ZTP Scripts

# CUMULUS-AUTOPROVISIONING

The script must contain the CUMULUS-AUTOPROVISIONING flag. You can include this flag in a comment or remark; you do not need to echo or write the flag to stdout.

You can write the script in any language that Cumulus Linux supports, such as:

• Perl
• Python
• Ruby
• Shell

The script must return an exit code of 0 upon success to mark the process as complete in the autoprovisioning configuration file.

The following script installs Cumulus Linux from a USB drive and applies a configuration:

#!/bin/bash
function error() {
  echo -e "\e[0;33mERROR: The ZTP script failed while running the command $BASH_COMMAND at line $BASH_LINENO.\e[0m" >&2
  exit 1
}

# Log all output from this script
exec >> /var/log/autoprovision 2>&1
date "+%FT%T ztp starting script $0"

trap error ERR

#Add Debian Repositories
echo "deb http://http.us.debian.org/debian buster main" >> /etc/apt/sources.list
echo "deb http://security.debian.org/ buster/updates main" >> /etc/apt/sources.list

#Update Package Cache
apt-get update -y

#Load interface config from usb
cp ${ZTP_USB_MOUNTPOINT}/interfaces /etc/network/interfaces

#Load port config from usb
#   (if breakout cables are used for certain interfaces)
cp ${ZTP_USB_MOUNTPOINT}/ports.conf /etc/cumulus/ports.conf

#Reload interfaces to apply loaded config
ifreload -a

# CUMULUS-AUTOPROVISIONING
exit 0

Continue Provisioning

Typically ZTP exits after executing the script locally and does not continue. To continue with provisioning so that you do not have to intervene manually or embed an Ansible callback into the script, you can add the CUMULUS-AUTOPROVISION-CASCADE directive.

Best Practices

ZTP scripts come in different forms and frequently perform the same tasks. As BASH is the most common language for ZTP scripts, use the following BASH snippets to perform common tasks with robust error checking.

Set the Default Cumulus User Password

The default cumulus user account password is cumulus. When you log into Cumulus Linux for the first time, you must provide a new password for the cumulus account, then log back into the system.

Add the following function to your ZTP script to change the default cumulus user account password to a clear-text password. The example changes the password cumulus to MyP4$$word.

function set_password(){
     # Unexpire the cumulus account
     passwd -x 99999 cumulus
     # Set the password
     echo 'cumulus:MyP4$$word' | chpasswd
}
set_password

If you have an insecure management network, set the password with an encrypted hash instead of a clear-text password.

• First, generate a sha-512 password hash with the following python commands. The example commands generate a sha-512 password hash for the password MyP4$$word.

  user@host:~$ python3 -c "import crypt; print(crypt.crypt('MyP4$$word',salt=crypt.mksalt()))"
  $6$hs7OPmnrfvLNKfoZ$iB3hy5N6Vv6koqDmxixpTO6lej6VaoKGvs5E8p5zNo4tPec0KKqyQnrFMII3jGxVEYWntG9e7Z7DORdylG5aR/
  

• Then, add the following function to the ZTP script to change the default cumulus user account password:

  function set_password(){
       # Unexpire the cumulus account
       passwd -x 99999 cumulus
       # Set the password
       usermod -p '$6$hs7OPmnrfvLNKfoZ$iB3hy5N6Vv6koqDmxixpTO6lej6VaoKGvs5E8p5zNo4tPec0KKqyQnrFMII3jGxVEYWntG9e7Z7DORdylG5aR/' cumulus
  }
  set_password
  

Test DNS Name Resolution

DNS names are frequently used in ZTP scripts. The ping_until_reachable function tests that each DNS name resolves into a reachable IP address. Call this function with each DNS target used in your script before you use the DNS name elsewhere in your script.

The following example shows how to call the ping_until_reachable function in the context of a larger task.

function ping_until_reachable(){
    last_code=1
    max_tries=30
    tries=0
    while [ "0" != "$last_code" ] && [ "$tries" -lt "$max_tries" ]; do
        tries=$((tries+1))
        echo "$(date) INFO: ( Attempt $tries of $max_tries ) Pinging $1 Target Until Reachable."
        ping $1 -c2 &> /dev/null
        last_code=$?
            sleep 1
    done
    if [ "$tries" -eq "$max_tries" ] && [ "$last_code" -ne "0" ]; then
        echo "$(date) ERROR: Reached maximum number of attempts to ping the target $1 ."
        exit 1
    fi
}

Check the Cumulus Linux Release

The following script segment demonstrates how to check which Cumulus Linux release is running and upgrades the node if the release is not the target release. If the release is the target release, normal ZTP tasks execute. This script calls the ping_until_reachable script (described above) to make sure the server holding the image server and the ZTP script is reachable.

function init_ztp(){
    #do normal ZTP tasks
}

CUMULUS_TARGET_RELEASE=5.1.0
CUMULUS_CURRENT_RELEASE=$(cat /etc/lsb-release  | grep RELEASE | cut -d "=" -f2)
IMAGE_SERVER_HOSTNAME=webserver.example.com
IMAGE_SERVER= "http:// "$IMAGE_SERVER_HOSTNAME "/ "$CUMULUS_TARGET_RELEASE ".bin "
ZTP_URL= "http:// "$IMAGE_SERVER_HOSTNAME "/ztp.sh "

if [ "$CUMULUS_TARGET_RELEASE" != "$CUMULUS_CURRENT_RELEASE" ]; then
    ping_until_reachable $IMAGE_SERVER_HOSTNAME
    /usr/cumulus/bin/onie-install -fa -i $IMAGE_SERVER -z $ZTP_URL && reboot
else
    init_ztp && reboot
fi
exit 0

Apply Management VRF Configuration

If you apply a management VRF in your script, either apply it last or reboot instead. If you do not apply a management VRF last, you need to prepend any commands that require eth0 to communicate out with /usr/bin/ip vrf exec mgmt; for example, /usr/bin/ip vrf exec mgmt apt-get update -y.

Perform Ansible Provisioning Callbacks

After initially configuring a node with ZTP, use Provisioning Callbacks to inform Ansible Tower or AWX that the node is ready for more detailed provisioning. The following example demonstrates how to use a provisioning callback:

/usr/bin/curl -H "Content-Type:application/json" -k -X POST --data '{"host_config_key":"'somekey'"}' -u username:password http://ansible.example.com/api/v2/job_templates/1111/callback/

Disable the DHCP Hostname Override Setting

Make sure to disable the DHCP hostname override setting in your script.

function set_hostname(){
    # Remove DHCP Setting of Hostname
    sed s/'SETHOSTNAME="yes"'/'SETHOSTNAME="no"'/g -i /etc/dhcp/dhclient-exit-hooks.d/dhcp-sethostname
    hostnamectl set-hostname $1
}

Test ZTP Scripts

Use these commands to test and debug your ZTP scripts.

You can use verbose mode to debug your script and see where your script fails. Include the -v option when you run ZTP:

cumulus@switch:~$ sudo ztp -v -r http://192.0.2.1/demo.sh
Attempting to provision via ZTP Manual from http://192.0.2.1/demo.sh

Broadcast message from root@dell-s6010-01 (ttyS0) (Tue May 10 22:44:17 2016):  

ZTP: Attempting to provision via ZTP Manual from http://192.0.2.1/demo.sh
ZTP Manual: URL response code 200
ZTP Manual: Found Marker CUMULUS-AUTOPROVISIONING
ZTP Manual: Executing http://192.0.2.1/demo.sh
error: ZTP Manual: Payload returned code 1
error: Script returned failure

To see results of the most recent ZTP execution, you can run the ztp -s command.

cumulus@switch:~$ ztp -s
ZTP INFO:

State              enabled
Version            1.0
Result             Script Failure
Date               Mon 20 May 2019 09:31:27 PM UTC
Method             ZTP DHCP
URL                http://192.0.2.1/demo.sh

If ZTP runs when the switch boots and not manually, you can run the systemctl -l status ztp.service then journalctl -l -u ztp.service to see if any failures occur:

cumulus@switch:~$ sudo systemctl -l status ztp.service
● ztp.service - Cumulus Linux ZTP
    Loaded: loaded (/lib/systemd/system/ztp.service; enabled)
    Active: failed (Result: exit-code) since Wed 2016-05-11 16:38:45 UTC; 1min 47s ago
        Docs: man:ztp(8)
    Process: 400 ExecStart=/usr/sbin/ztp -b (code=exited, status=1/FAILURE)
    Main PID: 400 (code=exited, status=1/FAILURE)

May 11 16:37:45 cumulus ztp[400]: ztp [400]: ZTP USB: Device not found
May 11 16:38:45 dell-s6010-01 ztp[400]: ztp [400]: ZTP DHCP: Looking for ZTP Script provided by DHCP
May 11 16:38:45 dell-s6010-01 ztp[400]: ztp [400]: Attempting to provision via ZTP DHCP from http://192.0.2.1/demo.sh
May 11 16:38:45 dell-s6010-01 ztp[400]: ztp [400]: ZTP DHCP: URL response code 200
May 11 16:38:45 dell-s6010-01 ztp[400]: ztp [400]: ZTP DHCP: Found Marker CUMULUS-AUTOPROVISIONING
May 11 16:38:45 dell-s6010-01 ztp[400]: ztp [400]: ZTP DHCP: Executing http://192.0.2.1/demo.sh
May 11 16:38:45 dell-s6010-01 ztp[400]: ztp [400]: ZTP DHCP: Payload returned code 1
May 11 16:38:45 dell-s6010-01 ztp[400]: ztp [400]: Script returned failure
May 11 16:38:45 dell-s6010-01 systemd[1]: ztp.service: main process exited, code=exited, status=1/FAILURE
May 11 16:38:45 dell-s6010-01 systemd[1]: Unit ztp.service entered failed state.
cumulus@switch:~$
cumulus@switch:~$ sudo journalctl -l -u ztp.service --no-pager
-- Logs begin at Wed 2016-05-11 16:37:42 UTC, end at Wed 2016-05-11 16:40:39 UTC. --
May 11 16:37:45 cumulus ztp[400]: ztp [400]: /var/lib/cumulus/ztp: Sate Directory does not exist. Creating it...
May 11 16:37:45 cumulus ztp[400]: ztp [400]: /var/run/ztp.lock: Lock File does not exist. Creating it...
May 11 16:37:45 cumulus ztp[400]: ztp [400]: /var/lib/cumulus/ztp/ztp_state.log: State File does not exist. Creating it...
May 11 16:37:45 cumulus ztp[400]: ztp [400]: ZTP LOCAL: Looking for ZTP local Script
May 11 16:37:45 cumulus ztp[400]: ztp [400]: ZTP LOCAL: Waterfall search for /var/lib/cumulus/ztp/cumulus-ztp-x86_64-dell_s6010_s1220-rUNKNOWN
May 11 16:37:45 cumulus ztp[400]: ztp [400]: ZTP LOCAL: Waterfall search for /var/lib/cumulus/ztp/cumulus-ztp-x86_64-dell_s6010_s1220
May 11 16:37:45 cumulus ztp[400]: ztp [400]: ZTP LOCAL: Waterfall search for /var/lib/cumulus/ztp/cumulus-ztp-x86_64-dell
May 11 16:37:45 cumulus ztp[400]: ztp [400]: ZTP LOCAL: Waterfall search for /var/lib/cumulus/ztp/cumulus-ztp-x86_64
May 11 16:37:45 cumulus ztp[400]: ztp [400]: ZTP LOCAL: Waterfall search for /var/lib/cumulus/ztp/cumulus-ztp
May 11 16:37:45 cumulus ztp[400]: ztp [400]: ZTP USB: Looking for unmounted USB devices
May 11 16:37:45 cumulus ztp[400]: ztp [400]: ZTP USB: Parsing partitions
May 11 16:37:45 cumulus ztp[400]: ztp [400]: ZTP USB: Device not found
May 11 16:38:45 dell-s6010-01 ztp[400]: ztp [400]: ZTP DHCP: Looking for ZTP Script provided by DHCP
May 11 16:38:45 dell-s6010-01 ztp[400]: ztp [400]: Attempting to provision via ZTP DHCP from http://192.0.2.1/demo.sh
May 11 16:38:45 dell-s6010-01 ztp[400]: ztp [400]: ZTP DHCP: URL response code 200
May 11 16:38:45 dell-s6010-01 ztp[400]: ztp [400]: ZTP DHCP: Found Marker CUMULUS-AUTOPROVISIONING
May 11 16:38:45 dell-s6010-01 ztp[400]: ztp [400]: ZTP DHCP: Executing http://192.0.2.1/demo.sh
May 11 16:38:45 dell-s6010-01 ztp[400]: ztp [400]: ZTP DHCP: Payload returned code 1
May 11 16:38:45 dell-s6010-01 ztp[400]: ztp [400]: Script returned failure
May 11 16:38:45 dell-s6010-01 systemd[1]: ztp.service: main process exited, code=exited, status=1/FAILURE
May 11 16:38:45 dell-s6010-01 systemd[1]: Unit ztp.service entered failed state.

Instead of running journalctl, you can see the log history by running:

cumulus@switch:~$ cat /var/log/syslog | grep ztp
2016-05-11T16:37:45.132583+00:00 cumulus ztp [400]: /var/lib/cumulus/ztp: State Directory does not exist. Creating it...
2016-05-11T16:37:45.134081+00:00 cumulus ztp [400]: /var/run/ztp.lock: Lock File does not exist. Creating it...
2016-05-11T16:37:45.135360+00:00 cumulus ztp [400]: /var/lib/cumulus/ztp/ztp_state.log: State File does not exist. Creating it...
2016-05-11T16:37:45.185598+00:00 cumulus ztp [400]: ZTP LOCAL: Looking for ZTP local Script
2016-05-11T16:37:45.485084+00:00 cumulus ztp [400]: ZTP LOCAL: Waterfall search for /var/lib/cumulus/ztp/cumulus-ztp-x86_64-dell_s6010_s1220-rUNKNOWN
2016-05-11T16:37:45.486394+00:00 cumulus ztp [400]: ZTP LOCAL: Waterfall search for /var/lib/cumulus/ztp/cumulus-ztp-x86_64-dell_s6010_s1220
2016-05-11T16:37:45.488385+00:00 cumulus ztp [400]: ZTP LOCAL: Waterfall search for /var/lib/cumulus/ztp/cumulus-ztp-x86_64-dell
2016-05-11T16:37:45.489665+00:00 cumulus ztp [400]: ZTP LOCAL: Waterfall search for /var/lib/cumulus/ztp/cumulus-ztp-x86_64
2016-05-11T16:37:45.490854+00:00 cumulus ztp [400]: ZTP LOCAL: Waterfall search for /var/lib/cumulus/ztp/cumulus-ztp
2016-05-11T16:37:45.492296+00:00 cumulus ztp [400]: ZTP USB: Looking for unmounted USB devices
2016-05-11T16:37:45.493525+00:00 cumulus ztp [400]: ZTP USB: Parsing partitions
2016-05-11T16:37:45.636422+00:00 cumulus ztp [400]: ZTP USB: Device not found
2016-05-11T16:38:43.372857+00:00 cumulus ztp [1805]: Found ZTP DHCP Request
2016-05-11T16:38:45.696562+00:00 cumulus ztp [400]: ZTP DHCP: Looking for ZTP Script provided by DHCP
2016-05-11T16:38:45.698598+00:00 cumulus ztp [400]: Attempting to provision via ZTP DHCP from http://192.0.2.1/demo.sh
2016-05-11T16:38:45.816275+00:00 cumulus ztp [400]: ZTP DHCP: URL response code 200
2016-05-11T16:38:45.817446+00:00 cumulus ztp [400]: ZTP DHCP: Found Marker CUMULUS-AUTOPROVISIONING
2016-05-11T16:38:45.818402+00:00 cumulus ztp [400]: ZTP DHCP: Executing http://192.0.2.1/demo.sh
2016-05-11T16:38:45.834240+00:00 cumulus ztp [400]: ZTP DHCP: Payload returned code 1
2016-05-11T16:38:45.835488+00:00 cumulus ztp [400]: Script returned failure
2016-05-11T16:38:45.876334+00:00 cumulus systemd[1]: ztp.service: main process exited, code=exited, status=1/FAILURE
2016-05-11T16:38:45.879410+00:00 cumulus systemd[1]: Unit ztp.service entered failed state.

If you see that the issue is a script failure, you can modify the script and then run ZTP manually using ztp -v -r <URL/path to that script>, as above.

cumulus@switch:~$ sudo ztp -v -r http://192.0.2.1/demo.sh
Attempting to provision via ZTP Manual from http://192.0.2.1/demo.sh

Broadcast message from root@dell-s6010-01 (ttyS0) (Tue May 10 22:44:17 2019):  

ZTP: Attempting to provision via ZTP Manual from http://192.0.2.1/demo.sh
ZTP Manual: URL response code 200
ZTP Manual: Found Marker CUMULUS-AUTOPROVISIONING
ZTP Manual: Executing http://192.0.2.1/demo.sh
error: ZTP Manual: Payload returned code 1
error: Script returned failure
cumulus@switch:~$ sudo ztp -s
State         enabled
Version       1.0
Result        Script Failure
Date          Mon 20 May 2019 09:31:27 PM UTC
Method        ZTP Manual
URL           http://192.0.2.1/demo.sh

Use the following command to check syslog for information about ZTP:

cumulus@switch:~$ sudo grep -i ztp /var/log/syslog

Common ZTP Script Errors

Could not find referenced script/interpreter in downloaded payload

cumulus@leaf01:~$ sudo cat /var/log/syslog | grep ztp
2018-04-24T15:06:08.887041+00:00 leaf01 ztp [13404]: Attempting to provision via ZTP Manual from http://192.168.0.254/ztp_oob_windows.sh
2018-04-24T15:06:09.106633+00:00 leaf01 ztp [13404]: ZTP Manual: URL response code 200
2018-04-24T15:06:09.107327+00:00 leaf01 ztp [13404]: ZTP Manual: Found Marker CUMULUS-AUTOPROVISIONING
2018-04-24T15:06:09.107635+00:00 leaf01 ztp [13404]: ZTP Manual: Executing http://192.168.0.254/ztp_oob_windows.sh
2018-04-24T15:06:09.132651+00:00 leaf01 ztp [13404]: ZTP Manual: Could not find referenced script/interpreter in downloaded payload.
2018-04-24T15:06:14.135521+00:00 leaf01 ztp [13404]: ZTP Manual: Retrying
2018-04-24T15:06:14.138915+00:00 leaf01 ztp [13404]: ZTP Manual: URL response code 200
2018-04-24T15:06:14.139162+00:00 leaf01 ztp [13404]: ZTP Manual: Found Marker CUMULUS-AUTOPROVISIONING
2018-04-24T15:06:14.139448+00:00 leaf01 ztp [13404]: ZTP Manual: Executing http://192.168.0.254/ztp_oob_windows.sh
2018-04-24T15:06:14.143261+00:00 leaf01 ztp [13404]: ZTP Manual: Could not find referenced script/interpreter in downloaded payload.
2018-04-24T15:06:24.147580+00:00 leaf01 ztp [13404]: ZTP Manual: Retrying
2018-04-24T15:06:24.150945+00:00 leaf01 ztp [13404]: ZTP Manual: URL response code 200
2018-04-24T15:06:24.151177+00:00 leaf01 ztp [13404]: ZTP Manual: Found Marker CUMULUS-AUTOPROVISIONING
2018-04-24T15:06:24.151374+00:00 leaf01 ztp [13404]: ZTP Manual: Executing http://192.168.0.254/ztp_oob_windows.sh
2018-04-24T15:06:24.155026+00:00 leaf01 ztp [13404]: ZTP Manual: Could not find referenced script/interpreter in downloaded payload.
2018-04-24T15:06:39.164957+00:00 leaf01 ztp [13404]: ZTP Manual: Retrying
2018-04-24T15:06:39.165425+00:00 leaf01 ztp [13404]: Script returned failure
2018-04-24T15:06:39.175959+00:00 leaf01 ztp [13404]: ZTP script failed. Exiting...

Errors in syslog for ZTP like those shown above often occur if you create or edit the script on a Windows machine. Check to make sure that the \r\n characters are not present in the end-of-line encodings.

Use the cat -v ztp.sh command to view the contents of the script and search for any hidden characters.

root@oob-mgmt-server:/var/www/html# cat -v ./ztp_oob_windows.sh 
#!/bin/bash^M
^M
###################^M
#   ZTP Script^M
###################^M
^M
/usr/cumulus/bin/cl-license -i http://192.168.0.254/license.txt^M
^M
# Clean method of performing a Reboot^M
nohup bash -c 'sleep 2; shutdown now -r "Rebooting to Complete ZTP"' &^M
^M
exit 0^M
^M
# The line below is required to be a valid ZTP script^M
#CUMULUS-AUTOPROVISIONING^M
root@oob-mgmt-server:/var/www/html#

The ^M characters in the output of your ZTP script, as shown above, indicate the presence of Windows end-of-line encodings that you need to remove.

Use the translate (tr) command on any Linux system to remove the '\r' characters from the file.

root@oob-mgmt-server:/var/www/html# tr -d '\r' < ztp_oob_windows.sh > ztp_oob_unix.sh
root@oob-mgmt-server:/var/www/html# cat -v ./ztp_oob_unix.sh 
#!/bin/bash
###################
#   ZTP Script
###################
/usr/cumulus/bin/cl-license -i http://192.168.0.254/license.txt
# Clean method of performing a Reboot
nohup bash -c 'sleep 2; shutdown now -r "Rebooting to Complete ZTP"' &
exit 0
# The line below is required to be a valid ZTP script
#CUMULUS-AUTOPROVISIONING
root@oob-mgmt-server:/var/www/html#

Manually Use the ztp Command

To enable ZTP, use the -e option:

cumulus@switch:~$ sudo ztp -e

To reset ZTP to its original state, use the -R option. This removes the ztp directory and ZTP runs the next time the switch reboots.

cumulus@switch:~$ sudo ztp -R

To disable ZTP, use the -d option:

cumulus@switch:~$ sudo ztp -d

To force provisioning to occur and ignore the status listed in the configuration file, use the -r option:

cumulus@switch:~$ sudo ztp -r cumulus-ztp.sh

To see the current ZTP state, use the -s option:

cumulus@switch:~$ sudo ztp -s
ZTP INFO:
State          disabled
Version        1.0
Result         success
Date           Mon May 20 21:51:04 2019 UTC
Method         Switch manually configured  
URL            None

Considerations

• While you are writing a provisioning script, you sometimes need to reboot the switch.
• You can use the Cumulus Linux onie-select -i command to reprovision the switch and install a network operating system again using ONIE.

This section describes how to configure the following system settings:

• Date and time
• Authentication, authorization, and accounting
• Access Control Lists (ACLs)
• Services and daemons
• switchd
• In Service System Upgrade (ISSU)
• NVIDIA User Experience (NVUE)

NVUE is an object-oriented, schema driven model of a complete Cumulus Linux system (hardware and software) providing a robust API that allows for multiple interfaces to both view (show) and configure (set and unset) any element within a system running the NVUE software.

• For a description of the NVUE object model, go to NVUE Object Model.
• For an overview of the NVUE CLI commands, go to NVUE CLI.
• For information on how to access and use the NVUE API, go to NVUE API.
• For information on how to use NVUE snippets, go to NVUE Snippets.

The NVUE object model definition uses the OpenAPI specification (OAS). Similar to YANG (RFC 6020 and RFC 7950), OAS is a data definition, manipulation, and modeling language (DML) that lets you build model-driven interfaces for both humans and machines. Although the computer networking and telecommunications industry commonly uses YANG (standardized by IETF) as a DML, the adoption of OpenAPI is broader, spanning cloud to compute to storage to IoT and even social media. The OpenAPI Initiative (OAI) consortium leads OpenAPI standardization, a chartered project under the Linux Foundation.

The OAS schema forms the management plane model with which you configure, monitor, and manage the Cumulus Linux switch. The v3.0.2 version of OAS defines the NVUE data model.

Like other systems that use OpenAPI, the NVUE OAS schema defines the endpoints (paths) exposed as RESTful APIs. With these REST APIs, you can perform various create, retrieve, update, delete, and eXecute (CRUDX) operations. The OAS schema also describes the API inputs and outputs (data models).

You can use the NVUE object model in these two ways:

• Through the NVUE REST API, where you run the GET, PATCH, DELETE, and other REST APIs on the NVUE object model endpoints to configure, monitor, and manage the switch. Because of the large user community and maturity of OAS, you can use several popular tools and libraries to create client-side bindings to use the NVUE REST API.
• Through the NVUE CLI, where you configure, monitor and manage the Cumulus Linux network elements. The CLI commands translate to their equivalent REST APIs, which Cumulus Linux then runs on the NVUE object model.

The CLI and the REST API are equivalent in functionality; you can run all management operations from the REST API or the CLI. The NVUE object model drives both the REST API and the CLI management operations. All operations are consistent; for example, the CLI nv show commands reflect any PATCH operation (create) you run through the REST API.

NVUE follows a declarative model, removing context-specific commands and settings. It is structured as a big tree that represents the entire state of a Cumulus Linux instance. At the base of the tree are high level branches representing objects, such as router and interface. Under each of these branches are further branches. As you navigate through the tree, you gain a more specific context. At the leaves of the tree are actual attributes, represented as key-value pairs. The path through the tree is similar to a filesystem path.

Cumulus Linux installs NVUE by default and enables the NVUE service nvued.

The NVUE CLI has a flat structure instead of a modal structure. Therefore, you can run all commands from the primary prompt instead of only in a specific mode.

Command Syntax

NVUE commands all begin with nv and fall into one of three syntax categories:

• Configuration (nv set and nv unset)
• Monitoring (nv show)
• Configuration management (nv config)
• Action commands (nv action)

Command Completion

As you enter commands, you can get help with the valid keywords or options using the tab key. For example, using tab completion with nv set displays the possible options for the command and returns you to the command prompt to complete the command.

cumulus@switch:~$ nv set <<press tab>>
acl        evpn       mlag       qos        service    vrf        
bridge     interface  nve        router     system

cumulus@switch:~$ nv set

Command Question Mark

You can type a question mark (?) after a command to display required information quickly and concisely. When you type ?, NVUE specifies the value type, range, and options with a brief description of each; for example:

cumulus@switch:~$ nv set interface swp1 link state ?
    [Enter]               
    down                   The interface is not ready
    up                     The interface is ready
cumulus@switch:~$ nv set interface swp1 link mtu ?
    <arg>                  (integer:552 - 9216)
cumulus@switch:~$ nv set interface swp1 link speed ?
    <arg>                  (string | enum:10M,100M,1G,10G,25G,40G,50G,100G,200G,40
                           0G,800G,auto)

NVUE also indicates if you need to provide specific values for the command:

cumulus@switch:~$ nv set interface swp1 bridge domain ?
    <domain-id>            Domain (bridge-name)

Command Abbreviation

NVUE supports command abbreviation, where you can type a certain number of characters instead of a whole command to speed up CLI interaction. For example, instead of typing nv show interface, you can type nv sh int.

If the command you type is ambiguous, NVUE shows the reason for the ambiguity so that you can correct the shortcut. For example:

cumulus@switch:~$ nv s i 
Ambiguous Command: 
   set interface 
   show interface 

Command Help

As you enter commands, you can get help with command syntax by entering -h or --help at various points within a command entry. For example, to examine the options available for nv set interface, enter nv set interface -h or nv set interface --help.

cumulus@switch:~$ nv set interface -h
usage: 
  nv [options] set interface <interface-id>

Description:
  interface             Update all interfaces

Identifiers:
  <interface-id>        Interface (interface-name)

Output Options:
  -o <format>, --output <format>
                        Supported formats: json, yaml, auto, constable, end-table, commands (default:auto)
  --color (on|off|auto)
                        Toggle coloring of output (default: auto)
  --paginate (on|off|auto)
                        Whether to send output to a pager (default: off)

General Options:
  -h, --help            Show help.

Command List

You can list all the NVUE commands by running nv list-commands. See List All NVUE Commands below.

Command History

At the command prompt, press the Up Arrow and Down Arrow keys to move back and forth through the list of commands you entered previously. When you find the command you want to use, you can run the command by pressing Enter. You can also modify the command before you run it.

Command Categories

The NVUE CLI has a flat structure; however, the commands are in three functional categories:

• Configuration
• Monitoring
• Configuration Management
• Action

Configuration Commands

The NVUE configuration commands modify switch configuration. You can set and unset configuration options.

The nv set and nv unset commands are in the following categories. Each command group includes subcommands. Use command completion (press the tab key) to list the subcommands.

Monitoring Commands

The NVUE monitoring commands show various parts of the network configuration. For example, you can show the complete network configuration or only interface configuration. The monitoring commands are in the following categories. Each command group includes subcommands. Use command completion (press the tab key) to list the subcommands.

The following example shows the nv show router commands after pressing the tab key, then shows the output of the nv show router bgp command.

cumulus@leaf01:mgmt:~$ nv show router <<tab>>
adaptive-routing  igmp              ospf              pim               ptm               vrrp              
bgp               nexthop-group     pbr               policy            vrr               

cumulus@leaf01:mgmt:~$ nv show router bgp
                                operational  applied  pending      description
------------------------------  -----------  -------  -----------  ----------------------------------------------------------------------
enable                                       off      on           Turn the feature 'on' or 'off'.  The default is 'off'.
autonomous-system                                     none         ASN for all VRFs, if a single AS is in use.  If "none", then ASN mu...
graceful-shutdown                                     off          Graceful shutdown enable will initiate the GSHUT community to be an...
policy-update-timer                                   5            Wait time in seconds before processing updates to policies to ensur...
router-id                                             none         BGP router-id for all VRFs, if a common one is used.  If "none", th...
wait-for-install                                      off          bgp waits for routes to be installed into kernel/asic before advert...
convergence-wait
  establish-wait-time                                 0            Maximum time to wait to establish BGP sessions. Any peers which do...
  time                                                0            Time to wait for peers to send end-of-RIB before router performs pa...
graceful-restart
  mode                                                helper-only  Role of router during graceful restart. helper-only, router is in h...
  path-selection-deferral-time                        360          Used by the restarter as an upper-bounds for waiting for peering es...
  restart-time                                        120          Amount of time taken to restart by router. It is advertised to the...
  stale-routes-time                                   360          Specifies an upper-bounds on how long we retain routes from a resta...
cumulus@leaf01:mgmt:~$ 

Additional options are available for the nv show commands. For example, you can choose the configuration you want to show (pending, applied, startup, or operational). You can also turn on colored output, and paginate specific output.

The following example shows pending BGP graceful restart configuration:

cumulus@switch:~$ nv show router bgp graceful-restart --pending
                              4                  description
----------------------------  -----------------  ----------------------------------------------------------------------
mode                          helper-only        Role of router during graceful restart. helper-only, router is in h...
path-selection-deferral-time  360                Used by the restarter as an upper-bounds for waiting for peeringes...
restart-time                  120                Amount of time taken to restart by router. It is advertised to the...
stale-routes-time             360                Specifies an upper-bounds on how long we retain routes from a resta...

Monitoring Commands and FRR Daemons

If you run an NVUE show command but the corresponding FRR routing daemons are not running on the switch, you see an error message; for example:

• If OSPF is not running when you run nv show vrf <vrf-id> ospf commands, NVUE returns Error: The requested item does not exist because the OSPF deamon is not running in FRR.
• If PIM and IGMP are not running when you run the nv show interface <interface> ip igmp -o json command, NVUE returns Error: The requested item does not exist because the PIM daemon is not running in FRR.
• If PIM is running but IGMP is not running when you the nv show interface <interface> ip igmp group -o json command, NVUE does not return an error message but shows an empty { } response.

Net Show commands

In addition to the nv show commands, Cumulus Linux continues to provide a subset of the NCLU net show commands. Use these commands to get additional views of various parts of your network configuration.

cumulus@leaf01:mgmt:~$ net show 
    bfd            :  Bidirectional forwarding detection
    bgp            :  Border Gateway Protocol
    bridge         :  a layer2 bridge
    clag           :  Multi-Chassis Link Aggregation
    commit         :  apply the commit buffer to the system
    configuration  :  settings, configuration state, etc
    counters       :  net show counters
    debugs         :  Debugs
    dhcp-snoop     :  DHCP snooping for IPv4
    dhcp-snoop6    :  DHCP snooping for IPv6
    dot1x          :  Configure, Enable, Delete or Show IEEE 802.1X EAPOL
    evpn           :  Ethernet VPN
    hostname       :  local hostname
    igmp           :  Internet Group Management Protocol
    interface      :  An interface, such as swp1, swp2, etc.
    ip             :  Internet Protocol version 4/6
    ipv6           :  Internet Protocol version 6
    lldp           :  Link Layer Discovery Protocol
    mpls           :  Multiprotocol Label Switching
    mroute         :  Static unicast routes in MRIB for multicast RPF lookup
    msdp           :  Multicast Source Discovery Protocol
    neighbor       :  A BGP, OSPF, PIM, etc neighbor
    ospf           :  Open Shortest Path First (OSPFv2)
    ospf6          :  Open Shortest Path First (OSPFv3)
    package        :  A Cumulus Linux package name
    pbr            :  Policy Based Routing
    pim            :  Protocol Independent Multicast
    port-mirror    :  port-mirror
    port-security  :  Port security
    ptp            :  Precision Time Protocol
    roce           :  Enable RoCE on all interfaces, default mode is lossless
    rollback       :  revert to a previous configuration state
    route          :  EVPN route information
    route-map      :  Route-map
    snmp-server    :  Configure the SNMP server
    system         :  System
    time           :  Time
    version        :  Version number
    vrf            :  Virtual routing and forwarding
    vrrp           :  Virtual Router Redundancy Protocol

Configuration Management Commands

The NVUE configuration management commands manage and apply configurations.

You can use the NVUE configuration management commands to back up and restore configuration when you upgrade Cumulus Linux on the switch. Refer to Upgrading Cumulus Linux.

Action Commands

The NVUE action commands clear counters, and provide system reboot and TACACS user disconnect options.

List All NVUE Commands

To show the full list of NVUE commands, run nv list-commands. For example:

cumulus@switch:~$ nv list-commands
nv show platform
nv show platform hardware
nv show platform hardware component
nv show platform hardware component <component-id>
nv show platform software
nv show platform software installed
nv show platform software installed <installed-id>
nv show platform capabilities
nv show platform environment
...

You can show the list of commands for a command grouping. For example, to show the list of interface commands:

cumulus@switch:~$ nv list-commands interface
nv show interface
nv show interface <interface-id>
nv show interface <interface-id> ip
nv show interface <interface-id> ip address
nv show interface <interface-id> ip address <ip-prefix-id>
nv show interface <interface-id> ip gateway
nv show interface <interface-id> ip gateway <ip-address-id>
...

Use the tab key to get help for the command lists you want to see. For example, to show the list of command options available for swp1, run the nv list-commands interface swp1 command and press the tab key:

cumulus@switch:~$ nv list-commands interface swp1 <<press tab>>
acl            counters       link           ptp            storm-control  
bond           evpn           lldp           qos            synce          
bridge         ip             pluggable      router         tunnel

To view the NVUE command reference for Cumulus Linux, which describes all the NVUE CLI commands and provides examples, go to the NVUE Command Reference.

NVUE Configuration File

When you save network configuration, NVUE writes the configuration to the /etc/nvue.d/startup.yaml file.

You can edit or replace the contents of the /etc/nvue.d/startup.yaml file. NVUE applies the configuration in the /etc/nvue.d/startup.yaml file during system boot only if the nvue-startup.service is running. If this service is not running, the switch reboots with the same configuration that is running before the reboot.

To start nvue-startup.service:

cumulus@switch:~$ sudo systemctl enable nvue-startup.service
cumulus@switch:~$ sudo systemctl start nvue-startup.service

When you apply a configuration with nv config apply, NVUE also writes to underlying Linux files such as /etc/network/interfaces and /etc/frr/frr.conf. You can view these configuration files; however, do not manually edit them while using NVUE. If you need to configure certain network settings manually or use automation such as Ansible to configure the switch, see Configure NVUE to Ignore Linux Files below.

Configuration Files that NVUE Manages

NVUE manages the following configuration files:

Search for a Specific Configuration

To search for a specific portion of the NVUE configuration, run the nv config find <search string> command. The search shows all items above and below the search string. For example, to search the entire NVUE object model configuration for any mention of ptm:

cumulus@switch:~$ nv config find ptm
- set:
    router:
      ptm:
        enable: off

Configure NVUE to Ignore Linux Files

You can configure NVUE to ignore certain underlying Linux files when applying configuration changes. For example, if you push certain configuration to the switch using Ansible and Jinja2 file templates or you want to use custom configuration for a particular service such as PTP, you can ensure that NVUE never writes to those configuration files.

The following example configures NVUE to ignore the Linux /etc/ptp4l.conf file when applying configuration changes and saves the configuration so it persists after a reboot.

cumulus@switch:~$ nv set system config apply ignore /etc/ptp4l.conf
cumulus@switch:~$ nv config apply
cumulus@switch:~$ nv config save

Configure Auto Save

By default, when you run the nv config apply command to apply a configuration setting, NVUE applies the pending configuration to become the applied configuration but does not update the startup configuration file (/etc/nvue.d/startup.yaml). To save the applied configuration to the startup configuration so that the changes persist after the reboot, you must run the nv config save command. The auto save option lets you save the pending configuration to the startup configuration automatically when you run nv config apply so that you do not have to run the nv config save command.

To enable auto save:

cumulus@switch:~$ nv set system config auto-save enable on
cumulus@switch:~$ nv config apply

To disable auto save, run the nv set system config auto-save enable off command.

Add Configuration Apply Messages

When you run the nv config apply command, you can add a message that describes the configuration updates you make. You can see the message when you run the nv config history command.

To add a configuration apply message, run the nv config apply -m <message> command. If the message includes more than one word, enclose the message in quotes.

cumulus@switch:~$ nv config apply -m "this is my message"

Reset NVUE Configuration to Default Values

To reset the NVUE configuration on the switch back to the default values, run the following command:

cumulus@switch:~$ nv config apply empty

Example Configuration Commands

This section provides examples of how to configure a Cumulus Linux switch using NVUE commands.

Configure the System Hostname

The example below shows the NVUE commands required to change the hostname for the switch to leaf01:

cumulus@switch:~$ nv set system hostname leaf01
cumulus@switch:~$ nv config apply

Configure the System DNS Server

The example below shows the NVUE commands required to define the DNS server for the switch:

cumulus@switch:~$ nv set service dns mgmt server 192.168.200.1
cumulus@switch:~$ nv config apply

Configure an Interface

The example below shows the NVUE commands required to bring up swp1.

cumulus@switch:~$ nv set interface swp1
cumulus@switch:~$ nv config apply 

Configure a Bond

The example below shows the NVUE commands required to configure the front panel port interfaces swp1 thru swp4 to be slaves in bond0.

cumulus@switch:~$ nv set interface bond0 bond member swp1-4
cumulus@switch:~$ nv config apply

Configure a Bridge

The example below shows the NVUE commands required to create a VLAN-aware bridge that contains two switch ports (swp1 and swp2) and includes 3 VLANs; tagged VLANs 10 and 20 and an untagged (native) VLAN of 1.

With NVUE, there is a default bridge called br_default, which has no ports assigned to it. The example below configures this default bridge.

cumulus@switch:~$ nv set interface swp1-2 bridge domain br_default
cumulus@switch:~$ nv set bridge domain br_default vlan 10,20
cumulus@switch:~$ nv set bridge domain br_default untagged 1
cumulus@switch:~$ nv config apply

Configure MLAG

The example below shows the NVUE commands required to configure MLAG on leaf01. The commands:

• Place swp1 into bond1 and swp2 into bond2.
• Configure the MLAG ID to 1 for bond1 and to 2 for bond2.
• Add bond1 and bond2 to the default bridge (br_default).
• Create the inter-chassis bond (swp49 and swp50) and the peer link (peerlink)
• Set the peer link IP address to linklocal, the MLAG system MAC address to 44:38:39:BE:EF:AA, and the backup interface to 10.10.10.2.

cumulus@leaf01:~$ nv set interface bond1 bond member swp1
cumulus@leaf01:~$ nv set interface bond2 bond member swp2
cumulus@leaf01:~$ nv set interface bond1 bond mlag id 1
cumulus@leaf01:~$ nv set interface bond2 bond mlag id 2
cumulus@switch:~$ nv set interface bond1-2 bridge domain br_default 
cumulus@leaf01:~$ nv set interface peerlink bond member swp49-50
cumulus@leaf01:~$ nv set mlag mac-address 44:38:39:BE:EF:AA
cumulus@leaf01:~$ nv set mlag backup 10.10.10.2
cumulus@leaf01:~$ nv set mlag peer-ip linklocal
cumulus@leaf01:~$ nv config apply

Configure BGP Unnumbered

The example below shows the NVUE commands required to configure BGP unnumbered on leaf01. The commands:

• Assign the ASN for this BGP node to 65101.
• Set the router ID to 10.10.10.1.
• Distribute routing information to the peer on swp51.
• Originate prefixes 10.10.10.1/32 from this BGP node.

cumulus@leaf01:~$ nv set router bgp autonomous-system 65101
cumulus@leaf01:~$ nv set router bgp router-id 10.10.10.1
cumulus@leaf01:~$ nv set vrf default router bgp neighbor swp51 remote-as external
cumulus@leaf01:~$ nv set vrf default router bgp address-family ipv4-unicast network 10.10.10.1/32
cumulus@leaf01:~$ nv config apply

Example Monitoring Commands

This section provides monitoring command examples.

Show Installed Software

The following example command lists the software installed on the switch:

cumulus@switch:~$ nv show platform software
Installed Software
=====================
Installed software            description                     package                      version                       
---------------------------   ---------------------------     --------------------------   -----------------------------
acpi                          displays information on ACPI    acpi                         1.7-1.1                       
                              devices                                                                                  
acpi-support-base             scripts for handling base       acpi-support-base            0.142-8                       
                              ACPI events such as the                                                                  
                              power button                                                                             
acpid                         Advanced Configuration and      acpid                        1:2.0.31-1                    
                              Power Interface event daemon                                                             
adduser                       add and remove users and        adduser                      3.118                         
                              groups                                                                                   
apt                           commandline package manager     apt                          1.8.2.3             
...

Show Interface Configuration

The following example command shows the running, applied, and pending swp1 interface configuration.

cumulus@leaf01:~$ nv show interface swp1
                          operational        applied   
------------------------  -----------------  ----------
type                      swp                swp       
[acl]                                                  
bridge                                                 
  [domain]                br_default         br_default
evpn                                                   
  multihoming                                          
    uplink                                   off       
ptp                                                    
  enable                                     off       
router                                                 
  adaptive-routing                                     
    enable                                   off       
  ospf                                                 
    enable                                   off       
  ospf6                                                
    enable                                   off       
  pbr                                                  
    [map]                                              
  pim            
...

Example Configuration Management Commands

This section provides examples of how to use the configuration management commands to apply, save, and detach configurations.

Apply and Save a Configuration

The following example command configures the front panel port interfaces swp1 thru swp4 to be slaves in bond0. The configuration is only in a pending configuration state. The configuration is not applied. NVUE has not yet made any changes to the running configuration.

cumulus@switch:~$ nv set interface bond0 bond member swp1-4

To apply the pending configuration to the running configuration, run the nv config apply command. The configuration does not persist after a reboot.

cumulus@switch:~$ nv config apply

To save the applied configuration to the startup configuration, run the nv config save command. This command overwrites the startup configuration with the applied configuration by writing to the /etc/nvue.d/startup.yaml file. The configuration persists after a reboot.

cumulus@switch:~$ nv config save

Detach a Pending Configuration

The following example configures the IP address of the loopback interface, then detaches the configuration from the current pending configuration. Cumulus Linux saves the detached configuration to a file with a numerical value to distinguish it from other pending configurations.

cumulus@switch:~$ nv set interface lo ip address 10.10.10.1/32
cumulus@switch:~$ nv config detach

View Differences Between Configurations

To view differences between configurations, run the nv config diff command.

To view differences between two detached pending configurations, run the nv config diff «tab» command to list all the current detached pending configurations, then run the nv config diff command with the pending configurations you want to diff.

cumulus@switch:~$ nv config diff <<press tab>>
1        2        3        4        5        6        applied  empty    startup

cumulus@switch:~$ nv config diff 2 3
- unset:
    system:
      wjh:
        channel:
          forwarding:
            trigger:
              l2:

To view differences between the applied configuration and the startup configuration:

cumulus@switch:~$ nv config diff applied startup
- unset:
    interface:
    system:
      wjh:

Replace and Patch a Pending Configuration

The following example replaces the pending configuration with the contents of the YAML configuration file called nv-02/13/2021.yaml located in the /deps directory:

cumulus@switch:~$ nv config replace /deps/nv-02/13/2021.yaml

The following example patches the pending configuration (runs the set or unset commands from the configuration in the nv-02/13/2021.yaml file located in the /deps directory):

cumulus@switch:~$ nv config patch /deps/nv-02/13/2021.yaml

This section discusses how to:

• Set the time zone, and the date and time on the software clock on the switch
• Configure NTP
• Configure PTP

NVUE supports both traditional snippets and flexible snippets:

• Use traditional snippets to add configuration to the /etc/network/interfaces, /etc/frr/frr.conf, /etc/frr/daemons, /etc/cumulus/switchd.conf, /etc/cumulus/datapath/traffic.conf or /etc/ssh/sshd_config files.
• Use flexible snippets to manage any other text file on the system.

Traditional Snippets

Use traditional snippets if you configure Cumulus Linux with NVUE commands, then want to configure a feature that does not yet support the NVUE Object Model. You create a snippet in yaml format, then add the configuration to the file with the nv config patch command.

/etc/frr/frr.conf Snippets

Example 1: Top Level Configuration

NVUE does not support configuring BGP to peer across the default route. The following example configures BGP to peer across the default route from the default VRF:

1. Create a .yaml file with the following traditional snippet:

   cumulus@switch:~$ sudo nano bgp_snippet.yaml
   - set:
       system:
         config:
           snippet:
             frr.conf: |
               ip nht resolve-via-default
   

2. Run the following command to patch the configuration:

   cumulus@switch:~$ nv config patch bgp_snippet.yaml
   

3. Run the nv config apply command to apply the configuration:

   cumulus@switch:~$ nv config apply
   

4. Verify that the configuration exists at the end of the /etc/frr/frr.conf file:

   cumulus@switch:~$ sudo cat /etc/frr/frr.conf
   ...
   ! end of router ospf block
   !---- CUE snippets ----
   ip nht resolve-via-default
   

Example 2: Nested Configuration

NVUE does not support configuring EVPN route targets using auto derived values from RFC 8365. The following example configures BGP to enable RFC 8365 derived router targets:

1. Create a .yaml file with the following traditional snippet:

   cumulus@switch:~$ sudo nano bgp_snippet.yaml
   - set:
       system:
         config:
           snippet:
             frr.conf: |
               router bgp 65517 vrf default
                 address-family l2vpn evpn
                   autort rfc8365-compatible
   

Make sure to use spaces not tabs; the parser expects spaces in yaml format.

2. Run the following command to patch the configuration:

   cumulus@switch:~$ nv config patch bgp_snippet.yaml
   

3. Run the nv config apply command to apply the configuration:

   cumulus@switch:~$ nv config apply
   

4. Verify that the configuration exists at the end of the /etc/frr/frr.conf file:

   cumulus@switch:~$ sudo cat /etc/frr/frr.conf
   ...
   ! end of router bgp 65517 vrf default
   !---- CUE snippets ----
   router bgp 65517 vrf default
   address-family l2vpn evpn
   autort rfc8365-compatible
   

The traditional snippets for FRR write content to the /etc/frr/frr.conf file. When you apply the configuration and snippet with the nv config apply command, the FRR service goes through and reads in the /etc/frr/frr.conf file.

Example 3: EVPN Multihoming FRR Debugging

NVUE does not support configuring FRR debugging for EVPN multihoming. The following example configures FRR debugging:

1. Create a .yaml file and add the following traditional snippet:

   cumulus@switch:~$ sudo nano mh_debug_snippet.yaml
   - set:
       system:
         config:
           snippet:
             frr.conf: |
               debug bgp evpn mh es
               debug bgp evpn mh route
               debug bgp zebra
               debug zebra evpn mh es
               debug zebra evpn mh mac
               debug zebra evpn mh neigh
               debug zebra evpn mh nh
               debug zebra vxlan
   

2. Run the following command to patch the configuration:

   cumulus@switch:~$ nv config patch mh_debug_snippet.yaml
   

3. Run the nv config apply command to apply the configuration:

   cumulus@switch:~$ nv config apply
   

4. Verify that the configuration exists in the /etc/frr/frr.conf file:

   cumulus@switch:~$ sudo cat /etc/frr/frr.conf
   ...
   !---- NVUE snippets ----
   debug bgp evpn mh es
   debug bgp evpn mh route
   debug bgp zebra
   debug zebra evpn mh es
   debug zebra evpn mh mac
   debug zebra evpn mh neigh
   debug zebra evpn mh nh
   debug zebra vxlan
   

The traditional snippets for FRR write content to the /etc/frr/frr.conf file. When you apply the configuration and snippet with the nv config apply command, the FRR service goes through and reads in the /etc/frr/frr.conf file.

/etc/network/interfaces Snippets

MLAG Timers Example

NVUE supports configuring only one of the MLAG service timeouts (initDelay). The following example configures the MLAG peer timeout to 400 seconds:

1. Create a .yaml file and add the following traditional snippet:

   cumulus@switch:~$ sudo nano mlag_snippet.yaml
   - set:
       system:
         config:
           snippet:
             ifupdown2_eni:
               peerlink.4094: |
                 clagd-args --peerTimeout 400
   

2. Run the following command to patch the configuration:

   cumulus@switch:~$ nv config patch mlag_snippet.yaml
   

3. Run the nv config apply command to apply the configuration:

   cumulus@switch:~$ nv config apply
   

4. Verify that the configuration exists in the peerlink.4094 stanza of the /etc/network/interfaces file:

   cumulus@switch:~$ sudo cat /etc/network/interfaces
   ...
   auto peerlink.4094
   iface peerlink.4094
    clagd-args --peerTimeout 400
    clagd-peer-ip linklocal
    clagd-backup-ip 10.10.10.2
    clagd-sys-mac 44:38:39:BE:EF:AA
    clagd-args --initDelay 180
   ...
   

Traditional Bridge Example

NVUE does not support configuring traditional bridges. The following example configures a traditional bridge called br0 with the IP address 11.0.0.10/24. swp1, swp2 are members of the bridge.

1. Create a .yaml file and add the following traditional snippet:

   cumulus@switch:~$ sudo nano bridge_snippet.yaml
   - set:
       system:
        config:
          snippet:
            ifupdown2_eni:
              eni_stanzas: |
                auto br0
                iface br0
                  address 11.0.0.10/24
                  bridge-ports swp1 swp2
                  bridge-vlan-aware no
   

2. Run the following command to patch the configuration:

   cumulus@switch:~$ nv config patch bridge_snippet.yaml
   

3. Run the nv config apply command to apply the configuration:

   cumulus@switch:~$ nv config apply
   

4. Verify that the configuration exists at the end of the /etc/network/interfaces file:

   cumulus@switch:~$ sudo cat /etc/network/interfaces
   ...
   auto br0
   iface br0
     address 11.0.0.10/24
     bridge-ports swp1 swp2
     bridge-vlan-aware no
   

VLAN-aware RSTP Timers Example

NVUE does not support configuring RSTP timers on VLAN-aware bridges. The following example configures non-default RSTP timers for the NVUE default bridge br_default:

1. Create a .yaml file and add the following traditional snippet:

   cumulus@switch:~$ sudo nano vlan-aware_bridge_snippet.yaml
   - set:
       system:
         config:
           snippet:
             ifupdown2_eni:
               br_default: |
                 mstpctl-maxage 10
                 mstpctl-hello 1
                 mstpctl-fdelay 8
   

2. Run the following command to patch the configuration:

   cumulus@switch:~$ nv config patch vlan-aware_bridge_snippet.yaml
   

3. Run the nv config apply command to apply the configuration:

   cumulus@switch:~$ nv config apply
   

4. Verify that the configuration exists at the end of the /etc/network/interfaces file:

   cumulus@switch:~$ sudo cat /etc/network/interfaces
   ...
   auto br_default
   iface br_default
       mstpctl-maxage 10
       mstpctl-hello 1
       mstpctl-fdelay 8
   ...
   

/etc/cumulus/switchd.conf Snippets

NVUE does not provide options to configure link flap detection settings. The following example configures the link flap window to 10 seconds and the link flap threshold to 5 seconds:

1. Create a .yaml file and add the following traditional snippet:

   cumulus@switch:~$ sudo nano switchd_snippet.yaml
   - set:
       system:
         config:
           snippet:
             switchd.conf: |
               link_flap_window = 10
               link_flap_threshold = 5
   

2. Run the following command to patch the configuration:

   cumulus@switch:~$ nv config patch switchd_snippet.yaml
   

3. Run the nv config apply command to apply the configuration:

   cumulus@switch:~$ nv config apply
   

4. Verify that the configuration exists at the end of the /etc/cumulus/switchd.conf file:

   cumulus@switch:~$ sudo cat /etc/cumulus/switchd.conf
   !---- NVUE snippets ----
   link_flap_window = 10
   link_flap_threshold = 5
   

/etc/cumulus/datapath/traffic.conf Snippets

To add data path configuration for the Cumulus Linux switchd module that NVUE does not yet support, create a traffic.conf snippet.

The following example creates a file called traffic_conf_snippet.yaml and enables the resilient hash setting.

1. Create a .yaml file and add the following traditional snippet:

   cumulus@switch:~$ sudo nano traffic_conf_snippet.yaml
   - set:
       system:
         config:
           snippet:
             traffic.conf: |
               resilient_hash_enable = TRUE
   

2. Run the following command to patch the configuration:

   cumulus@switch:~$ nv config patch traffic_conf_snippet.yaml
   

3. Run the nv config apply command to apply the configuration:

   cumulus@switch:~$ nv config apply
   

4. Verify that the configuration exists at the end of the /etc/cumulus/datapath/traffic.conf file:

   cumulus@switch:~$ sudo cat /etc/cumulus/datapath/traffic.conf
   ...
   !---- NVUE snippets ----
   resilient_hash_enable = TRUE
   

/etc/snmp/snmpd.conf Snippets

To add Cumulus Linux SNMP agent configuration not yet available with NVUE commands, create an snmpd.conf snippet.

The following example creates a file called snmpd.conf_snippet.yaml, and sets the read only community string and the listening address to run in the mgmt VRF.

1. Create a .yaml file and add the following traditional snippet:

   cumulus@switch:~$ sudo nano snmpd.conf_snippet.yaml
   - set:
       system:
         config:
           snippet:
             snmpd.conf: |
               rocommunity cumuluspassword default
               agentaddress udp:@mgmt:161
   

2. Run the following command to patch the configuration:

   cumulus@switch:~$ nv config patch snmpd.conf_snippet.yaml
   

3. Run the nv config apply command to apply the configuration:

   cumulus@switch:~$ nv config apply
   

4. Verify that the configuration exists at the end of the /etc/snmp/snmpd.conf file:

   cumulus@switch:~$ sudo cat /etc/snmp/snmpd.conf
   ...
   !---- NVUE SNMP Server Snippets ----
   rocommunity cumuluspassword default
   agentaddress udp:@mgmt:161
   

/etc/ssh/sshd_config Snippets

To add SSH service configuration not yet available with NVUE commands, create an sshd_config snippet.

The following example creates a file called sshd_config_snippet.yaml to allow root login and enable X11 forwarding for all users except user anoncvs. The snippet also disables TCP forwarding for the anoncvs user and runs the cvs server command when anoncvs logs in.

1. Create a .yaml file and add the following traditional snippet:

   cumulus@switch:~$ sudo nano sshd_config_snippet.yaml
   - set:
       system:
         config:
           snippet:
             sshd_config: |
               PermitRootLogin yes
               X11Forwarding yes
               Match User anoncvs
                  X11Forwarding no
                  AllowTcpForwarding no
                  ForceCommand cvs server
   

2. Run the following command to patch the configuration:

   cumulus@switch:~$ nv config patch sshd_config_snippet.yaml
   

3. Run the nv config apply command to apply the configuration:

   cumulus@switch:~$ nv config apply
   

4. Verify that the configuration exists at the end of the /etc/ssh/sshd_config file:

   cumulus@switch:~$ sudo cat /etc/ssh/sshd_config
   ...
   !---- NVUE snippets ----
   PermitRootLogin yes
   X11Forwarding yes
   Match User anoncvs
      X11Forwarding no
      AllowTcpForwarding no
      ForceCommand cvs server
   

Flexible Snippets

Flexible snippets are an extension of traditional snippets that let you manage any text file on the system.

• You can create new files or modify existing files that NVUE does not manage.
• You can add configuration to files that NVUE manages.

The account you use through the CLI or the REST API to configure and manage flexible snippets must be in the sudo group, which includes the NVUE system-admin role, or you must be the root user.

Files NVUE Manages

You can use flexible snippets to add configuration to the following files that NVUE manages:

Flexible snippets do not support:

• Binary files.
• Symbolic links.
• More than 1MB of content.
• More than one flexible snippet in the same destination file.

Create a Flexible Snippet

To create a flexible snippet:

1. Create a file in yaml format and add each flexible snippet you want to apply in the format shown below. NVUE appends the flexible snippet at the end of an existing file. If the file does not exist, NVUE creates the file, then adds the content.

   cumulus@leaf01:mgmt:~$ sudo nano <filename>.yaml>
   - set:
       system:
        config:
          snippet:
            <snippet-name>:
              file: "<filename>"
              permissions: "<umask-permissions>"
              content: |
                # This is my content
              services:
                 <name>:
                   service: <service-name>
                   action: <action>
   

   • You can only set the umast permissions to a new file that you create. Adding the permissions: line is optional. The default umask persmissions are 644.
   • You can add a service with an action, such as start, restart, or stop. Adding the services: lines is optional; however, if you add the service: line, you must specify at least one service.

2. Run the following command to patch the configuration:

   cumulus@switch:~$ nv config patch <filename>.yaml>
   

3. Run the nv config apply command to apply the configuration:

   cumulus@switch:~$ nv config apply
   

4. Verify the patched configuration.

Flexible Snippet Examples

The following example flexible snippet called crontab-flex-snippet appends the single line @daily /opt/utils/run-backup.sh to the existing /etc/crontab file, then restarts the cron service.

cumulus@leaf01:mgmt:~$ sudo nano crontab-flex-snippet.yaml
- set:
    system:
      config:
        snippet:
          crontab-flex-snippet:
            file: "/etc/crontab"
            content: |
              @daily /opt/utils/run-backup.sh
            services:
              schedule:
                service: cron
                action: restart

The following example flexible snippet called apt-flex-snippet creates a new file /etc/apt/sources.list.d/microsoft-prod.list with 0644 permissions and adds multi-line text:

cumulus@leaf01:mgmt:~$ sudo nano apt-flex-snippet.yaml
- set:
    system:
      config:
        snippet:
          apt-flexible-snippet:
            file: "/etc/apt/sources.list.d/microsoft-prod.list"
            content: |
              # Adding Microsoft SQL Server Sources
              deb [arch=amd64] https://packages.microsoft.com/debian/10/prod buster main
            permissions: "0644"

The following flexible snippet called lldp_config_snipppet disables LLDP on swp1 and swp2 using the configure system interface pattern-blacklist command:

cumulus@leaf01:mgmt:~$ sudo nano lldp_config_snipppet.yaml
- set:
    system:
      config:
        snippet:
          lldp-interfaces-config:
            file: "/etc/lldpd.d/lldp-interfaces.conf"
            content: |
              configure system interface pattern-blacklist swp1,swp2
              services:
                lldp:
                  service: lldpd
                  action: restart

The following flexible snippet disables LLDP on swp1 and swp2 using the system interface pattern keyword:

cumulus@leaf01:mgmt:~$ sudo nano lldp_config_snipppet.yaml
- set:
    system:
      config:
        snippet:
          lldp-interfaces-config:
            file: "/etc/lldpd.d/lldp-interfaces.conf"
            content: |
              configure system interface pattern eth*,swp*,!swp1,!swp2
            services:
              lldp:
                service: lldpd
                action: restart

After you patch and apply the configuration above, the snippet creates a new file in the /etc/lldp.d directory, then restarts the lldpd service to stop LLDP transmitting and receiving on swp1 and swp2. Other interfaces continue to participate in LLDP.

cumulus@leaf01:mgmt:~$ nv config apply
Invalid config [rev_id: 8]
  Flexible snippets are not allowed to be configured on the file '/etc/cumulus/ports.conf’.
  Flexible snippets are not allowed to be configured on the file '/etc/cumulus/ports_width.conf’.

cumulus@leaf01:mgmt:~$ nv config apply
Invalid config [rev_id: 1]
  Flexible snippet cannot be used to modify the file '/etc/ssh/sshd_config'. Traditional snippets (for e.g., 'sshd_config') are supported on this file. Consult NVIDIA NVUE documentation for further information on snippets.

You can also create a flexible snippet with the REST API. See NVUE API.

Remove a Snippet

To remove a traditional or flexible snippet, edit the snippet’s .yaml file to change set to unset, then patch and apply the configuration. Alternatively, you can use the REST API DELETE and PATCH methods.

The following example removes the MLAG timer traditional snippet created above to configure the MLAG peer timeout:

1. Edit the mlag_snippet.yaml file to change set to unset:

   cumulus@switch:~$ sudo nano mlag_snippet.yaml
   - unset:
       system:
         config:
           snippet:
             ifupdown2_eni:
   

2. Run the following command to patch the configuration:

   cumulus@switch:~$ nv config patch mlag_snippet.yaml
   

3. Run the nv config apply command to apply the configuration:

   cumulus@switch:~$ nv config apply
   

4. Verify that the peer timeout parameter no longer exists in the peerlink.4094 stanza of the /etc/network/interfaces file:

   cumulus@switch:~$ sudo cat /etc/network/interfaces
   ...
   auto peerlink.4094
   iface peerlink.4094
    clagd-peer-ip linklocal
    clagd-backup-ip 10.10.10.2
    clagd-sys-mac 44:38:39:BE:EF:AA
    clagd-args --initDelay 180
   ...
   

This section discusses how to set the time zone, and how to set the date and time on the software clock on the switch. To configure NTP, see Network Time Protocol - NTP. To configure PTP, see Precision Time Protocol - PTP.

Setting the time zone, and the date and time on the software clock requires root privileges; use sudo.

Set the Time Zone

You can use one of these methods to set the time zone on the switch:

• Run NVUE commands.
• Use the guided wizard.
• Edit the /etc/timezone file.

cumulus@switch:~$ nv set system timezone US/Eastern
cumulus@switch:~$ nv config apply

Set the Date and Time

The switch contains a battery backed hardware clock that maintains the time while the switch powers off and between reboots. When the switch is running, the Cumulus Linux operating system maintains its own software clock.

During boot up, the switch copies the time from the hardware clock to the operating system software clock. The software clock takes care of all the timekeeping. During system shutdown, the switch copies the software clock back to the battery backed hardware clock.

You can set the date and time on the software clock with the date command. First, determine your current time zone:

cumulus@switch:~$ date +%Z

To set the software clock according to the configured time zone:

cumulus@switch:~$ sudo date -s "Tue Jan 26 00:37:13 2021"

You can write the current value of the software clock to the hardware clock using the hwclock command:

cumulus@switch:~$ sudo hwclock -w

See man hwclock(8) for more information.

Related Information

• Debian System Administrator's Manual - Time

In addition to the CLI, NVUE supports a REST API. Instead of accessing Cumulus Linux using SSH, you can interact with the switch using an HTTP client, such as cURL or a web browser.

The nvued service provides access to the NVUE REST API. Cumulus Linux exposes the HTTP endpoint internally, which makes the NVUE REST API accessible locally within the Cumulus Linux switch. The NVUE CLI also communicates with the nvued service using internal APIs. To provide external access to the NVUE REST API, Cumulus Linux uses an HTTP reverse proxy server, and supports HTTPS and TLS connections from external REST API clients.

The following illustration shows the NVUE REST API architecture and illustrates how Cumulus Linux forwards the requests internally.

Supported HTTP Methods

The NVUE REST API supports the following methods:

• The GET method displays configuration and operational data, and is equivalent to the nv show commands.
• The POST method creates and submits operations. You typically use this method for nv action commands and for the nv config command to create revisions.
• The PATCH method replaces or unsets a configuration. You use this method for the nv set and nv config apply commands. You can either perform:
  • A targeted configuration patch to make a configuration change, where you run a specific NVUE REST API targeted at a particular OpenAPI end-point URI. Based on the NVUE schema definition, you need to direct the PATCH REST API request at a particular endpoint (for example, /nvue_v1/vrf/<vrf-id>/router/bgp) and provide the payload that conforms to the schema. With a targeted configuration patch, you can control individual resources.
  • A root patch, where you run the NVUE PATCH API on the root node of the schema so that a single PATCH operation can change one, some, or the entire configuration in a single payload. The payload of the PATCH method must be aware of the entire NVUE object model schema because you make the configuration changes relative to the root node /nvue_v1. You typically perform a root patch to push all configurations to the switch in bulk; for example, if you use an SDN controller or a network management system to push the entire switch configuration every time you need to make a change, regardless of how small or large. A root patch can also make configuration changes with fewer round trips to the switch.
  • The input payload in a PATCH request can have either a set or unset json object for the same resource, but not both. The order in which the API executes the set and unset objects is not deterministic and not supported.
• The DELETE method deletes a configuration and is equivalent to the nv unset commands.

Secure the API

The NVUE REST API supports HTTP basic authentication, and the same underlying authentication methods for username and password that the NVUE CLI supports. User accounts work the same on both the API and the CLI.

Certificates

Cumulus Linux includes a self-signed certificate and private key to use on the server so that it works out of the box. The switch generates the self-signed certificate and private key when it boots for the first time. The X.509 certificate with the public key is in /etc/ssl/certs/cumulus.pem and the corresponding private key is in /etc/ssl/private/cumulus.key.

NVIDIA recommends you use your own certificates and keys. Certificates must be in PEM format. For the steps to generate self-signed certificates and keys, and to install them on the switch, refer to the Ubuntu Certificates and Security documentation.

To use your own certificate chain:

1. Import the certificate and private key onto the Cumulus Linux switch using secure channels, such as SCP or SFTP.
2. Store the certificate and private key on the filesystem in a location of you choice or use the same location; for example, /etc/ssl/certs and /etc/ssl/private.
3. Update the /etc/nginx/sites-enabled/nvue.conf file to set the ssl_certificate and the ssl_certificate_key values to your keys.
4. Restart NGINX with the sudo systemctl restart nginx command.

API-only User

To create an API-only user without SSH permissions, use Linux group permissions. You can create the API-only user in the ZTP script.

# Create the dedicated automation user 
adduser --disabled-password --gecos "Automation User,,,," --shell /usr/bin/nologin automation

# Set the password
echo 'automation:password!' | chpasswd

# Add the user to nvapply group to make NVUE config changes
adduser automation nvapply

Control Plane ACLs

You can secure the API by configuring:

• A listening address; see API Port and Listening Address below.
• Control plane ACLs; see the following example.

This example shows how to create ACLs to allow users from the management subnet and the local switch to communicate with the switch using REST APIs, and restrict all other access.

cumulus@switch:~$ nv set acl API-PROTECT type ipv4 
cumulus@switch:~$ nv set acl API-PROTECT rule 10 action permit
cumulus@switch:~$ nv set acl API-PROTECT rule 10 match ip .protocol tcp .dest-port 8765 .source-ip 192.168.200.0/24
cumulus@switch:~$ nv set acl API-PROTECT rule 10 remark "Allow the Management Subnet to talk to API"

cumulus@switch:~$ nv set acl API-PROTECT rule 20 action permit
cumulus@switch:~$ nv set acl API-PROTECT rule 20 match ip .protocol tcp .dest-port 8765 .source-ip 127.0.0.1
cumulus@switch:~$ nv set acl API-PROTECT rule 20 remark "Allow the local switch to talk to the API"

cumulus@switch:~$ nv set acl API-PROTECT rule 30 action deny
cumulus@switch:~$ nv set acl API-PROTECT rule 30 match ip .protocol tcp .dest-port 8765
cumulus@switch:~$ nv set acl API-PROTECT rule 30 remark "Block everyone else from talking to the API"

cumulus@switch:~$ nv set system control-plane acl API-PROTECT inbound

Supported Objects

The NVUE object model supports most features on the Cumulus Linux switch. The following list shows the supported objects. The NVUE API supports more objects within each of these objects. You can find a full listing of the supported API endpoints here.

Use the API

The NVUE CLI and the REST API are equivalent in functionality; you can run all management operations from the REST API or from the CLI. The NVUE object model drives both the REST API and the CLI management operations. All operations are consistent; for example, the CLI nv show commands reflect any PATCH operation (create and update) you run through the REST API.

NVUE follows a declarative model, removing context-specific commands and settings. The structure of NVUE is like a big tree that represents the entire state of a Cumulus Linux instance. At the base of the tree are high level branches representing objects, such as router and interface. Under each of these branches are more branches. As you navigate through the tree, you gain a more specific context. At the leaves of the tree are actual attributes, represented as key-value pairs. The path through the tree is similar to a filesystem path.

Cumulus Linux enables the NVUE REST API by default. To disable the NVUE REST API, run the nv set system api state disabled command.

API Port and Listening Address

This section shows how to:

• Set the NVUE REST API port. If you do not set a port, Cumulus Linux uses the default port 8765.
• Specify the NVUE REST API listening address; you can specify an IPv4 address, IPv6 address, or localhost. If you do not specify a listening address, NGINX listens on all addresses for the target port.

cumulus@switch:~$ nv set system api port 8888
cumulus@switch:~$ nv config apply

cumulus@switch:~$ nv set system api listening-address 10.10.10.1
cumulus@switch:~$ nv set system api listening-address 10.10.20.1
cumulus@switch:~$ nv config apply

cumulus@switch:~$ nv set system api listening-address 172.0.24.0
cumulus@switch:~$ nv config apply

cumulus@switch:~$ nv set interface swp1 ip address 10.10.10.1/24
cumulus@switch:~$ nv set interface swp1 ip vrf BLUE
cumulus@switch:~$ nv config apply

cumulus@switch:~$ nv set system api listening-address 10.10.10.1
cumulus@switch:~$ nv config apply

cumulus@switch:~$ curl -u 'cumulus:cumulus' -k --request PATCH https://localhost:8765/nvue_v1/system/api?rev=2 -H 'Content-Type:application/json' -d '{"port": 8888 }'

cumulus@switch:~$ curl -u 'cumulus:cumulus' -k --request PATCH https://localhost:8765/nvue_v1/system/api/listening-address?rev=2 -H 'Content-Type:application/json' -d '{ "localhost": {}, "10.10.10.1": {}, "10.10.20.1": {}}'

cumulus@switch:~$ curl -u 'cumulus:cumulus' -k --request PATCH https://localhost:8765/nvue_v1/system/api/listening-address?rev=2 -H 'Content-Type:application/json' -d '{"172.0.24.0": {}}'

Show NVUE REST API Information

To show REST API port configuration, state (enabled or disabled), and connection information:

cumulus@switch:~$ nv show system api
                  operational     applied
--------------    -----------     -------
port                 8888         8888     
state                enabled      enabled  
[listening-address]  localhost    localhost
connections
  accepted        31
  active          1
  handled         33
  reading         0
  requests        28
  waiting         0
  writing         1

cumulus@switch:~$ nv show system api connections
          operational  applied
--------  -----------  -------
accepted  31                  
active    1                   
handled   33                  
reading   0                   
requests  28                   
waiting   0                   
writing   1     

cumulus@switch:~$ nv show system api listening-address

---------
localhost

cumulus@switch:~$ curl -u 'cumulus:cumulus' -k --request GET https://localhost:8765/nvue_v1/system/api?rev=2 -H "accept: application/json"
{
  "state": "enabled",
  "port": 8765,
  "listening-address": {
    "10.10.10.1": {},
    "10.10.20.1": {}
  },
  "connections": {
    "active": 1,
    "accepted": 31,
    "handled": 0,
    "requests": 28,
    "reading": 0,
    "writing": 1,
    "waiting": 0
  }
}

cumulus@switch:~$ curl -u 'cumulus:cumulus' -k --request GET https://localhost:8765/nvue_v1/system/api/listening-address?rev=2 -H "accept: application/json"
{
  "10.10.10.1": {},
  "10.10.20.1": {}
}

Run cURL Commands

You can run the cURL commands from the command line. Use the username and password for the switch. For example:

cumulus@switch:~$ curl  -u 'cumulus:cumulus' --insecure https://127.0.0.1:8765/nvue_v1/interface
{
  "eth0": {
    "ip": {
      "address": {
        "192.168.200.12/24": {}
      }
    },
    "link": {
      "mtu": 1500,
      "state": {
        "up": {}
      },
      "stats": {
        "carrier-transitions": 2,
        "in-bytes": 184151,
        "in-drops": 0,
        "in-errors": 0,
        "in-pkts": 2371,
        "out-bytes": 117506,
        "out-drops": 0,
        "out-errors": 0,
        "out-pkts": 762
      }
...

API Use Cases

The following examples show the primary API uses cases.

View a Configuration

Use the following example to obtain the current applied configuration on the switch. Change the rev argument to view any revision. Possible options for the rev argument include startup, pending, operational, and applied.

cumulus@switch:~$ curl -k -u cumulus:cumulus -X GET "https://127.0.0.1:8765/nvue_v1/?rev=applied&filled=false"
"acl": {}, 
  "bridge": { 
    "domain": { 
      "br_default": { 
        "encap": "802.1Q", 
        "mac-address": "auto", 
        "multicast": { 
          "snooping": { 
            "enable": "off" 
          } 
        }, 
        "stp": { 
          "priority": 32768, 
          "state": { 
            "up": {} 
          } 
        }, 
        "type": "vlan-aware", 
        "untagged": 1, 
        "vlan": { 
          "10": { 
            "multicast": { 
...  

#!/usr/bin/env python3

import requests
from requests.auth import HTTPBasicAuth
import json
import time

auth = HTTPBasicAuth(username="cumulus", password="password")
nvue_end_point = "https://127.0.0.1:8765/nvue_v1"
mime_header = {"Content-Type": "application/json"}

if __name__ == "__main__":
    r = requests.get(url=nvue_end_point + "/?rev=applied&filled=false",
                     auth=auth,
                     verify=False)
    print("=======Current Applied Revision=======")
    print(json.dumps(r.json(), indent=2))

cumulus@switch:~$ nv config show
- set: 
    bridge: 
      domain: 
        br_default: 
          type: vlan-aware 
          vlan: 
            '10': 
              vni: 
                '10': {} 
            '20': 
              vni: 
                '20': {} 
            '30': 
              vni: 
                '30': {} 
    evpn: 
      enable: on 
    mlag: 
      backup: 
        10.10.10.2: {} 
      enable: on 
      init-delay: 10 
      mac-address: 44:38:39:BE:EF:AA 
... 

Replace an Entire Configuration

To replace an entire configuration:

1. Create a new revision ID with a POST:

   cumulus@switch:~$ curl -u 'cumulus:cumulus' --insecure -X POST https://127.0.0.1:8765/nvue_v1/revision
   {
    "1": {
      "state": "pending",
      "transition": {
        "issue": {},
        "progress": ""
      }
    }
   }
   

2. Record the revision ID. In the above example, the revision ID is "1".

3. Do a root patch to delete the whole configuration.

   cumulus@switch:~$ curl -u 'cumulus:cumulus' -d '{}' -H 'Content-Type: application/json' -k -X DELETE https://127.0.0.1:8765/nvue_v1/?rev=1
   {}
   

4. Do a root patch to update the switch with the new configuration.

   cumulus@switch:~$ curl -u 'cumulus:cumulus' -d '{
      "system": {
        "hostname": "switch01"
      },
      "bridge": {
        "domain": {
          "br_default": {
            "type": "vlan-aware",
            "vlan": {
              "10": {
                "vni": {
                  "10": {}
                  }
                },
              "20": {
                "vni": {
                  "20": {}
                }
              },
              "30": {
                "vni": {
                  "30": {}
                }
              }
            }
          }
        }
      },
      "interface": {
        "eth0": {
          "ip": {
            "address": {
              "192.168.200.6/24": {}
            },
            "vrf": "mgmt"
          },
          "type": "eth"
        },
        "lo": {
          "ip": {
            "address": {
              "10.10.10.1/32": {}
            }
          },
          "type": "loopback"
        },
        "swp51": {
          "link": {
            "state": {
              "up": {}
            }
          },
          "type": "swp"
        },
        "swp52": {
          "link": {
            "state": {
              "up": {}
            }
          },
          "type": "swp"
        },
        "swp53": {
          "link": {
            "state": {
              "up": {}
            }
          },
          "type": "swp"
        },
        "swp54": {
          "link": {
            "state": {
              "up": {}
            }
          },
          "type": "swp"
        }
      },
      "mlag": {
        "backup": {
          "10.10.10.2": {}
        },
        "enable": "on",
        "init-delay": 10,
        "mac-address": "44:38:39:BE:EF:AA",
        "peer-ip": "linklocal",
        "priority": 1000
      }
      "router": {
        "bgp": {
          "enable": "on"
        },
        "vrr": {
          "enable": "on"
        }
      },
      "service": {},
      "vrf": {
        "mgmt": {
          "router": {
            "static": {
              "0.0.0.0/0": {
                "address-family": "ipv4-unicast",
                "via": {
                  "192.168.200.1": {
                    "type": "ipv4-address"
                  }
                }
              }
            }
          }
        }
      }
    }' -H 'Content-Type: application/json' -k -X PATCH https://127.0.0.1:8765/nvue_v1/?rev=1
   {}
   

5. Apply the changes with a PATCH to the revision changeset.

   Curl Command

   cumulus@switch:~$ curl -u 'cumulus:cumulus' -H 'Content-Type:application/json' -d '{"state": "apply", "auto-prompt": {"ays": "ays_yes"}}' -k -X PATCH https://127.0.0.1:8765/nvue_v1/revision/1
   {
     "state": "apply",
     "transition": {
       "issue": {},
       "progress": ""
     }
   }
   

   NVUE CLI

   cumulus@switch:~$ nv config apply
   

6. Review the status of the apply and the configuration:

   Curl Command

   cumulus@switch:~$ curl -u 'cumulus:cumulus' -k -X GET https://127.0.0.1:8765/nvue_v1/revision/1
   {
     "state": "applied",
     "transition": {
       "issue": {},
       "progress": ""
     }
   }
   

   cumulus@switch:~$ curl -u 'cumulus:cumulus' --insecure https://127.0.0.1:8765/nvue_v1/system
   {
    "build": "Cumulus Linux 5.4.0",
    "hostname": "switch01",
    "timezone": "Etc/UTC",
    "uptime": 763
   }
   cumulus@switch:~$ curl -u 'cumulus:cumulus' --insecure https://127.0.0.1:8765/nvue_v1/bridge/domain/br_default/vlan/10
   {
    "multicast": {
      "snooping": {
        "querier": {
          "source-ip": "0.0.0.0"
        }
      }
    },
    "ptp": {
      "enable": "off"
    },
    "vni": {
      "10": {
        "flooding": {
          "enable": "auto"
        },
        "mac-learning": "off"
      }
    }
   

   Python Code

   #!/usr/bin/env python3
   
   import requests
   from requests.auth import HTTPBasicAuth
   import json
   import time
   
   auth = HTTPBasicAuth(username="cumulus", password="password")
   nvue_end_point = "https://127.0.0.1:8765/nvue_v1"
   mime_header = {"Content-Type": "application/json"}
   
   DUMMY_SLEEP = 5  # In seconds
   POLL_APPLIED = 1  # in seconds
   RETRIES = 10
   
   def print_request(r: requests.Request):
       print("=======Request=======")
       print("URL:", r.url)
       print("Headers:", r.headers)
       print("Body:", r.body)
   
   def print_response(r: requests.Response):
       print("=======Response=======")
       print("Headers:", r.headers)
       print("Body:", json.dumps(r.json(), indent=2))
   
   def create_nvue_changest():
       r = requests.post(url=nvue_end_point + "/revision",
                         auth=auth,
                         verify=False)
       print_request(r.request)
       print_response(r)
       response = r.json()
       changeset = response.popitem()[0]
       return changeset
   
   def apply_nvue_changeset(changeset):
       apply_payload = {"state": "apply", "auto-prompt": {"ays": "ays_yes"}}
       url = nvue_end_point + "/revision/" + requests.utils.quote(changeset,
                                                                  safe="")
       r = requests.patch(url=url,
                          auth=auth,
                          verify=False,
                          data=json.dumps(apply_payload),
                          headers=mime_header)
       print_request(r.request)
       print_response(r)
   
   def is_config_applied(changeset) -> bool:
       # Check if the configuration was indeed applied
       global RETRIES
       global POLL_APPLIED
       retries = RETRIES
       while retries > 0:
           r = requests.get(url=nvue_end_point + "/revision/" + requests.utils.quote(changeset, safe=""),
                            auth=auth,
                            verify=False)
           response = r.json()
           print(response)
           if response["state"] == "applied":
               return True
           retries -= 1
           time.sleep(POLL_APPLIED)
   
       return False
   
   def apply_new_config(path,payload):
       # Create a new revision ID
       changeset = create_nvue_changest()
       print("Using NVUE Changeset: '{}'".format(changeset))
   
       # Delete existing configuration
       query_string = {"rev": changeset}
       r = requests.delete(url=nvue_end_point + path,
                          auth=auth,
                          verify=False,
                          params=query_string,
                          headers=mime_header)
       print_request(r.request)
       print_response(r)
   
       # Patch the new configuration
       
       query_string = {"rev": changeset}
       r = requests.patch(url=nvue_end_point + path,
                          auth=auth,
                          verify=False,
                          data=json.dumps(payload),
                          params=query_string,
                          headers=mime_header)
       print_request(r.request)
       print_response(r)
   
       # Apply the changes to the new revision changeset
       apply_nvue_changeset(changeset)
   
       # Check if the changeset was applied
       is_config_applied(changeset)
   
   def nvue_get(path):
       r = requests.get(url=nvue_end_point + path,
                        auth=auth,
                        verify=False)
       print_request(r.request)
       print_response(r)
   
   if __name__ == "__main__":
       payload = {
         "system": {
           "hostname": "switch01"
         },
         "bridge": {
           "domain": {
             "br_default": {
               "type": "vlan-aware",
               "vlan": {
                 "10": {
                   "vni": {
                     "10": {}
                     }
                   },
                 "20": {
                   "vni": {
                     "20": {}
                   }
                 },
                 "30": {
                   "vni": {
                     "30": {}
                   }
                 }
               }
             }
           }
         },
         "interface": {
           "eth0": {
             "ip": {
               "address": {
                 "192.168.200.6/24": {}
               },
               "vrf": "mgmt"
             },
             "type": "eth"
           },
           "lo": {
             "ip": {
               "address": {
                 "10.10.10.1/32": {}
               }
             },
             "type": "loopback"
           },
           "swp51": {
             "link": {
               "state": {
                 "up": {}
               }
             },
             "type": "swp"
           },
           "swp52": {
             "link": {
               "state": {
                 "up": {}
               }
             },
             "type": "swp"
           },
           "swp53": {
             "link": {
               "state": {
                 "up": {}
               }
             },
             "type": "swp"
           },
           "swp54": {
             "link": {
               "state": {
                 "up": {}
               }
             },
             "type": "swp"
           }
         },
         "mlag": {
           "backup": {
             "10.10.10.2": {}
           },
           "enable": "on",
           "init-delay": 10,
           "mac-address": "44:38:39:BE:EF:AA",
           "peer-ip": "linklocal",
           "priority": 1000
         }
         "router": {
           "bgp": {
             "enable": "on"
           },
           "vrr": {
             "enable": "on"
           }
         },
         "service": {},
         "vrf": {
           "mgmt": {
             "router": {
               "static": {
                 "0.0.0.0/0": {
                   "address-family": "ipv4-unicast",
                   "via": {
                     "192.168.200.1": {
                       "type": "ipv4-address"
                     }
                   }
                 }
               }
             }
           }
         }
       }
       apply_new_config("/",payload)
       time.sleep(DUMMY_SLEEP)
       print("=====Verifying some of the configurations=====")
       nvue_get("/system")
       nvue_get("/bridge/domain/br_default/vlan/10")
   

   NVUE CLI

   cumulus@switch:~$ nv show system
               operational          applied
   --------  -------------------  -------
   hostname  switch01             cumulus
   build     Cumulus Linux 5.4.0
   uptime    0:12:59
   timezone  Etc/UTC
   

   cumulus@switch:~$ nv show bridge domain br_default vlan 10
   
                    operational  applied  pending  description
   ---------------  -----------  -------  -------  ------------------------------------------------------
   [vni]            10           10       10       L2 VNI
   multicast
     snooping
       querier
         source-ip  0.0.0.0      0.0.0.0  0.0.0.0  Source IP to use when sending IGMP/MLD queries.
   ptp
     enable         off          off      off      Turn the feature 'on' or 'off'.  The default is 'off'.
   

Make a Configuration Change

To make a configuration change:

1. Create a new revision ID with a POST:

   cumulus@switch:~$ curl -u 'cumulus:cumulus' --insecure -X POST https://127.0.0.1:8765/nvue_v1/revision
   {
      "2": {
      "state": "pending",
      "transition": {
        "issue": {},
        "progress": ""
      }
    }
   }
   

2. Record the revision ID. In the above example, the revision ID is "2".

3. Make the change with a PATCH and link it to the revision ID:

   Curl Command

   cumulus@switch:~$ curl -u 'cumulus:cumulus' -d '{"99.99.99.99/32": {}}' -H 'Content-Type: application/json' -k -X PATCH https://127.0.0.1:876nvue_v1/interface/lo/ip/address?rev=2
   {
     "99.99.99.99/32": {}
   }
   

   NVUE CLI

   cumulus@switch:~$ nv set interface lo ip address 99.99.99.99/32
   

4. Apply the changes with a PATCH to the revision changeset:

   Curl Command

   cumulus@switch:~$ curl -u 'cumulus:cumulus' -H 'Content-Type:application/json' -k -X PATCH https://127.0.0.1:8765/nvue_v1/revision/2
   {
     "state": "apply",
     "transition": {
       "issue": {},
       "progress": ""
     }
   }
   

   NVUE CLI

   cumulus@switch:~$ nv config apply
   

5. Review the status of the apply and the configuration:

   Curl Command

   cumulus@switch:~$ curl -u 'cumulus:cumulus' -k -X GET https://127.0.0.1:8765/nvue_v1/revision/2
   {
     "state": "applied",
     "transition": {
       "issue": {},
       "progress": ""
     }
   }
   

   cumulus@switch:~$ curl -u 'cumulus:cumulus' --insecure https://127.0.0.1:8765/nvue_v1/interface/lo/ip/address
   {
     "127.0.0.1/8": {},
     "99.99.99.99/32": {},
     "::1/128": {}
   }
   

   Python Code

   #!/usr/bin/env python3
   
   import requests
   from requests.auth import HTTPBasicAuth
   import json
   import time
   
   auth = HTTPBasicAuth(username="cumulus", password="password")
   nvue_end_point = "https://127.0.0.1:8765/nvue_v1"
   mime_header = {"Content-Type": "application/json"}
   
   DUMMY_SLEEP = 5  # In seconds
   POLL_APPLIED = 1  # in seconds
   RETRIES = 10
   
   def print_request(r: requests.Request):
       print("=======Request=======")
       print("URL:", r.url)
       print("Headers:", r.headers)
       print("Body:", r.body)
   
   def print_response(r: requests.Response):
       print("=======Response=======")
       print("Headers:", r.headers)
       print("Body:", json.dumps(r.json(), indent=2))
   
   def create_nvue_changest():
       r = requests.post(url=nvue_end_point + "/revision",
                         auth=auth,
                         verify=False)
       print_request(r.request)
       print_response(r)
       response = r.json()
       changeset = response.popitem()[0]
       return changeset
   
   def apply_nvue_changeset(changeset):
       apply_payload = {"state": "apply", "auto-prompt": {"ays": "ays_yes"}}
       url = nvue_end_point + "/revision/" + requests.utils.quote(changeset,
                                                                  safe="")
       r = requests.patch(url=url,
                          auth=auth,
                          verify=False,
                          data=json.dumps(apply_payload),
                          headers=mime_header)
       print_request(r.request)
       print_response(r)
   
   def is_config_applied(changeset) -> bool:
       # Check if the configuration was indeed applied
       global RETRIES
       global POLL_APPLIED
       retries = RETRIES
       while retries > 0:
           r = requests.get(url=nvue_end_point + "/revision/" + requests.utils.quote(changeset, safe=""),
                            auth=auth,
                            verify=False)
           response = r.json()
           print(response)
           if response["state"] == "applied":
               return True
           retries -= 1
           time.sleep(POLL_APPLIED)
   
       return False
   
   def apply_new_config(path,payload):
       # Create a new revision ID
       changeset = create_nvue_changest()
       print("Using NVUE Changeset: '{}'".format(changeset))
   
       # Delete existing configuration
       query_string = {"rev": changeset}
       r = requests.delete(url=nvue_end_point + path,
                          auth=auth,
                          verify=False,
                          params=query_string,
                          headers=mime_header)
       print_request(r.request)
       print_response(r)
   
       # Patch the new configuration
       
       query_string = {"rev": changeset}
       r = requests.patch(url=nvue_end_point + path,
                          auth=auth,
                          verify=False,
                          data=json.dumps(payload),
                          params=query_string,
                          headers=mime_header)
       print_request(r.request)
       print_response(r)
   
       # Apply the changes to the new revision changeset
       apply_nvue_changeset(changeset)
   
       # Check if the changeset was applied
       is_config_applied(changeset)
   
   def nvue_get(path):
       r = requests.get(url=nvue_end_point + path,
                        auth=auth,
                        verify=False)
       print_request(r.request)
       print_response(r)
   
   if __name__ == "__main__":
       payload = {
           "99.99.99.99/32": {}
       }
       apply_new_config("/interface/lo/ip/address",payload)
       time.sleep(DUMMY_SLEEP)
       nvue_get("/interface/lo/ip/address")
   

   NVUE CLI

   cumulus@switch:~$ nv show interface lo ip address
      
   -------------
   99.99.99.99/32
   127.0.0.1/8
   ::1/128
   

Troubleshoot Configuration Changes

When a configuration change fails, you see an error in the change request.

Configuration Fails Because of a Dependency

If you stage a configuration but it fails because of a dependency, the failure shows the reason. In the following example, the change fails because the BGP router ID is not set.

cumulus@switch:~$ curl -u 'cumulus:cumulus' --insecure https://127.0.0.1:8765/nvue_v1/revision/6
{
  "state": "invalid",
  "transition": {
    "issue": {
      "0": {
        "code": "config_invalid",
        "data": {
          "location": "router.bgp.enable",
          "reason": "BGP requires router-id to be set globally or in the VRF.\n"
        },
        "message": "Config invalid at router.bgp.enable: BGP requires router-id to be set globally or in the VRF.\n",
        "severity": "error"
      }
    },
    "progress": "Invalid config"
  }
}

The staged configuration is missing router-id.

cumulus@switch:~$ curl -u 'cumulus:cumulus' --insecure https://127.0.0.1:8765/nvue_v1/vrf/default/router/bgp?rev=6
{
  "autonomous-system": 65999,
  "enable": "on"
}

Configuration Apply Fails with Warnings

In some cases, such as the first push with NVUE or if you change a file manually instead of using NVUE, you see a warning prompt and the apply fails.

cumulus@switch:~$ curl -u 'cumulus:cumulus' --insecure -X GET https://127.0.0.1:8765/nvue_v1/revision/6
{
  "6": {
    "state": "ays_fail",
    "transition": {
      "issue": {
        "0": {
          "code": "client_timeout",
          "data": {},
          "message": "Timeout while waiting for client response",
          "severity": "error"
        }
      },
      "progress": "Aborted apply after warnings"
    }
  }

To resolve this issue, observe the failures or errors, then inspect the configuration that you are trying to apply. After you resolve the errors, retry the API. If you prefer to overlook the errors and force an apply, add "auto-prompt":{"ays": "ays_yes"} to the configuration apply.

cumulus@switch:~$ curl -u 'cumulus:cumulus' -d '{"state":"apply","auto-prompt":{"ays": "ays_yes"}}' -H 'Content-Type:application/json' --insecure -X PATCH https://127.0.0.1:8765/nvue_v1/revision/6

Save a Configuration

To save an applied configuration change to the startup configuration file (/etc/nvue.d/startup.yaml) so that the changes persist after a reboot, use a PATCH to the applied revision with the save state.

cumulus@switch:~$ curl -u 'cumulus:cumulus' -k -X PATCH -d '{"state": "save", "auto-prompt": {"ays": "ays_yes"}}' -H 'Content-Type: application/json'  https://127.0.0.1:8765/nvue_v1/revision/applied 
{ 
  "state": "save",
  "transition": {
    "issue": {},
    "progress": ""
  }
}

#!/usr/bin/env python3

import requests
from requests.auth import HTTPBasicAuth
import json
import time

auth = HTTPBasicAuth(username="cumulus", password="password")
nvue_end_point = "https://127.0.0.1:8765/nvue_v1"
mime_header = {"Content-Type": "application/json"}

DUMMY_SLEEP = 5  # In seconds
POLL_APPLIED = 1  # in seconds
RETRIES = 10

def print_request(r: requests.Request):
    print("=======Request=======")
    print("URL:", r.url)
    print("Headers:", r.headers)
    print("Body:", r.body)

def print_response(r: requests.Response):
    print("=======Response=======")
    print("Headers:", r.headers)
    print("Body:", json.dumps(r.json(), indent=2))

def save_nvue_changeset():
    apply_payload = {"state": "save", "auto-prompt": {"ays": "ays_yes"}}
    url = nvue_end_point + "/revision/applied"
    r = requests.patch(url=url,
                       auth=auth,
                       verify=False,
                       data=json.dumps(apply_payload),
                       headers=mime_header)
    print_request(r.request)
    print_response(r)

if __name__ == "__main__":
    save_nvue_changeset()

cumulus@switch:~$ nv config save
saved

Unset a Configuration Change

To unset a configuration change, use the null value to the key. For example, to delete vlan100 from a switch, use the following syntax:

cumulus@switch:~$ curl -u 'cumulus:cumulus' -d '{"vlan100":null}' -H 'Content-Type: application/json' --insecure -X PATCH https://127.0.0.1:8765/nvue_v1/interface/rev=4

When you unset a change, you must still use the PATCH action. The value indicates removal of the entry. The data is {"vlan100":null} with the PATCH action.

Use the API for Active Monitoring

The example below fetches the counters for interface swp1.

cumulus@switch:~$ curl -u 'cumulus:cumulus' -k -X GET https://127.0.0.1:8765/nvue_v1/interface/swp1/link/stats
{
  "carrier-transitions": 6,
  "in-bytes": 293771538,
  "in-drops": 0,
  "in-errors": 0,
  "in-pkts": 2321737,
  "out-bytes": 366068936,
  "out-drops": 0,
  "out-errors": 0,
  "out-pkts": 3536629
}

#!/usr/bin/env python3

import requests
from requests.auth import HTTPBasicAuth
import json
import time

auth = HTTPBasicAuth(username="cumulus", password="password")
nvue_end_point = "https://127.0.0.1:8765/nvue_v1"
mime_header = {"Content-Type": "application/json"}

if __name__ == "__main__":
    r = requests.get(url=nvue_end_point + "/interface/swp1/link/stats",
                     auth=auth,
                     verify=False)
    print("=======Interface swp1 Statistics=======")
    print(json.dumps(r.json(), indent=2))

cumulus@switch:~$ nv show interface swp1 link stats
                     operational  applied  pending  description
-------------------  -----------  -------  -------  ----------------------------------------------------------------------
carrier-transitions  6                              Number of times the interface state has transitioned between up and...
in-bytes             280.15 MB                      total number of bytes received on the interface
in-drops             0                              number of received packets dropped
in-errors            0                              number of received packets with errors
in-pkts              2321659                        total number of packets received on the interface
out-bytes            349.10 MB                      total number of bytes transmitted out of the interface
out-drops            0                              The number of outbound packets that were chosen to be discarded eve...
out-errors           0                              The number of outbound packets that could not be transmitted becaus...
out-pkts             3536508                        total number of packets transmitted out of the interface

Convert CLI Changes to Use the API

You can take a configuration change from the CLI and use the API to configure the same set of changes.

1. Make your configuration changes on the system with the NVUE CLI.

   cumulus@switch:~$ nv set system hostname switch01
   cumulus@switch:~$ nv set interface lo ip address 99.99.99.99/32
   cumulus@switch:~$ nv set interface eth0 ip address 192.168.200.6/24
   cumulus@switch:~$ nv set interface bond0 bond member swp1-4
   

2. View the changes as a JSON blob.

   cumulus@switch:~$ nv config diff -o json
   [
     {
       "set": {
         "interface": {
           "bond0": {
             "bond": {
               "member": {
                 "swp1": {},
                 "swp2": {},
                 "swp3": {},
                 "swp4": {}
               }
             },
             "type": "bond"
           },
           "lo": {
             "ip": {
               "address": {
                 "99.99.99.99/32": {}
               }
             }
           }
         },
         "system": {
           "hostname": "switch01"
         }
       }
     }
   ]
   

3. Staple the JSON blob to a root patch request as the payload.

   cumulus@switch:~$ curl -u 'cumulus:cumulus' -d '{
         "interface": {
           "bond0": {
             "bond": {
               "member": {
                 "swp1": {},
                 "swp2": {},
                 "swp3": {},
                 "swp4": {}
               }
             },
             "type": "bond"
           },
           "lo": {
             "ip": {
               "address": {
                 "99.99.99.99/32": {}
               }
             }
           }
         },
         "system": {
           "hostname": "switch01"
         }
       }' -k -X PATCH https://127.0.0.1:8765/nvue_v1/?rev=3
   
   {
     "bridge": {
       "domain": {
         "br_default": {
           "type": "vlan-aware",
           "vlan": {
             "10": {
               "vni": {
                 "10": {}
               }
             },
             "20": {
               "vni": {
                 "20": {}
               }
             },
             "30": {
               "vni": {
                 "30": {}
               }
             }
           }
         }
       }
     },
     "evpn": {
       "enable": "on"
     },
     "interface": {
       "bond1": {
         "bond": {
           "lacp-bypass": "on",
           "member": {
             "swp1": {}
           },
   ...
   

4. Apply the changes with a PATCH to the revision changeset.

   cumulus@switch:~$ curl -u 'cumulus:cumulus' -H 'Content-Type:application/json' -k -d '{"state": "apply", "auto-prompt": {"ays": "ays_yes"}}' -X PATCH https://127.0.0.1:8765/nvue_v1/revision/3
   {
     "state": "apply",
     "transition": {
       "issue": {},
       "progress": ""
     }
   }
   

5. Review the status of the apply and the configuration:

   Curl Command

   cumulus@switch:~$ curl -u 'cumulus:cumulus' -k -X GET https://127.0.0.1:8765/nvue_v1/revision/3
   {
     "state": "applied",
     "transition": {
       "issue": {},
       "progress": ""
     }
   }
   

   Python Code

   #!/usr/bin/env python3
   
   import requests
   from requests.auth import HTTPBasicAuth
   import json
   import time
   
   auth = HTTPBasicAuth(username="cumulus", password="password")
   nvue_end_point = "https://127.0.0.1:8765/nvue_v1"
   mime_header = {"Content-Type": "application/json"}
   
   DUMMY_SLEEP = 5  # In seconds
   POLL_APPLIED = 1  # in seconds
   RETRIES = 10
   
   def print_request(r: requests.Request):
       print("=======Request=======")
       print("URL:", r.url)
       print("Headers:", r.headers)
       print("Body:", r.body)
   
   def print_response(r: requests.Response):
       print("=======Response=======")
       print("Headers:", r.headers)
       print("Body:", json.dumps(r.json(), indent=2))
   
   def create_nvue_changest():
       r = requests.post(url=nvue_end_point + "/revision",
                         auth=auth,
                         verify=False)
       print_request(r.request)
       print_response(r)
       response = r.json()
       changeset = response.popitem()[0]
       return changeset
   
   def apply_nvue_changeset(changeset):
       # apply_payload = {"state": "apply"}
       apply_payload = {"state": "apply", "auto-prompt": {"ays": "ays_yes"}}
       url = nvue_end_point + "/revision/" + requests.utils.quote(changeset,
                                                                  safe="")
       r = requests.patch(url=url,
                          auth=auth,
                          verify=False,
                          data=json.dumps(apply_payload),
                          headers=mime_header)
       print_request(r.request)
       print_response(r)
   
   def is_config_applied(changeset) -> bool:
       # Check if the configuration was indeed applied
       global RETRIES
       global POLL_APPLIED
       retries = RETRIES
       while retries > 0:
           r = requests.get(url=nvue_end_point + "/revision/" + requests.utils.quote(changeset, safe=""),
                            auth=auth,
                            verify=False)
           response = r.json()
           print(response)
   
           if response["state"] == "applied":
               return True
           retries -= 1
           time.sleep(POLL_APPLIED)
   
       return False
   
   def apply_new_config(path,payload):
       # Create a new revision ID
       changeset = create_nvue_changest()
       print("Using NVUE Changeset: '{}'".format(changeset))
   
       # Delete existing configuration
       query_string = {"rev": changeset}
       r = requests.delete(url=nvue_end_point + path,
                          auth=auth,
                          verify=False,
                          params=query_string,
                          headers=mime_header)
       print_request(r.request)
       print_response(r)
   
       # Patch the new configuration
       
       query_string = {"rev": changeset}
       r = requests.patch(url=nvue_end_point + path,
                          auth=auth,
                          verify=False,
                          data=json.dumps(payload),
                          params=query_string,
                          headers=mime_header)
       print_request(r.request)
       print_response(r)
   
       # Apply the changes to the new revision changeset
       apply_nvue_changeset(changeset)
   
       # Check if the changeset was applied
       is_config_applied(changeset)
   
   def nvue_get(path):
       r = requests.get(url=nvue_end_point + path,
                        auth=auth,
                        verify=False)
       print_request(r.request)
       print_response(r)
   
   if __name__ == "__main__":
       payload = {
         "interface": {
           "bond0": {
             "bond": {
               "member": {
                 "swp1": {},
                 "swp2": {},
                 "swp3": {},
                 "swp4": {}
               }
             },
             "type": "bond"
           },
           "lo": {
             "ip": {
               "address": {
                 "99.99.99.99/32": {}
               }
             }
           }
         },
         "system": {
           "hostname": "switch01"
         }
       }
       apply_new_config("/",payload)
       time.sleep(DUMMY_SLEEP)
       nvue_get("/interface/bond0")
       nvue_get("/interface/lo")
       nvue_get("/system")
   
   

API Examples

The following section provides practical API examples.

Configure the System

To set the system hostname, pre-login or post-login message, and time zone on the switch, send a targeted API request to /nvue_v1/system.

cumulus@switch:~$ curl -u 'cumulus:cumulus' -d '{"system": {"hostname":"switch01","timezone":"America/Los_Angeles","message":{"pre-login":"Welcome to NVIDIA Cumulus Linux","post-login":"You have successfully logged in to switch01"}}}' -k -X PATCH https://127.0.0.1:8765/nvue_v1/?rev=4

#!/usr/bin/env python3

import requests
from requests.auth import HTTPBasicAuth
import json
import time

auth = HTTPBasicAuth(username="cumulus", password="password")
nvue_end_point = "https://127.0.0.1:8765/nvue_v1"
mime_header = {"Content-Type": "application/json"}

DUMMY_SLEEP = 5  # In seconds
POLL_APPLIED = 1  # in seconds
RETRIES = 10

def print_request(r: requests.Request):
    print("=======Request=======")
    print("URL:", r.url)
    print("Headers:", r.headers)
    print("Body:", r.body)

def print_response(r: requests.Response):
    print("=======Response=======")
    print("Headers:", r.headers)
    print("Body:", json.dumps(r.json(), indent=2))

def create_nvue_changest():
    r = requests.post(url=nvue_end_point + "/revision",
                      auth=auth,
                      verify=False)
    print_request(r.request)
    print_response(r)
    response = r.json()
    changeset = response.popitem()[0]
    return changeset

def apply_nvue_changeset(changeset):
    # apply_payload = {"state": "apply"}
    apply_payload = {"state": "apply", "auto-prompt": {"ays": "ays_yes"}}
    url = nvue_end_point + "/revision/" + requests.utils.quote(changeset,
                                                               safe="")
    r = requests.patch(url=url,
                       auth=auth,
                       verify=False,
                       data=json.dumps(apply_payload),
                       headers=mime_header)
    print_request(r.request)
    print_response(r)

def is_config_applied(changeset) -> bool:
    # Check if the configuration was indeed applied
    global RETRIES
    global POLL_APPLIED
    retries = RETRIES
    while retries > 0:
        r = requests.get(url=nvue_end_point + "/revision/" + requests.utils.quote(changeset, safe=""),
                         auth=auth,
                         verify=False)
        response = r.json()
        print(response)

        if response["state"] == "applied":
            return True
        retries -= 1
        time.sleep(POLL_APPLIED)

    return False

def apply_new_config(path,payload):
    # Create a new revision ID
    changeset = create_nvue_changest()
    print("Using NVUE Changeset: '{}'".format(changeset))

    # Delete existing configuration
    query_string = {"rev": changeset}
    r = requests.delete(url=nvue_end_point + path,
                       auth=auth,
                       verify=False,
                       params=query_string,
                       headers=mime_header)
    print_request(r.request)
    print_response(r)

    # Patch the new configuration
    
    query_string = {"rev": changeset}
    r = requests.patch(url=nvue_end_point + path,
                       auth=auth,
                       verify=False,
                       data=json.dumps(payload),
                       params=query_string,
                       headers=mime_header)
    print_request(r.request)
    print_response(r)

    # Apply the changes to the new revision changeset
    apply_nvue_changeset(changeset)

    # Check if the changeset was applied
    is_config_applied(changeset)

def nvue_get(path):
    r = requests.get(url=nvue_end_point + path,
                     auth=auth,
                     verify=False)
    print_request(r.request)
    print_response(r)

if __name__ == "__main__":
    payload = {
      "system": 
      {
        "hostname":"switch01",
        "timezone":"America/Los_Angeles",
        "message":
        {
          "pre-login":"Welcome to NVIDIA Cumulus Linux",
          "post-login:"You have successfully logged in to switch01"
        }
      }
    }
    apply_new_config("/",payload) # Root patch
    time.sleep(DUMMY_SLEEP)
    nvue_get("/system")

cumulus@switch:~$ nv set system hostname switch01
cumulus@switch:~$ nv set system timezone America/Los_Angeles
cumulus@switch:~$ nv set system message pre-login "Welcome to NVIDIA Cumulus Linux"
cumulus@switch:~$ nv set system message post-login "You have successfully logged into switch01"

Configure Services

To set up NTP, DNS, and SNMP on the switch, send a targeted API request to /nvue_v1/service.

cumulus@switch:~$ curl -u 'cumulus:cumulus' -d '{"service": { "ntp": {"default":{"server":{"4.cumulusnetworks.pool.ntp.org":{"iburst":"on"}}}}, "dns": {"mgmt":{"server":{"192.168.1.100":{}}}}, "syslog": {"mgmt":{"server":{"192.168.1.120":{"port":8000}}}}}}' -k -X PATCH https://127.0.0.1:8765/nvue_v1/?rev=5

#!/usr/bin/env python3

import requests
from requests.auth import HTTPBasicAuth
import json
import time

auth = HTTPBasicAuth(username="cumulus", password="password")
nvue_end_point = "https://127.0.0.1:8765/nvue_v1"
mime_header = {"Content-Type": "application/json"}

DUMMY_SLEEP = 5  # In seconds
POLL_APPLIED = 1  # in seconds
RETRIES = 10

def print_request(r: requests.Request):
    print("=======Request=======")
    print("URL:", r.url)
    print("Headers:", r.headers)
    print("Body:", r.body)

def print_response(r: requests.Response):
    print("=======Response=======")
    print("Headers:", r.headers)
    print("Body:", json.dumps(r.json(), indent=2))

def create_nvue_changest():
    r = requests.post(url=nvue_end_point + "/revision",
                      auth=auth,
                      verify=False)
    print_request(r.request)
    print_response(r)
    response = r.json()
    changeset = response.popitem()[0]
    return changeset

def apply_nvue_changeset(changeset):
    # apply_payload = {"state": "apply"}
    apply_payload = {"state": "apply", "auto-prompt": {"ays": "ays_yes"}}
    url = nvue_end_point + "/revision/" + requests.utils.quote(changeset,
                                                               safe="")
    r = requests.patch(url=url,
                       auth=auth,
                       verify=False,
                       data=json.dumps(apply_payload),
                       headers=mime_header)
    print_request(r.request)
    print_response(r)

def is_config_applied(changeset) -> bool:
    # Check if the configuration was indeed applied
    global RETRIES
    global POLL_APPLIED
    retries = RETRIES
    while retries > 0:
        r = requests.get(url=nvue_end_point + "/revision/" + requests.utils.quote(changeset, safe=""),
                         auth=auth,
                         verify=False)
        response = r.json()
        print(response)

        if response["state"] == "applied":
            return True
        retries -= 1
        time.sleep(POLL_APPLIED)

    return False

def apply_new_config(path,payload):
    # Create a new revision ID
    changeset = create_nvue_changest()
    print("Using NVUE Changeset: '{}'".format(changeset))

    # Delete existing configuration
    query_string = {"rev": changeset}
    r = requests.delete(url=nvue_end_point + path,
                       auth=auth,
                       verify=False,
                       params=query_string,
                       headers=mime_header)
    print_request(r.request)
    print_response(r)

    # Patch the new configuration
    
    query_string = {"rev": changeset}
    r = requests.patch(url=nvue_end_point + path,
                       auth=auth,
                       verify=False,
                       data=json.dumps(payload),
                       params=query_string,
                       headers=mime_header)
    print_request(r.request)
    print_response(r)

    # Apply the changes to the new revision changeset
    apply_nvue_changeset(changeset)

    # Check if the changeset was applied
    is_config_applied(changeset)

def nvue_get(path):
    r = requests.get(url=nvue_end_point + path,
                     auth=auth,
                     verify=False)
    print_request(r.request)
    print_response(r)

if __name__ == "__main__":
    payload = {
      "service":
      {
        "ntp":
        {
          "default":
          {
            "server:
            {
              "4.cumulusnetworks.pool.ntp.org":
              {
                "iburst":"on"
              }
            }
          }
        },
        "dns":
        {
          "mgmt":
          {
            "server:
            {
              "192.168.1.100":{}
            }
          }
        },
        "syslog":
        {
          "mgmt":
          {
            "server:
            {
              "192.168.1.120":
              {
                "port":8000
              }
            }
          }
        }
      }
    }
    apply_new_config("/",payload) # Root patch
    time.sleep(DUMMY_SLEEP)
    nvue_get("/service/ntp")
    nvue_get("/service/dns")
    nvue_get("/service/syslog")

cumulus@switch:~$ nv set service ntp default server 4.cumulusnetworks.pool.ntp.org iburst on
cumulus@switch:~$ nv set service dns mgmt server 192.168.1.100 
cumulus@switch:~$ nv set service syslog mgmt server 192.168.1.120 port 8000

Configure Users

The following example creates a new user, then deletes the user.

cumulus@switch:~$ curl -u 'cumulus:cumulus' -d '{"system": {"aaa": {"user": {"test1": {"hashed-password":"72b28582708d749c6c82f3b3f226041f1bd37090281641eaeba8d44bd915d0042d609a92759d9f6fb96475cb0601cf428cd22613df8a53a09461e0b426cf0a35","role": "nvue-monitor","enable": "on","full-name": "Test User"}}}}}' -k -X PATCH https://127.0.0.1:8765/nvue_v1/?rev=5

cumulus@switch:~$ curl -u 'cumulus:cumulus' -k -X DELETE https://127.0.0.1:8765/nvue_v1/system/aaa/user/test1?rev=6

#!/usr/bin/env python3

import requests
from requests.auth import HTTPBasicAuth
import json
import time

auth = HTTPBasicAuth(username="cumulus", password="password")
nvue_end_point = "https://127.0.0.1:8765/nvue_v1"
mime_header = {"Content-Type": "application/json"}

DUMMY_SLEEP = 5  # In seconds
POLL_APPLIED = 1  # in seconds
RETRIES = 10

def print_request(r: requests.Request):
    print("=======Request=======")
    print("URL:", r.url)
    print("Headers:", r.headers)
    print("Body:", r.body)

def print_response(r: requests.Response):
    print("=======Response=======")
    print("Headers:", r.headers)
    print("Body:", json.dumps(r.json(), indent=2))

def create_nvue_changest():
    r = requests.post(url=nvue_end_point + "/revision",
                      auth=auth,
                      verify=False)
    print_request(r.request)
    print_response(r)
    response = r.json()
    changeset = response.popitem()[0]
    return changeset

def apply_nvue_changeset(changeset):
    # apply_payload = {"state": "apply"}
    apply_payload = {"state": "apply", "auto-prompt": {"ays": "ays_yes"}}
    url = nvue_end_point + "/revision/" + requests.utils.quote(changeset,
                                                               safe="")
    r = requests.patch(url=url,
                       auth=auth,
                       verify=False,
                       data=json.dumps(apply_payload),
                       headers=mime_header)
    print_request(r.request)
    print_response(r)

def is_config_applied(changeset) -> bool:
    # Check if the configuration was indeed applied
    global RETRIES
    global POLL_APPLIED
    retries = RETRIES
    while retries > 0:
        r = requests.get(url=nvue_end_point + "/revision/" + requests.utils.quote(changeset, safe=""),
                         auth=auth,
                         verify=False)
        response = r.json()
        print(response)

        if response["state"] == "applied":
            return True
        retries -= 1
        time.sleep(POLL_APPLIED)

    return False

def apply_new_config(path,payload):
    # Create a new revision ID
    changeset = create_nvue_changest()
    print("Using NVUE Changeset: '{}'".format(changeset))

    # Delete existing configuration
    query_string = {"rev": changeset}
    r = requests.delete(url=nvue_end_point + path,
                       auth=auth,
                       verify=False,
                       params=query_string,
                       headers=mime_header)
    print_request(r.request)
    print_response(r)

    # Patch the new configuration
    
    query_string = {"rev": changeset}
    r = requests.patch(url=nvue_end_point + path,
                       auth=auth,
                       verify=False,
                       data=json.dumps(payload),
                       params=query_string,
                       headers=mime_header)
    print_request(r.request)
    print_response(r)

    # Apply the changes to the new revision changeset
    apply_nvue_changeset(changeset)

    # Check if the changeset was applied
    is_config_applied(changeset)

def delete_config(path):
    # Create an NVUE changeset
    changeset = create_nvue_changest()
    print("Using NVUE Changeset: '{}'".format(changeset))

    # Equivalent to JSON `null`
    payload = None

    # Stage the change
    query_string = {"rev": changeset}
    r = requests.delete(url=nvue_end_point + path,
                        auth=auth,
                        verify=False,
                        data=json.dumps(payload),
                        params=query_string,
                        headers=mime_header)
    print_request(r.request)
    print_response(r)

    # Apply the staged changeset
    apply_nvue_changeset(changeset)

    # Check if the changeset was applied
    is_config_applied(changeset)

def nvue_get(path):
    r = requests.get(url=nvue_end_point + path,
                     auth=auth,
                     verify=False)
    print_request(r.request)
    print_response(r)

if __name__ == "__main__":

    # Need to create a hashed password - The supported password
    # hashes are documented here:
    # https://docs.nvidia.com/networking-ethernet-software/cumulus-linux-55/System-Configuration/Authentication-Authorization-and-Accounting/User-Accounts/#hashed-passwords  # noqa
    # Here in this example, we use SHA-512
    import crypt
    hashed_password = crypt.crypt("hello$world#2023", salt=crypt.METHOD_SHA512)
    payload = {
        "system": {
            "aaa": {
                "user": {
                    "test1": {
                        "hashed-password": hashed_password,
                        "role": "nvue-monitor",
                        "enable": "on",
                        "full-name": "Test User",
                    }
                }
            }
        }
    }
    apply_new_config("/",payload) # Root patch
    time.sleep(DUMMY_SLEEP)
    nvue_get("/system/user/aaa")

    """Delete an existing user account using the AAA API."""
    delete_config("/system/aaa/user/test1")
    time.sleep(DUMMY_SLEEP)
    nvue_get("/system/user/aaa")

cumulus@switch:~$ nv set system aaa user test1
cumulus@switch:~$ nv set system aaa user test1 full-name "Test User" 
cumulus@switch:~$ nv set system aaa user test1 password "abcd@test"
cumulus@switch:~$ nv set system aaa user test1 role nvue-monitor
cumulus@switch:~$ nv set system aaa user test1 enable on

cumulus@switch:~$ nv unset system aaa user test1

Configure an Interface

The following example configures an interface.

cumulus@switch:~$ curl -u 'cumulus:cumulus' -k -d '{"swp1": {"link":{"state":{"up": {}}}}}' -H 'Content-Type: application/json' -k -X PATCH https://127.0.0.1:8765/nvue_v1/interface?rev=21

#!/usr/bin/env python3

import requests
from requests.auth import HTTPBasicAuth
import json
import time

auth = HTTPBasicAuth(username="cumulus", password="password")
nvue_end_point = "https://127.0.0.1:8765/nvue_v1"
mime_header = {"Content-Type": "application/json"}

DUMMY_SLEEP = 5  # In seconds
POLL_APPLIED = 1  # in seconds
RETRIES = 10

def print_request(r: requests.Request):
    print("=======Request=======")
    print("URL:", r.url)
    print("Headers:", r.headers)
    print("Body:", r.body)

def print_response(r: requests.Response):
    print("=======Response=======")
    print("Headers:", r.headers)
    print("Body:", json.dumps(r.json(), indent=2))

def create_nvue_changest():
    r = requests.post(url=nvue_end_point + "/revision",
                      auth=auth,
                      verify=False)
    print_request(r.request)
    print_response(r)
    response = r.json()
    changeset = response.popitem()[0]
    return changeset

def apply_nvue_changeset(changeset):
    # apply_payload = {"state": "apply"}
    apply_payload = {"state": "apply", "auto-prompt": {"ays": "ays_yes"}}
    url = nvue_end_point + "/revision/" + requests.utils.quote(changeset,
                                                               safe="")
    r = requests.patch(url=url,
                       auth=auth,
                       verify=False,
                       data=json.dumps(apply_payload),
                       headers=mime_header)
    print_request(r.request)
    print_response(r)

def is_config_applied(changeset) -> bool:
    # Check if the configuration was indeed applied
    global RETRIES
    global POLL_APPLIED
    retries = RETRIES
    while retries > 0:
        r = requests.get(url=nvue_end_point + "/revision/" + requests.utils.quote(changeset, safe=""),
                         auth=auth,
                         verify=False)
        response = r.json()
        print(response)

        if response["state"] == "applied":
            return True
        retries -= 1
        time.sleep(POLL_APPLIED)

    return False

def apply_new_config(path,payload):
    # Create a new revision ID
    changeset = create_nvue_changest()
    print("Using NVUE Changeset: '{}'".format(changeset))

    # Delete existing configuration
    query_string = {"rev": changeset}
    r = requests.delete(url=nvue_end_point + path,
                       auth=auth,
                       verify=False,
                       params=query_string,
                       headers=mime_header)
    print_request(r.request)
    print_response(r)

    # Patch the new configuration
    
    query_string = {"rev": changeset}
    r = requests.patch(url=nvue_end_point + path,
                       auth=auth,
                       verify=False,
                       data=json.dumps(payload),
                       params=query_string,
                       headers=mime_header)
    print_request(r.request)
    print_response(r)

    # Apply the changes to the new revision changeset
    apply_nvue_changeset(changeset)

    # Check if the changeset was applied
    is_config_applied(changeset)

def nvue_get(path):
    r = requests.get(url=nvue_end_point + path,
                     auth=auth,
                     verify=False)
    print_request(r.request)
    print_response(r)

if __name__ == "__main__":
    payload = {
      "swp1":
      {
        "type":"swp",
        "link":
        {
          "state":"up"
          }
        }
      }
    apply_new_config("/interface",payload)
    time.sleep(DUMMY_SLEEP)
    nvue_get("/interface/swp1")

cumulus@switch:~$ nv set interface swp1

Configure a Bond

The following example configures a bond.

cumulus@switch:~$ curl -u 'cumulus:cumulus' -d '{"bond0": {"type":"bond","bond":{"member":{"swp1":{},"swp2":{},"swp3":{},"swp4":{}}}}}' -H 'Content-Type: application/json' -k -X PATCH https://127.0.0.1:8765/nvue_v1/interface?rev=7
{
  "bond0": {
    "bond": {
      "member": {
        "swp1": {},
        "swp2": {},
        "swp3": {},
        "swp4": {}
      }
    },
    "type": "bond"
  },
  "bond1": {
    "bond": {
      "lacp-bypass": "on",
      "member": {
        "swp1": {}
      },
      "mlag": {
        "enable": "on",
        "id": 1
      },
      "mode": "lacp"
    },
    "bridge": {
      "domain": {
        "br_default": {
          "access": 10,
          "stp": {
            "admin-edge": "on",
            "auto-edge": "on",
            "bpdu-guard": "on"
          }
        }
      }
    },
    "link": {
      "mtu": 9000
    },
    "type": "bond"
  },
  "eth0": {
    "ip": {
      "address": {
        "192.168.200.6/24": {}
      },
      "vrf": "mgmt"
    },
    "type": "eth"
  },
  "lo": {
    "ip": {
      "address": {
        "10.10.10.1/32": {}
      }
    },
    "type": "loopback"
  }
}

#!/usr/bin/env python3

import requests
from requests.auth import HTTPBasicAuth
import json
import time

auth = HTTPBasicAuth(username="cumulus", password="password")
nvue_end_point = "https://127.0.0.1:8765/nvue_v1"
mime_header = {"Content-Type": "application/json"}

DUMMY_SLEEP = 5  # In seconds
POLL_APPLIED = 1  # in seconds
RETRIES = 10

def print_request(r: requests.Request):
    print("=======Request=======")
    print("URL:", r.url)
    print("Headers:", r.headers)
    print("Body:", r.body)

def print_response(r: requests.Response):
    print("=======Response=======")
    print("Headers:", r.headers)
    print("Body:", json.dumps(r.json(), indent=2))

def create_nvue_changest():
    r = requests.post(url=nvue_end_point + "/revision",
                      auth=auth,
                      verify=False)
    print_request(r.request)
    print_response(r)
    response = r.json()
    changeset = response.popitem()[0]
    return changeset

def apply_nvue_changeset(changeset):
    # apply_payload = {"state": "apply"}
    apply_payload = {"state": "apply", "auto-prompt": {"ays": "ays_yes"}}
    url = nvue_end_point + "/revision/" + requests.utils.quote(changeset,
                                                               safe="")
    r = requests.patch(url=url,
                       auth=auth,
                       verify=False,
                       data=json.dumps(apply_payload),
                       headers=mime_header)
    print_request(r.request)
    print_response(r)

def is_config_applied(changeset) -> bool:
    # Check if the configuration was indeed applied
    global RETRIES
    global POLL_APPLIED
    retries = RETRIES
    while retries > 0:
        r = requests.get(url=nvue_end_point + "/revision/" + requests.utils.quote(changeset, safe=""),
                         auth=auth,
                         verify=False)
        response = r.json()
        print(response)

        if response["state"] == "applied":
            return True
        retries -= 1
        time.sleep(POLL_APPLIED)

    return False

def apply_new_config(path,payload):
    # Create a new revision ID
    changeset = create_nvue_changest()
    print("Using NVUE Changeset: '{}'".format(changeset))

    # Delete existing configuration
    query_string = {"rev": changeset}
    r = requests.delete(url=nvue_end_point + path,
                       auth=auth,
                       verify=False,
                       params=query_string,
                       headers=mime_header)
    print_request(r.request)
    print_response(r)

    # Patch the new configuration
    
    query_string = {"rev": changeset}
    r = requests.patch(url=nvue_end_point + path,
                       auth=auth,
                       verify=False,
                       data=json.dumps(payload),
                       params=query_string,
                       headers=mime_header)
    print_request(r.request)
    print_response(r)

    # Apply the changes to the new revision changeset
    apply_nvue_changeset(changeset)

    # Check if the changeset was applied
    is_config_applied(changeset)

def nvue_get(path):
    r = requests.get(url=nvue_end_point + path,
                     auth=auth,
                     verify=False)
    print_request(r.request)
    print_response(r)

if __name__ == "__main__":
    payload = {
      "bond0":
      {
        "type":"bond",
        "bond":
        {
          "member":
          {
            "swp1":{},
            "swp2":{},
            "swp3":{},
            "swp4":{}
          }
        }
      }
    }
    apply_new_config("/interface",payload)
    time.sleep(DUMMY_SLEEP)
    nvue_get("/interface/bond0")

cumulus@switch:~$ nv set interface bond0 bond member swp1-4

Configure a Bridge

The following example configures a bridge.

cumulus@switch:~$ curl -u 'cumulus:cumulus' -d '{"swp1": {"bridge":{"domain":{"br_default":{}}}},"swp2": {"bridge":{"domain":{"br_default":{}}}}}' -H 'Content-Type: application/json' -k -X PATCH https://127.0.0.1:8765/nvue_v1/interface?rev=21
cumulus@switch:~$ curl -u 'cumulus:cumulus' -d '{"untagged":1,"vlan":{"10":{},"20":{}}}' -H 'Content-Type: application/json' -k -X PATCH https://127.0.0.1:8765/nvue_v1/bridge/domain/br_default?rev=8

#!/usr/bin/env python3

import requests
from requests.auth import HTTPBasicAuth
import json
import time

auth = HTTPBasicAuth(username="cumulus", password="password")
nvue_end_point = "https://127.0.0.1:8765/nvue_v1"
mime_header = {"Content-Type": "application/json"}

DUMMY_SLEEP = 5  # In seconds
POLL_APPLIED = 1  # in seconds
RETRIES = 10

def print_request(r: requests.Request):
    print("=======Request=======")
    print("URL:", r.url)
    print("Headers:", r.headers)
    print("Body:", r.body)

def print_response(r: requests.Response):
    print("=======Response=======")
    print("Headers:", r.headers)
    print("Body:", json.dumps(r.json(), indent=2))

def create_nvue_changest():
    r = requests.post(url=nvue_end_point + "/revision",
                      auth=auth,
                      verify=False)
    print_request(r.request)
    print_response(r)
    response = r.json()
    changeset = response.popitem()[0]
    return changeset

def apply_nvue_changeset(changeset):
    # apply_payload = {"state": "apply"}
    apply_payload = {"state": "apply", "auto-prompt": {"ays": "ays_yes"}}
    url = nvue_end_point + "/revision/" + requests.utils.quote(changeset,
                                                               safe="")
    r = requests.patch(url=url,
                       auth=auth,
                       verify=False,
                       data=json.dumps(apply_payload),
                       headers=mime_header)
    print_request(r.request)
    print_response(r)

def is_config_applied(changeset) -> bool:
    # Check if the configuration was indeed applied
    global RETRIES
    global POLL_APPLIED
    retries = RETRIES
    while retries > 0:
        r = requests.get(url=nvue_end_point + "/revision/" + requests.utils.quote(changeset, safe=""),
                         auth=auth,
                         verify=False)
        response = r.json()
        print(response)

        if response["state"] == "applied":
            return True
        retries -= 1
        time.sleep(POLL_APPLIED)

    return False

def apply_new_config(path,payload):
    # Create a new revision ID
    changeset = create_nvue_changest()
    print("Using NVUE Changeset: '{}'".format(changeset))

    # Delete existing configuration
    query_string = {"rev": changeset}
    r = requests.delete(url=nvue_end_point + path,
                       auth=auth,
                       verify=False,
                       params=query_string,
                       headers=mime_header)
    print_request(r.request)
    print_response(r)

    # Patch the new configuration
    
    query_string = {"rev": changeset}
    r = requests.patch(url=nvue_end_point + path,
                       auth=auth,
                       verify=False,
                       data=json.dumps(payload),
                       params=query_string,
                       headers=mime_header)
    print_request(r.request)
    print_response(r)

    # Apply the changes to the new revision changeset
    apply_nvue_changeset(changeset)

    # Check if the changeset was applied
    is_config_applied(changeset)

def nvue_get(path):
    r = requests.get(url=nvue_end_point + path,
                     auth=auth,
                     verify=False)
    print_request(r.request)
    print_response(r)

if __name__ == "__main__":
    int_payload = {
      "swp1":
      {
        "bridge":
        {
          "domain":
          {
            "br_default":{}
          }
        },
        "swp2": 
        {
          "bridge":
          {
            "domain":
            {
              "br_default":{}
            }
          }
        }
      }
    }
    apply_new_config("/interface",int_payload)
    br_payload = {
      "untagged":1,
      "vlan":
      {
        "10":{},
        "20":{}
      }
    }
    apply_new_config("/bridge/domain/br_default",br_payload)
    time.sleep(DUMMY_SLEEP)
    nvue_get("/interface/swp1")
    nvue_get("/bridge/domain/br_default")

cumulus@switch:~$ nv set interface swp1-2 bridge domain br_default
cumulus@switch:~$ nv set bridge domain br_default vlan 10,20
cumulus@switch:~$ nv set bridge domain br_default untagged 1

Configure BGP

The following example configures BGP.

cumulus@switch:~$ curl -u 'cumulus:cumulus' -d '{"bgp": {"autonomous-system": 65101,"router-id":"10.10.10.1"}}' -H 'Content-Type: application/json' -k -X PATCH https://127.0.0.1:8765/nvue_v1/router?rev=9
cumulus@switch:~$ curl -u 'cumulus:cumulus' -d '{"bgp":{"neighbor":{"swp51":{"remote-as":"external"}},"address-family":{"ipv4-unicast":{"network":{"10.10.10.1/32":{}}}}}}' -H 'Content-Type: application/json' -k -X PATCH https://127.0.0.1:8765/nvue_v1/vrf/default/router?rev=9

#!/usr/bin/env python3

import requests
from requests.auth import HTTPBasicAuth
import json
import time

auth = HTTPBasicAuth(username="cumulus", password="password")
nvue_end_point = "https://127.0.0.1:8765/nvue_v1"
mime_header = {"Content-Type": "application/json"}

DUMMY_SLEEP = 5  # In seconds
POLL_APPLIED = 1  # in seconds
RETRIES = 10

def print_request(r: requests.Request):
    print("=======Request=======")
    print("URL:", r.url)
    print("Headers:", r.headers)
    print("Body:", r.body)

def print_response(r: requests.Response):
    print("=======Response=======")
    print("Headers:", r.headers)
    print("Body:", json.dumps(r.json(), indent=2))

def create_nvue_changest():
    r = requests.post(url=nvue_end_point + "/revision",
                      auth=auth,
                      verify=False)
    print_request(r.request)
    print_response(r)
    response = r.json()
    changeset = response.popitem()[0]
    return changeset

def apply_nvue_changeset(changeset):
    # apply_payload = {"state": "apply"}
    apply_payload = {"state": "apply", "auto-prompt": {"ays": "ays_yes"}}
    url = nvue_end_point + "/revision/" + requests.utils.quote(changeset,
                                                               safe="")
    r = requests.patch(url=url,
                       auth=auth,
                       verify=False,
                       data=json.dumps(apply_payload),
                       headers=mime_header)
    print_request(r.request)
    print_response(r)

def is_config_applied(changeset) -> bool:
    # Check if the configuration was indeed applied
    global RETRIES
    global POLL_APPLIED
    retries = RETRIES
    while retries > 0:
        r = requests.get(url=nvue_end_point + "/revision/" + requests.utils.quote(changeset, safe=""),
                         auth=auth,
                         verify=False)
        response = r.json()
        print(response)

        if response["state"] == "applied":
            return True
        retries -= 1
        time.sleep(POLL_APPLIED)

    return False

def apply_new_config(path,payload):
    # Create a new revision ID
    changeset = create_nvue_changest()
    print("Using NVUE Changeset: '{}'".format(changeset))

    # Delete existing configuration
    query_string = {"rev": changeset}
    r = requests.delete(url=nvue_end_point + path,
                       auth=auth,
                       verify=False,
                       params=query_string,
                       headers=mime_header)
    print_request(r.request)
    print_response(r)

    # Patch the new configuration
    
    query_string = {"rev": changeset}
    r = requests.patch(url=nvue_end_point + path,
                       auth=auth,
                       verify=False,
                       data=json.dumps(payload),
                       params=query_string,
                       headers=mime_header)
    print_request(r.request)
    print_response(r)

    # Apply the changes to the new revision changeset
    apply_nvue_changeset(changeset)

    # Check if the changeset was applied
    is_config_applied(changeset)

def nvue_get(path):
    r = requests.get(url=nvue_end_point + path,
                     auth=auth,
                     verify=False)
    print_request(r.request)
    print_response(r)

if __name__ == "__main__":
    rt_payload = {
      "bgp":
      {
        "autonomous-system": 65101,
        "router-id":"10.10.10.1"
      }
    }
    apply_new_config("/router",rt_payload)
    vrf_payload = {
      "bgp":
      {
        "neighbor":
        {
          "swp51":
          {
            "remote-as":"external"
          }
        },
        "address-family":
        {
          "ipv4-unicast":
          {
            "network":
            {
              "10.10.10.1/32":{}
            }
          }
        }
      }
    }
    apply_new_config("/vrf/default/router",vrf_payload)
    time.sleep(DUMMY_SLEEP)
    nvue_get("/router")
    nvue_get("/vrf/default/router")

cumulus@switch:~$ nv set router bgp autonomous-system 65101
cumulus@switch:~$ nv set router bgp router-id 10.10.10.1
cumulus@switch:~$ nv set vrf default router bgp neighbor swp51 remote-as external
cumulus@switch:~$ nv set vrf default router bgp address-family ipv4-unicast network 10.10.10.1/32

Action Operations

The NVUE action operations are ephemeral operations that do not modify the state of the configuration; they reset counters for interfaces, BGP, QoS buffers and pools, and remove conflicts from protodown MLAG bonds.

cumulus@switch:~$ curl -u 'cumulus:cumulus' -H 'Content-Type:application/json' -d '{"@clear": {"state": "start", "parameters": {}}}' -k -X POST https://127.0.0.1:8765/nvue_v1/interface/swp1/counters
1
cumulus@switch:~$ curl -u 'cumulus:cumulus' -X GET https://127.0.0.1:8765/nvue_v1/action/1 -k
{"detail":"swp1 counters cleared.","http_status":200,"issue":[],"state":"action_success","status":"swp1 counters cleared.","timeout":60,"type":""}

cumulus@switch:~$ curl -u 'cumulus:cumulus' -H 'Content-Type:application/json' -d '{"@clear": {"state": "start", "parameters": {}}}' -k -X POST https://127.0.0.1:8765/nvue_v1/interface/swp1/qos/buffer
2
cumulus@switch:~$ curl -u 'cumulus:cumulus'  -X GET https://127.0.0.1:8765/nvue_v1/action/2 -k
{"detail":"QoS buffers cleared on swp1.","http_status":200,"issue":[],"state":"action_success","status":"QoS buffers cleared on swp1.","timeout":60,"type":""}

#!/usr/bin/env python3

import requests
from requests.auth import HTTPBasicAuth
import json
import time

auth = HTTPBasicAuth(username="cumulus", password="password")
nvue_end_point = "https://127.0.0.1:8765/nvue_v1"
mime_header = {"Content-Type": "application/json"}

DUMMY_SLEEP = 5  # In seconds
POLL_APPLIED = 1  # in seconds
RETRIES = 10

def print_request(r: requests.Request):
    print("=======Request=======")
    print("URL:", r.url)
    print("Headers:", r.headers)
    print("Body:", r.body)

def print_response(r: requests.Response):
    print("=======Response=======")
    print("Headers:", r.headers)
    print("Body:", json.dumps(r.json(), indent=2))

def nvue_action():
    r = requests.post(url=nvue_end_point + path,
                      auth=auth,
                      verify=False,
                      data=json.dumps(apply_payload),
                      headers=mime_header)
    print_request(r.request)
    print_response(r)
    return response

def nvue_get(path):
    r = requests.get(url=nvue_end_point + path,
                     auth=auth,
                     verify=False)
    print_request(r.request)
    print_response(r)

if __name__ == "__main__":
    payload = {
      "@clear": 
      {
        "state": "start", 
        "parameters": {}
      }
    }
    action_id=nvue_action("/interface/swp1/qos/counter",payload)
    time.sleep(DUMMY_SLEEP)
    nvue_get(f"/action/{action_id}")
   

cumulus@switch:~$ nv action clear interface swp1 qos counter

Example Python Script

In the following python example, the full_config_example() method sets the system pre-login message, enables BGP globally, and changes a few other configuration settings in a single bulk operation. The API end-point goes to the root node /nvue_v1. The bridge_config_example() method performs a targeted API request to /nvue_v1/bridge/domain/<domain-id> to set the vlan-vni-offset attribute.

#!/usr/bin/env python3

import requests
from requests.auth import HTTPBasicAuth
import json
import time

auth = HTTPBasicAuth(username="vagrant", password="vagrant")
nvue_end_point = "https://127.0.0.1:8765/nvue_v1"
mime_header = {"Content-Type": "application/json"}

DUMMY_SLEEP = 5  # In seconds
POLL_APPLIED = 1  # in seconds
RETRIES = 10

def print_request(r: requests.Request):
    print("=======Request=======")
    print("URL:", r.url)
    print("Headers:", r.headers)
    print("Body:", r.body)

def print_response(r: requests.Response):
    print("=======Response=======")
    print("Headers:", r.headers)
    print("Body:", json.dumps(r.json(), indent=2))

def sanity():
    # Basic retrieval to check connectivity
    r = requests.get(url=nvue_end_point + "/system",
                     auth=auth,
                     verify=False)
    print_request(r.request)
    print_response(r)

def create_nvue_changest():
    r = requests.post(url=nvue_end_point + "/revision",
                      auth=auth,
                      verify=False)
    print_request(r.request)
    print_response(r)
    response = r.json()
    changeset = response.popitem()[0]
    return changeset

def apply_nvue_changeset(changeset):
    # apply_payload = {"state": "apply"}
    apply_payload = {"state": "apply", "auto-prompt": {"ays": "ays_yes"}}
    url = nvue_end_point + "/revision/" + requests.utils.quote(changeset,
                                                               safe="")
    r = requests.patch(url=url,
                       auth=auth,
                       verify=False,
                       data=json.dumps(apply_payload),
                       headers=mime_header)
    print_request(r.request)
    print_response(r)

def full_config_example():
    # Create an NVUE changeset
    changeset = create_nvue_changest()
    print("Using NVUE Changeset: '{}'".format(changeset))

    # https://www.asciiart.eu/comics/batman
    pre_login_message = u"""
                   ,.ood888888888888boo.,
              .od888P^""            ""^Y888bo.
          .od8P''   ..oood88888888booo.    ``Y8bo.
       .odP'"  .ood8888888888888888888888boo.  "`Ybo.
     .d8'   od8'd888888888f`8888't888888888b`8bo   `Yb.
    d8'  od8^   8888888888[  `'  ]8888888888   ^8bo  `8b
  .8P  d88'     8888888888P      Y8888888888     `88b  Y8.
 d8' .d8'       `Y88888888'      `88888888P'       `8b. `8b
.8P .88P            """"            """"            Y88. Y8.
88  888                                              888  88
88  888                                              888  88
88  888.        ..                        ..        .888  88
`8b `88b,     d8888b.od8bo.      .od8bo.d8888b     ,d88' d8'
 Y8. `Y88.    8888888888888b    d8888888888888    .88P' .8P
  `8b  Y88b.  `88888888888888  88888888888888'  .d88P  d8'
    Y8.  ^Y88bod8888888888888..8888888888888bod88P^  .8P
     `Y8.   ^Y888888888888888LS888888888888888P^   .8P'
       `^Yb.,  `^^Y8888888888888888888888P^^'  ,.dP^'
          `^Y8b..   ``^^^Y88888888P^^^'    ..d8P^'
              `^Y888bo.,            ,.od888P^'
                   "`^^Y888888888888P^^'"
"""

    # https://www.asciiart.eu/comics/superman
    post_login_message = u'''
        _____________________________________________
      //:::::::::::::::::::::::::::::::::::::::::::::\\
    //:::_______:::::::::________::::::::::_____:::::::\\
  //:::_/   _-"":::_--"""        """--_::::\_  ):::::::::\\
 //:::/    /:::::_"                    "-_:::\/:::::|^\:::\\
//:::/   /~::::::I__                      \:::::::::|  \:::\\
\\:::\   (::::::::::""""---___________     "--------"  /::://
 \\:::\  |::::::::::::::::::::::::::::""""==____      /::://
  \\:::"\/::::::::::::::::::::::::::::::::::::::\   /~::://
    \\:::::::::::::::::::::::::::::::::::::::::::)/~::://
      \\::::\""""""------_____::::::::::::::::::::::://
        \\:::"\               """""-----_____:::::://
          \\:::"\    __----__                )::://
            \\:::"\/~::::::::~\_         __/~:://
              \\::::::::::::::::""----""":::://
                \\::::::::::::::::::::::::://
                  \\:::\^""--._.--""^/::://
                    \\::"\         /":://
                      \\::"\     /":://
                        \\::"\_/":://
                          \\::::://
                            \\_//
                              "
'''

    # Prepare payload which configures a few
    # different switch configurations
    payload = {
        "interface":{
            "eth0":{
                "description": "management port"
            }
        },
        "router":{
            "bgp":{
                "enable":"on"
            }
        },
        "system":{
            "message":{
                "pre-login": pre_login_message,
                "post-login": post_login_message
            },
            "timezone": "Europe/Paris",
            "config": {
               "snippet": {
                   "test-flexible-snippet": {
                       "file": "/tmp/blah",
                       "content": "NVIDIA rocks"
                   },
                   "frr.conf": "hello world"
               }
            }
        },
        "service": {
            "ntp": {
                "mgmt": {
                    "listen": "eth0"
                }
            }
        }
    }
    # Stage the change
    query_string = {"rev": changeset}
    r = requests.patch(url=nvue_end_point + "/",  # Root patch
                       auth=auth,
                       verify=False,
                       data=json.dumps(payload),
                       params=query_string,
                       headers=mime_header)
    print_request(r.request)
    print_response(r)

    # Apply the staged changeset
    apply_nvue_changeset(changeset)

    # Check if the changeset was applied
    is_config_applied(changeset)

def bridge_config_example(domain_id):
    # Create an NVUE changeset
    changeset = create_nvue_changest()
    print("Using NVUE Changeset: '{}'".format(changeset))
    payload = {
        "vlan-vni-offset": 1000
    }

    # Stage the change
    query_string = {"rev": changeset}
    r = requests.patch(url=nvue_end_point + f"/bridge/domain/{domain_id}",
                       auth=auth,
                       verify=False,
                       data=json.dumps(payload),
                       params=query_string,
                       headers=mime_header)
    print_request(r.request)
    print_response(r)

    # Apply the staged changeset
    apply_nvue_changeset(changeset)

    # Check if the changeset was applied
    is_config_applied(changeset)

def message_get():
    # Get the system pre-login/post-login
    # message that was configured.
    r = requests.get(url=nvue_end_point + "/system/message",
                     auth=auth,
                     verify=False)
    print_request(r.request)
    print_response(r)

def is_config_applied(changeset) -> bool:
    # Check if the configuration was indeed applied
    global RETRIES
    global POLL_APPLIED
    retries = RETRIES
    while retries > 0:
        r = requests.get(url=nvue_end_point + "/revision/" + requests.utils.quote(changeset, safe=""),
                         auth=auth,
                         verify=False)
        response = r.json()
        print(response)

        if response["state"] == "applied":
            return True
        retries -= 1
        time.sleep(POLL_APPLIED)

    return False

if __name__ == "__main__":
    sanity()
    time.sleep(DUMMY_SLEEP)
    full_config_example()
    time.sleep(DUMMY_SLEEP)
    bridge_config_example("br_default")
    time.sleep(DUMMY_SLEEP)
    message_get()

Try the API

To try out the NVUE REST API, use the NVUE API Lab available on NVIDIA Air. The lab provides a basic example to help you get started. You can also try out the other examples in this document.

Resources

For information about using the NVUE REST API, refer to the NVUE API Swagger documentation. The full object model download is available here.

Considerations

• Unlike the NVUE CLI, the NVUE API does not support configuring a plain text password for a user account; you must configure a hashed password for a user account with the NVUE API.
• If you need to make multiple updates on the switch, NVIDIA recommends you use a root patch, which can make configuration changes with fewer round trips to the switch. Running many specific NVUE PATCH APIs to set or unset objects requires many round trips to the switch to set up the HTTP connection, transfer payload and responses, manage network utilization, and so on.

Related Information

• NGINX documentation
• Ubuntu Certificates and Security documentation
• Python requests module

The ntpd daemon running on the switch implements the NTP protocol. It synchronizes the system time with time servers in the /etc/ntp.conf file. The ntpd daemon starts at boot by default.

Configure NTP Servers

The default NTP configuration includes the following servers, which are in the /etc/ntp.conf file:

• server 0.cumulusnetworks.pool.ntp.org iburst
• server 1.cumulusnetworks.pool.ntp.org iburst
• server 2.cumulusnetworks.pool.ntp.org iburst
• server 3.cumulusnetworks.pool.ntp.org iburst

To add the NTP servers you want to use, run the following commands. Include the iburst option to increase the sync speed.

cumulus@switch:~$ nv set service ntp default server 4.cumulusnetworks.pool.ntp.org iburst on
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo nano /etc/ntp.conf
# pool.ntp.org maps to about 1000 low-stratum NTP servers.  Your server will
# pick a different set every time it starts up.  Please consider joining the
# pool: <http://www.pool.ntp.org/join.html>
server 0.cumulusnetworks.pool.ntp.org iburst
server 1.cumulusnetworks.pool.ntp.org iburst
server 2.cumulusnetworks.pool.ntp.org iburst
server 3.cumulusnetworks.pool.ntp.org iburst
server 4.cumulusnetworks.pool.ntp.org iburst

To verify that ntpd is running on the system:

cumulus@switch:~$ ps -ef | grep ntp
ntp       4074     1  0 Jun20 ?        00:00:33 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 101:102

To check the NTP peer status:

cumulus@switch:~$ nv show service ntp default server

cumulus@switch:~$ ntpq -p
      remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
+ec2-34-225-6-20 129.6.15.30      2 u   73 1024  377   70.414   -2.414   4.110
+lax1.m-d.net    132.163.96.1     2 u   69 1024  377   11.676    0.155   2.736
*69.195.159.158  199.102.46.72    2 u  133 1024  377   48.047   -0.457   1.856
-2.time.dbsinet. 198.60.22.240    2 u 1057 1024  377   63.973    2.182   2.692

The following example commands remove some of the default NTP servers:

cumulus@switch:~$ nv unset service ntp default server 0.cumulusnetworks.pool.ntp.org
cumulus@switch:~$ nv unset service ntp default server 1.cumulusnetworks.pool.ntp.org
cumulus@switch:~$ nv unset service ntp default server 2.cumulusnetworks.pool.ntp.org
cumulus@switch:~$ nv unset service ntp default server 3.cumulusnetworks.pool.ntp.org
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo nano /etc/ntp.conf
...
# pool.ntp.org maps to about 1000 low-stratum NTP servers.  Your server will
# pick a different set every time it starts up.  Please consider joining the
# pool: <http://www.pool.ntp.org/join.html>
server 4.cumulusnetworks.pool.ntp.org iburst
...

Specify the NTP Source Interface

By default, the source interface that NTP uses is eth0. The following example command configures the NTP source interface to be swp10.

cumulus@switch:~$ nv set service ntp default listen swp10
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo nano /etc/ntp.conf
...
# Specify interfaces
interface listen swp10
...

Use NTP in a DHCP Environment

You can use DHCP to specify your NTP servers. Ensure that the DHCP-generated configuration file /run/ntp.conf.dhcp exists. The /etc/dhcp/dhclient-exit-hooks.d/ntp script generates this file, which is a copy of the default /etc/ntp.conf file with a modified server list from the DHCP server. If this file does not exist and you plan on using DHCP in the future, you can copy your current /etc/ntp.conf file to the location of the DHCP file.

To use DHCP to specify your NTP servers, run the sudo -E systemctl edit ntp.service command and add the ExecStart= line:

cumulus@switch:~$ sudo -E systemctl edit ntp.service
[Service]
ExecStart=
ExecStart=/usr/sbin/ntpd -n -u ntp:ntp -g -c /run/ntp.conf.dhcp

To validate that your configuration, run these commands:

cumulus@switch:~$ sudo systemctl restart ntp
cumulus@switch:~$ sudo systemctl status -n0 ntp.service

If the state is not Active, or the alternate configuration file does not appear in the ntp command line, it is likely that you made a configuration mistake. Correct the mistake and rerun the commands above to verify.

Configure NTP with Authorization Keys

For added security, you can configure NTP to use authorization keys.

Configure the NTP Server

1. Create a .keys file, such as /etc/ntp.keys. Specify a key identifier (a number between 1 and 65535), an encryption method (M for MD5), and the password. The following provides an example:

   #
   # PLEASE DO NOT USE THE DEFAULT VALUES HERE.
   #
   #65535  M  akey
   #1      M  pass
   
   1  M  CumulusLinux!
   

2. In the /etc/ntp.conf file, add a pointer to the /etc/ntp.keys file you created above and specify the key identifier. For example:

   keys /etc/ntp/ntp.keys
   trustedkey 1
   controlkey 1
   requestkey 1
   

3. Restart NTP with the sudo systemctl restart ntp command.

Configure the NTP Client

The NTP client is the Cumulus Linux switch.

1. Create the same .keys file you created on the NTP server (/etc/ntp.keys). For example:

   cumulus@switch:~$  sudo nano /etc/ntp.keys
   #
   # DO NOT USE THE DEFAULT VALUES HERE.
   #
   #65535  M  akey
   #1      M  pass
   
   1  M  CumulusLinux!
   

2. Edit the /etc/ntp.conf file to specify the server you want to use, the key identifier, and a pointer to the /etc/ntp.keys file you created in step 1. For example:

   cumulus@switch:~$ sudo nano /etc/ntp.conf
   ...
   # You do need to talk to an NTP server or two (or three).
   #pool ntp.your-provider.example
   # OR
   #server ntp.your-provider.example
   
   # pool.ntp.org maps to about 1000 low-stratum NTP servers.  Your server will
   # pick a different set every time it starts up.  Please consider joining the
   # pool: <http://www.pool.ntp.org/join.html>
   #server 0.cumulusnetworks.pool.ntp.org iburst
   #server 1.cumulusnetworks.pool.ntp.org iburst
   #server 2.cumulusnetworks.pool.ntp.org iburst
   #server 3.cumulusnetworks.pool.ntp.org iburst
   server 10.50.23.121 key 1
   
   #keys
   keys /etc/ntp.keys
   trustedkey 1
   controlkey 1
   requestkey 1
   ...
   

3. Restart NTP in the active VRF (default or management). For example:

   cumulus@switch:~$ systemctl restart ntp@mgmt.service
   

4. Wait a few minutes, then run the ntpq -c as command to verify the configuration:

   cumulus@switch:~$ ntpq -c as
   
   ind assid status  conf reach auth condition  last_event cnt
   ===========================================================
     1 40828  f014   yes   yes   ok     reject   reachable  1
   

   After a successful authorization, you see the following command output:

   cumulus@switch:~$ ntpq -c as
   
   ind assid status  conf reach auth condition  last_event cnt
   ===========================================================
     1 40828  f61a   yes   yes   ok   sys.peer    sys_peer  1
   

Considerations

NTP in Cumulus Linux uses the /usr/share/zoneinfo/leap-seconds.list file, which expires periodically and results in generated log messages about the expiration. When the file expires, update it from https://www.ietf.org/timezones/data/leap-seconds.list or upgrade the tzdata package to the newest version.

Related Information

• NTP website
• Wikipedia - Network Time Protocol
• Debian wiki - NTP

Cumulus Linux supports IEEE 1588-2008 Precision Timing Protocol (PTPv2), which defines the algorithm and method for synchronizing clocks of various devices across packet-based networks, including Ethernet switches and IP routers.

PTP is capable of sub-microsecond accuracy. The clocks are in a master-slave hierarchy, where the slaves synchronize to their masters, which can be slaves to their own masters. The best master clock (BMC) algorithm, which runs on every clock, creates and updates the hierarchy automatically. The grandmaster clock is the top-level master. To provide a high-degree of accuracy, a Global Positioning System (GPS) time source typically synchronizes the grandmaster clock.

In the following example:

• Boundary clock 2 receives time from Master 1 (the grandmaster) on a PTP slave port, sets its clock and passes the time down from the PTP master port to Boundary clock 1.
• Boundary clock 1 receives the time on a PTP slave port, sets its clock and passes the time down the hierarchy through the PTP master ports to the hosts that receive the time.

Cumulus Linux and PTP

PTP in Cumulus Linux uses the linuxptp package that includes the following programs:

• ptp4l provides the PTP protocol and state machines
• phc2sys provides PTP Hardware Clock and System Clock synchronization
• timemaster provides System Clock and PTP synchronization

Cumulus Linux supports:

• PTP boundary clock mode only (the switch provides timing to downstream servers; it is a slave to a higher-level clock and a master to downstream clocks).
• UDPv4, UDPv6, and 802.3 encapsulation.
• Only a single PTP domain per network.
• PTP on layer 3 interfaces, layer 3 bonds, trunk ports, and switch ports belonging to a VLAN.
• Multicast, unicast, and mixed message mode.
• End-to-End delay mechanism only. Cumulus Linux does not support Peer-to-Peer.
• One-step and two-step clock timestamp mode.
• Hardware timestamping for PTP packets. This allows PTP to avoid inaccuracies caused by message transfer delays and improves the accuracy of time synchronization.

Basic Configuration

Basic PTP configuration requires you:

• Enable PTP on the switch.
• Configure PTP on at least one interface; this can be a layer 3 routed port, switch port, or trunk port. You do not need to specify which is a master interface and which is a slave interface; the PTP Best Master Clock Algorithm (BMCA) determines the master and slave.

The basic configuration shown below uses the default PTP settings:

• The clock mode is Boundary. This is the only clock mode that Cumulus Linux supports.
• The PTP clock domain is 0.
• PTP Priority1 and Priority2 are both 128.
• The DSCP is 46 for both general and event messages.
• The PTP interface transport mode is IPv4.
• Announce messages from any master are accepted.
• The PTP Interface Message Mode is multicast.
• The delay mechanism is End-to-End (E2E), where the slave measures the delay between itself and the master. The master and slave send delay request and delay response messages between each other to measure the delay.
• The clock timestamp mode is two-step.

To configure other settings, such as the PTP profile, domain, priority, and DSCP, the PTP interface transport mode and timers, and PTP monitoring, see the Optional Configuration sections below.

cumulus@switch:~$ nv set service ptp 1 enable on
cumulus@switch:~$ nv set interface swp1 ip address 10.0.0.9/32
cumulus@switch:~$ nv set interface swp2 ip address 10.0.0.10/32
cumulus@switch:~$ nv set interface swp1 ptp enable on
cumulus@switch:~$ nv set interface swp2 ptp enable on
cumulus@switch:~$ nv config apply

cumulus@switch:~$ nv set service ptp 1 enable on
cumulus@switch:~$ nv set bridge domain br_default
cumulus@switch:~$ nv set bridge domain br_default type vlan-aware
cumulus@switch:~$ nv set bridge domain br_default vlan 10-30
cumulus@switch:~$ nv set bridge domain br_default vlan 10 ptp enable on
cumulus@switch:~$ nv set interface vlan10 type svi
cumulus@switch:~$ nv set interface vlan10 ip address 10.1.10.2/24
cumulus@switch:~$ nv set interface vlan10 ptp enable on
cumulus@switch:~$ nv set interface swp1 bridge domain br_default
cumulus@switch:~$ nv set interface swp1 bridge domain br_default vlan 10
cumulus@switch:~$ nv set interface swp1 ptp enable on
cumulus@switch:~$ nv config apply

cumulus@switch:~$ nv set service ptp 1 enable on
cumulus@switch:~$ nv set bridge domain br_default
cumulus@switch:~$ nv set bridge domain br_default type vlan-aware
cumulus@switch:~$ nv set bridge domain br_default vlan 10-30
cumulus@switch:~$ nv set bridge domain br_default vlan 10 ptp enable on
cumulus@switch:~$ nv set interface vlan10 type svi
cumulus@switch:~$ nv set interface vlan10 ip address 10.1.10.2/24
cumulus@switch:~$ nv set interface swp2 bridge domain br_default
cumulus@switch:~$ nv set interface swp2 bridge domain br_default access 10
cumulus@switch:~$ nv set interface swp2 ptp enable on
cumulus@switch:~$ nv config apply

Global Configuration

Cumulus Linux provides several ways to modify the default basic global configuration. You can:

• Use profiles.
• Modify the parameters directly with NVUE commands.
• Modify the Linux /etc/ptp4l.conf file.

When a predefined profile is set, NVUE does not allow you to configure global parameters. Do not edit the Linux /etc/ptp4l.conf file to modify the global parameters when a predefined profile is in use. For information about profiles, see PTP Profiles.

Clock Domains

PTP domains allow different independent timing systems to be present in the same network without confusing each other. A PTP domain is a network or a portion of a network within which all the clocks synchronize. Every PTP message contains a domain number. A PTP instance works in only one domain and ignores messages that contain a different domain number. Cumulus Linux supports only one domain in the system.

You can specify multiple PTP clock domains. PTP isolates each domain from other domains so that each domain is a different PTP network. You can specify a number between 0 and 127.

The following example commands configure domain 3 when a profile is not set:

cumulus@switch:~$ nv set service ptp 1 domain 3
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo nano /etc/ptp4l.conf
[global]
#
# Default Data Set
#
slaveOnly               0
priority1               128
priority2               128
domainNumber            3
...

cumulus@switch:~$ sudo systemctl restart ptp4l.service

Clock Timestamp Mode

The Cumulus Linux switch provides the following clock timestamp modes:

• One-step, where PTP adds the precise time that the Sync packet egresses the port to the packet. There is no need for a follow up packet.
• Two-step, where PTP notes the precise time when the Sync packet egresses the port and sends it in a separate follow up message.

One-step mode significantly reduces the number of PTP messages. Two-step mode is the default configuration.

cumulus@switch:~$ nv set service ptp 1 two-step off
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo nano /etc/ptp4l.conf
[global]
#
# Default Data Set
#
slaveOnly               0
priority1               254
priority2               254
domainNumber            3

twoStepFlag             0
dscp_event              43
dscp_general            43
...

cumulus@switch:~$ sudo systemctl restart ptp4l.service

PTP Priority

The BMC selects the PTP master according to the criteria in the following order:

1. Priority 1
2. Clock class
3. Clock accuracy
4. Clock variance
5. Priority 2
6. Port ID

Use the PTP priority to select the best master clock. You can set priority 1 and 2:

• Priority 1 overrides the clock class and quality selection criteria to select the best master clock.
• Priority 2 identifies primary and backup clocks among identical redundant Grandmasters.

The range for both priority1 and priority2 is between 0 and 255. The default priority is 128. For the boundary clock, use a number above 128. The lower priority applies first.

The following example commands set priority 1 and priority 2 to 200 when a profile is not set:

cumulus@switch:~$ nv set service ptp 1 priority1 200
cumulus@switch:~$ nv set service ptp 1 priority2 200
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo nano /etc/ptp4l.conf
[global]
#
# Default Data Set
#
slaveOnly               0
priority1               200
priority2               200
domainNumber            3
...

cumulus@switch:~$ sudo systemctl restart ptp4l.service

Local Priority

Use the local priority when you create a custom profile based on a Telecom profile (ITU 8275-1 or ITU 8275-2). Modify the local priority in a custom profile to set the local priority of the local clock. You can set a value between 0 and 255. The default priority is 128.

The following example command configures the local priority to 10 for the custom profile called CUSTOM1, based on ITU 8275-2:

cumulus@switch:~$ nv set service ptp 1 profile CUSTOM1 local-priority 10
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo nano /etc/ptp4l.conf
[global]
#
# Default Data Set
#
slaveOnly                      0
priority1                      128
priority2                      128
domainNumber                   28

twoStepFlag                    1
dscp_event                     46
dscp_general                   46
network_transport              L2
dataset_comparison             G.8275.x
G.8275.defaultDS.localPriority 10
ptp_dst_mac                    01:80:C2:00:00:0E
...

cumulus@switch:~$ sudo systemctl restart ptp4l.service

Optional Global Configuration

Optional global PTP configuration includes configuring the DiffServ code point (DSCP). You can configure the DSCP value for all PTP IPv4 packets originated locally. You can set a value between 0 and 63.

cumulus@switch:~$ nv set service ptp 1 ip-dscp 22
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo nano /etc/ptp4l.conf
[global]
#
# Default Data Set
#
slaveOnly               0
priority1               200
priority2               200
domainNumber            3

twoStepFlag             1
dscp_event              22
dscp_general            22
...

cumulus@switch:~$ sudo systemctl restart ptp4l.service

PTP Interface Configuration

Cumulus Linux provides several ways to modify the default basic interface configuration. You can:

• Use profiles
• Modify the parameters directly with NVUE commands
• Modify the Linux /etc/ptp4l.conf configuration file.

When a profile is in use, avoid configuring the following interface configuration parameters with NVUE or in the Linux configuration file so that the interface retains its profile settings.

Transport Mode

By default, Cumulus Linux encapsulates PTP messages in UDP IPV4 frames. To encapsulate PTP messages on an interface in UDP IPV6 frames:

cumulus@switch:~$ nv set interface swp1 ptp transport ipv6
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo nano /etc/ptp4l.conf
...
# Default interface options
#
time_stamping           hardware

# Interfaces in which ptp should be enabled
# these interfaces should be routed ports
# if an interface does not have an ip address
# the ptp4l will not work as expected.

[swp1]
udp_ttl                 1
masterOnly              0
delay_mechanism         E2E
network_transport       UDPv6

[swp2]
udp_ttl                 1
masterOnly              0
delay_mechanism         E2E
network_transport       UDPv6
...

cumulus@switch:~$ sudo systemctl restart ptp4l.service

Message Mode

Cumulus Linux supports the following PTP message modes:

• Multicast, where the ports subscribe to two multicast addresses, one for event messages with timestamps and the other for general messages without timestamps. The Sync message that the master sends is a multicast message; all slave ports receive this message because the slaves need the time from the master. The slave ports in turn generate a Delay Request to the master. This is a multicast message that the intended master for the message and other slave ports receive. Similarly, all slave ports in addition to the intended slave port receive the master’s Delay Response. The slave ports receiving the unintended Delay Requests and Responses need to drop the packets. This can affect network bandwidth if there are hundreds of slave ports.
• Mixed, where Sync and Announce messages are multicast messages but Delay Request and Response messages are unicast. This avoids the issue seen in multicast message mode where every slave port sees Delay Requests and Responses from every other slave port.
• Unicast, where you configure the port as a unicast client or server. See Unicast Mode.

Multicast mode is the default setting; when you enable PTP on an interface, the message mode is multicast.

To change the message mode to mixed on swp1:

cumulus@switch:~$ nv set interface swp1 ptp mixed-multicast-unicast on
cumulus@switch:~$ nv config apply

cumulus@switch:~$ nv set interface swp1 ptp mixed-multicast-unicast off
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo nano /etc/ptp4l.conf
...
# Default interface options
#
time_stamping           hardware

# Interfaces in which ptp should be enabled
# these interfaces should be routed ports
# if an interface does not have an ip address
# the ptp4l will not work as expected.

[swp1]
hybrid_e2e              1
...

cumulus@switch:~$ sudo systemctl restart ptp4l.service

PTP Interface Timers

You can set the following timers for PTP messages.

• To set the timers with NVUE, run the nv set interface <interface> ptp timers <timer> <value> command.
• To set the timers with Linux commands, edit the /etc/ptp4l.conf file and set the timers in the Default interface options section.

cumulus@switch:~$ nv set interface swp1 ptp timers announce-interval -1
cumulus@switch:~$ nv config apply

cumulus@switch:~$ nv set interface swp1 ptp timers sync-interval -5
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo nano /etc/ptp4l.conf
...
# Default interface options
#
time_stamping           hardware

# Interfaces in which ptp should be enabled
# these interfaces should be routed ports
# if an interface does not have an ip address
# the ptp4l will not work as expected.

[swp1]
logAnnounceInterval     -1
logSyncInterval         -5
udp_ttl                 20
masterOnly              1
delay_mechanism         E2E
...

cumulus@switch:~$ sudo systemctl restart ptp4l.service

Local Priority

Set the local priority on an interface for a profile that uses ITU 8275-1 or ITU 8275-2. You can set a value between 0 and 255. The default priority is 128.

The following example sets the local priority on swp1 to 10.

cumulus@switch:~$ nv set interface swp1 ptp local-priority 10
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo nano /etc/ptp4l.conf
...
[swp1]
udp_ttl                      1
hybrid_e2e                   1
masterOnly                   0
delay_mechanism              E2E
network_transport            UDPv6
G.8275.portDS.localPriority  10
...

cumulus@switch:~$ sudo systemctl restart ptp4l.service

Optional PTP Interface Configuration

Forced Master Mode

By default, PTP ports are in auto mode, where the BMC algorithm determines the state of the port.

You can configure Forced Master mode on a PTP port so that it is always in a master state and the BMC algorithm does not run for this port. This port ignores any Announce messages it receives.

cumulus@switch:~$ nv set interface swp1 ptp forced-master on
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo nano /etc/ptp4l.conf
...
# Default interface options
#
time_stamping           hardware

# Interfaces in which ptp should be enabled
# these interfaces should be routed ports
# if an interface does not have an ip address
# the ptp4l will not work as expected.

[swp1]
udp_ttl                 1
masterOnly              1
delay_mechanism         E2E
...

cumulus@switch:~$ sudo systemctl restart ptp4l.service

TTL for a PTP Message

To restrict the number of hops a PTP message can travel, set the TTL on the PTP interface. You can set a value between 1 and 255.

cumulus@switch:~$ nv set interface swp1 ptp ttl 20
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo nano /etc/ptp4l.conf
...
# Default interface options
#
time_stamping           hardware

# Interfaces in which ptp should be enabled
# these interfaces should be routed ports
# if an interface does not have an ip address
# the ptp4l will not work as expected.

[swp1]
udp_ttl                 20
masterOnly              1
delay_mechanism         E2E
...

cumulus@switch:~$ sudo systemctl restart ptp4l.service

Unicast Mode

Cumulus Linux supports unicast mode so that a unicast client can perform Unicast Discover and Negotiation with servers. Unlike the default multicast mode, where both the server(master) and client(slave) start sending out announce requests and discover each other, in unicast mode, the client starts by sending out requests for unicast transmission. The client sends this to every server address in its Unicast Master Table. The server responds with an accept or deny to the request.

Global Unicast Configuration

Unicast clients need a unicast master table for unicast negotiation; you must configure at least one unicast master table on the switch.

To configure unicast globally:

• Set the unicast table ID; a unique ID that identifies the unicast master table.
• Set the unicast master address. You can set more than one unicast master address, which can be an IPv4, IPv6, or MAC address.
• Optional: Set the unicast master query interval, which is the mean interval between requests for Announce messages. Specify this value as a power of two in seconds. You can specify a value between -3 and 4. The default value is -0 (2 power).
  NVUE Commands

  cumulus@switch:~$ nv set service ptp 1 unicast-master 1 address 10.10.10.1
  cumulus@switch:~$ nv set service ptp 1 unicast-master 1 query-interval 4
  cumulus@switch:~$ nv set interface swp1 ptp unicast-master-table-id 1
  cumulus@switch:~$ nv config apply
  

  Linux Commands

  1. Add the following lines at the end of the # Default interface options section of the /etc/ptp4l.conf file:

     cumulus@switch:~$ sudo nano /etc/ptp4l.conf
     ...
     # Default interface options
     ...
     [unicast_master_table]
     table_id               1
     logQueryInterval       4
     UDPv4                  10.10.10.1
     ...
     

  2. Restart the ptp4l service.

     cumulus@switch:~$ sudo systemctl restart ptp4l.service
     

Interface Unicast Configuration

For interface unicast configuration, in addition to enabling PTP on an interface, you also need to configure the PTP interface to be either a unicast client or a unicast server.

When configuring multiple PTP interfaces on the switch to be unicast clients, you must configure a unicast table ID on every interface set as a unicast client. Each client must have a different table ID.

To configure a PTP interface to be the unicast client:

cumulus@switch:~$ nv set interface swp1 ptp unicast-service-mode client
cumulus@switch:~$ nv config apply

To configure a PTP interface to be the unicast server:

cumulus@switch:~$ nv set interface swp1 ptp unicast-service-mode server
cumulus@switch:~$ nv config apply

To configure a unicast table ID:

cumulus@switch:~$ nv set interface swp1 ptp unicast-master-table-id 1
cumulus@switch:~$ nv config apply

To show the unicast master table configuration on the switch, run the nv show service ptp <instance-id> unicast-master <table-id> command.

Optional Unicast Interface Configuration

You can set the unicast request duration for unicast clients, which is the service time in seconds requested by the unicast client during unicast negotiation. The default value is 300 seconds.

cumulus@switch:~$ nv set interface swp1 ptp unicast-request-duration 20
cumulus@switch:~$ nv config apply

PTP Profiles

PTP profiles are a standardized set of configurations and rules intended to meet the requirements of a specific application. Profiles define required, allowed, and restricted PTP options, network restrictions, and performance requirements.

Cumulus Linux supports the following predefined profiles:

Set a Predefined Profile

To set a predefined profile:

cumulus@switch:~$ nv set service ptp 1 current-profile default-itu-8275-1
cumulus@switch:~$ nv config apply

cumulus@switch:~$ nv set service ptp 1 current-profile default-1588
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo nano /etc/ptp4l.conf
...
...
[global]
#
# Default Data Set
#
slaveOnly                      0
priority1                      128
priority2                      128
domainNumber                   24

twoStepFlag                    1
dscp_event                     46
dscp_general                   46
dataset_comparison             G.8275.x
G.8275.defaultDS.localPriority 128
ptp_dst_mac                    01:80:C2:00:00:0E
...

cumulus@switch:~$ sudo systemctl restart ptp4l.service

cumulus@switch:~$ sudo nano /etc/ptp4l.conf
...
...
[global]
#
# Default Data Set
#
slaveOnly                      0
priority1                      128
priority2                      128
domainNumber                   24

twoStepFlag                    1 
dscp_event                     46
dscp_general                   46
network_transport              UDPv4
dataset_comparison             G.8275.x
G.8275.defaultDS.localPriority 128
hybrid_e2e                     1
inhibit_multicast_service      1
unicast_listen                 1
unicast_req_duration           60
...

cumulus@switch:~$ sudo systemctl restart ptp4l.service

cumulus@switch:~$ sudo nano /etc/ptp4l.conf
[global]
#
# Default Data Set
#
slaveOnly                      0
priority1                      128
priority2                      128
domainNumber                   0

twoStepFlag                    1
dscp_event                     46
dscp_general                   46
network_transport              UDPv4
dataset_comparison             ieee1588
...

cumulus@switch:~$ sudo systemctl restart ptp4l.service

Create a Custom Profile

To create a custom profile:

cumulus@switch:~$  nv set service ptp 1 profile CUSTOM1 
cumulus@switch:~$  nv set service ptp 1 profile CUSTOM1 profile-type itu-g-8275-1  
cumulus@switch:~$  nv set service ptp 1 profile CUSTOM1 domain 28
cumulus@switch:~$  nv set service ptp 1 profile CUSTOM1 announce-timeout 3
cumulus@switch:~$  nv set service ptp 1 current-profile CUSTOM1
cumulus@switch:~$  nv config apply

cumulus@switch:~$ sudo nano /etc/ptp4l.conf
[global]
#
# Default Data Set
#
slaveOnly                      0
priority1                      128
priority2                      128
domainNumber                   28

twoStepFlag                    1
dscp_event                     46
dscp_general                   46
network_transport              L2
dataset_comparison             G.8275.x
G.8275.defaultDS.localPriority 128
ptp_dst_mac                    01:80:C2:00:00:0E

#
# Port Data Set
#
logAnnounceInterval            5
logSyncInterval                -4
logMinDelayReqInterval         -4
announceReceiptTimeout         3
delay_mechanism                E2E

offset_from_master_min_threshold   -50
offset_from_master_max_threshold   50
mean_path_delay_threshold          200
tsmonitor_num_ts                   100
tsmonitor_num_log_sets             3
tsmonitor_num_log_entries          4
tsmonitor_log_wait_seconds         1

#
# Run time options
#
logging_level                  6
path_trace_enabled             0
use_syslog                     1
verbose                        0
summary_interval               0

#
# servo parameters
#
pi_proportional_const          0.000000
pi_integral_const              0.000000
pi_proportional_scale          0.700000
pi_proportional_exponent       -0.300000
pi_proportional_norm_max       0.700000
pi_integral_scale              0.300000
pi_integral_exponent           0.400000
pi_integral_norm_max           0.300000
step_threshold                 0.000002
first_step_threshold           0.000020
max_frequency                  900000000
sanity_freq_limit              0

#
# Default interface options
#
time_stamping                  software


# Interfaces in which ptp should be enabled
# these interfaces should be routed ports
# if an interface does not have an ip address
# the ptp4l will not work as expected.

[swp1]
udp_ttl                 1
masterOnly              0
delay_mechanism         E2E

[swp2]
udp_ttl                 1
masterOnly              0
delay_mechanism         E2E

cumulus@switch:~$ sudo systemctl restart ptp4l.service

To show the current PTP profile setting, run the nv show service ptp <ptp-instance> command:

cumulus@switch:~$ nv show service ptp 1
                             operational  applied             description
---------------------------  -----------  ------------------  --------------------------------------------------------------------
enable                       on           on                  Turn the feature 'on' or 'off'.  The default is 'off'.
current-profile                           default-itu-8275-1  Current PTP profile index
domain                       24           0                   Domain number of the current syntonization
ip-dscp                      46           46                  Sets the Diffserv code point for all PTP packets originated locally.
priority1                    128          128                 Priority1 attribute of the local clock
priority2                    128          128                 Priority2 attribute of the local clock
...

To show the settings for a profile, run the nv show service ptp <instance> profile <profile-name> command:

cumulus@switch:~$ nv show service ptp 1 profile CUSTOM1
                             operational  applied           
---------------------------  -----------  ------------------
enable                                    on                
current-profile                           default-itu-8275-1
domain                                    0                 
ip-dscp                                   46                
logging-level                             info              
priority1                                 128               
priority2                                 128               
[acceptable-master]    
monitor                                                     
  max-offset-threshold                    50                
  max-timestamp-entries                   100               
  max-violation-log-entries               4                 
  max-violation-log-sets                  3                 
  min-offset-threshold                    -50               
  path-delay-threshold                    200               
  violation-log-interval                  1                 

Optional Acceptable Master Table

The acceptable master table option is a security feature that prevents a rogue player from pretending to be the grandmaster clock to take over the PTP network. To use this feature, you configure the clock IDs of known grandmaster clocks in the acceptable master table and set the acceptable master table option on a PTP port. The BMC algorithm checks if the grandmaster clock received in the Announce message is in this table before proceeding with the master selection. Cumulus Linux disables this option by default on PTP ports.

The following example command adds the grandmaster clock ID 24:8a:07:ff:fe:f4:16:06 to the acceptable master table and enables the PTP acceptable master table option for swp1:

cumulus@switch:~$ nv set service ptp 1 acceptable-master 24:8a:07:ff:fe:f4:16:06
cumulus@switch:~$ nv config apply

cumulus@switch:~$ nv set service ptp 1 acceptable-master 24:8a:07:ff:fe:f4:16:06 alt-priority 2

cumulus@switch:~$ nv set interface swp1 ptp acceptable-master on
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo nano /etc/ptp4l.conf
...
#
# Default interface options
#
time_stamping           hardware


[acceptable_master_table]
maxTableSize 16
acceptable_master_clockIdentity 248a07.fffe.f41606
...

cumulus@switch:~$ sudo nano /etc/ptp4l.conf
...
#
# Default interface options
#
time_stamping           hardware


[acceptable_master_table]
maxTableSize 16
acceptable_master_clockIdentity 248a07.fffe.f41606 2

...
# Default interface options
#
time_stamping           hardware

# Interfaces in which ptp should be enabled
# these interfaces should be routed ports
# if an interface does not have an ip address
# the ptp4l will not work as expected.

[swp1]
udp_ttl                 20
masterOnly              1
delay_mechanism         E2E
acceptable_master       on
...

cumulus@switch:~$ sudo systemctl restart ptp4l.service

Optional Monitor Configuration

Cumulus Linux provides the following optional PTP monitoring configuration.

Configure Clock TimeStamp and Path Delay Thresholds

Cumulus Linux monitors clock timestamp and path delay against thresholds, and generates counters when PTP reaches the set thresholds. You can see the counters in the NVUE nv show command output and in log messages.

You can configure the following monitor settings:

cumulus@switch:~$ nv set service ptp 1 monitor min-offset-threshold -1000
cumulus@switch:~$ nv set service ptp 1 monitor max-offset-threshold 1000
cumulus@switch:~$ nv set service ptp 1 monitor path-delay-threshold 300
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo nano /etc/ptp4l.conf
...
[global]
#
# Default Data Set
#
slaveOnly               0
priority1               128
priority2               128
domainNumber            0

twoStepFlag             1
dscp_event              46
dscp_general            46

offset_from_master_min_threshold   -1000
offset_from_master_max_threshold   1000
mean_path_delay_threshold          300
...

Configure PTP Logging

A log set contains the log entries for clock timestamp and path delay violations at different times. You can set the number of entries to log and the interval between successive violation logs.

cumulus@switch:~$ nv set service ptp 1 monitor max-violation-log-sets 4
cumulus@switch:~$ nv set service ptp 1 monitor max-violation-log-entries 6
cumulus@switch:~$ nv set service ptp 1 monitor violation-log-interval 10
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo nano /etc/ptp4l.conf
...
[global]
#
# Default Data Set
#
slaveOnly               0
priority1               128
priority2               128
domainNumber            0

twoStepFlag             1
dscp_event              46
dscp_general            46

offset_from_master_min_threshold   -50
offset_from_master_max_threshold   50
mean_path_delay_threshold          300
tsmonitor_num_ts                   100
tsmonitor_num_log_sets             4
tsmonitor_num_log_entries          6
tsmonitor_log_wait_seconds         10
...

Show PTP Logs

PTP monitoring provides commands to show counters for violations as well as the timestamp log entries for a violation.

The following example shows the threshold violation count and the last time a minimum offset threshold violation occurred:

cumulus@switch:~$ nv show service ptp 1 monitor violations
                  operational                  applied
----------------  ---------------------------  -------
last-max-offset
last-min-offset   2023-04-24T15:22:01.312295Z
last-path-delay
max-offset-count  0
min-offset-count  2
path-delay-count  0

Clear PTP Violation Logs

• To clear the maximum offset violation logs, run the nv action clear service ptp <instance> monitor violations log max-offset command.
• To clear the minimum offset violation logs, run the nv action clear service ptp <instance> monitor violations log min-offset command.
• To clear the path delay violation logs, run the nv action clear service ptp <instance> monitor violations log path-delay command.

cumulus@leaf01:mgmt:~$ nv action clear service ptp 1 monitor violations log path-delay
Action succeeded

Delete PTP Configuration

To delete PTP configuration, delete the PTP master and slave interfaces. The following example commands delete the PTP interfaces swp1, swp2, and swp3.

cumulus@switch:~$ nv unset interface swp1 ptp
cumulus@switch:~$ nv unset interface swp2 ptp
cumulus@switch:~$ nv unset interface swp3 ptp
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo nano /etc/ptp4l.conf
...
# Default interface options
#
time_stamping           hardware

# Interfaces in which ptp should be enabled
# these interfaces should be routed ports
# if an interface does not have an ip address
# the ptp4l will not work as expected.

cumulus@switch:~$ sudo systemctl restart ptp4l.service

To disable PTP on the switch and stop the ptp4l and phc2sys processes:

cumulus@switch:~$ nv set service ptp 1 enable off
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo systemctl stop ptp4l.service phc2sys.service
cumulus@switch:~$ sudo systemctl disable ptp4l.service phc2sys.service

Troubleshooting

Show PTP Configuration

To show a summary of the PTP configuration on the switch, run the nv show service ptp <instance> command:

cumulus@switch:~$ nv show service ptp 1
                             operational  applied
---------------------------  -----------  ------------------
enable                       on           on
current-profile                            default-itu-8275-2
domain                                    0
ip-dscp                                   46
logging-level                             info
priority1                                 128
priority2                                 128
[acceptable-master]
monitor
  max-offset-threshold                     50
  max-timestamp-entries                   100
  max-violation-log-entries               4
  max-violation-log-sets                  2
  min-offset-threshold                     -50
  path-delay-threshold                    200
  violation-log-interval                  1
[profile]                                  abc
[profile]                                  default-1588
[profile]                                  default-itu-8275-1
[profile]                                  default-itu-8275-2
[unicast-master]                          1
[unicast-master]                          2
[unicast-master]                          3
[unicast-master]                          4
...

You can drill down with the following nv show service ptp <instance> commands:

• nv show service ptp <instance> acceptable-master shows acceptable master configuration.
• nv show service ptp <instance> clock-quality shows the clock quality status.
• nv show service ptp <instance> current shows the local states learned during PTP message exchange.
• nv show service ptp <instance> domain shows the domain configuration.
• nv show service ptp <instance> ip-dscp shows PTP DSCP configuration.
• nv show service ptp <instance> monitor shows PTP monitor configuration.
• nv show service ptp <instance> profile shows PTP profile configuration.
• nv show service ptp <instance> parent shows the local states learned during PTP message exchange.
• nv show service ptp <instance> priority1 shows PTP priority1 configuration.
• nv show service ptp <instance> priority2 shows PTP priority2 configuration.
• nv show service ptp <instance> status shows the status of all PTP interfaces.
• nv show service ptp <instance> time-properties shows the clock time attributes.
• nv show service ptp <instance> unicast-master shows the unicast master configuration.

Show PTP Interface Configuration

To check configuration for a PTP interface, run the nv show interface <interface> ptp command.

cumulus@switch:~$ nv show interface swp1 ptp
                           operational  applied     description
-------------------------  -----------  ----------  ----------------------------------------------------------------------
enable                                  on          Turn the feature 'on' or 'off'.  The default is 'off'.
acceptable-master                       off         Determines if acceptable master check is enabled for this interface.
delay-mechanism            end-to-end   end-to-end  Mode in which PTP message is transmitted.
forced-master              off          off         Configures PTP interfaces to forced master state.
instance                                1           PTP instance number.
mixed-multicast-unicast                 off         Enables Multicast for Announce, Sync and Followup and Unicast for D...
transport                  ipv4         ipv4        Transport method for the PTP messages.
ttl                        1            1           Maximum number of hops the PTP messages can make before it gets dro...
unicast-request-duration                300         The service time in seconds to be requested during discovery.
timers
  announce-interval        0            0           Mean time interval between successive Announce messages.  It's spec...
  announce-timeout         3            3           The number of announceIntervals that have to pass without receipt o...
  delay-req-interval       -3           -3          The minimum permitted mean time interval between successive Delay R...
  sync-interval            -3           -3          The mean SyncInterval for multicast messages.  It's specified as a...
peer-mean-path-delay       0                        An estimate of the current one-way propagation delay on the link wh...
port-state                 master                   State of the port
protocol-version           2                        The PTP version in use on the port

Show PTP Counters

To show all PTP counters, run the nv show service ptp <instance> counters command:

cumulus@switch:~$ nv show service ptp 1 counters
Packet Type              Received       Transmitted    
---------------------    ------------   ------------   
Port swp4
  Announce                 0              10370            
  Sync                     0              20731             
  Follow-up                0              20731            
  Delay Request            0              0              
  Delay Response           0              0              
  Peer Delay Request       0              0              
  Peer Delay Response      0              0              
  Management               0              0              
  Signaling                0              0

To show PTP counters for an interface, run the nv show interface <interface> counters ptp command.

To clear PTP counters for an interface, run the nv action clear interface <interface> counters ptp command:

cumulus@switch:~$ nv action clear interface swp1 counters ptp
Action succeeded

Show the Status of All PTP Interfaces

To show the status of all PTP interfaces, run the nv show service ptp <instance> status command. The command output shows the PTP enabled ports, the PTP port mode (unicast or multicast), the state of the port based on BMCA, the unicast state, and identifies the server address to which the client connects.

cumulus@switch:~$ nv show service ptp 1 status
Port   Mode   State    Ustate                           Server
-----  -----  -------  -------------------------------  -------
swp9   Ucast  SLAVE    Sync and Delay Granted (H_SYDY)  9.9.9.2
swp10  Ucast  PASSIVE  Initial State (WAIT)
swp11  Ucast  PASSIVE  Initial State (WAIT)
swp12  Ucast  PASSIVE  Initial State (WAIT)

Show the List of NVUE PTP Commands

• To see a full list of NVUE show commands for PTP, run the nv list-commands service ptp command.
• To show a full list of show commands for a PTP interface, run the nv list-commands | grep 'nv show interface <interface-id> ptp' command.

cumulus@switch:~$ nv list-commands service ptp
nv show service ptp
nv show service ptp <instance-id>
nv show service ptp <instance-id> status
nv show service ptp <instance-id> domain
nv show service ptp <instance-id> priority1
nv show service ptp <instance-id> priority2
nv show service ptp <instance-id> ip-dscp
nv show service ptp <instance-id> acceptable-master
...

cumulus@switch:~$ nv list-commands | grep 'nv show interface <interface-id> ptp'
...
nv show interface <interface-id> ptp
nv show interface <interface-id> ptp timers
nv show interface <interface-id> ptp shaper
...

Example Configuration

In the following example, the boundary clock on the switch receives time from Master 1 (the grandmaster) on PTP slave port swp1, sets its clock and passes the time down through PTP master ports swp2, swp3, and swp4 to the hosts that receive the time.

The following example configuration assumes that you have already configured the layer 3 routed interfaces (swp1, swp2, swp3, and swp4) you want to use for PTP.

cumulus@switch:~$ nv set service ptp 1 enable on
cumulus@switch:~$ nv set service ptp 1 priority2 254
cumulus@switch:~$ nv set service ptp 1 priority1 254
cumulus@switch:~$ nv set service ptp 1 domain 3
cumulus@switch:~$ nv set interface swp1 ptp enable on
cumulus@switch:~$ nv set interface swp2 ptp enable on
cumulus@switch:~$ nv set interface swp3 ptp enable on
cumulus@switch:~$ nv set interface swp4 ptp enable on
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo cat /etc/nvue.d/startup.yaml
- set:
    interface:
      lo:
        ip:
          address:
            10.10.10.1/32: {}
        type: loopback
      swp1:
        ptp:
          enable: on
        type: swp
      swp2:
        ptp:
          enable: on
        type: swp
      swp3:
        ptp:
          enable: on
        type: swp
      swp4:
        ptp:
          enable: on
        type: swp
    service:
      ptp:
        '1':
          domain: 3
          enable: on
          priority1: 254
          priority2: 254

cumulus@switch:~$ sudo cat /etc/ptp4l.conf
...
[global]
#
# Default Data Set
#
slaveOnly                      0
priority1                      254
priority2                      254
domainNumber                   3

twoStepFlag                    1
dscp_event                     46
dscp_general                   46

offset_from_master_min_threshold   -50
offset_from_master_max_threshold   50
mean_path_delay_threshold          200
tsmonitor_num_ts                   100
tsmonitor_num_log_sets             2
tsmonitor_num_log_entries          4
tsmonitor_log_wait_seconds         1

#
# Run time options
#
logging_level                  6
path_trace_enabled             0
use_syslog                     1
verbose                        0
summary_interval               0

#
# servo parameters
#
pi_proportional_const          0.000000
pi_integral_const              0.000000
pi_proportional_scale          0.700000
pi_proportional_exponent       -0.300000
pi_proportional_norm_max       0.700000
pi_integral_scale              0.300000
pi_integral_exponent           0.400000
pi_integral_norm_max           0.300000
step_threshold                 0.000002
first_step_threshold           0.000020
max_frequency                  900000000
sanity_freq_limit              0

#
# Default interface options
#
time_stamping                  software


# Interfaces in which ptp should be enabled
# these interfaces should be routed ports
# if an interface does not have an ip address
# the ptp4l will not work as expected.

[swp1]
udp_ttl                      1
masterOnly                   0
delay_mechanism              E2E
network_transport            UDPv4

[swp2]
udp_ttl                      1
masterOnly                   0
delay_mechanism              E2E
network_transport            UDPv4

[swp3]
udp_ttl                      1
masterOnly                   0
delay_mechanism              E2E
network_transport            UDPv4

[swp4]
udp_ttl                      1
masterOnly                   0
delay_mechanism              E2E
network_transport            UDPv4

Considerations

PTP Traffic Shaping

To improve performance on the NVIDIA Spectrum 1 switch for PTP-enabled ports with speeds lower than 100G, you can enable a pre-defined traffic shaping profile. For example, if you see that the PTP timing offset varies widely and does not stabilize, enable PTP shaping on all PTP enabled ports to reduce the bandwidth on the ports slightly and improve timing stabilization.

cumulus@switch:~$ nv set interface swp1 ptp shaper enable on
cumulus@switch:~$ nv set interface swp2 ptp shaper enable on
cumulus@switch:~$ nv config apply

cumulus@switch:~$ nv show interface swp1 ptp shaper
        operational  applied  
------  -----------  -------  
enable               on   

cumulus@switch:~$ sudo nano /etc/cumulus/switchd.d/ptp_shaper.conf
## Per-port configuration for PTP shaper
ptp_shaper.port_group_list = [enable-group]
ptp_shaper.enable-group.port_set = swp1,swp2
ptp_shaper.enable-group.ptp_shaper_enable = true

cumulus@switch:~$ sudo systemctl reload switchd.service

Spanning Tree and PTP

PTP frames are affected by STP filtering; events, such as an STP topology change (where ports temporarily go into the blocking state), can cause interruptions to PTP communications.

If you configure PTP on bridge ports, NVIDIA recommends that the bridge ports are spanning tree edge ports or in a bridge domain where spanning tree is disabled.

This section describes how to set up user accounts and ssh for remote access, and configure LDAP authentication, TACACS+, and RADIUS AAA.

Cumulus Linux uses the OpenSSH package to provide access to the system using the Secure Shell (SSH) protocol.

Configure SSH

You can configure SSH to provide login access to the root user and to specific user accounts, limit SSH to listen on a specific VRF, and configure timeouts and session options.

Root User Settings

By default, the root account cannot use SSH to log in.

You can configure the root account to use SSH to log into the switch with:

• A password
• A public key or any allowed mechanism that is not a password and not keyboard interactive. This is the default setting.
• A set of commands defined in the authorized_keys file.

cumulus@switch:~$ nv set system ssh-server permit-root-login enabled
cumulus@switch:~$ nv config apply

cumulus@switch:~$ nv set system ssh-server permit-root-login prohibit-password
cumulus@switch:~$ nv config apply

cumulus@switch:~$ nv set system ssh-server permit-root-login forced-commands-only
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo cat /etc/ssh/sshd_config
...
# Authentication:
LoginGraceTime 2m
PermitRootLogin yes
...

Allow and Deny Users

cumulus@switch:~$ nv set system ssh-server allow-users user1
cumulus@switch:~$ nv config apply

cumulus@switch:~$ nv set system ssh-server deny-users user4
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo cat /etc/ssh/sshd_config
...
...
# Example of overriding settings on a per-user basis
#Match User anoncvs
# X11Forwarding no
# AllowTcpForwarding no
# PermitTTY no
# ForceCommand cvs server
AllowUsers = user1

cumulus@switch:~$ sudo cat /etc/ssh/sshd_config
...
# Example of overriding settings on a per-user basis
#Match User anoncvs
# X11Forwarding no
# AllowTcpForwarding no
# PermitTTY no
# ForceCommand cvs server
AllowUsers = user1
DenyUsers  = user4

SSH and VRFs

The SSH service runs in the default VRF on the switch but listens on all interfaces in all VRFs. You can limit SSH to listen on specific VRFs.

cumulus@switch:~$ nv set system ssh-server vrf mgmt
cumulus@switch:~$ nv config apply

cumulus@switch:~$ nv set system ssh-server vrf mgmt
cumulus@switch:~$ nv set system ssh-server vrf RED
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo systemctl stop ssh.service
cumulus@switch:~$ sudo systemctl disable ssh.service
cumulus@switch:~$ sudo systemctl start ssh@mgmt.service
cumulus@switch:~$ sudo systemctl enable ssh@mgmt.service

cumulus@switch:~$ sudo systemctl stop ssh.service
cumulus@switch:~$ sudo systemctl disable ssh.service
cumulus@switch:~$ sudo systemctl start ssh@mgmt.service
cumulus@switch:~$ sudo systemctl enable ssh@mgmt.service
cumulus@switch:~$ sudo systemctl start ssh@RED.service
cumulus@switch:~$ sudo systemctl enable ssh@RED.service

cumulus@switch:~$ sudo cat /etc/ssh/sshd_config
...

#Port 22
#AddressFamily any
ListenAddress 10.10.10.6
#ListenAddress ::

Enable and Disable the SSH Server

Cumulus Linux enables the SSH server by default. To disable the SSH server:

cumulus@switch:~$ nv set system ssh-server state disabled
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo systemctl stop ssh.service
cumulus@switch:~$ sudo systemctl disable ssh.service

cumulus@switch:~$ sudo systemctl start ssh.service
cumulus@switch:~$ sudo systemctl enable ssh.service

Configure Timeouts and Sessions

You can configure the following SSH timeout and session options:

• The number of login attempts allowed before rejecting the SSH session. You can specify a value between 3 and 100. The default value is 3 login attempts.
• The number of seconds allowed before login times out. You can specify a value between 1 and 600. The default value is 120 seconds.
• The TCP port numbers that listen for incoming SSH sessions. You can specify a value between 1 and 65535.
• The number of minutes a session can be inactive before the SSH server terminates the connection. The default value is 0 minutes.
• The maximum number of SSH sessions allowed per TCP connection. You can specify a value between 1 and 100. The default value is 10.
• Unauthenticated SSH sessions:
  • The maximum number of unauthenticated SSH sessions allowed. You can specify a value between 1 and 10000. The default value is 100.
  • The number of unauthenticated SSH sessions allowed before throttling starts. You can specify a value between 1 and 10000. The default value is 10.
  • The starting percentage of connections to reject above the throttle start count before reaching the session count limit. You can specify a value between 1 and 100. The default value is 30.

The following example configures the number of login attempts allowed before rejecting the SSH session to 10 and the number of seconds allowed before login times out to 200:

cumulus@switch:~$ nv set system ssh-server authentication-retries 10
cumulus@switch:~$ nv set system ssh-server login-timeout 200
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo nano /etc/ssh/sshd_config
...
# Authentication:

LoginGraceTime 200s
PermitRootLogin prohibit-password
#StrictModes yes
MaxAuthTries 10
MaxSessions 10

The following example configures the TCP port that listens for incoming SSH sessions to 443:

cumulus@switch:~$ nv set system ssh-server port 443
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo nano /etc/ssh/sshd_config
...
Port 443
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::
...

The following example configures the amount of time a session can be inactive before the SSH server terminates the connection to 5 minutes (300 seconds) and the maximum number of SSH sessions allowed per TCP connection to 5:

cumulus@switch:~$ nv set system ssh-server inactive-timeout 5
cumulus@switch:~$ nv set system ssh-server max-sessions-per-connection 5
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo nano /etc/ssh/sshd_config
...
# Authentication:

LoginGraceTime 120s
PermitRootLogin prohibit-password
#StrictModes yes
MaxAuthTries 10
MaxSessions 5
...
#AllowAgentForwarding yes
#AllowTcpForwarding yes
#GatewayPorts no
X11Forwarding yes
#X11DisplayOffset 10
#X11UseLocalhost yes
#PermitTTY yes
PrintMotd no
#PrintLastLog yes
#TCPKeepAlive yes
#PermitUserEnvironment no
#Compression delayed
ClientAliveInterval 300
...

The following example configures:

• The number of unauthenticated SSH sessions allowed before throttling starts to 5.
• The starting percentage of connections to reject above the throttle start count before reaching the session count limit to 22.
• The maximum number of unauthenticated SSH sessions allowed to 20.

cumulus@switch:~$ nv set system ssh-server max-unauthenticated throttle-start 5
cumulus@switch:~$ nv set system ssh-server max-unauthenticated throttle-percent 22
cumulus@switch:~$ nv set system ssh-server max-unauthenticated session-count 20
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo nano /etc/ssh/sshd_config
...
MaxStartups 5:22:20
...

Generate and Install an SSH Key Pair

This section describes how to generate an SSH key pair on one system and install the key as an authorized key on another system.

Generate an SSH Key Pair

To generate an SSH key pair, run the ssh-keygen command and follow the prompts.

cumulus@host01:~$ ssh-keygen 
Generating public/private rsa key pair. 
Enter file in which to save the key (/home/cumulus/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/cumulus/.ssh/id_rsa. 
Your public key has been saved in /home/cumulus/.ssh/id_rsa.pub. 
The key fingerprint is: 
5a:b4:16:a0:f9:14:6b:51:f6:f6:c0:76:1a:35:2b:bb cumulus@leaf04 
The key's randomart image is: 
+---[RSA 2048]----+ 
|      +.o   o    | 
|     o * o . o   | 
|    o + o O o    | 
|     + . = O     | 
|      . S o .    | 
|       +   .     | 
|      .   E      | 
|                 | 
|                 | 
+-----------------+ 

Install an Authorized SSH Key

To install an authorized SSH key, you take the contents of an SSH public key and add it to the SSH authorized key file (~/.ssh/authorized_keys) of the user.

A public key is a text file with three space separated fields:

<type> <key string> <comment>

The procedure to install an authorized SSH key is different based on whether the user is an NVUE managed user or a non-NVUE managed user.

cumulus@leaf01:~$ nv set system aaa user admin2 ssh authorized-key prod_key key XABDB3NzaC1yc2EAAAADAQABAAABgQCvjs/RFPhxLQMkckONg+1RE1PTIO2JQhzFN9TRg7ox7o0tfZ+IzSB99lr2dmmVe8FRWgxVjc...
cumulus@leaf01:~$ nv set system aaa user admin2 ssh authorized-key prod_key type ssh-rsa
cumulus@leaf01:~$ nv config apply

Troubleshooting

To show all the current SSH server configuration settings, run the NVUE nv show system ssh-server command:

cumulus@switch:~$ nv show system ssh-server
                             applied          
---------------------------  -----------------
authentication-retries       6                
inactive-timeout             0                
login-timeout                120              
max-sessions-per-connection  10               
permit-root-login            prohibit-password
state                        enabled          
max-unauthenticated                           
  session-count              100              
  throttle-percent           30               
  throttle-start             10

To show the current number of active SSH sessions, run the NVUE nv show system ssh-server active-sessions command or the Linux w command:

cumulus@switch:~$ nv show system ssh-server active-sessions
Peer Address:Port    Local Address:Port      State
-------------------  ----------------------  -----
192.168.200.1:46528  192.168.200.11%mgmt:22  ESTAB

cumulus@switch:~$ w
 11:10:46 up 19:19,  4 users,  load average: 0.08, 0.05, 0.05
USER     TTY      FROM             LOGIN@   IDLE   JCPU   PCPU WHAT
cumulus  ttyS0    -                Wed15   19:19m  0.03s  0.02s -bash
cumulus  pts/0    192.168.200.1    07:27    3:43m  0.03s  0.03s -bash
cumulus  pts/1    192.168.200.1    10:01    1:09m  0.02s  0.02s -bash
cumulus  pts/2    192.168.200.1    11:10    1.00s  0.03s  0.00s w

To show which users can establish an SSH session, run the nv show system ssh-server allow-users command. To show which users cannot establish an SSH session, run the nv show system ssh-server deny-users command. You can also show information for a specific user with the nv show system ssh-server allow-users <user> command and the nv show system ssh-server deny-users <user> command.

To show the TCP port numbers that listen for incoming SSH sessions, run the nv show system ssh-server port command. You can also show information for a specific port with the nv show system ssh-server port <port> command.

To show the SSH timer and session information, run the nv show system ssh-server max-unauthenticated command:

cumulus@switch:~$ nv show system ssh-server max-unauthenticated
                  applied
----------------  -------
session-count     20     
throttle-percent  22     
throttle-start    5

By default, Cumulus Linux has two user accounts: cumulus and root.

The cumulus account:

• Uses the default password cumulus. You must change the default password when you log into Cumulus Linux for the first time.
• Is a user account in the sudo group with sudo privileges.
• Can log in to the system through all the usual channels, such as console and SSH.
• Includes permissions to run NVUE nv show, nv set, nv unset, and nv apply commands.

The root account:

• Has the default password disabled by default and prevents you from using SSH, telnet, FTP, and so on, to log in to the switch.
• Has the standard Linux root user access to everything on the switch.

Add a New User Account

You can add additional user accounts as needed.

• You control local user account access to NVUE commands by changing the group membership (role) for a user. Like the cumulus account, these accounts must be in the sudo group or include the NVUE system-admin role to execute privileged commands.
• You can set a plain text password or a hashed password for the local user account. To access the switch without a password, you need to boot into single user mode.
• You can provide a full name for the local user account (optional).

cumulus@switch:~$ nv set system aaa user admin2 role system-admin
cumulus@switch:~$ nv set system aaa user admin2 password
Enter new password:
Confirm password:
cumulus@switch:~$ nv set system aaa user admin2 full-name "FIRST LAST"
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo useradd admin2 -m -c "First Last"
cumulus@switch:~$ sudo passwd admin2
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully
cumulus@switch:~$ sudo adduser admin2 sudo
cumulus@switch:~$ sudo adduser admin2 nvapply

Hashed Passwords

Instead of a plain text password, you can provide a hashed password for a local user.

You must specify the hashed password in Linux crypt format; the password must be a minimum of 15 to 20 characters long and must include special characters, digits, lower case alphabetic letters, and more. Typically, the password format is set to $id$salt$hashed, where $id is the hashing algorithm. In GNU or Linux:

• $1$ is MD5
• $2a$ is Blowfish
• $2y$ is Blowfish
• $5$ is SHA-256
• $6$ is SHA-512

To generate a hashed password on the switch, you can either run a python3 command or install and use the mkpasswd utility:

cumulus@switch:~$ python3 -c "import crypt; import getpass; print(crypt.crypt(getpass.getpass(), salt=crypt.METHOD_SHA512))"                    
Password:                                                                                                                                                                 
$6$MIDE.sdxwxuAMGHd$XFXSpHV4NRJymUpeCKz.SYEMUfGGEtLbcqK0fBw3d96ZzegP3sw6ppl5Atx9xLS3UHLLTWS/BOwjkeBJJaRx10

cumulus@switch:~$ sudo -E apt-get update
cumulus@switch:~$ sudo -E apt-get install whois

To set the hashed password for the local user:

cumulus@switch:~$ nv set system aaa user admin2 hashed-password '$1$/ETjhZMJ$P73qhBZEYP20mKnRkhBol0'
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo useradd admin2 -c "First Last" -p '$1$/ETjhZMJ$P73qhBZEYP20mKnRkhBol0'

Delete a User Account

To delete a user account:

cumulus@switch:~$ nv unset system aaa user admin2
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo userdel admin2

Show User Accounts

To show the user accounts configured on the system, run the NVUE nv show system aaa command or the linux sudo cat /etc/passwd command.

cumulus@switch:~$ nv show system aaa
Username          Full-name                           Role          enable
----------------  ----------------------------------  ------------  ------
Debian-snmp                                           Unknown       system
_apt                                                  Unknown       system
_lldpd                                                Unknown       system
admin2            FIRST LAST                          system-admin  on    
...

To show information about a specific user account, run the NVUE nv show system aaa user <user> command:

cumulus@switch:~$ nv show system aaa user admin2
                 operational   applied     
---------------  ------------  ------------
full-name        FIRST LAST    FIRST LAST  
hashed-password  *             *           
role             system-admin  system-admin
enable           on            on  

Enable the root User

The root user does not have a password and cannot log into a switch using SSH. This default account behavior is consistent with Debian.

Enable Console Access

To log into the switch using root from the console, you must set the password for the root account:

cumulus@switch:~$ sudo passwd root
Enter new password:
...

Enable SSH Access

To log into the switch using root with SSH, either:

• Install an SSH authorized key; refer to Install an Authorized SSH Key.

• Follow these steps to set a password and enable password authentication for root in sshd:

  1. Run the following command:

     cumulus@switch:~$ sudo passwd root 
     

  2. In the /etc/ssh/sshd_config file, change the PermitRootLogin setting from without-password to yes:

     cumulus@switch:~$ sudo nano /etc/ssh/sshd_config
     ...
     # Authentication: 
     LoginGraceTime 120 
     PermitRootLogin yes 
     StrictModes yes
     ...
     

  3. Restart the ssh service:

     cumulus@switch:~$ sudo systemctl reload ssh.service
     

Related Information

• crypt man page
• Understanding /etc/shadow file format on Linux

By default, Cumulus Linux has two user accounts: root and cumulus. The cumulus account is a normal user and is in the group sudo.

You can add more user accounts as needed. Like the cumulus account, these accounts must use sudo to execute privileged commands.

sudo Basics

sudo allows you to execute a command as superuser or another user as specified by the security policy.

The default security policy is sudoers, which you configure in the /etc/sudoers file. Use /etc/sudoers.d/ to add to the default sudoers policy.

By default, users in the sudo group can use sudo to execute privileged commands. To add users to the sudo group, use the useradd(8) or usermod(8) command. To see which users belong to the sudo group, see /etc/group (man group(5)).

You can run any command as sudo, including su. You must enter a password.

The example below shows how to use sudo as a non-privileged user cumulus to bring up an interface:

cumulus@switch:~$ ip link show dev swp1
3: swp1: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master br0 state DOWN mode DEFAULT qlen 500
link/ether 44:38:39:00:27:9f brd ff:ff:ff:ff:ff:ff

cumulus@switch:~$ ip link set dev swp1 up
RTNETLINK answers: Operation not permitted

cumulus@switch:~$ sudo ip link set dev swp1 up
Password:

umulus@switch:~$ ip link show dev swp1
3: swp1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP mode DEFAULT qlen 500
link/ether 44:38:39:00:27:9f brd ff:ff:ff:ff:ff:ff

sudoers Examples

The following examples show how you grant as few privileges as necessary to a user or group of users to allow them to perform the required task. Each example uses the system group noc; groups include the prefix %.

When an unprivileged user runs a command, the command must include the sudo prefix.

Related Information

• Debian wiki - sudo

Cumulus Linux uses Pluggable Authentication Modules (PAM) and Name Service Switch (NSS) for user authentication. NSS enables PAM to use LDAP to provide user authentication, group mapping, and information for other services on the system.

• NSS specifies the order of the information sources that resolve names for each service. Using NSS with authentication and authorization provides the order and location for user lookup and group mapping on the system.
• PAM handles the interaction between the user and the system, providing login handling, session setup, authentication of users, and authorization of user actions.

Install libnss-ldapd

The libldap-2.4-2 and libldap-common LDAP packages are already installed on the Cumulus Linux image; however you need to install these additional packages to use LDAP authentication:

• libnss-ldapd
• libpam-ldapd
• ldap-utils

To install the additional packages, run the following command:

cumulus@switch:~$ sudo apt-get install libnss-ldapd libpam-ldapd ldap-utils nslcd

You can also install these packages even if the switch does not connect to the internet, as they are in the cumulus-local-apt-archive repository that is embedded in the Cumulus Linux image.

Follow the interactive prompts to specify the LDAP URI, search base distinguished name (DN), and services that must have LDAP lookups enabled. You need to select at least the passwd, group, and shadow services (press space to select a service). When done, select OK. This creates a basic LDAP configuration using anonymous bind and initiates user search under the base DN specified.

After the dialog closes, the install process prints information similar to the following:

/etc/nsswitch.conf: enable LDAP lookups for group
/etc/nsswitch.conf: enable LDAP lookups for passwd
/etc/nsswitch.conf: enable LDAP lookups for shadow

After the installation is complete, the name service caching daemon (nslcd) runs. This service handles all the LDAP protocol interactions and caches information that returns from the LDAP server. nslcd appends ldap to the /etc/nsswitch.conf file, as well as the secondary information source for passwd, group, and shadow. nslcd references the local files (/etc/passwd, /etc/groups and /etc/shadow) first, as specified by the compat source.

passwd: compat ldap
group: compat ldap
shadow: compat ldap

Entering incorrect information during the installation process produces configuration errors. You can correct the information after installation by editing certain configuration files.

• Edit the /etc/nslcd.conf file to update the LDAP URI and search base DN (see Update the nslcd.conf File, below).
• Edit the /etc/nssswitch.conf file to update the service selections.

After editing the files, restart the NVUE and nginx-authenticator services with the sudo systemctl restart nvued.service command and the sudo systemctl restart nginx-authenticator.service command.

Update the nslcd.conf File

After installation, update the main configuration file (/etc/nslcd.conf) to accommodate the expected LDAP server settings.

This section documents some of the more important options that relate to security and queries. For details on all the available configuration options, read the nslcd.conf man page.

Connection

The LDAP client starts a session by connecting to the LDAP server on TCP and UDP port 389 or on port 636 for LDAPS. Depending on the configuration, this connection establishes without authentication (anonymous bind); otherwise, the client must provide a bind user and password. The variables you use to define the connection to the LDAP server are the URI and bind credentials.

The URI is mandatory and specifies the LDAP server location using the FQDN or IP address. The URI also designates whether to use ldap:// for clear text transport, or ldaps:// for SSL/TLS encrypted transport. You can also specify an alternate port in the URI. In production environments, use the LDAPS protocol so that all communications are secure.

After the connection to the server is complete, the BIND operation authenticates the session. The BIND credentials are optional; if you do not specify the credentials, the switch assumes an anonymous bind. Configure authenticated (Simple) BIND by specifying the user (binddn) and password (bindpw) in the configuration. Another option is to use SASL (Simple Authentication and Security Layer) BIND, which provides authentication services using other mechanisms, like Kerberos. Contact your LDAP server administrator for this information as it depends on the configuration of the LDAP server and the credentials for the client device.

# The location at which the LDAP server(s) should be reachable.
uri ldaps://ldap.example.com
# The DN to bind with for normal lookups.
binddn cn=CLswitch,ou=infra,dc=example,dc=com
bindpw CuMuLuS

Search Function

When an LDAP client requests information about a resource, it must connect and bind to the server. Then, it performs one or more resource queries depending on the lookup. All search queries to the LDAP server use the configured search base, filter, and the desired entry (uid=myuser). If the LDAP directory is large, this search takes a long time. Define a more specific search base for the common maps (passwd and group).

# The search base that will be used for all queries.
base dc=example,dc=com
# Mapped search bases to speed up common queries.
base passwd ou=people,dc=example,dc=com
base group ou=groups,dc=example,dc=com

Search Filters

To limit the search scope when authenticating users, use search filters to specify criteria when searching for objects within the directory. The default filters applied are:

filter passwd (objectClass=posixAccount)
filter group (objectClass=posixGroup)

Attribute Mapping

The map configuration allows you to override the attributes pushed from LDAP. To override an attribute for a given map, specify the attribute name and the new value. This is useful to ensure that the shell is bash and the home directory is /home/cumulus:

map    passwd homeDirectory "/home/cumulus"
map    passwd shell "/bin/bash"

Create Home Directory on Login

If you want to use unique home directories, run the sudo pam-auth-update command and select Create home directory on login in the PAM configuration dialog (press the space bar to select the option). Select OK, then press Enter to save the update and close the dialog.

cumulus@switch:~$ sudo pam-auth-update

The home directory for any user that logs in (using LDAP or not) populates with the standard dotfiles from /etc/skel.

nslcd[5816]: unable to dlopen /usr/lib/x86_64-linux-gnu/sasl2/libsasldb.so: libdb-5.3.so: cannot open
shared object file: No such file or directory

Example Configuration

Here is an example configuration using Cumulus Linux.

# /etc/nslcd.conf
# nslcd configuration file. See nslcd.conf(5)
# for details.

# The user and group nslcd should run as.
uid nslcd
gid nslcd

# The location at which the LDAP server(s) should be reachable.
uri ldaps://myadserver.rtp.example.test

# The search base that will be used for all queries.
base ou=support,dc=rtp,dc=example,dc=test

# The LDAP protocol version to use.
#ldap_version 3

# The DN to bind with for normal lookups.
# defconf-set-selections doesn't seem to set this. so have to manually set this.
binddn CN=cumulus admin,CN=Users,DC=rtp,DC=example,DC=test
bindpw 1Q2w3e4r!

# The DN used for password modifications by root.
#rootpwmoddn cn=admin,dc=example,dc=com

# SSL options
#ssl off (default)
# Not good does not prevent man in the middle attacks
#tls_reqcert demand(default)
tls_cacertfile /etc/ssl/certs/rtp-example-ca.crt

# The search scope.
#scope sub

# Add nested group support
# Supported in nslcd 0.9 and higher.
# default wheezy install of nslcd supports on 0.8. wheezy-backports has 0.9
nss_nested_groups yes

# Mappings for Active Directory
# (replace the SIDs in the objectSid mappings with the value for your domain)
# "dsquery * -filter (samaccountname=testuser1) -attr ObjectSID" where cn == 'testuser1'
pagesize 1000
referrals off
idle_timelimit 1000

# Do not allow uids lower than 100 to login (aka Administrator)
# not needed as pam already has this support
# nss_min_uid 1000

# This filter says to get all users who are part of the cumuluslnxadm group. Supports nested groups.
# Example, mary is part of the snrnetworkadm group which is part of cumuluslnxadm group
# Ref: http://msdn.microsoft.com/en-us/library/aa746475%28VS.85%29.aspx (LDAP_MATCHING_RULE_IN_CHAIN)
filter passwd (&(Objectclass=user)(!(objectClass=computer))(memberOf:1.2.840.113556.1.4.1941:=cn=cumuluslnxadm,ou=groups,ou=support,dc=rtp,dc=example,dc=test))
map    passwd uid           sAMAccountName
map    passwd uidNumber     objectSid:S-1-5-21-1391733952-3059161487-1245441232
map    passwd gidNumber     objectSid:S-1-5-21-1391733952-3059161487-1245441232
map    passwd homeDirectory "/home/$sAMAccountName"
map    passwd gecos         displayName
map    passwd loginShell    "/bin/bash"

# Filter for any AD group or user in the baseDN. the reason for filtering for the
# user to make sure group listing for user files don't say '<user> <gid>'. instead will say '<user> <user>'
# So for cosmetic reasons..nothing more.
filter group (&(|(objectClass=group)(Objectclass=user))(!(objectClass=computer)))
map    group gidNumber     objectSid:S-1-5-21-1391733952-3059161487-1245441232
map    group cn            sAMAccountName

Configure LDAP Authorization

Linux uses the sudo command to allow non-administrator users (such as the default cumulus user account) to perform privileged operations. To control the users that can use sudo, define a series of rules in the /etc/sudoers file and files in the /etc/sudoers.d/ directory. The rules apply to groups but you can also define specific users. You can add sudo rules using the group names from LDAP. For example, if a group of users are in the group netadmin, you can add a rule to give those users sudo privileges. Refer to the sudoers manual (man sudoers) for a complete usage description. The following shows an example in the /etc/sudoers file:

# The basic structure of a user specification is "who where = (as_whom) what ".
%sudo ALL=(ALL:ALL) ALL
%netadmin ALL=(ALL:ALL) ALL

Active Directory Configuration

Active Directory (AD) is a fully featured LDAP-based NIS server create by Microsoft. It offers unique features that classic OpenLDAP servers do not have. AD can be more complicated to configure on the client and each version works a little differently with Linux-based LDAP clients. Some more advanced configuration examples, from testing LDAP clients on Cumulus Linux with Active Directory (AD/LDAP), are available in the knowledge base.

LDAP Verification Tools

The LDAP client daemon retrieves and caches password and group information from LDAP. To verify the LDAP interaction, use these command-line tools to trigger an LDAP query from the device.

Identify a User with the id Command

The id command performs a username lookup by following the lookup information sources in NSS for the passwd service. This returns the user ID, group ID and the group list retrieved from the information source. In the following example, the user cumulus is locally defined in /etc/passwd, and myuser is on LDAP. The NSS configuration has the passwd map configured with the sources compat ldap:

cumulus@switch:~$ id cumulus
uid=1000(cumulus) gid=1000(cumulus) groups=1000(cumulus),24(cdrom),25(floppy),27(sudo),29(audio),30(dip),44(video),46(plugdev)
cumulus@switch:~$ id myuser
uid=1230(myuser) gid=3000(Development) groups=3000(Development),500(Employees),27(sudo)

getent

The getent command retrieves all records found with NSS for a given map. It can also retrieve a specific entry under that map. You can perform tests with the passwd, group, shadow, or any other map in the /etc/nsswitch.conf file. The output from this command formats according to the map requested. For the passwd service, the structure of the output is the same as the entries in /etc/passwd. The group map outputs the same structure as /etc/group.

In this example, looking up a specific user in the passwd map, the user cumulus is locally defined in /etc/passwd, and myuser is only in LDAP.

cumulus@switch:~$ getent passwd cumulus
cumulus:x:1000:1000::/home/cumulus:/bin/bash
cumulus@switch:~$ getent passwd myuser
myuser:x:1230:3000:My Test User:/home/myuser:/bin/bash

In the next example, looking up a specific group in the group service, the group cumulus is locally defined in /etc/groups, and netadmin is on LDAP.

cumulus@switch:~$ getent group cumulus
cumulus:x:1000:
cumulus@switch:~$ getent group netadmin
netadmin:*:502:larry,moe,curly,shemp

Running the command getent passwd or getent group without a specific request returns all local and LDAP entries for the passwd and group maps.

LDAP search

The ldapsearch command performs LDAP operations directly on the LDAP server. This does not interact with NSS. This command displays the information that the LDAP daemon process receives back from the server. The command has several options. The simplest option uses anonymous bind to the host and specifies the search DN and the attribute to look up.

cumulus@switch:~$ ldapsearch -H ldap://ldap.example.com -b dc=example,dc=com -x uid=myuser

# extended LDIF
#
# LDAPv3
# base <dc=example,dc=com> with scope subtree
# filter: uid=myuser
# requesting: ALL
#
# myuser, people, example.com
dn: uid=myuser,ou=people,dc=example,dc=com
cn: My User
displayName: My User
gecos: myuser
gidNumber: 3000
givenName: My
homeDirectory: /home/myuser
initials: MU
loginShell: /bin/bash
mail: myuser@example.com
objectClass: inetOrgPerson
objectClass: posixAccount
objectClass: shadowAccount
objectClass: top
shadowExpire: -1
shadowFlag: 0
shadowMax: 999999
shadowMin: 8
shadowWarning: 7
sn: User
uid: myuser
uidNumber: 1234

# search result
search: 2
result: 0 Success

# numResponses: 2
# numEntries: 1

LDAP Browsers

The GUI LDAP clients are free tools that show the structure of the LDAP database graphically.

• Apache Directory Studio
• LDAPManager

Troubleshooting

nslcd Debug Mode

When setting up LDAP authentication for the first time, turn off the nslcd service using the systemctl stop nslcd.service command (or the systemctl stop nslcd@mgmt.service if you are running the service in a management VRF) and run it in debug mode. Debug mode works whether you are using LDAP over SSL (port 636) or an unencrypted LDAP connection (port 389).

cumulus@switch:~$ sudo systemctl stop nslcd.service
cumulus@switch:~$ sudo nslcd -d

After you enable debug mode, run the following command to test LDAP queries:

cumulus@switch:~$ getent passwd

If you configure LDAP correctly, the following messages appear after you run the getent command:

nslcd: DEBUG: accept() failed (ignored): Resource temporarily unavailable
nslcd: [8e1f29] DEBUG: connection from pid=11766 uid=0 gid=0
nslcd: [8e1f29] <passwd(all)> DEBUG: myldap_search(base="dc=example,dc=com", filter="(objectClass=posixAccount)")
nslcd: [8e1f29] <passwd(all)> DEBUG: ldap_result(): uid=myuser,ou=people,dc=example,dc=com
nslcd: [8e1f29] <passwd(all)> DEBUG: ldap_result(): ... 152 more results
nslcd: [8e1f29] <passwd(all)> DEBUG: ldap_result(): end of results (162 total)

In the example output above, <passwd(all)> shows a query of the entire directory structure.

You can query a specific user with the following command:

cumulus@switch:~$ getent passwd myuser

You can replace myuser with any username on the switch. The following debug output indicates that user myuser exists:

nslcd: DEBUG: add_uri(ldap://10.50.21.101)
nslcd: version 0.8.10 starting
nslcd: DEBUG: unlink() of /var/run/nslcd/socket failed (ignored): No such file or directory
nslcd: DEBUG: setgroups(0,NULL) done
nslcd: DEBUG: setgid(110) done
nslcd: DEBUG: setuid(107) done
nslcd: accepting connections
nslcd: DEBUG: accept() failed (ignored): Resource temporarily unavailable
nslcd: [8b4567] DEBUG: connection from pid=11369 uid=0 gid=0
nslcd: [8b4567] <passwd="myuser"> DEBUG: myldap_search(base="dc=cumulusnetworks,dc=com", filter="(&(objectClass=posixAccount)(uid=myuser))")
nslcd: [8b4567] <passwd="myuser"> DEBUG: ldap_initialize(ldap://<ip_address>)
nslcd: [8b4567] <passwd="myuser"> DEBUG: ldap_set_rebind_proc()
nslcd: [8b4567] <passwd="myuser"> DEBUG: ldap_set_option(LDAP_OPT_PROTOCOL_VERSION,3)
nslcd: [8b4567] <passwd="myuser"> DEBUG: ldap_set_option(LDAP_OPT_DEREF,0)
nslcd: [8b4567] <passwd="myuser"> DEBUG: ldap_set_option(LDAP_OPT_TIMELIMIT,0)
nslcd: [8b4567] <passwd="myuser"> DEBUG: ldap_set_option(LDAP_OPT_TIMEOUT,0)
nslcd: [8b4567] <passwd="myuser"> DEBUG: ldap_set_option(LDAP_OPT_NETWORK_TIMEOUT,0)
nslcd: [8b4567] <passwd="myuser"> DEBUG: ldap_set_option(LDAP_OPT_REFERRALS,LDAP_OPT_ON)
nslcd: [8b4567] <passwd="myuser"> DEBUG: ldap_set_option(LDAP_OPT_RESTART,LDAP_OPT_ON)
nslcd: [8b4567] <passwd="myuser"> DEBUG: ldap_simple_bind_s(NULL,NULL) (uri="ldap://<ip_address>")
nslcd: [8b4567] <passwd="myuser"> DEBUG: ldap_result(): end of results (0 total)

Common Problems

SSL/TLS

• The FQDN of the LDAP server URI does not match the FQDN in the CA-signed server certificate.
• nslcd cannot read the SSL certificate and reports a Permission denied error in the debug during server connection negotiation. Check the permission on each directory in the path of the root SSL certificate. Ensure that it is readable by the nslcd user.

NSCD

• If the nscd cache daemon is also enabled and you make some changes to the user from LDAP, you can clear the cache using the following commands:

  nscd --invalidate = passwd
  nscd --invalidate = group
  

• The nscd package works with nslcd to cache name entries returned from the LDAP server. This sometimes causes authentication failures. To work around these issues, disable nscd, restart the nslcd service, then retry authentication:

  cumulus@switch:~$ sudo nscd -K
  cumulus@switch:~$ sudo systemctl restart nslcd.service
  

cumulus@switch:~$ sudo nscd -K
cumulus@switch:~$ sudo systemctl restart nslcd@mgmt.service

LDAP

• The search filter returns incorrect results. Check for typos in the search filter. Use ldapsearch to test your filter.

• Optionally, configure the basic LDAP connection and search parameters in /etc/ldap/ldap.conf.

  # ldapsearch -D 'cn=CLadmin' -w 'CuMuLuS' "(&(ObjectClass=inetOrgUser)(uid=myuser))"
  

Related Information

• Debian - configuring LDAP authentication
• Debian - configuring PAM to use LDAP
• GitHub - Arthur de Jong nslcd.conf file
• Debian backports

Cumulus Linux implements TACACS+ client AAA in a transparent way with minimal configuration. The client implements the TACACS+ protocol as described in this IETF document. There is no need to create accounts or directories on the switch. Accounting records go to all configured TACACS+ servers by default. Using per-command authorization requires additional setup on the switch.

TACACS+ in Cumulus Linux:

• Uses PAM authentication and includes login, ssh, sudo and su.
• Allows users with privilege level 15 to run any command with sudo.
• Allows users with privilege level 15 to run NVUE nv set, nv unset, and nv apply commands in addition to nv show commands. TACACS+ users with a lower privilege level can only execute nv show commands.
• Supports up to seven TACACS+ servers. Be sure to configure your TACACS+ servers in addition to the TACACS+ client. Refer to your TACACS+ server documentation.

Install the TACACS+ Client Packages

'tacplus-client' package needs to be installed to enable tacacs

You can install the TACACS+ packages even if the switch is not connected to the internet; the packages are in the cumulus-local-apt-archive repository in the Cumulus Linux image.

To install all required packages, run these commands:

cumulus@switch:~$ sudo -E apt-get update
cumulus@switch:~$ sudo -E apt-get install tacplus-client

Required TACACS+ Client Configuration

After you install the required TACACS+ packages, configure the following required settings on the switch (the TACACS+ client).

• Set the IP address or hostname of at least one TACACS+ server.
• Set the secret (key) shared between the TACACS+ server and client.
• Set the VRF you want to use to communicate with the TACACS+ server. This is typically the management VRF (mgmt), which is the default VRF on the switch.

If you use NVUE commands to configure TACACS+, you must also set the priority for the authentication order for local and TACACS+ users, and enable TACACS+.

cumulus@switch:~$ nv set system aaa tacacs server 5 host 192.168.0.30
cumulus@switch:~$ nv set system aaa tacacs server 5 secret 'mytacac$key'
cumulus@switch:~$ nv set system aaa tacacs vrf mgmt 
cumulus@switch:~$ nv set system aaa authentication-order 5 tacacs
cumulus@switch:~$ nv set system aaa authentication-order 10 local
cumulus@switch:~$ nv set system aaa tacacs enable on
cumulus@switch:~$ nv config apply

cumulus@switch:~$ nv set system aaa tacacs server 5 host server5
cumulus@switch:~$ nv set system aaa tacacs server 5 prefer-ip-version 6
...

cumulus@switch:~$ nv set system aaa tacacs server 5 host 192.168.0.30
cumulus@switch:~$ nv set system aaa tacacs server 5 secret 'mytacac$key' 
cumulus@switch:~$ nv set system aaa tacacs server 10 host 192.168.1.30
cumulus@switch:~$ nv set system aaa tacacs server 10 secret 'mytacac$key2'
cumulus@switch:~$ nv config apply

Optional TACACS+ Configuration

You can configure the following optional TACACS+ settings:

• The port to use for communication between the TACACS+ server and client. By default, Cumulus Linux uses IP port 49.
• The TACACS timeout value, which is the number of seconds to wait for a response from the TACACS+ server before trying the next TACACS+ server. You can specify a value between 0 and 60. The default is 5 seconds.
• The source IP address to use when communicating with the TACACS+ server so that the server can identify the client switch. You must specify an IPv4 address, which must be valid for the interface you use. This source IP address is typically the loopback address on the switch.

• The TACACS+ authentication type. You can specify PAP to send clear text between the user and the server, CHAP to establish a PPP connection between the user and the server, or login. The default is PAP.

• The users you do not want to send to the TACACS+ server for authentication; for example, local user accounts that exist on the switch, such as the cumulus user.
• A separate home directory for each TACACS+ user when the TACACS+ user first logs in. By default, the switch uses the home directory in the mapping accounts in /etc/passwd. If the home directory does not exist, the mkhomedir_helper program creates it. This option does not apply to accounts with restricted shells (users mapped to a TACACS privilege level that has enforced per-command authorization).

cumulus@switch:~$ nv set system aaa tacacs timeout 10
cumulus@switch:~$ nv set system aaa tacacs server 5 port 32
cumulus@switch:~$ nv config apply

cumulus@switch:~$ nv set system aaa tacacs source-ip 10.10.10.1
cumulus@switch:~$ nv set system aaa tacacs authentication mode chap
cumulus@switch:~$ nv config apply

cumulus@switch:~$ nv set system aaa tacacs exclude-user USER1
cumulus@switch:~$ nv set system aaa tacacs authentication per-user-homedir on
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo nano /etc/tacplus_servers
...
secret=mytacac$key
server=192.168.0.30:32
...
# Sets the IPv4 address used as the source IP address when communicating with
# the TACACS+ server.  IPv6 addresses are not supported, nor are hostnames.
# The address must work when passsed to the bind() system call, that is, it must
# be valid for the interface being used.
source_ip=10.10.10.1
...
# If user_homedir=1, then tacacs users will be set to have a home directory
# based on their login name, rather than the mapped tacacsN home directory.
# mkhomedir_helper is used to create the directory if it does not exist (similar
# to use of pam_mkhomedir.so). This flag is ignored for users with restricted
# shells, e.g., users mapped to a tacacs privilege level that has enforced
# per-command authorization (see the tacplus-restrict man page).
user_homedir=1
...
login=chap

cumulus@switch:~$ sudo systemctl restart auditd

cumulus@switch:~$ sudo nano /etc/tacplus_nss.conf
...
# The connection timeout for an NSS library should be short, since it is
# invoked for many programs and daemons, and a failure is usually not
# catastrophic.  Not set or set to a negative value disables use of poll().
# This follows the include of tacplus_servers, so it can override any
# timeout value set in that file.
# It's important to have this set in this file, even if the same value
# as in tacplus_servers, since tacplus_servers should not be readable
# by users other than root.
timeout=10
...
# This is a comma separated list of usernames that are never sent to
# a tacacs server, they cause an early not found return.
#
# "*" is not a wild card.  While it's not a legal username, it turns out
# that during pathname completion, bash can do an NSS lookup on "*"
# To avoid server round trip delays, or worse, unreachable server delays
# on filename completion, we include "*" in the exclusion list.
exclude_users=root,daemon,nobody,cron,radius_user,radius_priv_user,sshd,cumulus,quagga,frr,snmp,www-data,ntp,man,_lldpd,USER1,*

TACACS+ Accounting

When you install the TACACS+ packages and configure the basic TACACS+ settings (set the server and shared secret), accounting is on and there is no additional configuration required.

TACACS+ accounting uses the audisp module, with an additional plugin for auditd and audisp. The plugin maps the auid in the accounting record to a TACACS login, which it bases on the auid and sessionid. The audisp module requires libnss_tacplus and uses the libtacplus_map.so library interfaces as part of the modified libpam_tacplus package.

Communication with the TACACS+ servers occurs with the libsimple-tacact1 library, through dlopen(). A maximum of 240 bytes of command name and arguments send in the accounting record, due to the TACACS+ field length limitation of 255 bytes.

By default, Cumulus Linux sends accounting records to all servers. You can change this setting to send accounting records to the server that is first to respond:

cumulus@switch:~$ nv set system aaa tacacs accounting send-records first-response
cumulus@switch:~$ nv config apply

To disable TACACS+ accounting:

cumulus@switch:~$ nv set system aaa tacacs accounting enable off
cumulus@switch:~$ nv config apply

Local Fallback Authentication

You can configure the switch to allow local fallback authentication for a user when the TACACS servers are unreachable, do not include the user for authentication, or have the user in the exclude user list.

To allow local fallback authentication for a user, add a local privileged user account on the switch with the same username as a TACACS user. A local user is always active even when the TACACS service is not running.

To configure local fallback authentication:

1. Edit the /etc/nsswitch.conf file to remove the keyword tacplus from the line starting with passwd. (You need to add the keyword back in step 3.)

   The following example shows the /etc/nsswitch.conf file with no tacplus keyword in the line starting with passwd.

   cumulus@switch:~$ sudo nano /etc/nsswitch.conf
   #
   # Example configuration of GNU Name Service Switch functionality.
   # If you have the `glibc-doc-reference' and `info' packages installed, try:
   # `info libc "Name Service Switch"' for information about this file.
   
   passwd:         files
   group:          tacplus files
   shadow:         files
   gshadow:        files
   ...
   

2. To enable the local privileged user to run sudo and NVUE commands, run the adduser commands shown below. In the example commands, the TACACS account name is tacadmin.

   The first adduser command prompts for information and a password. You can skip most of the requested information by pressing ENTER.

   cumulus@switch:~$ sudo adduser --ingroup tacacs tacadmin
   cumulus@switch:~$ sudo adduser tacadmin nvset
   cumulus@switch:~$ sudo adduser tacadmin nvapply
   cumulus@switch:~$ sudo adduser tacadmin sudo
   

3. Edit the /etc/nsswitch.conf file to add the keyword tacplus back to the line starting with passwd (the keyword you removed in the first step).

    cumulus@switch:~$ sudo nano /etc/nsswitch.conf
    #
    # Example configuration of GNU Name Service Switch functionality.
    # If you have the `glibc-doc-reference' and `info' packages installed, try:
    # `info libc "Name Service Switch"' for information about this file.
    passwd:         tacplus files
    group:          tacplus files
    shadow:         files
    gshadow:        files
    ...
   

4. Restart the nvued service with the following command:

   cumulus@switch:~$ sudo systemctl restart nvued
   

TACACS+ Per-command Authorization

TACACS+ per-command authorization lets you configure the commands that TACACS+ users at different privilege levels can run.

The following command allows TACACS+ users at privilege level 0 to run the nv and ip commands (if authorized by the TACACS+ server):

cumulus@switch:~$ nv set system aaa tacacs authorization 0 command ip 
cumulus@switch:~$ nv set system aaa tacacs authorization 0 command nv
cumulus@switch:~$ nv config apply

cumulus@switch:~$ nv show system aaa tacacs authorization
Privilege Level  role          command
---------------  ------------  -------
0                nvue-monitor  ip     
                               nv  

tacuser0@switch:~$ sudo tacplus-restrict -i -u tacacs0 -a ip nv

cumulus@switch:~$ sudo ls -lR ~tacacs0
total 12
lrwxrwxrwx 1 root root 22 Nov 21 22:07 ip -> /usr/sbin/tacplus-auth
lrwxrwxrwx 1 root root 22 Nov 21 22:07 nv -> /usr/sbin/tacplus-auth

cumulus@switch:~$ sudo rm ~tacacs0/bin/nv

cumulus@switch:~$ sudo rm ~tacacs0/bin/*

Remove the TACACS+ Client Packages

To remove all the TACACS+ client packages, use the following commands:

cumulus@switch:~$ sudo -E apt-get remove tacplus-client
cumulus@switch:~$ sudo -E apt-get autoremove

To remove the TACACS+ client configuration files as well as the packages (recommended), use this command:

cumulus@switch:~$ sudo -E apt-get autoremove --purge

Troubleshooting

Show TACACS+ Configuration

Run the following commands to show TACACS+ configuration:

• To show all TACACS+ configuration (NVUE hides server secret keys), run the nv show aaa tacacs command.
• To show TACACS+ authentication configuration , run the nv show system aaa tacacs authentication command.
• To show TACACS+ accounting configuration , run the nv show system aaa tacacs accounting command.
• To show TACACS+ server configuration, run the nv show system aaa tacacs server command.
• To show TACACS+ server priority configuration, run the nv show system aaa tacacs server <priority-id> command.
• To show the list of users excluded from TACACS+ server authentication, run the nv show system aaa tacacs exclude-user command.

The following example command shows all TACACS+ configuration:

cumulus@switch:~$ nv show system aaa tacacs
                    applied
------------------  -------
enable              off    
debug-level         0      
timeout             5      
vrf                 mgmt   
accounting                 
  enable            off    
authentication             
  mode              pap    
  per-user-homedir  off    
[server]            5      
[server]            10 

The following command shows the list of users excluded from TACACS+ server authentication:

cumulus@switch:~$ nv show system aaa tacacs exclude-user
          applied
--------  -------
username  USER1  

Basic Server Connectivity or NSS Issues

You can use the getent command to determine if you configured TACACS+ correctly and if the local password is in the configuration files. In the example commands below, the cumulus user represents the local user, while cumulusTAC represents the TACACS user.

To look up the username within all NSS methods:

cumulus@switch:~$ sudo getent passwd cumulusTAC
cumulusTAC:x:1016:1001:TACACS+ mapped user at privilege level 15,,,:/home/tacacs15:/bin/bash

To look up the user within the local database only:

cumulus@switch:~$ sudo getent -s compat passwd cumulus
cumulus:x:1000:1000:cumulus,,,:/home/cumulus:/bin/bash

To look up the user within the TACACS+ database only:

cumulus@switch:~$ sudo getent -s tacplus passwd cumulusTAC
cumulusTAC:x:1016:1001:TACACS+ mapped user at privilege level 15,,,:/home/tacacs15:/bin/bash

If TACACS+ is not working correctly, you can use debugging. Add the debug=1 parameter to the /etc/tacplus_servers and /etc/tacplus_nss.conf files; see the Linux Commands under Optional TACACS+ Configuration above. You can also add debug=1 to individual pam_tacplus lines in /etc/pam.d/common*.

All log messages are in /var/log/syslog.

Incorrect Shared Key

The TACACS client on the switch and the TACACS server must have the same shared secret key. If this key is incorrect, the following message prints to syslog:

2017-09-05T19:57:00.356520+00:00 leaf01 sshd[3176]: nss_tacplus: TACACS+ server 192.168.0.254:49 read failed with protocol error (incorrect shared secret?) user cumulus

Debug Issues with Per-command Authorization

To debug TACACS user command authorization, have the TACACS+ user enter the following command at a shell prompt, then try the command again:

tacuser0@switch:~$ export TACACSAUTHDEBUG=1

When you enable debugging, the command authorization conversation with the TACACS+ server shows additional information.

To disable debugging:

tacuser0@switch:~$ export -n TACACSAUTHDEBUG

Debug Issues with Accounting Records

If you add or delete TACACS+ servers from the configuration files, make sure you notify the audisp plugin with this command:

cumulus@switch:~$ sudo killall -HUP audisp-tacplus

If accounting records do not send, add debug=1 to the /etc/audisp/audisp-tac_plus.conf file, then run the command above to notify the plugin. Ask the TACACS+ user to run a command and examine the end of /var/log/syslog for messages from the plugin. You can also check the auditing log file /var/log/audit audit.log to be sure the auditing records exist. If the auditing records do not exist, restart the audit daemon with:

cumulus@switch:~$ sudo systemctl restart auditd.service

TACACS+ Package Descriptions

Cumulus Linux uses the following packages for TACACS.

TACACS+ Client Configuration Files

The following table describes the TACACS+ client configuration files that Cumulus Linux uses.

Considerations

Multiple TACACS+ Users

If two or more TACACS+ users log in simultaneously with the same privilege level, while the accounting records are correct, a lookup on either name matches both users, while a UID lookup only returns the user that logs in first.

As a result, any processes that either user runs apply to both and all files either user creates apply to the first name matched. This is similar to adding two local users to the password file with the same UID and GID and is an inherent limitation of using the UID for the base user from the password file.

To work around this issue, you can use the switch audit log or the TACACS server accounting logs to determine which processes and files each user creates.

• For commands that do not execute other commands (for example, changes to configurations in an editor or actions with tools like clagctl and vtysh), there is no additional accounting.
• Per-command authorization is at the most basic level (Cumulus Linux uses standard Linux user permissions for the local TACACS users and only privilege level 15 users can run sudo commands by default).

The Linux auditd system does not always generate audit events for processes when terminated with a signal (with the kill system call or internal errors such as SIGSEGV). As a result, processes that exit on a signal that you do not handle, generate a STOP accounting record.

Issues with the deluser Command

TACACS+ and other non-local users that run the deluser command with the --remove-home option see the following error:

tacuser0@switch: deluser --remove-home USERNAME
userdel: cannot remove entry 'USERNAME' from /etc/passwd
/usr/sbin/deluser: `/usr/sbin/userdel USERNAME' returned error code 1. Exiting

The command does remove the home directory. The user can still log in on that account but does not have a valid home directory. This is a known upstream issue with the deluser command for all non-local users.

Only use the --remove-home option with the user_homedir=1 configuration command.

Both TACACS+ and RADIUS AAA Clients

When you install both the TACACS+ and the RADIUS AAA client, Cumulus Linux does not attempt RADIUS login. As a workaround, do not install both the TACACS+ and the RADIUS AAA client on the same switch.

TACACS+ and PAM

PAM modules and an updated version of the libpam-tacplus package configure authentication initially. When you install the package, the pam-auth-update command updates the PAM configuration in /etc/pam.d. If you make changes to your PAM configuration, you need to integrate these changes. If you also use LDAP with the libpam-ldap package, you need to edit the PAM configuration with the LDAP and TACACS ordering you prefer. The libpam-tacplus package ignore rules and the values in success=2 require adjustments to ignore LDAP rules.

The TACACS+ privilege attribute priv_lvl determines the privilege level for the user that the TACACS+ server returns during the user authorization exchange. The client accepts the attribute in either the mandatory or optional forms and also accepts priv-lvl as the attribute name. The attribute value must be a numeric string in the range 0 to 15, with 15 the most privileged level.

NSS Plugin

With pam_tacplus, TACACS+ authenticated users can log in without a local account on the system using the NSS plugin that comes with the tacplus_nss package. The plugin uses the mapped tacplus information if the user is not in the local password file, provides the getpwnam() and getpwuid()entry points, and uses the TACACS+ authentication functions.

The plugin asks the TACACS+ server if it knows the user, and then for relevant attributes to determine the privilege level of the user. When you install the libnss_tacplus package, nsswitch.conf changes to set tacplus as the first lookup method for passwd. If you change the order, lookups return the local accounts, such as tacacs0

If TACACS+ server does not find the user, it uses the libtacplus.so exported functions to do a mapped lookup. The privilege level appends to tacacs and the lookup searches for the name in the local password file. For example, privilege level 15 searches for the tacacs15 user. If the TACACS+ server finds the user, it adds information for the user in the password structure.

If the TACACS+ server does not find the user, it decrements the privilege level and checks again until it reaches privilege level 0 (user tacacs0). This allows you to use only the two local users tacacs0 and tacacs15, for minimal configuration.

TACACS+ Client Sequencing

Cumulus Linux requires the following information at the beginning of the AAA sequence:

• Whether the user is a valid TACACS+ user
• The user privilege level

For non-local users (users not in the local password file) you need to send a TACACS+ authorization request as the first communication with the TACACS+ server, before authentication and before the user logging in requests a password.

You need to configure certain TACACS+ servers to allow authorization requests before authentication. Contact your TACACS+ server vendor for information.

Multiple Servers with Different User Accounts

If you configure multiple TACACS+ servers that have different user accounts:

• TACACS+ authentication allows for fall through; if the first reachable server does not authenticate the user, the client tries the second server, and so on.
• TACACS authorization does not fall through. If the first reachable server returns an unauthorized result, the command is unauthorized and the client does not try the next server.

Various add-on packages enable RADIUS users to log in to Cumulus Linux switches in a transparent way with minimal configuration. There is no need to create accounts or directories on the switch. Authentication uses PAM and includes login, ssh, sudo and su.

Install the RADIUS Packages

You can install the RADIUS packages even if the switch is not connected to the internet, as they are in the cumulus-local-apt-archive repository, which is embedded in the Cumulus Linux image.

To install the RADIUS packages:

cumulus@switch:~$ sudo apt-get update
cumulus@switch:~$ sudo apt-get install libnss-mapuser libpam-radius-auth

After installation is complete, either reboot the switch or run the sudo systemctl restart nvued command.

The libpam-radius-auth package supplied with the Cumulus Linux RADIUS client is a newer version than the one in Debian Buster. This package contains support for IPv6, the src_ip option described below, as well as bug fixes and minor features. The package also includes VRF support, provides man pages describing the PAM and RADIUS configuration, and sets the SUDO_PROMPT environment variable to the login name for RADIUS mapping support.

The libnss-mapuser package is specific to Cumulus Linux and supports the getgrent, getgrnam and getgrgid library interfaces. These interfaces add logged in RADIUS users to the group member list for groups that contain the mapped_user (radius_user) if the RADIUS account does not have privileges, and add privileged RADIUS users to the group member list for groups that contain the mapped_priv_user (radius_priv_user) during the group lookups.

During package installation:

• The PAM configuration updates automatically using pam-auth-update (8), and the NSS configuration file /etc/nsswitch.conf adds the mapuser and mapuid plugins. If you remove or purge the packages, these files remove the configuration for these plugins.
• The radius_shell package installs the /sbin/radius_shell and setcap cap_setuid program for the login shell for RADIUS accounts. The package adjusts the UID when needed, then runs the bash shell with the same arguments. When installed, the package changes the shell of the RADIUS accounts to /sbin//radius_shell, and to /bin/shell if you remove the package. You need this package to enable privileged RADIUS users. You do not need this package for regular RADIUS clients.
• The nvshow group includes the radius_user account, the nvset and nvapply groups and sudo groups include the radius_priv_user account. This change enables all RADUS logins to run NVUE nv show commands and all privileged RADIUS users to also run nv set, nv unset, and nv apply commands, and to use sudo.

Configure the RADIUS Client

To configure the RADIUS client, edit the /etc/pam_radius_auth.conf file:

1. Add the hostname or IP address of at least one RADIUS server (such as a freeradius server on Linux), and the shared secret used to authenticate and encrypt communication with each server.

   You must be able to resolve the hostname of the switch to an IP address. If for some reason you cannot find the hostname in DNS, you can add the hostname to the /etc/hosts file manually. However, this can cause problems because DHCP assigns the IP address, which can change at any time.

   Multiple server configuration lines are verified in the order listed. Other than memory, there is no limit to the number of RADIUS servers you can use.

   The server port number or name is optional. The system looks up the port in the /etc/services file. However, you can override the ports in the /etc/pam_radius_auth.conf file.

2. If the server is slow or latencies are high, change the timeout setting. The setting defaults to 3 seconds.

3. If you want to use a specific interface to reach the RADIUS server, specify the src_ip option. You can specify the hostname of the interface, an IPv4, or an IPv6 address. If you specify the src_ip option, you must also specify the timeout option.

4. Set the vrf-name field. This is typically set to mgmt if you are using a management VRF. You cannot specify more than one VRF.

The configuration file includes the mapped_priv_user field that sets the account used for privileged RADIUS users and the priv-lvl field that sets the minimum value for the privilege level to be a privileged login (the default value is 15). If you edit these fields, make sure the values match those set in the /etc/nss_mapuser.conf file.

The following example provides a sample /etc/pam_radius_auth.conf file configuration:

mapped_priv_user   radius_priv_user
# server[:port]    shared_secret   timeout (secs)  src_ip
192.168.0.254      secretkey
other-server       othersecret     3               192.168.1.10
# when mgmt vrf is in use
vrf-name mgmt

As an optional step, you can set PAM configuration keywords by editing the /usr/share/pam-configs/radius file. After you edit the file, you must run the pam-auth-update --package command. The pam_radius_auth (8) man page describes the PAM configuration keywords.

Service-Type = Administrative-User,
Cisco-AVPair = "shell:roles=network-administrator",
Cisco-AVPair += "shell:priv-lvl=15"

Enable Login without Local Accounts

LDAP is not commonly used with switches and adding accounts locally is cumbersome, Cumulus Linux includes a mapping capability with the libnss-mapuser package.

Mapping uses two NSS (Name Service Switch) plugins, one for account name, and one for UID lookup. The installation process configures these accounts automatically in the /etc/nsswitch.conf file and removes them when you delete the package. See the nss_mapuser (8) man page for the full description of this plugin.

A username is mapped at login to a fixed account specified in the configuration file, with the fields of the fixed account used as a template for the user that is logging in.

For example, if you look up the name dave and the fixed account in the configuration file is radius\_user, and that entry in /etc/passwd is:

radius_user:x:1017:1002:radius user:/home/radius_user:/bin/bash

then the matching line that returns when you run getent passwd dave is:

cumulus@switch:~$ getent passwd dave
dave:x:1017:1002:dave mapped user:/home/dave:/bin/bash

The login process creates the home directory /home/dave if it does not already exist and populates it with the standard skeleton files by the mkhomedir_helper command.

The configuration file /etc/nss_mapuser.conf configures the plugins. The file includes the mapped account name, which is radius_user by default. You can change the mapped account name by editing the file. The nss_mapuser (5) man page describes the configuration file.

A flat file mapping derives from the session number assigned during login, which persists across su and sudo. Cumulus Linux removes the mapping at logout.

Local Fallback Authentication

If a site wants to allow local fallback authentication for a user when none of the RADIUS servers are reachable, you can add a privileged user account as a local account on the switch. The local account must have the same unique identifier as the privileged user and the shell must be the same.

To configure local fallback authentication:

1. Add a local privileged user account. For example, if the radius_priv_user account in the /etc/passwd file is radius_priv_user:x:1002:1001::/home/radius_priv_user:/sbin/radius_shell, run the following command to add a local privileged user account named johnadmin:

   cumulus@switch:~$ sudo useradd -u 1002 -g 1001 -o -s /sbin/radius_shell johnadmin
   

2. To enable the local privileged user to run sudo and NVUE commands, run the following commands:

   cumulus@switch:~$ sudo adduser johnadmin nvset
   cumulus@switch:~$ sudo adduser johnadmin nvapply
   cumulus@switch:~$ sudo adduser johnadmin sudo
   cumulus@switch:~$ sudo systemctl restart nvued
   

3. Edit the /etc/passwd file to move the local user line before to the radius_priv_user line:

   cumulus@switch:~$ sudo vi /etc/passwd
   ...
   johnadmin:x:1002:1001::/home/johnadmin:/sbin/radius_shell
   radius_priv_user:x:1002:1001::/home/radius_priv_user:/sbin/radius_shell
   

4. To set the local password for the local user, run the following command:

   cumulus@switch:~$ sudo passwd johnadmin
   

Verify RADIUS Client Configuration

To verify that you configured the RADIUS client correctly, log in as a non-privileged user and run a nv set interface command.

In this example, the ops user is not a privileged RADIUS user so the ops user cannot add an interface.

ops@leaf01:~$ nv set interface swp1
ERROR: User ops does not have permission to make networking changes.

In this example, the admin user is a privileged RADIUS user (with privilege level 15) so is able to add interface swp1.

admin@leaf01:~$ nv set interface swp1
admin@leaf01:~$ nv apply

Remove RADIUS Client Packages

Remove the RADIUS packages with the following command:

cumulus@switch:~$ sudo apt-get remove libnss-mapuser libpam-radius-auth

When you remove the packages, Cumulus Linux deletes the plugins from the /etc/nsswitch.conf file and from the PAM files.

To remove all configuration files for these packages, run:

cumulus@switch:~$ sudo apt-get purge libnss-mapuser libpam-radius-auth

To remove the home directories of the RADIUS users, first get the list by running:

cumulus@switch:~$ sudo ls -l /home | grep radius

For all users listed, except the radius_user, run this command to remove the home directories:

cumulus@switch:~$ sudo deluser --remove-home USERNAME

where USERNAME is the account name (the home directory relative portion). This command gives the following warning because the user is not listed in the /etc/passwd file.

userdel: cannot remove entry 'USERNAME' from /etc/passwd
/usr/sbin/deluser: `/usr/sbin/userdel USERNAME' returned error code 1. Exiting.

After you remove all the RADIUS users, run the command to remove the fixed account. If there are changes to the account in the /etc/nss_mapuser.conf file, use that account name instead of radius_user.

cumulus@switch:~$ sudo deluser --remove-home radius_user
cumulus@switch:~$ sudo deluser --remove-home radius_priv_user
cumulus@switch:~$ sudo delgroup radius_users

Considerations

• If two or more RADIUS users log in simultaneously, a UID lookup only returns the user that logs in first. Any process that either user runs applies to both, and all files that either user creates apply to the first name matched. This process is similar to adding two local users to the password file with the same UID and GID, and is an inherent limitation of using the UID for the fixed user from the password file. The current algorithm returns the first name matching the UID from the mapping file, which is either the first or second user that logs in.

• When you install both the TACACS+ and the RADIUS AAA client, Cumulus Linux does not attempt the RADIUS login. As a workaround, do not install both the TACACS+ and the RADIUS AAA client on the same switch.

• When the RADIUS server is reachable outside of the management VRF, such as in the default VRF, you might see the following error message when you try to run sudo:

  2008-10-31T07:06:36.191359+00:00 SW01 sudo: pam_radius_auth(sudo:auth): Bind for server 10.1.1.25 failed: Cannot assign requested address
  2008-10-31T07:06:36.192307+00:00 sw01 sudo: pam_radius_auth(sudo:auth): No valid server found in configuration file /etc/pam_radius_auth.conf
  

  The error occurs because sudo tries to authenticate to the RADIUS server through the management VRF. Before you run sudo, you must set the shell to the correct VRF:

  cumulus@switch:~$ vrf exec default bash
  cumulus@switch:~$ sudo 
  

Netfilter is the packet filtering framework in Cumulus Linux and other Linux distributions. You can use several different tools to configure ACLs in Cumulus Linux:

• iptables, ip6tables, and ebtables are Linux userspace tools you use to administer filtering rules for IPv4 packets, IPv6 packets, and Ethernet frames (layer 2 using MAC addresses).
• cl-acltool is a Cumulus Linux-specific userspace tool you use to administer filtering rules and configure default ACLs. cl-acltool operates on various configuration files and uses iptables, ip6tables, and ebtables to install rules into the kernel. In addition, cl-acltool programs rules in hardware for switch port interfaces, which iptables, ip6tables and ebtables cannot do on their own.
• NVUE is a Cumulus Linux-specific userspace tool you can use to configure custom ACLs.

Traffic Rules

Chains

Netfilter describes the way that the Linux kernel classifies and controls packets to, from, and across the switch. Netfilter does not require a separate software daemon to run; it is part of the Linux kernel. Netfilter asserts policies at layer 2, 3 and 4 of the OSI model by inspecting packet and frame headers according to a list of rules. The iptables, ip6tables, and ebtables userspace applications provide syntax you use to define rules.

The rules inspect or operate on packets at several points (chains) in the life of the packet through the system:

• PREROUTING touches packets before the switch routes them.
• INPUT touches packets after the switch determines that the packets are for the local system but before the control plane software receives them.
• FORWARD touches transit traffic as it moves through the switch.
• OUTPUT touches packets from the control plane software before they leave the switch.
• POSTROUTING touches packets immediately before they leave the switch but after a routing decision.

Tables

When you build rules to affect the flow of traffic, tables can access the individual chains. Linux provides three tables by default:

• Filter classifies traffic or filters traffic
• NAT applies Network Address Translation rules
• Mangle alters packets as they move through the switch

Each table has a set of default chains that modify or inspect packets at different points of the path through the switch. Chains contain the individual rules to influence traffic.

Rules

Rules classify the traffic you want to control. You apply rules to chains, which attach to tables.

Rules have several different components:

• Table: The first argument is the table.
• Chain: The second argument is the chain. Each table supports several different chains. See Tables above.
• Matches: The third argument is the match. You can specify multiple matches in a single rule. However, the more matches you use in a rule, the more memory the rule consumes.
• Jump: The jump specifies the target of the rule; what action to take if the packet matches the rule. If you omit this option in a rule, matching the rule has no effect on the packet, but the counters on the rule increment.
• Targets: The target is a user-defined chain (other than the one this rule is in), one of the special built-in targets that decides the fate of the packet immediately (like DROP), or an extended target. See Supported Rule Types below for different target examples.

How Rules Parse and Apply

The switch reads all the rules from each chain from iptables, ip6tables, and ebtables and enters them in order into either the filter table or the mangle table. The switch reads the rules from the kernel in the following order:

• IPv6 (ip6tables)
• IPv4 (iptables)
• ebtables

When you combine and put rules into one table, the order determines the relative priority of the rules; iptables and ip6tables have the highest precedence and ebtables has the lowest.

The Linux packet forwarding construct is an overlay for how the silicon underneath processes packets. Be aware of the following:

• The switch silicon reorders rules when switchd writes to the ASIC, whereas traditional iptables execute the list of rules in order.

• All rules, except for POLICE and SETCLASS rules, are terminating; after a rule matches, the action occurs and no more rules process.

• When processing traffic, rules affecting the FORWARD chain that specify an ingress interface process before rules that match on an egress interface. As a workaround, rules that only affect the egress interface can have an ingress interface wildcard (only swp+ and bond+) that matches any interface you apply so that you can maintain order of operations with other input interface rules. For example, with the following rules:

  -A FORWARD -i swp1 -j ACCEPT
  -A FORWARD -o swp1 -j ACCEPT   <-- This rule processes LAST (because of egress interface matching)
  -A FORWARD -i swp2 -j DROP
  

  If you modify the rules like this, they process in order:

  -A FORWARD -i swp1 -j ACCEPT
  -A FORWARD -i swp+ -o $PORTA -j ACCEPT   <-- These rules are performed in order (because of wildcard match on the ingress interface)
  -A FORWARD -i swp2 -j DROP
  

• When using rules that do a mangle and a filter lookup for a packet, Cumulus Linux processes them in parallel and combines the action.

• If there is no ingress interface or egress interface match, Cumulus Linux installs FORWARD chain rules in ingress by default.

• When using the OUTPUT chain, you must assign rules to the source. For example, if you assign a rule to the switch port in the direction of traffic but the source is a bridge (VLAN), the rule does not affect the traffic and you must apply it to the bridge.

• If you need to apply a rule to all transit traffic, use the FORWARD chain, not the OUTPUT chain.

• The switch puts ebtable rules into either the IPv4 or IPv6 memory space depending on whether the rule uses IPv4 or IPv6 to make a decision. The switch only puts layer 2 rules that match the MAC address into the IPv4 memory space.

Rule Placement in Memory

INPUT and ingress (FORWARD -i) rules occupy the same memory space. A rule counts as ingress if you set the -i option. If you set both input and output options (-i and -o), the switch considers the rule as ingress and occupies that memory space. For example:

-A FORWARD -i swp1 -o swp2 -s 10.0.14.2 -d 10.0.15.8 -p tcp -j ACCEPT

-A FORWARD,INPUT -i swp1 -o swp2 -s 10.0.14.2 -d 10.0.15.8 -p tcp -j ACCEPT
error: line 2 : output interface specified with INPUT chain error processing rule '-A FORWARD,INPUT -i swp1 -o swp2 -s 10.0.14.2 -d 10.0.15.8 -p tcp -j ACCEPT'

Nonatomic Update Mode and Atomic Update Mode

Cumulus Linux enables atomic update mode by default. However, this mode limits the number of ACL rules that you can configure.

To increase the number of configurable ACL rules, configure the switch to operate in nonatomic mode.

Instead of reserving 50% of your TCAM space for atomic updates, incremental update uses the available free space to write the new TCAM rules and swap over to the new rules after this is complete. Cumulus Linux then deletes the old rules and frees up the original TCAM space. If there is insufficient free space to complete this task, the original nonatomic update runs, which interrupts traffic.

You can enable nonatomic updates for switchd, which offer better scaling because all TCAM resources actively impact traffic. With atomic updates, half of the hardware resources are on standby and do not actively impact traffic.

Incremental nonatomic updates are table based, so they do not interrupt network traffic when you install new rules. The rules map to the following tables and update in this order:

• mirror (ingress only)
• ipv4-mac (can be both ingress and egress)
• ipv6 (ingress only)

The incremental nonatomic update operation follows this order:

1. Updates are incremental, one table at a time without stopping traffic.
2. Cumulus Linux checks if the rules in a table are different from installation time; if a table does not have any changes, it does not reinstall the rules.
3. If there are changes in a table, the new rules populate in new groups or slices in hardware, then that table switches over to the new groups or slices.
4. Finally, old resources for that table free up. This process repeats for each of the tables listed above.
5. If there are insufficient resources to hold both the new rule set and old rule set, Cumulus Linux tries the regular nonatomic mode, which interrupts network traffic.
6. If the regular nonatomic update fails, Cumulus Linux reverts back to the previous rules.

To always reload switchd with nonatomic updates:

1. Edit /etc/cumulus/switchd.conf.

2. Add the following line to the file:

   acl.non_atomic_update_mode = TRUE
   

3. Reload switchd with the sudo systemctl reload switchd.service command for the changes to take effect. The reload does not interrupt network services.

Use iptables, ip6tables, and ebtables Directly

Do not use iptables, ip6tables, ebtables directly; installed rules only apply to the Linux kernel and Cumulus Linux does not hardware accelerate. When you run cl-acltool -i, Cumulus Linux resets all rules and deletes anything that is not in /etc/cumulus/acl/policy.conf.

For example, the following rule appears to work:

cumulus@switch:~$ sudo iptables -A INPUT -p icmp --icmp-type echo-request -j DROP

The cl-acltool -L command shows the rule:

cumulus@switch:~$ sudo cl-acltool -L ip
-------------------------------
Listing rules of type iptables:
-------------------------------

TABLE filter :
Chain INPUT (policy ACCEPT 72 packets, 5236 bytes)
pkts bytes target prot opt in out source destination
0 0 DROP icmp -- any any anywhere anywhere icmp echo-request

However, Cumulus Linux does not synchronize the rule to hardware. Running cl-acltool -i or reboot removes the rule without replacing it. To ensure that Cumulus Linux hardware accelerates all rules that can be in hardware, add them to /etc/cumulus/acl/policy.conf and install them with the cl-acltool -i command.

Estimate the Number of Rules

To estimate the number of rules you can create from an ACL entry, first determine if that entry is an ingress or an egress. Then, determine if it is an IPv4-mac or IPv6 type rule. This determines the slice to which the rule belongs. Use the following to determine how many entries the switch uses for each type.

By default, each entry occupies one double wide entry, except if the entry is one of the following:

• An entry with multiple comma-separated input interfaces splits into one rule for each input interface. For example, this entry splits into two rules:

  -A FORWARD -i swp1s0,swp1s1 -p icmp -j ACCEPT
  

• An entry with multiple comma-separated output interfaces splits into one rule for each output interface. This entry splits into two rules:

  -A FORWARD -i swp+ -o swp1s0,swp1s1 -p icmp -j ACCEPT
  

• An entry with both input and output comma-separated interfaces splits into one rule for each combination of input and output interface This entry splits into four rules:

  -A FORWARD -i swp1s0,swp1s1 -o swp1s2,swp1s3 -p icmp -j ACCEPT
  

• An entry with multiple layer 4 port ranges splits into one rule for each range. For example, this entry splits into two rules:

  -A FORWARD -i swp+ -p tcp -m multiport --dports 1050:1051,1055:1056 -j ACCEPT
  

  You can only use port ranges for ingress rules.

Match on VLAN IDs on Layer 2 Interfaces

You can match on VLAN IDs on layer 2 interfaces for ingress rules. The following example matches on a VLAN and DSCP class, and sets the internal class of the packet. For extended matching on IP fields, combine this rule with ingress iptable rules.

[ebtables]
-A FORWARD -p 802_1Q --vlan-id 100 -j mark --mark-set 102

[iptables]
-A FORWARD -i swp31 -m mark --mark 102 -m dscp --dscp-class CS1 -j SETCLASS --class 2

Install and Manage ACL Rules with NVUE

Instead of crafting a rule by hand, then installing it with cl-acltool, you can use NVUE commands. Cumulus Linux converts the commands to the /etc/cumulus/acl/policy.d/50_nvue.rules file. The rules you create with NVUE are independent of the default files /etc/cumulus/acl/policy.d/00control_plane.rules and 99control_plane_catch_all.rules.

Consider the following iptables rule:

-t mangle -A PREROUTING -i swp1 -s 10.0.14.2/32 -d 10.0.15.8/32 -p tcp -j ACCEPT

To create this rule with NVUE, follow the steps below. NVUE adds all options in the rule automatically.

1. Set the rule type, the matching protocol, source IP address and port, destination IP address and port, and the action. You must provide a name for the rule (EXAMPLE1 in the commands below):

   cumulus@switch:~$ nv set acl EXAMPLE1 type ipv4
   cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 match ip protocol tcp
   cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 match ip source-ip 10.0.14.2/32
   cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 match ip source-port ANY
   cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 match ip dest-ip 10.0.15.8/32
   cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 match ip dest-port ANY
   cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 action permit
   

2. Apply the rule to an inbound or outbound interface with the nv set interface <interface> acl command.

   • For rules affecting the -t mangle -A PREROUTING chain (-A FORWARD in previous releases), apply the rule to an inbound or outbound interface: For example:

   cumulus@switch:~$ nv set interface swp1 acl EXAMPLE1 inbound
   cumulus@switch:~$ nv config apply
   

   • For rules affecting the INPUT or OUPUT chain (-A INPUT or -A OUTPUT), apply the rule to a control plane interface. For example:

   cumulus@switch:~$ nv set interface swp1 acl EXAMPLE1 inbound control-plane
   cumulus@switch:~$ nv config apply
   

To see all installed rules, examine the /etc/cumulus/acl/policy.d/50_nvue.rules file:

cumulus@switch:~$ sudo cat /etc/cumulus/acl/policy.d/50_nvue.rules
[iptables]

## ACL EXAMPLE1 in dir inbound on interface swp1 ##
-t mangle -A PREROUTING -i swp1 -s 10.0.14.2/32 -d 10.0.15.8/32 -p tcp -j ACCEPT
...

To remove this rule, run the nv unset acl <acl-name> and nv unset interface <interface> acl <acl-name> commands. These commands delete the rule from the /etc/cumulus/acl/policy.d/50_nvue.rules file.

cumulus@switch:~$ nv unset acl EXAMPLE1
cumulus@switch:~$ nv unset interface swp1 acl EXAMPLE1
cumulus@switch:~$ nv config apply

To show ACL statistics per interface, such as the total number of bytes that match the ACL rule, run the nv show interface <interface-id> acl <acl-id> statistics or nv show interface <interface-id> acl <acl-id> statistics <rule-id> command.

To see the list of all NVUE ACL commands, run the nv list-commands acl command.

Install and Manage ACL Rules with cl-acltool

You can manage Cumulus Linux ACLs with cl-acltool. Rules write first to the iptables chains, as described above, and then synchronize to hardware through switchd.

To examine the current state of chains and list all installed rules, run:

cumulus@switch:~$ sudo cl-acltool -L all
 -------------------------------
Listing rules of type iptables:
-------------------------------

TABLE filter :
Chain INPUT (policy ACCEPT 90 packets, 14456 bytes)
pkts bytes target prot opt in out source destination
0 0 DROP all -- swp+ any 240.0.0.0/5 anywhere
0 0 DROP all -- swp+ any loopback/8 anywhere
0 0 DROP all -- swp+ any base-address.mcast.net/8 anywhere
0 0 DROP all -- swp+ any 255.255.255.255 anywhere ...

To list installed rules using native iptables, ip6tables and ebtables, use the -L option with the respective commands:

cumulus@switch:~$ sudo iptables -L
cumulus@switch:~$ sudo ip6tables -L
cumulus@switch:~$ sudo ebtables -L

To remove all installed rules, run:

cumulus@switch:~$ sudo cl-acltool -F all

To remove only the IPv4 iptables rules, run:

cumulus@switch:~$ sudo cl-acltool -F ip

If the install fails, ACL rules in the kernel and hardware roll back to the previous state. You also see errors from programming rules in the kernel or ASIC.

Install Packet Filtering (ACL) Rules

cl-acltool takes access control list (ACL) rule input in files. Each ACL policy file includes iptables, ip6tables and ebtables categories under the tags [iptables], [ip6tables] and [ebtables]. You must assign each rule in an ACL policy to one of the rule categories.

See man cl-acltool(5) for ACL rule details. For iptables rule syntax, see man iptables(8). For ip6tables rule syntax, see man ip6tables(8). For ebtables rule syntax, see man ebtables(8).

See man cl-acltool(5) and man cl-acltool(8) for more details on using cl-acltool.

Here is an example ACL policy file:

[iptables]
-A INPUT -i swp1 -p tcp --dport 80 -j ACCEPT
-A FORWARD -i swp1 -p tcp --dport 80 -j ACCEPT

[ip6tables]
-A INPUT -i swp1 -p tcp --dport 80 -j ACCEPT
-A FORWARD -i swp1 -p tcp --dport 80 -j ACCEPT

[ebtables]
-A INPUT -p IPv4 -j ACCEPT
-A FORWARD -p IPv4 -j ACCEPT

You can use wildcards or variables to specify chain and interface lists.

INGRESS = swp+
INPUT_PORT_CHAIN = INPUT,FORWARD

[iptables]
-A $INPUT_PORT_CHAIN -i $INGRESS -p tcp --dport 80 -j ACCEPT

[ip6tables]
-A $INPUT_PORT_CHAIN -i $INGRESS -p tcp --dport 80 -j ACCEPT

[ebtables]
-A INPUT -p IPv4 -j ACCEPT

You can write ACL rules for the system into multiple files under the default /etc/cumulus/acl/policy.d/ directory. The ordering of rules during installation follows the sort order of the files according to their file names.

Use multiple files to stack rules. The example below shows two rule files that separate rules for management and datapath traffic:

cumulus@switch:~$ ls /etc/cumulus/acl/policy.d/
00sample_mgmt.rules 01sample_datapath.rules
cumulus@switch:~$ cat /etc/cumulus/acl/policy.d/00sample_mgmt.rules

INGRESS_INTF = swp+
INGRESS_CHAIN = INPUT

[iptables]
# protect the switch management
-A $INGRESS_CHAIN -i $INGRESS_INTF -s 10.0.14.2 -d 10.0.15.8 -p tcp -j ACCEPT
-A $INGRESS_CHAIN -i $INGRESS_INTF -s 10.0.11.2 -d 10.0.12.8 -p tcp -j ACCEPT
-A $INGRESS_CHAIN -i $INGRESS_INTF -d 10.0.16.8 -p udp -j DROP

cumulus@switch:~$ cat /etc/cumulus/acl/policy.d/01sample_datapath.rules
INGRESS_INTF = swp+
INGRESS_CHAIN = INPUT, FORWARD

[iptables]
-A $INGRESS_CHAIN -i $INGRESS_INTF -s 192.0.2.5 -p icmp -j ACCEPT
-A $INGRESS_CHAIN -i $INGRESS_INTF -s 192.0.2.6 -d 192.0.2.4 -j DROP
-A $INGRESS_CHAIN -i $INGRESS_INTF -s 192.0.2.2 -d 192.0.2.8 -j DROP

Install all ACL policies under a directory:

cumulus@switch:~$ sudo cl-acltool -i -P ./rules
Reading files under rules
Reading rule file ./rules/01_http_rules.txt ...
Processing rules in file ./rules/01_http_rules.txt ...
Installing acl policy ...
Done.

Apply all rules and policies included in /etc/cumulus/acl/policy.conf:

cumulus@switch:~$ sudo cl-acltool -i

Specify the Policy Files to Install

By default, Cumulus Linux installs any .rules file you configure in /etc/cumulus/acl/policy.d/. To add other policy files to an ACL, you need to include them in /etc/cumulus/acl/policy.conf. For example, for Cumulus Linux to install a rule in a policy file called 01_new.datapathacl, add include /etc/cumulus/acl/policy.d/01_new.rules to policy.conf:

cumulus@switch:~$ sudo nano /etc/cumulus/acl/policy.conf

#
# This file is a master file for acl policy file inclusion
#
# Note: This is not a file where you list acl rules.
#
# This file can contain:
# - include lines with acl policy files
#   example:
#     include <filepath>
#
# see manpage cl-acltool(5) and cl-acltool(8) for how to write policy files 
#

include /etc/cumulus/acl/policy.d/01_new.datapathacl

Hardware Limitations on Number of Rules

The maximum number of rules that the hardware process depends on:

• The mix of IPv4 and IPv6 rules; Cumulus Linux does not support the maximum number of rules for both IPv4 and IPv6 simultaneously.
• The number of default rules that Cumulus Linux provides.
• Whether the rules apply on ingress or egress.
• Whether the rules are in atomic or nonatomic mode; Cumulus Linux uses nonatomic mode rules when you enable nonatomic updates (see above).

If you exceed the maximum number of rules for a particular table, cl-acltool -i generates the following error:

error: hw sync failed (sync_acl hardware installation failed) Rolling back .. failed.

In the table below, the default rules count toward the limits listed. The raw limits below assume only one ingress and one egress table are present.

The NVIDIA Spectrum ASIC has one common TCAM for both ingress and egress, which you can use for other non-ACL-related resources. However, the number of supported rules varies with the TCAM profile for the switch.