NVIDIA® Cumulus Linux is the first full-featured Debian Buster-based, Linux operating system for the networking industry.

This user guide provides in-depth documentation on the Cumulus Linux installation process, system configuration and management, network solutions, and monitoring and troubleshooting recommendations. In addition, the quick start guide provides an end-to-end setup process to get you started.

Cumulus Linux 5.2 includes the NVIDIA NetQ agent and CLI. You can use NetQ to monitor and manage your data center network infrastructure and operational health. Refer to the NVIDIA NetQ documentation for details.

For a list of the new features in this release, see What's New. For bug fixes and known issues present in this release, refer to the Cumulus Linux 5.2 Release Notes.

Try It Pre-built Demos

The Cumulus Linux documentation includes pre-built Try It demos for certain Cumulus Linux features. The Try It demos run a simulation in NVIDIA Air; a cloud hosted platform that works exactly like a real world production deployment. Use the Try It demos to examine switch configuration for a feature. For more information, see Try It Pre-built Demos.

Open Source Contributions

To implement various Cumulus Linux features, NVIDIA has forked various software projects, like CFEngine Netdev and some Puppet Labs packages. Some of the forked code resides in the NVIDIA Networking GitHub repository and some is available as part of the Cumulus Linux repository as Debian source packages.

NVIDIA has also developed and released new applications as open source. The list of open source projects is on the Cumulus Linux packages page.

Download the User Guide

You can view the complete Cumulus Linux 5.2 user guide as a single page to print to PDF here.

This document supports the Cumulus Linux 5.2 release, and lists new platforms, features, and enhancements.

• For a list of open and fixed issues in Cumulus Linux 5.2, see the Cumulus Linux 5.2 Release Notes.
• To upgrade to Cumulus Linux 5.2, follow the steps in Upgrading Cumulus Linux.

What’s New in Cumulus Linux 5.2.1

Cumulus Linux 5.2.1 provides bug fixes.

What’s New in Cumulus Linux 5.2.0

Cumulus Linux 5.2.0 supports new platforms, provides bug fixes, and contains several new features and improvements.

Platforms

• NVIDIA SN2201 (100G Spectrum-1)

New Features and Enhancements

• Support for signed images on secured switches and support for SecureApt to update individual packages
• ZTP on front panel ports
• PTP enhancements include:
  • Pre-defined PTP profiles and custom profiles
  • PTP unicast message mode
• NVUE enhancements include:
  • Command abbreviation
  • Command question mark (?) to show required information quickly and concisely, such as the command value type, range, and options with a brief description of each. ? also indicates if you need to provide specific values for the command.
  • TEID-based ECMP hashing and TEID-based bond hashing configuration commands available
  • Show ACL statistics per interface commands
  • New commands:
    show commands

    nv show interface <interface-id> acl <acl-id> statistics
    nv show interface <interface-id> acl <acl-id> statistics <rule-id>
    nv show service ptp <instance-id> unicast-master
    nv show service ptp <instance-id> unicast-master <table-id>
    nv show service ptp <instance-id> unicast-master <table-id> address
    nv show service ptp <instance-id> unicast-master <table-id> address <ip-mac-address-id>
    nv show service ptp <instance-id> profile
    nv show service ptp <instance-id> profile <profile-id>
    nv show system forwarding
    nv show system forwarding lag-hash
    nv show system forwarding ecmp-hash
    

    set commands

    nv set router policy route-map <route-map-id> rule <rule-id> match evpn-default-route (on|off)
    nv set router policy route-map <route-map-id> rule <rule-id> set as-path-prepend as 1-4294967295
    nv set router policy route-map <route-map-id> rule <rule-id> set as-path-prepend as <asn-range>
    nv set router policy route-map <route-map-id> rule <rule-id> set originator-id <ipv4>
    nv set router policy route-map <route-map-id> rule <rule-id> set label-index 0-1048560
    nv set router policy route-map <route-map-id> rule <rule-id> set forwarding-address <ipv6>
    nv set router policy route-map <route-map-id> rule <rule-id> description none
    nv set interface <interface-id> link breakout (1x|2x20G|2x40G|2x50G|2x100G|2x200G|4x10G|4x25G|4x50G|4x100G|8x50G|disabled|loopback)
    nv set interface <interface-id> link breakout (1x|2x|4x|8x|2x10G|2x25G|2x40G|2x50G|2x100G|2x200G|4x10G|4x25G|4x50G|4x100G|8x50G|disabled|loopback)
    nv set interface <interface-id> link speed (auto|10M|100M|1G|10G|25G|40G|50G|100G|200G|400G)
    nv set interface <interface-id> link speed (auto|10M|100M|1G|10G|25G|40G|50G|100G|200G|400G|800G)
    nv set interface <interface-id> link lanes (1|2|4|8)
    nv set interface <interface-id> ptp message-mode (multicast|unicast|mixed)
    nv set interface <interface-id> ptp mixed-multicast-unicast (on|off)
    nv set interface <interface-id> ptp unicast-service-mode (client|server)
    nv set interface <interface-id> ptp unicast-request-duration 10-1000
    nv set interface <interface-id> ptp unicast-master-table-id <integer>
    nv set service ptp <instance-id> unicast-master <table-id>
    nv set service ptp <instance-id> unicast-master <table-id> address <ip-mac-address-id>
    nv set service ptp <instance-id> unicast-master <table-id> query-interval -3-4
    nv set service ptp <instance-id> unicast-master <table-id> peer-address (<ipv4>|<ipv6>)
    nv set service ptp <instance-id> profile <profile-id>
    nv set service ptp <instance-id> profile <profile-id> profile-type (ieee-1588|itu-g-8275-1)
    nv set service ptp <instance-id> profile <profile-id> priority1 <value>
    nv set service ptp <instance-id> profile <profile-id> priority2 <value>
    nv set service ptp <instance-id> profile <profile-id> local-priority <value>
    nv set service ptp <instance-id> profile <profile-id> domain 0-255
    nv set service ptp <instance-id> profile <profile-id> delay-mechanism end-to-end
    nv set service ptp <instance-id> profile <profile-id> transport (ipv4|ipv6|802.3)
    nv set service ptp <instance-id> profile <profile-id> announce-interval -7-7
    nv set service ptp <instance-id> profile <profile-id> sync-interval -7-7
    nv set service ptp <instance-id> profile <profile-id> delay-req-interval -7-7
    nv set service ptp <instance-id> profile <profile-id> announce-timeout 2-255
    nv set service ptp <instance-id> current-profile <value>
    nv set system control-plane policer <policer-id> rate 10-10000
    nv set system control-plane policer <policer-id> rate 10-50000
    nv set system forwarding
    nv set system forwarding lag-hash
    nv set system forwarding lag-hash ip-protocol (on|off)
    nv set system forwarding lag-hash source-mac (on|off)
    nv set system forwarding lag-hash destination-mac (on|off)
    nv set system forwarding lag-hash source-ip (on|off)
    nv set system forwarding lag-hash destination-ip (on|off)
    nv set system forwarding lag-hash source-port (on|off)
    nv set system forwarding lag-hash destination-port (on|off)
    nv set system forwarding lag-hash ether-type (on|off)
    nv set system forwarding lag-hash vlan (on|off)
    nv set system forwarding lag-hash gtp-teid (on|off)
    nv set system forwarding ecmp-hash
    nv set system forwarding ecmp-hash ip-protocol (on|off)
    nv set system forwarding ecmp-hash source-ip (on|off)
    nv set system forwarding ecmp-hash destination-ip (on|off)
    nv set system forwarding ecmp-hash source-port (on|off)
    nv set system forwarding ecmp-hash destination-port (on|off)
    nv set system forwarding ecmp-hash ipv6-label (on|off)
    nv set system forwarding ecmp-hash ingress-interface (on|off)
    nv set system forwarding ecmp-hash gtp-teid (on|off)
    nv set system forwarding ecmp-hash inner-ip-protocol (on|off)
    nv set system forwarding ecmp-hash inner-source-ip (on|off)
    nv set system forwarding ecmp-hash inner-destination-ip (on|off)
    nv set system forwarding ecmp-hash inner-source-port (on|off)
    nv set system forwarding ecmp-hash inner-destination-port (on|off)
    nv set system forwarding ecmp-hash inner-ipv6-label (on|off)
    nv set system forwarding hash-seed 0-4294967295
    nv set vrf <vrf-id> router bgp address-family ipv4-unicast network <static-network-id> route-map (none|<instance-name>)
    nv set vrf <vrf-id> router bgp address-family ipv4-unicast network <static-network-id> route-map <instance-name>
    nv set vrf <vrf-id> router bgp address-family ipv6-unicast network <static-network-id> route-map (none|<instance-name>)
    nv set vrf <vrf-id> router bgp address-family ipv6-unicast network <static-network-id> route-map <instance-name>
    

    unset commands

    nv unset router policy route-map <route-map-id> rule <rule-id> match evpn-default-route
    nv unset router policy route-map <route-map-id> rule <rule-id> set originator-id
    nv unset router policy route-map <route-map-id> rule <rule-id> set label-index
    nv unset router policy route-map <route-map-id> rule <rule-id> set forwarding-address
    nv unset router policy route-map <route-map-id> rule <rule-id> description
    nv unset interface <interface-id> link lanes
    nv unset interface <interface-id> ptp message-mode
    nv unset interface <interface-id> ptp mixed-multicast-unicast
    nv unset interface <interface-id> ptp unicast-service-mode
    nv unset interface <interface-id> ptp unicast-request-duration
    nv unset interface <interface-id> ptp unicast-master-table-id
    nv unset service ptp <instance-id> unicast-master
    nv unset service ptp <instance-id> unicast-master <table-id>
    nv unset service ptp <instance-id> unicast-master <table-id> address
    nv unset service ptp <instance-id> unicast-master <table-id> address <ip-mac-address-id>
    nv unset service ptp <instance-id> unicast-master <table-id> query-interval
    nv unset service ptp <instance-id> unicast-master <table-id> peer-address
    nv unset service ptp <instance-id> profile
    nv unset service ptp <instance-id> profile <profile-id>
    nv unset service ptp <instance-id> profile <profile-id> profile-type
    nv unset service ptp <instance-id> profile <profile-id> priority1
    nv unset service ptp <instance-id> profile <profile-id> priority2
    nv unset service ptp <instance-id> profile <profile-id> local-priority
    nv unset service ptp <instance-id> profile <profile-id> domain
    nv unset service ptp <instance-id> profile <profile-id> delay-mechanism
    nv unset service ptp <instance-id> profile <profile-id> transport
    nv unset service ptp <instance-id> profile <profile-id> announce-interval
    nv unset service ptp <instance-id> profile <profile-id> sync-interval
    nv unset service ptp <instance-id> profile <profile-id> delay-req-interval
    nv unset service ptp <instance-id> profile <profile-id> announce-timeout
    nv unset service ptp <instance-id> current-profile
    nv unset system forwarding
    nv unset system forwarding lag-hash
    nv unset system forwarding lag-hash ip-protocol
    nv unset system forwarding lag-hash source-mac
    nv unset system forwarding lag-hash destination-mac
    nv unset system forwarding lag-hash source-ip
    nv unset system forwarding lag-hash destination-ip
    nv unset system forwarding lag-hash source-port
    nv unset system forwarding lag-hash destination-port
    nv unset system forwarding lag-hash ether-type
    nv unset system forwarding lag-hash vlan
    nv unset system forwarding lag-hash gtp-teid
    nv unset system forwarding ecmp-hash
    nv unset system forwarding ecmp-hash ip-protocol
    nv unset system forwarding ecmp-hash source-ip
    nv unset system forwarding ecmp-hash destination-ip
    nv unset system forwarding ecmp-hash source-port
    nv unset system forwarding ecmp-hash destination-port
    nv unset system forwarding ecmp-hash ipv6-label
    nv unset system forwarding ecmp-hash ingress-interface
    nv unset system forwarding ecmp-hash gtp-teid
    nv unset system forwarding ecmp-hash inner-ip-protocol
    nv unset system forwarding ecmp-hash inner-source-ip
    nv unset system forwarding ecmp-hash inner-destination-ip
    nv unset system forwarding ecmp-hash inner-source-port
    nv unset system forwarding ecmp-hash inner-destination-port
    nv unset system forwarding ecmp-hash inner-ipv6-label
    nv unset system forwarding hash-seed
    nv config history [<revision>]
    

This quick start guide provides an end-to-end setup process for installing and running Cumulus Linux.

Prerequisites

This guide assumes you have intermediate-level Linux knowledge. You need to be familiar with basic text editing, Unix file permissions, and process monitoring. A variety of text editors are pre-installed, including vi and nano.

You must have access to a Linux or UNIX shell. If you are running Windows, use a Linux environment like Cygwin as your command line tool for interacting with Cumulus Linux.

Get Started

Cumulus Linux is installed on the switch by default. To upgrade to a different Cumulus Linux release or re-install Cumulus Linux, refer to Installation Management. To show the Cumulus Linux release installed on the switch, run the NVUE nv show system command.

When starting Cumulus Linux for the first time, the management port makes a DHCPv4 request. To determine the IP address of the switch, you can cross reference the MAC address of the switch with your DHCP server. The MAC address is typically located on the side of the switch or on the box in which the unit ships.

To get started:

• Log in to Cumulus Linux on the switch and change the default credentials.
• Configure Cumulus Linux. This quick start guide provides instructions on changing the hostname of the switch, setting the date and time, and configuring switch ports and a loopback interface.

Login Credentials

The default installation includes two accounts:

• The system account (root) has full system privileges. Cumulus Linux locks the root account password by default (which prohibits login).

• The user account (cumulus) has sudo privileges. The cumulus account uses the default password cumulus.

  When you log in for the first time with the cumulus account, Cumulus Linux prompts you to change the default password. After you provide a new password, the SSH session disconnects and you have to reconnect with the new password.

In this quick start guide, you use the cumulus account to configure Cumulus Linux.

All accounts except root can use remote SSH login; you can use sudo to grant a non-root account root-level access. Commands that change the system configuration require this elevated level of access.

For more information about sudo, see Using sudo to Delegate Privileges.

Serial Console Management

NVIDIA recommends you perform management and configuration over the network, either in band or out of band. A serial console is fully supported.

Typically, switches ship from the manufacturer with a mating DB9 serial cable. Switches with ONIE are always set to a 115200 baud rate.

Wired Ethernet Management

A Cumulus Linux switch always provides at least one dedicated Ethernet management port called eth0. This interface is specifically for out-of-band management use. The management interface uses DHCPv4 for addressing by default.

To set a static IP address:

cumulus@switch:~$ nv set interface eth0 ip address 192.0.2.42/24
cumulus@switch:~$ nv set interface eth0 ip gateway 192.0.2.1
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo nano /etc/network/interfaces
# Management interface
auto eth0
iface eth0
    address 192.0.2.42/24
    gateway 192.0.2.1

Configure the Hostname

The hostname identifies the switch; make sure you configure the hostname to be unique and descriptive.

To change the hostname:

cumulus@switch:~$ nv set system hostname leaf01
cumulus@switch:~$ nv config apply

Configure the Time Zone

The default time zone on the switch is UTC (Coordinated Universal Time). Change the time zone on your switch to be the time zone for your location.

To update the time zone:

cumulus@switch:~$ nv set system timezone US/Eastern
cumulus@switch:~$ nv config apply

Verify the System Time

Verify that the date and time on the switch are correct with the Linux date command:

cumulus@switch:~$ date
Mon 21 Nov 2022 06:30:37 PM UTC

If the date and time are incorrect, the switch does not synchronize with automation tools, such as Puppet, and returns errors after you restart switchd.

To set the software clock according to the configured time zone, run the Linux sudo date -s command; for example:

cumulus@switch:~$ sudo date -s "Tue Jan 26 00:37:13 2021"

For more information about setting the system time, see Setting the Date and Time.

NTP and PTP

• NTP starts at boot by default on the switch and the NTP configuration includes default servers. To customize NTP, see NTP.
• PTP is off by default on the switch. To configure PTP, see PTP.

Configure Breakout Ports with Splitter Cables

If you are using 4x10G DAC or AOC cables, or you want to break out 100G or 40G switch ports, configure the breakout ports. For more details, see Switch Port Attributes.

Test Cable Connectivity

By default, Cumulus Linux disables all data plane ports (every Ethernet port except the management interface, eth0). To test cable connectivity, administratively enable physical ports.

cumulus@switch:~$ nv set interface swp1
cumulus@switch:~$ nv config apply

cumulus@switch:~$ nv set interface swp1-52
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo ip link set swp1 up

cumulus@switch:~$ sudo su -
cumulus@switch:~$ for i in /sys/class/net/*; do iface=`basename $i`; if [[ $iface == swp* ]]; then ip link set $iface up fi done

Configure Layer 2 Ports

Cumulus Linux does not put all ports into a bridge by default. To create a bridge and configure one or more front panel ports as members of the bridge:

cumulus@switch:~$ nv set interface swp1 bridge domain br_default
cumulus@switch:~$ nv config apply

cumulus@switch:~$ nv set interface swp1-3,swp6,swp14-20 bridge domain br_default
cumulus@switch:~$ nv config apply

...
auto br_default
iface br_default
    bridge-ports swp1
...

...
auto br_default
iface br_default
    bridge-ports glob swp1-10 swp12 glob swp14-20
...

cumulus@switch:~$ sudo ifquery -a

cumulus@switch:~$ sudo ifup -a

For more information about Ethernet bridges, see Ethernet Bridging - VLANs.

Configure Layer 3 Ports

You can configure a front panel port or bridge interface as a layer 3 port.

cumulus@switch:~$ nv set interface swp1 ip address 10.0.0.0/31
cumulus@switch:~$ nv config apply

cumulus@switch:~$ nv set interface swp1-2 bridge domain br_default
cumulus@switch:~$ nv set bridge domain br_default vlan 10
cumulus@switch:~$ nv set interface vlan10 ip address 10.1.10.2/24
cumulus@switch:~$ nv set bridge domain br_default untagged 1
cumulus@switch:~$ nv config apply

auto swp1
iface swp1
  address 10.0.0.0/31

auto br_default
iface br_default
    address 10.1.10.2/24
    bridge-ports swp1 swp2
    bridge-pvid 1

cumulus@switch:~$ sudo ifquery -a

cumulus@switch:~$ sudo ifup -a

Configure a Loopback Interface

Cumulus Linux has a preconfigured loopback interface. When the switch boots up, the loopback interface, called lo, is up and assigned an IP address of 127.0.0.1.

To add an IP address to a loopback interface, configure the lo interface:

cumulus@switch:~$ nv set interface lo ip address 10.10.10.1/32
cumulus@switch:~$ nv config apply

auto lo
iface lo inet loopback
    address 10.10.10.1

You can add multiple loopback addresses. For more information, see Interface Configuration and Management.

cumulus@switch:~$ nv config save

Show Platform and System Settings

• To show the hostname of the switch, the time zone, and the version of Cumulus Linux running on the switch, run the NVUE nv show system command.
• To show switch platform information, such as the platform model, RAM, serial number, and system MAC address, run the NVUE nv show platform hardware command.

Next Steps

You are now ready to configure the switch according to your needs. This guide provides separate sections that describe how to configure system, layer 1, layer 2, layer 3, and network virtualization settings. Each section includes example configurations and pre-built demos.

For a deep dive into the NVUE object model that provides a CLI to simplify configuration, see NVUE.

This section describes how to manage, install, and upgrade Cumulus Linux on your switch.

The Cumulus Linux operating system resides on a switch as a disk image. This section discusses how to manage the image.

To install a new Cumulus Linux image, refer to Installing a New Cumulus Linux Image. To upgrade Cumulus Linux, refer to Upgrading Cumulus Linux.

Reprovision the System (Restart the Installer)

Reprovisioning the system deletes all system data from the switch.

To stage an ONIE installer from the network (where ONIE automatically locates the installer), run the onie-select -i command. You must reboot the switch to start the install process.

cumulus@switch:~$ sudo onie-select -i
WARNING:
WARNING: Operating System install requested.
WARNING: This will wipe out all system data.
WARNING:
Are you sure (y/N)? y
Enabling install at next reboot...done.
Reboot required to take effect.

To cancel a pending reinstall operation, run the onie-select -c command:

cumulus@switch:~$ sudo onie-select -c
Cancelling pending install at next reboot...done.

To stage an installer located in a specific location, run the onie-install -i <location> command. You can specify a local, absolute or relative path, an HTTP or HTTPS server, SCP or FTP server. You can also stage a Zero Touch Provisioning (ZTP) script along with the installer. You typically use the onie-install command with the -a option to activate installation. If you do not specify the -a option, you must reboot the switch to start the installation process.

The following example stages the installer located at http://203.0.113.10/image-installer together with the ZTP script located at http://203.0.113.10/ztp-script and activates installation and ZTP:

cumulus@switch:~$ sudo onie-install -i http://203.0.113.10/image-installer
cumulus@switch:~$ sudo onie-install -z http://203.0.113.10/ztp-script
cumulus@switch:~$ sudo onie-install -a

You can also specify these options together in the same command. For example:

cumulus@switch:~$ sudo onie-install -i http://203.0.113.10/image-installer -z http://203.0.113.10/ztp-script -a

To see more onie-install options, run man onie-install.

Migrate from Cumulus Linux to ONIE (Uninstall All Images and Remove the Configuration)

To remove all installed images and configurations, and return the switch to its factory defaults, run the onie-select -k command.

cumulus@switch:~$ sudo onie-select -k
WARNING:
WARNING: Operating System uninstall requested.
WARNING: This will wipe out all system data.
WARNING:
Are you sure (y/N)? y
Enabling uninstall at next reboot...done.
Reboot required to take effect.

You must reboot the switch to start the uninstallation process.

cumulus@switch:~$ sudo onie-select -c
Cancelling pending uninstall at next reboot...done.

Boot Into Rescue Mode

If your system becomes unresponsive, you can correct certain issues by booting into ONIE rescue mode, which uses unmounted file systems. You can use various Cumulus Linux utilities to try and resolve a problem.

To reboot the system into ONIE rescue mode, run the onie-select -r command:

cumulus@switch:~$ sudo onie-select -r
WARNING:
WARNING: Rescue boot requested.
WARNING:
Are you sure (y/N)? y
Enabling rescue at next reboot...done.
Reboot required to take effect.

cumulus@switch:~$ sudo onie-select -c
Cancelling pending rescue at next reboot...done.

Inspect the Image File

The Cumulus Linux image file is executable. From a running switch, you can display, extract, and verify the contents of the image file.

To display the contents of the Cumulus Linux image file, pass the info option to the image file. For example, to display the contents of an image file called onie-installer located in the /var/lib/cumulus/installer directory:

cumulus@switch:~$ sudo /var/lib/cumulus/installer/onie-installer info
Verifying image checksum ...OK.
Preparing image archive ... OK.
Control File Contents
=====================
Description: Cumulus Linux 4.1.0
Release: 4.1.0
Architecture: amd64
Switch-Architecture: bcm-amd64
Build-Id: dirtyz224615f
Build-Date: 2019-05-17T16:34:22+00:00
Build-User: clbuilder
Homepage: http://www.cumulusnetworks.com/
Min-Disk-Size: 1073741824
Min-Ram-Size: 536870912
mkimage-version: 0.11.111_gbcf0

To extract the contents of the image file, use with the extract <path> option. For example, to extract an image file called onie-installer located in the /var/lib/cumulus/installer directory to the mypath directory:

cumulus@switch:~$ sudo /var/lib/cumulus/installer/onie-installer extract mypath
total 181860
-rw-r--r-- 1 4000 4000       308 May 16 19:04 control
drwxr-xr-x 5 4000 4000      4096 Apr 26 21:28 embedded-installer
-rw-r--r-- 1 4000 4000  13273936 May 16 19:04 initrd
-rw-r--r-- 1 4000 4000   4239088 May 16 19:04 kernel
-rw-r--r-- 1 4000 4000 168701528 May 16 19:04 sysroot.tar

To verify the contents of the image file, use with the verify option. For example, to verify the contents of an image file called onie-installer located in the /var/lib/cumulus/installer directory:

cumulus@switch:~$ sudo /var/lib/cumulus/installer/onie-installer verify
Verifying image checksum ...OK.
Preparing image archive ... OK.
./cumulus-linux-bcm-amd64.bin.1: 161: ./cumulus-linux-bcm-amd64.bin.1: onie-sysinfo: not found
Verifying image compatibility ...OK.
Verifying system ram ...OK.

Related Information

You can install a new Cumulus Linux image using ONIE, an open source project (equivalent to PXE on servers) that enables the installation of network operating systems (NOS) on bare metal switches.

Before you install Cumulus Linux, the switch can be in two different states:

• The switch does not contain an image (the switch is only running ONIE).
• Cumulus Linux is already on the switch but you want to use ONIE to reinstall Cumulus Linux or upgrade to a newer version.

The sections below describe some of the different ways you can install the Cumulus Linux image. Steps show how to install directly from ONIE (if no image is on the switch) and from Cumulus Linux (if the image is already on the switch). For additional methods to find and install the Cumulus Linux image, see the ONIE Design Specification.

You can download a Cumulus Linux image from the NVIDIA Enterprise support portal.

In the following procedures:

• You can name your Cumulus Linux image using any of the ONIE naming schemes mentioned here.
• Run the sudo onie-install -h command to show the ONIE installer options.

Install Using a DHCP/Web Server With DHCP Options

To install Cumulus Linux using a DHCP or web server with DHCP options, set up a DHCP/web server on your laptop and connect the eth0 management port of the switch to your laptop. After you connect the cable, the installation proceeds as follows:

1. The switch boots up and requests an IP address (DHCP request).

2. The DHCP server acknowledges and responds with DHCP option 114 and the location of the installation image.

3. ONIE downloads the Cumulus Linux image, installs, and reboots.

   You are now running Cumulus Linux.

Here is an example DHCP configuration with an ISC DHCP server:

subnet 172.0.24.0 netmask 255.255.255.0 {
  range 172.0.24.20 172.0.24.200;
  option default-url = "http://172.0.24.14/onie-installer-x86_64";
}

Here is an example DHCP configuration with dnsmasq (static address assignment):

dhcp-host=sw4,192.168.100.14,6c:64:1a:00:03:ba,set:sw4
dhcp-option=tag:sw4,114,"http://roz.rtplab.test/onie-installer-x86_64"

If you do not have a web server, you can use this free Apache example.

Install Using a DHCP/Web Server without DHCP Options

Follow the steps below if you can log into the switch on a serial console (ONIE), or log in on the console or with ssh (Install from Cumulus Linux).

Install Using a Web Server With no DHCP

Follow the steps below if you can log into the switch on a serial console (ONIE), or you can log in on the console or with ssh (Install from Cumulus Linux) but no DHCP server is available.

Install Using FTP Without a Web Server

Follow the steps below if your laptop is on the same network as the switch eth0 interface but no DHCP server is available.

Install Using a Local File

Follow the steps below to install the Cumulus Linux image referencing a local file.

Install Using a USB Drive

Follow the steps below to install the Cumulus Linux image using a USB drive.

Prepare for USB Installation

1. From the NVIDIA Enterprise support portal, download the appropriate Cumulus Linux image for your platform.

2. From a computer, prepare your USB drive by formatting it using one of the supported formats: FAT32, vFAT or EXT2.

   ▼ Optional: Prepare a USB Drive inside Cumulus Linux

   a. Insert your USB drive into the USB port on the switch running Cumulus Linux and log in to the switch. Examine output from cat /proc/partitions and sudo fdisk -l [device] to determine the location of your USB drive. For example, sudo fdisk -l /dev/sdb.

   These instructions assume your USB drive is the /dev/sdb device, which is typical if you insert the USB drive after the machine is already booted. However, if you insert the USB drive during the boot process, it is possible that your USB drive is the /dev/sda device. Make sure to modify the commands below to use the proper device for your USB drive.

   b. Create a new partition table on the USB drive. If the parted utility is not on the system, install it with sudo -E apt-get install parted.

   sudo parted /dev/sdb mklabel msdos
   

   c. Create a new partition on the USB drive:

   sudo parted /dev/sdb -a optimal mkpart primary 0% 100%
   

   d. Format the partition to your filesystem of choice using one of the examples below:

   sudo mkfs.ext2 /dev/sdb1
   sudo mkfs.msdos -F 32 /dev/sdb1
   sudo mkfs.vfat /dev/sdb1
   

   To use mkfs.msdos or mkfs.vfat, you need to install the dosfstools package from the Debian software repositories, as they are not included by default.

   e. To continue installing Cumulus Linux, mount the USB drive to move files:

   sudo mkdir /mnt/usb
   sudo mount /dev/sdb1 /mnt/usb
   

3. Copy the Cumulus Linux image to the USB drive, then rename the image file to onie-installer-x86_64.

   You can also use any of the ONIE naming schemes mentioned here.

   When using a MAC or Windows computer to rename the installation file, the file extension can still be present. Make sure you remove the file extension so that ONIE can detect the file.

4. Insert the USB drive into the switch, then prepare the switch for installation:

   • If the switch is offline, connect to the console and power on the switch.
   • If the switch is already online in ONIE, use the reboot command.

   SSH sessions to the switch get dropped after this step. To complete the remaining instructions, connect to the console of the switch. Cumulus Linux switches display their boot process to the console; you need to monitor the console specifically to complete the next step.

5. Monitor the console and select the ONIE option from the first GRUB screen shown below.

   

6. Cumulus Linux on x86 uses GRUB chainloading to present a second GRUB menu specific to the ONIE partition. No action is necessary in this menu to select the default option ONIE: Install OS.

   

7. The switch recognizes the USB drive and mounts it automatically. Cumulus Linux installation begins.

8. After installation completes, the switch automatically reboots into the newly installed instance of Cumulus Linux.

ONIE Installation Options

You can run several installer command line options from ONIE to perform basic switch configuration automatically after installation completes and Cumulus Linux boots for the first time. These options enable you to:

• Set a unique password for the cumulus user
• Provide an initial network configuration
• Execute a ZTP script to perform necessary configuration

ONIE:/ # wget http://myserver.datacenter.com/cumulus-linux-4.4.0-mlx-amd64.bin
ONIE:/ # chmod 755 cumulus-linux-4.4.0-mlx-amd64.bin
ONIE:/ # ./cumulus-linux-4.4.0-mlx-amd64.bin --password 'MyP4$$word'

You can run more than one option in the same command.

Set the cumulus User Password

The default cumulus user account password is cumulus. When you log into Cumulus Linux for the first time, you must provide a new password for the cumulus account, then log back into the system.

To automate this process, you can specify a new password from the command line of the installer with the --password '<clear text-password>' option. For example, to change the default cumulus user password to MyP4$$word:

ONIE:/ # ./cumulus-linux-4.4.0-mlx-amd64.bin --password 'MyP4$$word'

To provide a hashed password instead of a clear text password, use the --hashed-password '<hash>' option. An encrypted hash maintains a secure management network.

1. Generate a sha-512 password hash with the following openssl command. The example command generates a sha-512 password hash for the password MyP4$$word.

   user@host:~$ openssl passwd -6 'MyP4$$word'
   6$LXOrvmOkqidBGqu7$dy0dpYYllekNKOY/9LLrobWA4iGwL4zHsgG97qFQWAMZ3ZzMeyz11JcqtgwKDEgYR6RtjfDtdPCeuj8eNzLnS.
   

2. Specify the new password from the command line of the installer with the --hashed-password '<hash>' command:

   ONIE:/ # ./cumulus-linux-4.4.0-mlx-amd64.bin  --hashed-password '6$LXOrvmOkqidBGqu7$dy0dpYYllekNKOY/9LLrobWA4iGwL4zHsgG97qFQWAMZ3ZzMeyz11JcqtgwKDEgYR6RtjfDtdPCeuj8eNzLnS.'
   

Provide Initial Network Configuration

To provide initial network configuration automatically when Cumulus Linux boots for the first time after installation, use the --interfaces-file <filename> option. For example, to copy the contents of a file called network.intf into the /etc/network/interfaces file and run the ifreload -a command:

ONIE:/ # ./cumulus-linux-4.4.0-mlx-amd64.bin  --interfaces-file network.intf

Execute a ZTP Script

To run a ZTP script that contains commands to execute after Cumulus Linux boots for the first time after installation, use the --ztp <filename> option. For example, to run a ZTP script called initial-conf.ztp:

ONIE:/ # ./cumulus-linux-4.4.0-mlx-amd64.bin --ztp initial-conf.ztp

The ZTP script must contain the CUMULUS-AUTOPROVISIONING string near the beginning of the file and must reside on the ONIE filesystem. Refer to Zero Touch Provisioning - ZTP.

If you use the --ztp option together with any of the other command line options, the ZTP script takes precedence and the switch ignores other command line options.

Change the Default BIOS Password

To provide a layer of security and to prevent unauthorized access to the switch, NVIDIA recommends you change the default BIOS password. The default BIOS password is admin.

To change the default BIOS password:

1. During system boot, press Ctrl+B through the serial console while the BIOS version prints.

   

2. From the Security menu, select Administrator Password.

3. Follow the prompts.

Edit the Cumulus Linux Image (Advanced)

The Cumulus Linux disk image file contains a BASH script that includes a set of variables. You can set these variables to be able to install a fully configured system with a single image file.

...
CL_INSTALLER_PASSWORD=''
CL_INSTALLER_HASHED_PASSWORD=''
CL_INSTALLER_LICENSE=''
CL_INSTALLER_INTERFACES_FILENAME=''
CL_INSTALLER_INTERFACES_CONTENT=''
CL_INSTALLER_ZTP_FILENAME=''
CL_INSTALLER_QUIET=""
CL_INSTALLER_FORCEINST=""
CL_INSTALLER_INTERACTIVE=""
CL_INSTALLER_EXTRACTDIR=""
CL_INSTALLER_PAYLOAD_SHA256="72a8c3da28cda7a610e272b67fa1b3a54c50248bf6abf720f73ff3d10e79ae76"

head -20 cumulus-linux-4.4.0-mlx-amd64.bin > cumulus-linux-4.4.0-mlx-amd64.bin.1

sed -e '1,20d' cumulus-linux-4.4.0-mlx-amd64.bin > cumulus-linux-4.4.0-mlx-amd64.bin.2

cat cumulus-linux-4.4.0-mlx-amd64.bin.1 cumulus-linux-4.4.0-mlx-amd64.bin.2 > cumulus-linux-4.4.0-mlx-amd64.bin.final

...
CL_INSTALLER_PAYLOAD_SHA256='d14a028c2a3a2bc9476102bb288234c415a2b01f828ea62ac332e42f'
CL_INSTALLER_PASSWORD='MyP4$$word'
CL_INSTALLER_HASHED_PASSWORD=''
CL_INSTALLER_LICENSE='customer@datacenter.com|4C3YMCACDiK0D/EnrxlXpj71FBBNAg4Yrq+brza4ZtJFCInvalid'
CL_INSTALLER_INTERFACES_FILENAME=''
CL_INSTALLER_INTERFACES_CONTENT='# This file describes the network interfaces available on your system and how to activate them.

source /etc/network/interfaces.d/*.intf

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet dhcp
	vrf mgmt

auto bridge
iface bridge
    bridge-ports swp1 swp2
    bridge-pvid 1
    bridge-vids 10 11
    bridge-vlan-aware yes

auto mgmt
iface mgmt
	address 127.0.0.1/8
	address ::1/128
	vrf-table auto
'
CL_INSTALLER_ZTP_FILENAME=''
...

Secure Boot

Secure Boot validates each binary image loaded during system boot with key signatures that correspond to a stored trusted key in firmware.

Secure Boot settings are in the BIOS Security menu. To access BIOS, press Ctrl+B through the serial console during system boot while the BIOS version prints:

To access the BIOS menu, use admin which is the default BIOS password:

NVIDIA recommends changing the default BIOS password; navigate to Security and select Administrator Password.

To validate or change the Secure Boot mode, navigate to Security and select Secure Boot:

In the Secure Boot menu, you can enable and disable Secure Boot mode. To install an unsigned version of Cumulus Linux or access ONIE without a prompt for a username and password, set Secure Boot to disabled:

To access ONIE when Secure Boot is enabled, authentication is necessary. The default username and password are both root:

​ONIE: Rescue Mode ...
Platform  : x86_64-mlnx_x86-r0
Version   : 2021.02-5.3.0006-rc3-115200
Build Date: 2021-05-20T14:27+03:00
Info: Mounting kernel filesystems... done.

Info: Mounting ONIE-BOOT on /mnt/onie-boot ...
[   17.011057] ext4 filesystem being mounted at /mnt/onie-boot supports timestamps until 2038 (0x7fffffff)
Info: Mounting EFI System on /boot/efi ...
Info: BIOS mode: UEFI
Info: Using eth0 MAC address: b8:ce:f6:3c:62:06
Info: eth0:  Checking link... up.
Info: Trying DHCPv4 on interface: eth0
ONIE: Using DHCPv4 addr: eth0: 10.20.84.226 / 255.255.255.0
Starting: klogd... done.
Starting: dropbear ssh daemon... done.
Starting: telnetd... done.
discover: Rescue mode detected.  Installer disabled.

Please press Enter to activate this console. To check the install status inspect /var/log/onie.log.
Try this:  tail -f /var/log/onie.log

** Rescue Mode Enabled **
login: root
Password: root
ONIE:~ #

To validate the Secure Boot status of a system from Cumulus Linux, run the mokutil --sb-state command.

cumulus@leaf01:mgmt:~$ mokutil --sb-state
SecureBoot enabled

Related Information

• ONIE Design Specification
• NVIDIA Enterprise support portal
• Managing Cumulus Linux Disk Images

This topic describes how to upgrade Cumulus Linux on your switch.

Consider deploying, provisioning, configuring, and upgrading switches using automation, even with small networks or test labs. During the upgrade process, you can upgrade dozens of devices in a repeatable manner. Using tools like Ansible, Chef, or Puppet for configuration management greatly increases the speed and accuracy of the next major upgrade; these tools also enable you to quickly swap failed switch hardware.

Before You Upgrade

Back up Configuration Files

Understanding the location of configuration data is important for successful upgrades, migrations, and backup. As with other Linux distributions, the /etc directory is the primary location for all configuration data in Cumulus Linux. The following list contains the files you need to back up and migrate to a new release. Make sure you examine any changed files. Make the following files and directories part of a backup strategy.

The following commands verify which files have changed compared to the previous Cumulus Linux install. Be sure to back up any changed files.

• Run the sudo dpkg --verify command to show a list of changed files.
• Run the egrep -v '^$|^#|=""$' /etc/default/isc-dhcp-* command to see if any of the generated /etc/default/isc-* files have changed.

Back Up and Restore Configuration with NVUE

To back up and restore the configuration on the switch with NVUE, you can either:

• Back up and restore the NVUE configuration file (available when upgrading from 4.4 and later).
• Back up and restore the NVUE configuration commands (available when upgrading from 5.0 and later).

You can back up and restore the configuration with NVUE only if you used NVUE commands to configure the switch you want to upgrade.

To back up and restore the configuration file:

1. Save the configuration to the /etc/nvue.d/startup.yaml file with the nv config save command:

   cumulus@switch:~$ nv config save
   saved
   

2. Copy the /etc/nvue.d/startup.yaml file off the switch to a different location.

3. After upgrade is complete, restore the configuration. Copy the /etc/nvue.d/startup.yaml file to the switch, then run the nv config apply startup command:

   cumulus@switch:~$ nv config apply startup
   applied
   

To back up and restore the configuration commands:

1. Run the nv config show -o commands > backup.config command to save the commands to the backup.config file:

   cumulus@switch:~$ nv config show -o commands > backup.config
   

2. Copy the backup.config file off the switch to a different location.

3. After upgrade is complete, restore the configuration. Copy the backup.config file to the switch, then run the source backup.config command to run all the commands in the file.

   cumulus@switch:~$ source backup.config
   

   If the backup configuration contains an obfuscated password, you need to reconfigure the password after you run the source backup.config command; otherwise authentication fails.

4. Verify the configuration on the switch, then run the nv config save command to save the configuration to the /etc/nvue.d/startup.yaml file.

For information about the NVUE object model and commands, see NVIDIA User Experience - NVUE.

Create a cl-support File

Before and after you upgrade the switch, run the cl-support script to create a cl-support archive file. The file is a compressed archive of useful information for troubleshooting. If you experience any issues during upgrade, you can send this archive file to the Cumulus Linux support team to investigate.

1. Create the cl-support archive file with the cl-support command:

   cumulus@switch:~$ sudo cl-support
   

2. Copy the cl-support file off the switch to a different location.

3. After upgrade is complete, run the cl-support command again to create a new archive file:

   cumulus@switch:~$ sudo cl-support
   

Upgrade Cumulus Linux

ONIE is an open source project (equivalent to PXE on servers) that enables the installation of network operating systems (NOS) on a bare metal switch.

You can upgrade Cumulus Linux in one of two ways:

• Install a Cumulus Linux image of the new release, using ONIE.
• Upgrade only the changed packages using the sudo -E apt-get update and sudo -E apt-get upgrade command.

Cumulus Linux also provides ISSU to upgrade an active switch with minimal disruption to the network. See ISSU.

Install a Cumulus Linux Image or Upgrade Packages?

The decision to upgrade Cumulus Linux by either installing a Cumulus Linux image or upgrading packages depends on your environment and your preferences. Here are some recommendations for each upgrade method.

Install a Cumulus Linux image if you are performing a rolling upgrade in a production environment and if you are using up-to-date and comprehensive automation scripts. This upgrade method enables you to choose the exact release to which you want to upgrade and is the only method available to upgrade your switch to a new release train (for example, from 4.4.3 to 5.2.0).

Be aware of the following when installing the Cumulus Linux image:

• Installing a Cumulus Linux image is destructive; any configuration files on the switch are not saved; copy them to a different server before you start the Cumulus Linux image install.
• You must move configuration data to the new OS using ZTP or automation while the OS is first booted, or soon afterwards using out-of-band management.
• Moving a configuration file can cause issues.
• Identifying all the locations of configuration data is not always an easy task. See Before You Upgrade Cumulus Linux above.
• Merge conflicts with configuration file changes in the new release sometimes go undetected.
• If configuration files do not restore correctly, you cannot ssh to the switch from in-band management. Use out-of-band connectivity (eth0 or console).
• You must reinstall and reconfigure third-party applications after upgrade.

Run package upgrade if you are upgrading from Cumulus Linux 5.0.0 to a later 5.x release, or if you use third-party applications (package upgrade does not replace or remove third-party applications, unlike the Cumulus Linux image install).

Be aware of the following when upgrading packages:

• You cannot upgrade the switch to a new release train. For example, you cannot upgrade the switch from 4.x to 5.x.
• You can only use package upgrade to upgrade a switch with an image install to a maximum of two releases; for example, you can package upgrade a switch running the Cumulus Linux 5.0 image to 5.1 or 5.2 (5.0 plus two releases).
• The sudo -E apt-get upgrade command might restart or stop services as part of the upgrade process.
• The sudo -E apt-get upgrade command might disrupt core services by changing core service dependency packages.
• After you upgrade, account UIDs and GIDs created by packages might be different on different switches, depending on the configuration and package installation history.
• Cumulus Linux does not support the sudo -E apt-get dist-upgrade command. Be sure to use sudo -E apt-get upgrade when upgrading packages.
• The supported upgrade path is the base image plus two. For example, if the starting image is Cumulus Linux 5.2, the latest release that package upgrade supports is Cumulus Linux 5.4 (5.2 + 2 = 5.4).
• You can check the base image with the grep RELEASE /etc/image-release syntax.
• Occasionally, a release contains a base OS upgrade and does not support package upgrade; release notes indicate when a release does not support package upgrade.

Cumulus Linux Image Install (ONIE)

ONIE is an open source project (equivalent to PXE on servers) that enables the installation of network operating systems (NOS) on a bare metal switch.

To upgrade the switch:

1. Back up the configurations off the switch.

2. Download the Cumulus Linux image.

3. Install the Cumulus Linux image with the onie-install -a -i <image-location> command, which boots the switch into ONIE. The following example command installs the image from a web server, then reboots the switch. There are additional ways to install the Cumulus Linux image, such as using FTP, a local file, or a USB drive. For more information, see Installing a New Cumulus Linux Image.

   cumulus@switch:~$ sudo onie-install -a -i http://10.0.1.251/cumulus-linux-4.1.0-mlx-amd64.bin && sudo reboot
   

4. Restore the configuration files to the new release (restoring files with automation is not recommended).

5. Verify correct operation with the old configurations on the new release.

6. Reinstall third party applications and associated configurations.

Package Upgrade

Cumulus Linux completely embraces the Linux and Debian upgrade workflow, where you use an installer to install a base image, then perform any upgrades within that release train with sudo -E apt-get update and sudo -E apt-get upgrade commands. Any packages that have changed after the base install get upgraded in place from the repository. All switch configuration files remain untouched, or in rare cases merged (using the Debian merge function) during the package upgrade.

When you use package upgrade to upgrade your switch, configuration data stays in place during the upgrade. If the new release updates a previously changed configuration file, the upgrade process prompts you to either specify the version you want to use or evaluate the differences.

To upgrade the switch using package upgrade:

1. Back up the configurations from the switch.

2. Fetch the latest update metadata from the repository.

   cumulus@switch:~$ sudo -E apt-get update
   

3. Review potential upgrade issues (in some cases, upgrading new packages might also upgrade additional existing packages due to dependencies).

   cumulus@switch:~$ sudo -E apt-get upgrade --dry-run
   

4. Upgrade all the packages to the latest distribution.

   cumulus@switch:~$ sudo -E apt-get upgrade
   

   If you do not need to reboot the switch after the upgrade completes, the upgrade ends, restarts all upgraded services, and logs messages in the /var/log/syslog file similar to the ones shown below. In the examples below, the process only upgrades the frr package.

   Policy: Service frr.service action stop postponed
   Policy: Service frr.service action start postponed
   Policy: Restarting services: frr.service
   Policy: Finished restarting services
   Policy: Removed /usr/sbin/policy-rc.d
   Policy: Upgrade is finished
   

   If the upgrade process encounters changed configuration files that have new versions in the release to which you are upgrading, you see a message similar to this:

   Configuration file '/etc/frr/daemons'
   ==> Modified (by you or by a script) since installation.
   ==> Package distributor has shipped an updated version.
   What would you like to do about it ? Your options are:
   Y or I : install the package maintainer's version
   N or O : keep your currently-installed version
   D : show the differences between the versions
   Z : start a shell to examine the situation
   The default action is to keep your current version.
   *** daemons (Y/I/N/O/D/Z) [default=N] ?
   

   • To see the differences between the currently installed version and the new version, type D.
   • To keep the currently installed version, type N. The new package version installs with the suffix .dpkg-dist (for example, /etc/frr/daemons.dpkg-dist). When the upgrade completes and before you reboot, merge your changes with the changes from the newly installed file.
   • To install the new version, type I. Your currently installed version has the suffix .dpkg-old.
   • Cumulus Linux includes /etc/apt/sources.list in the cumulus-archive-keyring package. During upgrade, you must select if you want the new version from the package or the existing file.

   When the upgrade is complete, you can search for the files with the sudo find / -mount -type f -name '*.dpkg-*' command.

   If you see errors for expired GPG keys that prevent you from upgrading packages, follow the steps in Upgrading Expired GPG Keys.

5. Reboot the switch if the upgrade messages indicate that you need to perform a system restart.

   cumulus@switch:~$ sudo -E apt-get upgrade
   ... upgrade messages here ...
   
   *** Caution: Service restart prior to reboot could cause unpredictable behavior
   *** System reboot required ***
   cumulus@switch:~$ sudo reboot
   

6. Verify correct operation with the old configurations on the new version.

Upgrade Notes

Package upgrade always updates to the latest available release in the Cumulus Linux repository. For example, if you are currently running Cumulus Linux 5.0.0 and run the sudo -E apt-get upgrade command on that switch, the packages upgrade to the latest releases in the latest 5.x release.

Because Cumulus Linux is a collection of different Debian Linux packages, be aware of the following:

• The /etc/os-release and /etc/lsb-release files update to the currently installed Cumulus Linux release when you upgrade the switch using either package upgrade or Cumulus Linux image install. For example, if you run sudo -E apt-get upgrade and the latest Cumulus Linux release on the repository is 5.2.0, these two files display the release as 5.2.0 after the upgrade.
• The /etc/image-release file updates only when you run a Cumulus Linux image install. Therefore, if you run a Cumulus Linux image install of Cumulus Linux 5.0.0, followed by a package upgrade to 5.2.0 using sudo -E apt-get upgrade, the /etc/image-release file continues to display Cumulus Linux 5.0.0, which is the originally installed base image.

Upgrade Switches in an MLAG Pair

If you are using MLAG to dual connect two switches in your environment, follow the steps below to upgrade the switches.

Roll Back a Cumulus Linux Installation

Even the most well planned and tested upgrades can result in unforeseen problems and sometimes the best solution is to roll back to the previous state. These main strategies require detailed planning and execution:

• Flatten and rebuild. If the OS becomes unusable, you can use orchestration tools to reinstall the previous OS release from scratch and then rebuild the configuration automatically.
• Restore to a previous state using a backup configuration captured before the upgrade.

The method you employ is specific to your deployment strategy. Providing detailed steps for each scenario is outside the scope of this document.

Third Party Packages

If you install any third party applications on a Cumulus Linux switch, configuration data is typically installed in the /etc directory, but it is not guaranteed. It is your responsibility to understand the behavior and configuration file information of any third party packages installed on the switch.

After you upgrade using a full Cumulus Linux image install, you need to reinstall any third party packages or any Cumulus Linux add-on packages.

Related Information

• Upgrades: Network Device Worldview and Linux Host Worldview Comparison
• Automation Solutions
• ONIE Design Specification
• Multi-Chassis Link Aggregation - MLAG
• Zero Touch Provisioning - ZTP

To manage additional applications in the form of packages and to install the latest updates, use the Advanced Packaging Tool (apt).

Update the Package Cache

To work correctly, apt relies on a local cache listing of the available packages. You must populate the cache initially, then periodically update it with sudo -E apt-get update:

cumulus@switch:~$ sudo -E apt-get update
Ign:1 copy:/var/lib/cumulus/cumulus-local-apt-archive cumulus-local-apt-archive InRelease
Get:2 copy:/var/lib/cumulus/cumulus-local-apt-archive cumulus-local-apt-archive Release [1,115 B]
Ign:3 copy:/var/lib/cumulus/cumulus-local-apt-archive cumulus-local-apt-archive Release.gpg
Get:4 http://security.debian.org buster/updates InRelease [65.4 kB]                 
Hit:5 http://deb.debian.org/debian buster InRelease                                 
Get:6 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
Get:7 http://deb.debian.org/debian buster-backports InRelease [46.7 kB]
Get:8 http://deb.debian.org/debian buster-updates/main Sources.diff/Index [8,608 B] 
Get:9 http://deb.debian.org/debian buster-updates/main amd64 Packages.diff/Index [8,608 B]
Get:10 http://deb.debian.org/debian buster-updates/main Sources 2021-09-28-1420.03.pdiff [185 B]
Get:10 http://deb.debian.org/debian buster-updates/main Sources 2021-09-28-1420.03.pdiff [185 B]
Get:11 http://deb.debian.org/debian buster-updates/main amd64 Packages 2021-09-28-1420.03.pdiff [184 B]               
Get:11 http://deb.debian.org/debian buster-updates/main amd64 Packages 2021-09-28-1420.03.pdiff [184 B]               
Get:12 http://deb.debian.org/debian buster-backports/main Sources.diff/Index [27.8 kB]                     
Get:13 http://deb.debian.org/debian buster-backports/main amd64 Packages.diff/Index [27.8 kB]                         
Hit:14 http://apps3.cumulusnetworks.com/repos/deb CumulusLinux-4 InRelease                                            
Get:15 http://security.debian.org buster/updates/main Sources [200 kB]                             
Get:16 http://security.debian.org buster/updates/main amd64 Packages [305 kB]              
Hit:17 http://apt.cumulusnetworks.com/repo CumulusLinux-4-latest InRelease                       
Get:18 http://deb.debian.org/debian buster-backports/main Sources 2021-10-02-0801.17.pdiff [681 B]
Get:19 http://deb.debian.org/debian buster-backports/main Sources 2021-10-02-1405.24.pdiff [31 B]
Get:19 http://deb.debian.org/debian buster-backports/main Sources 2021-10-02-1405.24.pdiff [31 B]
Get:20 http://deb.debian.org/debian buster-backports/main amd64 Packages 2021-10-02-1405.24.pdiff [178 B]
Get:20 http://deb.debian.org/debian buster-backports/main amd64 Packages 2021-10-02-1405.24.pdiff [178 B]
Fetched 744 kB in 1s (982 kB/s)
Reading package lists... Done

List Available Packages

After the cache populates, use the apt-cache command to search the cache and find the packages of interest or to get information about an available package.

Here are examples of the search and show sub-commands:

cumulus@switch:~$ apt-cache search tcp
collectd-core - statistics collection and monitoring daemon (core system)
fakeroot - tool for simulating superuser privileges
iperf - Internet Protocol bandwidth measuring tool
iptraf-ng - Next Generation Interactive Colorful IP LAN Monitor
libfakeroot - tool for simulating superuser privileges - shared libraries
libfstrm0 - Frame Streams (fstrm) library
libibverbs1 - Library for direct userspace use of RDMA (InfiniBand/iWARP)
libnginx-mod-stream - Stream module for Nginx
libqt4-network - Qt 4 network module
librtr-dev - Small extensible RPKI-RTR-Client C library - development files
librtr0 - Small extensible RPKI-RTR-Client C library
libwiretap8 - network packet capture library -- shared library
libwrap0 - Wietse Venema's TCP wrappers library
libwrap0-dev - Wietse Venema's TCP wrappers library, development files
netbase - Basic TCP/IP networking system
nmap-common - Architecture independent files for nmap
nuttcp - network performance measurement tool
openssh-client - secure shell (SSH) client, for secure access to remote machines
openssh-server - secure shell (SSH) server, for secure access from remote machines
openssh-sftp-server - secure shell (SSH) sftp server module, for SFTP access from remote machines
python-dpkt - Python 2 packet creation / parsing module for basic TCP/IP protocols
rsyslog - reliable system and kernel logging daemon
socat - multipurpose relay for bidirectional data transfer
tcpdump - command-line network traffic analyzer

cumulus@switch:~$ apt-cache show tcpdump
Package: tcpdump
Version: 4.9.3-1~deb10u1
Installed-Size: 1109
Maintainer: Romain Francoise <rfrancoise@debian.org>
Architecture: amd64
Replaces: apparmor-profiles-extra (<< 1.12~)
Depends: libc6 (>= 2.14), libpcap0.8 (>= 1.5.1), libssl1.1 (>= 1.1.0)
Suggests: apparmor (>= 2.3)
Breaks: apparmor-profiles-extra (<< 1.12~)
Size: 400060
SHA256: 3a63be16f96004bdf8848056f2621fbd863fadc0baf44bdcbc5d75dd98331fd3
SHA1: 2ab9f0d2673f49da466f5164ecec8836350aed42
MD5sum: 603baaf914de63f62a9f8055709257f3
Description: command-line network traffic analyzer
 This program allows you to dump the traffic on a network. tcpdump
 is able to examine IPv4, ICMPv4, IPv6, ICMPv6, UDP, TCP, SNMP, AFS
 BGP, RIP, PIM, DVMRP, IGMP, SMB, OSPF, NFS and many other packet
 types.
 .
 It can be used to print out the headers of packets on a network
 interface, filter packets that match a certain expression. You can
 use this tool to track down network problems, to detect attacks
 or to monitor network activities.
Description-md5: f01841bfda357d116d7ff7b7a47e8782
Homepage: http://www.tcpdump.org/
Multi-Arch: foreign
Section: net
Priority: optional
Filename: pool/upstream/t/tcpdump/tcpdump_4.9.3-1~deb10u1_amd64.deb

List Packages Installed on the System

The apt-cache command shows information about all the packages available in the repository. To see which packages are actually installed on your system with the version, run the following command.

cumulus@switch:~$ nv show platform software installed
                                       description                                                                                                                   package                                version
-------------------------------------  ----------------------------------------------------------------------------------------------------------------------------  -------------------------------------  ----------------------------------------------
acpi                                   displays information on ACPI devices                                                                                          acpi                                   1.7-1.1
acpi-support-base                      scripts for handling base ACPI events such as the power button                                                                acpi-support-base                      0.142-8
acpid                                  Advanced Configuration and Power Interface event daemon                                                                       acpid                                  1:2.0.31-1
...

cumulus@switch:~$ dpkg -l
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                Version                   Architecture Description
+++-===================-=========================-============-=================================
ii  acpi                1.7-1.1                   amd64        displays information on ACPI devices
ii  acpi-support-base   0.142-8                   all          scripts for handling base ACPI events such as th
ii  acpid               1:2.0.31-1                amd64        Advanced Configuration and Power Interface event
ii  adduser             3.118                     all          add and remove users and groups
ii  apt                 1.8.2                     amd64        commandline package manager
ii  arping              2.19-6                    amd64        sends IP and/or ARP pings (to the MAC address)
ii  arptables           0.0.4+snapshot20181021-4  amd64        ARP table administration
...

Show the Version of a Package

To show the version of a specific package installed on the system:

cumulus@switch:~$ nv show platform software installed vrf
             running              applied  pending  description
-----------  -------------------  -------  -------  -----------
description  Linux tools for VRF                    Description
package      vrf                                    Package
version      1.0-cl4.4.0u0                         Version

cumulus@switch:~$ dpkg -l vrf
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name       Version      Architecture Description
+++-==========-============-============-=================================
ii  vrf        1.0-cl4.4.0u0    amd64        Linux tools for VRF

Upgrade Packages

To upgrade all the packages installed on the system to their latest versions, run the following commands:

cumulus@switch:~$ sudo -E apt-get update
cumulus@switch:~$ sudo -E apt-get upgrade

The system lists the packages for upgrade and prompts you to continue.

The above commands upgrade all installed versions with their latest versions but do not install any new packages.

Add New Packages

To add a new package, first ensure the package is not already on the system:

cumulus@switch:~$ dpkg -l | grep <name of package>

• If the package is already on the system, you can update the package from the Cumulus Linux repository as part of the package upgrade process, which upgrades all packages on the system. See Upgrade Packages above.
• If the package is not already on the system, add it by running sudo -E apt-get install <name of package>. This retrieves the package from the Cumulus Linux repository and installs it on your system together with any other dependent packages. The following example adds the tcpreplay package to the system:

cumulus@switch:~$ sudo -E apt-get update
cumulus@switch:~$ sudo -E apt-get install tcpreplay
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
tcpreplay
0 upgraded, 1 newly installed, 0 to remove and 1 not upgraded.
Need to get 436 kB of archives.
After this operation, 1008 kB of additional disk space will be used
...

You can install several packages at the same time:

cumulus@switch:~$ sudo -E apt-get install <package1> <package2> <package3>

Add Packages From Another Repository

As shipped, Cumulus Linux searches the Cumulus Linux repository for available packages. You can add additional repositories to search by adding them to the list of sources that apt-get consults. See man sources.list for more information.

NVIDIA adds features or makes bug fixes to certain packages; do not replace these packages with versions from other repositories.

If you want to install packages that are not in the Cumulus Linux repository, the procedure is the same as above, but with one additional step.

Installing packages outside of the Cumulus Linux repository requires the use of sudo -E apt-get; however, depending on the package, you can use easy-install and other commands.

To install a new package, complete the following steps:

1. Run the dpkg command to ensure that the package is not already installed on the system:

   cumulus@switch:~$ dpkg -l | grep <name of package>
   

2. If the package is already on the system, ensure it is the version you need. If it is an older version, update the package from the Cumulus Linux repository:

   cumulus@switch:~$ sudo -E apt-get update
   cumulus@switch:~$ sudo -E apt-get install <name of package>
   cumulus@switch:~$ sudo -E apt-get upgrade
   

3. If the package is not on the system, the package source location is not in the /etc/apt/sources.list file. Edit and add the appropriate source to the file. For example, add the following if you want a package from the Debian repository that is not in the Cumulus Linux repository:

   deb http://http.us.debian.org/debian buster main
   deb http://security.debian.org/ buster/updates main
   

   Otherwise, /etc/apt/sources.list lists the repository but comments it out. To uncomment the repository, remove the # at the start of the line, then save the file.

4. Run sudo -E apt-get update, then install the package and upgrade:

   cumulus@switch:~$ sudo -E apt-get update
   cumulus@switch:~$ sudo -E apt-get install <name of package>
   cumulus@switch:~$ sudo -E apt-get upgrade
   

Add Packages from the Cumulus Linux Local Archive

Cumulus Linux contains a local archive embedded in the Cumulus Linux image. This archive, cumulus-local-apt-archive, contains the packages you need to install ifplugd, LDAP, RADIUS or TACACS+ without a network connection.

The archive contains the following packages:

• audisp-tacplus
• ifplugd
• libdaemon0
• libnss-ldapd
• libnss-mapuser
• libnss-tacplus
• libpam-ldapd
• libpam-radius-auth
• libpam-tacplus
• libtac2
• libtacplus-map1
• nslcd

Add these packages with apt-get update && apt-get install, as described above.

Related Information

• Debian GNU/Linux FAQ, Ch 8 Package management tools
• man pages for apt-get, dpkg, sources.list, apt_preferences

Use ZTP to deploy network devices in large-scale environments. On first boot, Cumulus Linux runs ZTP, which executes the provisioning automation that deploys the device for its intended role in the network.

The provisioning framework allows you to execute a one-time, user-provided script. You can develop this script using a variety of automation tools and scripting languages. You can also use it to add the switch to a configuration management (CM) platform such as Puppet, Chef, CFEngine or a custom, proprietary tool.

While developing and testing the provisioning logic, you can use the ztp command in Cumulus Linux to run your provisioning script manually on a device.

ZTP in Cumulus Linux can run automatically in one of the following ways, in this order:

1. Through a local file
2. Using a USB drive inserted into the switch (ZTP-USB)
3. Through DHCP

Use a Local File

ZTP only looks one time for a ZTP script on the local file system when the switch boots. ZTP searches for an install script that matches an ONIE-style waterfall in /var/lib/cumulus/ztp, looking for the most specific name first, and ending at the most generic:

• 'cumulus-ztp-' + architecture + '-' + vendor + '_' + model + '-r' + revision
• 'cumulus-ztp-' + architecture + '-' + vendor + '_' + model
• 'cumulus-ztp-' + vendor + '_' + model
• 'cumulus-ztp-' + architecture
• 'cumulus-ztp'

You can also trigger the ZTP process manually by running the ztp --run <URL> command, where the URL is the path to the ZTP script.

Use a USB Drive

At minimum, the script must:

• Install the Cumulus Linux operating system.
• Copy over a basic configuration to the switch.
• Restart the switch or the relevant services to get switchd up and running with that configuration.

Follow these steps to perform ZTP using a USB drive:

1. Copy the installation image to the USB drive.
2. The ztp process searches the root filesystem of the newly mounted drive for filenames matching an ONIE-style waterfall (see the patterns and examples above), looking for the most specific name first, and ending at the most generic.
3. ZTP parses the contents of the script to ensure it contains the CUMULUS-AUTOPROVISIONING flag (see example scripts).

ZTP Over DHCP

If the ztp process does not discover a local ONIE script or applicable USB drive, it checks DHCP every ten seconds for up to five minutes for the presence of a ZTP URL specified in /var/run/ztp.dhcp. The URL can be any of HTTP, HTTPS, FTP, or TFTP.

For ZTP using DHCP, provisioning initially takes place over the management network and initiates through a DHCP hook. A DHCP option specifies a configuration script. The ZTP process requests this script from the Web server and the script executes locally.

The ZTP process over DHCP follows these steps:

1. The first time you boot Cumulus Linux, eth0 makes a DHCP request. By default, Cumulus Linux sends DHCP option 60 (the vendor class identifier) with the value cumulus-linux x86_64 to identify itself to the DHCP server.
2. The DHCP server offers a lease to the switch.
3. If option 239 is in the response, the ZTP process starts.
4. The ZTP process requests the contents of the script from the URL, sending additional HTTP headers containing details about the switch.
5. ZTP parses the contents of the script to ensure it contains the CUMULUS-AUTOPROVISIONING flag (see example scripts).
6. If provisioning is necessary, the script executes locally on the switch with root privileges.
7. ZTP examines the return code of the script. If the return code is 0, ZTP marks the provisioning state as complete in the autoprovisioning configuration file.

Trigger ZTP Over DHCP

If you have not yet provisioned the switch, you can trigger the ZTP process over DHCP when eth0 uses DHCP and one of the following events occur:

• The switch boots.
• You plug a cable into or unplug a cable from the eth0 port.
• You disconnect, then reconnect the switch power cord.

You can also run the ztp --run <URL> command, where the URL is the path to the ZTP script.

Configure the DHCP Server

During the DHCP process over eth0, Cumulus Linux requests DHCP option 239. This option specifies the custom provisioning script.

For example, the /etc/dhcp/dhcpd.conf file for an ISC DHCP server looks like:

option cumulus-provision-url code 239 = text;

  subnet 192.0.2.0 netmask 255.255.255.0 {
  range 192.0.2.100 192.168.0.200;
  option cumulus-provision-url "http://192.0.2.1/demo.sh";
}

In addition, you can specify the hostname of the switch with the host-name option:

subnet 192.168.0.0 netmask 255.255.255.0 {
  range 192.168.0.100 192.168.0.200;
  option cumulus-provision-url "http://192.0.2.1/demo.sh";
  host dc1-tor-sw1 { hardware ethernet 44:38:39:00:1a:6b; fixed-address 192.168.0.101; option host-name "dc1-tor-sw1"; }
}

DHCP on Front Panel Ports

ZTP runs DHCP on all the front panel switch ports and on any active interface. ZTP assesses the list of active ports on every retry cycle. When it receives the DHCP lease and option 239 is present in the response, ZTP starts to execute the script.

Inspect HTTP Headers

The following HTTP headers in the request to the web server retrieve the provisioning script:

Header                        Value                 Example
------                        -----                 -------
User-Agent                                          CumulusLinux-AutoProvision/0.4
CUMULUS-ARCH                  CPU architecture      x86_64
CUMULUS-BUILD                                       5.1.0
CUMULUS-MANUFACTURER                                odm
CUMULUS-PRODUCTNAME                                 switch_model
CUMULUS-SERIAL                                      XYZ123004
CUMULUS-BASE-MAC                                    44:38:39:FF:40:94
CUMULUS-MGMT-MAC                                    44:38:39:FF:00:00
CUMULUS-VERSION                                     5.1.0
CUMULUS-PROV-COUNT                                  0
CUMULUS-PROV-MAX                                    32

Write ZTP Scripts

# CUMULUS-AUTOPROVISIONING

The script must contain the CUMULUS-AUTOPROVISIONING flag. You can include this flag in a comment or remark; you do not need to echo or write the flag to stdout.

You can write the script in any language that Cumulus Linux supports, such as:

• Perl
• Python
• Ruby
• Shell

The script must return an exit code of 0 upon success to mark the process as complete in the autoprovisioning configuration file.

The following script installs Cumulus Linux from a USB drive and applies a configuration:

#!/bin/bash
function error() {
  echo -e "\e[0;33mERROR: The ZTP script failed while running the command $BASH_COMMAND at line $BASH_LINENO.\e[0m" >&2
  exit 1
}

# Log all output from this script
exec >> /var/log/autoprovision 2>&1
date "+%FT%T ztp starting script $0"

trap error ERR

#Add Debian Repositories
echo "deb http://http.us.debian.org/debian buster main" >> /etc/apt/sources.list
echo "deb http://security.debian.org/ buster/updates main" >> /etc/apt/sources.list

#Update Package Cache
apt-get update -y

#Load interface config from usb
cp ${ZTP_USB_MOUNTPOINT}/interfaces /etc/network/interfaces

#Load port config from usb
#   (if breakout cables are used for certain interfaces)
cp ${ZTP_USB_MOUNTPOINT}/ports.conf /etc/cumulus/ports.conf

#Reload interfaces to apply loaded config
ifreload -a

# CUMULUS-AUTOPROVISIONING
exit 0

Continue Provisioning

Typically ZTP exits after executing the script locally and does not continue. To continue with provisioning so that you do not have to intervene manually or embed an Ansible callback into the script, you can add the CUMULUS-AUTOPROVISION-CASCADE directive.

Best Practices

ZTP scripts come in different forms and frequently perform the same tasks. As BASH is the most common language for ZTP scripts, use the following BASH snippets to perform common tasks with robust error checking.

Set the Default Cumulus User Password

The default cumulus user account password is cumulus. When you log into Cumulus Linux for the first time, you must provide a new password for the cumulus account, then log back into the system.

Add the following function to your ZTP script to change the default cumulus user account password to a clear-text password. The example changes the password cumulus to MyP4$$word.

function set_password(){
     # Unexpire the cumulus account
     passwd -x 99999 cumulus
     # Set the password
     echo 'cumulus:MyP4$$word' | chpasswd
}
set_password

If you have an insecure management network, set the password with an encrypted hash instead of a clear-text password.

• First, generate a sha-512 password hash with the following python commands. The example commands generate a sha-512 password hash for the password MyP4$$word.

  user@host:~$ python3 -c "import crypt; print(crypt.crypt('MyP4$$word',salt=crypt.mksalt()))"
  $6$hs7OPmnrfvLNKfoZ$iB3hy5N6Vv6koqDmxixpTO6lej6VaoKGvs5E8p5zNo4tPec0KKqyQnrFMII3jGxVEYWntG9e7Z7DORdylG5aR/
  

• Then, add the following function to the ZTP script to change the default cumulus user account password:

  function set_password(){
       # Unexpire the cumulus account
       passwd -x 99999 cumulus
       # Set the password
       usermod -p '$6$hs7OPmnrfvLNKfoZ$iB3hy5N6Vv6koqDmxixpTO6lej6VaoKGvs5E8p5zNo4tPec0KKqyQnrFMII3jGxVEYWntG9e7Z7DORdylG5aR/' cumulus
  }
  set_password
  

Test DNS Name Resolution

DNS names are frequently used in ZTP scripts. The ping_until_reachable function tests that each DNS name resolves into a reachable IP address. Call this function with each DNS target used in your script before you use the DNS name elsewhere in your script.

The following example shows how to call the ping_until_reachable function in the context of a larger task.

function ping_until_reachable(){
    last_code=1
    max_tries=30
    tries=0
    while [ "0" != "$last_code" ] && [ "$tries" -lt "$max_tries" ]; do
        tries=$((tries+1))
        echo "$(date) INFO: ( Attempt $tries of $max_tries ) Pinging $1 Target Until Reachable."
        ping $1 -c2 &> /dev/null
        last_code=$?
            sleep 1
    done
    if [ "$tries" -eq "$max_tries" ] && [ "$last_code" -ne "0" ]; then
        echo "$(date) ERROR: Reached maximum number of attempts to ping the target $1 ."
        exit 1
    fi
}

Check the Cumulus Linux Release

The following script segment demonstrates how to check which Cumulus Linux release is running and upgrades the node if the release is not the target release. If the release is the target release, normal ZTP tasks execute. This script calls the ping_until_reachable script (described above) to make sure the server holding the image server and the ZTP script is reachable.

function init_ztp(){
    #do normal ZTP tasks
}

CUMULUS_TARGET_RELEASE=5.1.0
CUMULUS_CURRENT_RELEASE=$(cat /etc/lsb-release  | grep RELEASE | cut -d "=" -f2)
IMAGE_SERVER_HOSTNAME=webserver.example.com
IMAGE_SERVER= "http:// "$IMAGE_SERVER_HOSTNAME "/ "$CUMULUS_TARGET_RELEASE ".bin "
ZTP_URL= "http:// "$IMAGE_SERVER_HOSTNAME "/ztp.sh "

if [ "$CUMULUS_TARGET_RELEASE" != "$CUMULUS_CURRENT_RELEASE" ]; then
    ping_until_reachable $IMAGE_SERVER_HOSTNAME
    /usr/cumulus/bin/onie-install -fa -i $IMAGE_SERVER -z $ZTP_URL && reboot
else
    init_ztp && reboot
fi
exit 0

Apply Management VRF Configuration

If you apply a management VRF in your script, either apply it last or reboot instead. If you do not apply a management VRF last, you need to prepend any commands that require eth0 to communicate out with /usr/bin/ip vrf exec mgmt; for example, /usr/bin/ip vrf exec mgmt apt-get update -y.

Perform Ansible Provisioning Callbacks

After initially configuring a node with ZTP, use Provisioning Callbacks to inform Ansible Tower or AWX that the node is ready for more detailed provisioning. The following example demonstrates how to use a provisioning callback:

/usr/bin/curl -H "Content-Type:application/json" -k -X POST --data '{"host_config_key":"'somekey'"}' -u username:password http://ansible.example.com/api/v2/job_templates/1111/callback/

Disable the DHCP Hostname Override Setting

Make sure to disable the DHCP hostname override setting in your script.

function set_hostname(){
    # Remove DHCP Setting of Hostname
    sed s/'SETHOSTNAME="yes"'/'SETHOSTNAME="no"'/g -i /etc/dhcp/dhclient-exit-hooks.d/dhcp-sethostname
    hostnamectl set-hostname $1
}

Test ZTP Scripts

Use these commands to test and debug your ZTP scripts.

You can use verbose mode to debug your script and see where your script fails. Include the -v option when you run ZTP:

cumulus@switch:~$ sudo ztp -v -r http://192.0.2.1/demo.sh
Attempting to provision via ZTP Manual from http://192.0.2.1/demo.sh

Broadcast message from root@dell-s6010-01 (ttyS0) (Tue May 10 22:44:17 2016):  

ZTP: Attempting to provision via ZTP Manual from http://192.0.2.1/demo.sh
ZTP Manual: URL response code 200
ZTP Manual: Found Marker CUMULUS-AUTOPROVISIONING
ZTP Manual: Executing http://192.0.2.1/demo.sh
error: ZTP Manual: Payload returned code 1
error: Script returned failure

To see results of the most recent ZTP execution, you can run the ztp -s command.

cumulus@switch:~$ ztp -s
ZTP INFO:

State              enabled
Version            1.0
Result             Script Failure
Date               Mon 20 May 2019 09:31:27 PM UTC
Method             ZTP DHCP
URL                http://192.0.2.1/demo.sh

If ZTP runs when the switch boots and not manually, you can run the systemctl -l status ztp.service then journalctl -l -u ztp.service to see if any failures occur:

cumulus@switch:~$ sudo systemctl -l status ztp.service
● ztp.service - Cumulus Linux ZTP
    Loaded: loaded (/lib/systemd/system/ztp.service; enabled)
    Active: failed (Result: exit-code) since Wed 2016-05-11 16:38:45 UTC; 1min 47s ago
        Docs: man:ztp(8)
    Process: 400 ExecStart=/usr/sbin/ztp -b (code=exited, status=1/FAILURE)
    Main PID: 400 (code=exited, status=1/FAILURE)

May 11 16:37:45 cumulus ztp[400]: ztp [400]: ZTP USB: Device not found
May 11 16:38:45 dell-s6010-01 ztp[400]: ztp [400]: ZTP DHCP: Looking for ZTP Script provided by DHCP
May 11 16:38:45 dell-s6010-01 ztp[400]: ztp [400]: Attempting to provision via ZTP DHCP from http://192.0.2.1/demo.sh
May 11 16:38:45 dell-s6010-01 ztp[400]: ztp [400]: ZTP DHCP: URL response code 200
May 11 16:38:45 dell-s6010-01 ztp[400]: ztp [400]: ZTP DHCP: Found Marker CUMULUS-AUTOPROVISIONING
May 11 16:38:45 dell-s6010-01 ztp[400]: ztp [400]: ZTP DHCP: Executing http://192.0.2.1/demo.sh
May 11 16:38:45 dell-s6010-01 ztp[400]: ztp [400]: ZTP DHCP: Payload returned code 1
May 11 16:38:45 dell-s6010-01 ztp[400]: ztp [400]: Script returned failure
May 11 16:38:45 dell-s6010-01 systemd[1]: ztp.service: main process exited, code=exited, status=1/FAILURE
May 11 16:38:45 dell-s6010-01 systemd[1]: Unit ztp.service entered failed state.
cumulus@switch:~$
cumulus@switch:~$ sudo journalctl -l -u ztp.service --no-pager
-- Logs begin at Wed 2016-05-11 16:37:42 UTC, end at Wed 2016-05-11 16:40:39 UTC. --
May 11 16:37:45 cumulus ztp[400]: ztp [400]: /var/lib/cumulus/ztp: Sate Directory does not exist. Creating it...
May 11 16:37:45 cumulus ztp[400]: ztp [400]: /var/run/ztp.lock: Lock File does not exist. Creating it...
May 11 16:37:45 cumulus ztp[400]: ztp [400]: /var/lib/cumulus/ztp/ztp_state.log: State File does not exist. Creating it...
May 11 16:37:45 cumulus ztp[400]: ztp [400]: ZTP LOCAL: Looking for ZTP local Script
May 11 16:37:45 cumulus ztp[400]: ztp [400]: ZTP LOCAL: Waterfall search for /var/lib/cumulus/ztp/cumulus-ztp-x86_64-dell_s6010_s1220-rUNKNOWN
May 11 16:37:45 cumulus ztp[400]: ztp [400]: ZTP LOCAL: Waterfall search for /var/lib/cumulus/ztp/cumulus-ztp-x86_64-dell_s6010_s1220
May 11 16:37:45 cumulus ztp[400]: ztp [400]: ZTP LOCAL: Waterfall search for /var/lib/cumulus/ztp/cumulus-ztp-x86_64-dell
May 11 16:37:45 cumulus ztp[400]: ztp [400]: ZTP LOCAL: Waterfall search for /var/lib/cumulus/ztp/cumulus-ztp-x86_64
May 11 16:37:45 cumulus ztp[400]: ztp [400]: ZTP LOCAL: Waterfall search for /var/lib/cumulus/ztp/cumulus-ztp
May 11 16:37:45 cumulus ztp[400]: ztp [400]: ZTP USB: Looking for unmounted USB devices
May 11 16:37:45 cumulus ztp[400]: ztp [400]: ZTP USB: Parsing partitions
May 11 16:37:45 cumulus ztp[400]: ztp [400]: ZTP USB: Device not found
May 11 16:38:45 dell-s6010-01 ztp[400]: ztp [400]: ZTP DHCP: Looking for ZTP Script provided by DHCP
May 11 16:38:45 dell-s6010-01 ztp[400]: ztp [400]: Attempting to provision via ZTP DHCP from http://192.0.2.1/demo.sh
May 11 16:38:45 dell-s6010-01 ztp[400]: ztp [400]: ZTP DHCP: URL response code 200
May 11 16:38:45 dell-s6010-01 ztp[400]: ztp [400]: ZTP DHCP: Found Marker CUMULUS-AUTOPROVISIONING
May 11 16:38:45 dell-s6010-01 ztp[400]: ztp [400]: ZTP DHCP: Executing http://192.0.2.1/demo.sh
May 11 16:38:45 dell-s6010-01 ztp[400]: ztp [400]: ZTP DHCP: Payload returned code 1
May 11 16:38:45 dell-s6010-01 ztp[400]: ztp [400]: Script returned failure
May 11 16:38:45 dell-s6010-01 systemd[1]: ztp.service: main process exited, code=exited, status=1/FAILURE
May 11 16:38:45 dell-s6010-01 systemd[1]: Unit ztp.service entered failed state.

Instead of running journalctl, you can see the log history by running:

cumulus@switch:~$ cat /var/log/syslog | grep ztp
2016-05-11T16:37:45.132583+00:00 cumulus ztp [400]: /var/lib/cumulus/ztp: State Directory does not exist. Creating it...
2016-05-11T16:37:45.134081+00:00 cumulus ztp [400]: /var/run/ztp.lock: Lock File does not exist. Creating it...
2016-05-11T16:37:45.135360+00:00 cumulus ztp [400]: /var/lib/cumulus/ztp/ztp_state.log: State File does not exist. Creating it...
2016-05-11T16:37:45.185598+00:00 cumulus ztp [400]: ZTP LOCAL: Looking for ZTP local Script
2016-05-11T16:37:45.485084+00:00 cumulus ztp [400]: ZTP LOCAL: Waterfall search for /var/lib/cumulus/ztp/cumulus-ztp-x86_64-dell_s6010_s1220-rUNKNOWN
2016-05-11T16:37:45.486394+00:00 cumulus ztp [400]: ZTP LOCAL: Waterfall search for /var/lib/cumulus/ztp/cumulus-ztp-x86_64-dell_s6010_s1220
2016-05-11T16:37:45.488385+00:00 cumulus ztp [400]: ZTP LOCAL: Waterfall search for /var/lib/cumulus/ztp/cumulus-ztp-x86_64-dell
2016-05-11T16:37:45.489665+00:00 cumulus ztp [400]: ZTP LOCAL: Waterfall search for /var/lib/cumulus/ztp/cumulus-ztp-x86_64
2016-05-11T16:37:45.490854+00:00 cumulus ztp [400]: ZTP LOCAL: Waterfall search for /var/lib/cumulus/ztp/cumulus-ztp
2016-05-11T16:37:45.492296+00:00 cumulus ztp [400]: ZTP USB: Looking for unmounted USB devices
2016-05-11T16:37:45.493525+00:00 cumulus ztp [400]: ZTP USB: Parsing partitions
2016-05-11T16:37:45.636422+00:00 cumulus ztp [400]: ZTP USB: Device not found
2016-05-11T16:38:43.372857+00:00 cumulus ztp [1805]: Found ZTP DHCP Request
2016-05-11T16:38:45.696562+00:00 cumulus ztp [400]: ZTP DHCP: Looking for ZTP Script provided by DHCP
2016-05-11T16:38:45.698598+00:00 cumulus ztp [400]: Attempting to provision via ZTP DHCP from http://192.0.2.1/demo.sh
2016-05-11T16:38:45.816275+00:00 cumulus ztp [400]: ZTP DHCP: URL response code 200
2016-05-11T16:38:45.817446+00:00 cumulus ztp [400]: ZTP DHCP: Found Marker CUMULUS-AUTOPROVISIONING
2016-05-11T16:38:45.818402+00:00 cumulus ztp [400]: ZTP DHCP: Executing http://192.0.2.1/demo.sh
2016-05-11T16:38:45.834240+00:00 cumulus ztp [400]: ZTP DHCP: Payload returned code 1
2016-05-11T16:38:45.835488+00:00 cumulus ztp [400]: Script returned failure
2016-05-11T16:38:45.876334+00:00 cumulus systemd[1]: ztp.service: main process exited, code=exited, status=1/FAILURE
2016-05-11T16:38:45.879410+00:00 cumulus systemd[1]: Unit ztp.service entered failed state.

If you see that the issue is a script failure, you can modify the script and then run ZTP manually using ztp -v -r <URL/path to that script>, as above.

cumulus@switch:~$ sudo ztp -v -r http://192.0.2.1/demo.sh
Attempting to provision via ZTP Manual from http://192.0.2.1/demo.sh

Broadcast message from root@dell-s6010-01 (ttyS0) (Tue May 10 22:44:17 2019):  

ZTP: Attempting to provision via ZTP Manual from http://192.0.2.1/demo.sh
ZTP Manual: URL response code 200
ZTP Manual: Found Marker CUMULUS-AUTOPROVISIONING
ZTP Manual: Executing http://192.0.2.1/demo.sh
error: ZTP Manual: Payload returned code 1
error: Script returned failure
cumulus@switch:~$ sudo ztp -s
State         enabled
Version       1.0
Result        Script Failure
Date          Mon 20 May 2019 09:31:27 PM UTC
Method        ZTP Manual
URL           http://192.0.2.1/demo.sh

Use the following command to check syslog for information about ZTP:

cumulus@switch:~$ sudo grep -i ztp /var/log/syslog

Common ZTP Script Errors

Could not find referenced script/interpreter in downloaded payload

cumulus@leaf01:~$ sudo cat /var/log/syslog | grep ztp
2018-04-24T15:06:08.887041+00:00 leaf01 ztp [13404]: Attempting to provision via ZTP Manual from http://192.168.0.254/ztp_oob_windows.sh
2018-04-24T15:06:09.106633+00:00 leaf01 ztp [13404]: ZTP Manual: URL response code 200
2018-04-24T15:06:09.107327+00:00 leaf01 ztp [13404]: ZTP Manual: Found Marker CUMULUS-AUTOPROVISIONING
2018-04-24T15:06:09.107635+00:00 leaf01 ztp [13404]: ZTP Manual: Executing http://192.168.0.254/ztp_oob_windows.sh
2018-04-24T15:06:09.132651+00:00 leaf01 ztp [13404]: ZTP Manual: Could not find referenced script/interpreter in downloaded payload.
2018-04-24T15:06:14.135521+00:00 leaf01 ztp [13404]: ZTP Manual: Retrying
2018-04-24T15:06:14.138915+00:00 leaf01 ztp [13404]: ZTP Manual: URL response code 200
2018-04-24T15:06:14.139162+00:00 leaf01 ztp [13404]: ZTP Manual: Found Marker CUMULUS-AUTOPROVISIONING
2018-04-24T15:06:14.139448+00:00 leaf01 ztp [13404]: ZTP Manual: Executing http://192.168.0.254/ztp_oob_windows.sh
2018-04-24T15:06:14.143261+00:00 leaf01 ztp [13404]: ZTP Manual: Could not find referenced script/interpreter in downloaded payload.
2018-04-24T15:06:24.147580+00:00 leaf01 ztp [13404]: ZTP Manual: Retrying
2018-04-24T15:06:24.150945+00:00 leaf01 ztp [13404]: ZTP Manual: URL response code 200
2018-04-24T15:06:24.151177+00:00 leaf01 ztp [13404]: ZTP Manual: Found Marker CUMULUS-AUTOPROVISIONING
2018-04-24T15:06:24.151374+00:00 leaf01 ztp [13404]: ZTP Manual: Executing http://192.168.0.254/ztp_oob_windows.sh
2018-04-24T15:06:24.155026+00:00 leaf01 ztp [13404]: ZTP Manual: Could not find referenced script/interpreter in downloaded payload.
2018-04-24T15:06:39.164957+00:00 leaf01 ztp [13404]: ZTP Manual: Retrying
2018-04-24T15:06:39.165425+00:00 leaf01 ztp [13404]: Script returned failure
2018-04-24T15:06:39.175959+00:00 leaf01 ztp [13404]: ZTP script failed. Exiting...

Errors in syslog for ZTP like those shown above often occur if you create or edit the script on a Windows machine. Check to make sure that the \r\n characters are not present in the end-of-line encodings.

Use the cat -v ztp.sh command to view the contents of the script and search for any hidden characters.

root@oob-mgmt-server:/var/www/html# cat -v ./ztp_oob_windows.sh 
#!/bin/bash^M
^M
###################^M
#   ZTP Script^M
###################^M
^M
/usr/cumulus/bin/cl-license -i http://192.168.0.254/license.txt^M
^M
# Clean method of performing a Reboot^M
nohup bash -c 'sleep 2; shutdown now -r "Rebooting to Complete ZTP"' &^M
^M
exit 0^M
^M
# The line below is required to be a valid ZTP script^M
#CUMULUS-AUTOPROVISIONING^M
root@oob-mgmt-server:/var/www/html#

The ^M characters in the output of your ZTP script, as shown above, indicate the presence of Windows end-of-line encodings that you need to remove.

Use the translate (tr) command on any Linux system to remove the '\r' characters from the file.

root@oob-mgmt-server:/var/www/html# tr -d '\r' < ztp_oob_windows.sh > ztp_oob_unix.sh
root@oob-mgmt-server:/var/www/html# cat -v ./ztp_oob_unix.sh 
#!/bin/bash
###################
#   ZTP Script
###################
/usr/cumulus/bin/cl-license -i http://192.168.0.254/license.txt
# Clean method of performing a Reboot
nohup bash -c 'sleep 2; shutdown now -r "Rebooting to Complete ZTP"' &
exit 0
# The line below is required to be a valid ZTP script
#CUMULUS-AUTOPROVISIONING
root@oob-mgmt-server:/var/www/html#

Manually Use the ztp Command

To enable ZTP, use the -e option:

cumulus@switch:~$ sudo ztp -e

To reset ZTP to its original state, use the -R option. This removes the ztp directory and ZTP runs the next time the switch reboots.

cumulus@switch:~$ sudo ztp -R

To disable ZTP, use the -d option:

cumulus@switch:~$ sudo ztp -d

To force provisioning to occur and ignore the status listed in the configuration file, use the -r option:

cumulus@switch:~$ sudo ztp -r cumulus-ztp.sh

To see the current ZTP state, use the -s option:

cumulus@switch:~$ sudo ztp -s
ZTP INFO:
State          disabled
Version        1.0
Result         success
Date           Mon May 20 21:51:04 2019 UTC
Method         Switch manually configured  
URL            None

Considerations

• While you are writing a provisioning script, you sometimes need to reboot the switch.
• You can use the Cumulus Linux onie-select -i command to reprovision the switch and install a network operating system again using ONIE.

This section describes how to configure the following system settings:

• Date and time
• Authentication, authorization, and accounting
• Access Control Lists (ACLs)
• Services and daemons
• switchd
• In Service System Upgrade (ISSU)
• NVIDIA User Experience (NVUE)

NVUE is an object-oriented, schema driven model of a complete Cumulus Linux system (hardware and software) providing a robust API that allows for multiple interfaces to both view (show) and configure (set and unset) any element within a system running the NVUE software.

The NVUE object model definition uses the OpenAPI specification (OAS). Similar to YANG (RFC 6020 and RFC 7950), OAS is a data definition, manipulation, and modeling language (DML) that lets you build model-driven interfaces for both humans and machines. Although the computer networking and telecommunications industry commonly uses YANG (standardized by IETF) as a DML, the adoption of OpenAPI is broader, spanning cloud to compute to storage to IoT and even social media. The OpenAPI Initiative (OAI) consortium leads OpenAPI standardization, a chartered project under the Linux Foundation.

The OAS schema forms the management plane model with which you configure, monitor, and manage the Cumulus Linux switch. The v3.0.2 version of OAS defines the NVUE data model.

Like other systems that use OpenAPI, the NVUE OAS schema defines the endpoints (paths) exposed as RESTful APIs. With these REST APIs, you can perform various create, retrieve, update, delete, and eXecute (CRUDX) operations. The OAS schema also describes the API inputs and outputs (data models).

You can use the NVUE object model in these two ways:

• Through the NVUE REST API, where you run the GET, PATCH, DELETE, and other REST APIs on the NVUE object model endpoints to configure, monitor, and manage the switch. Because of the large user community and maturity of OAS, you can use several popular tools and libraries to create client-side bindings to use the NVUE REST API.
• Through the NVUE CLI, where you configure, monitor and manage the Cumulus Linux network elements. The CLI commands translate to their equivalent REST APIs, which Cumulus Linux then runs on the NVUE object model.

The CLI and the REST API are equivalent in functionality; you can run all management operations from the REST API or the CLI. The NVUE object model drives both the REST API and the CLI management operations. All operations are consistent; for example, the CLI nv show commands reflect any PATCH operation (create) you run through the REST API.

NVUE follows a declarative model, removing context-specific commands and settings. It is structured as a big tree that represents the entire state of a Cumulus Linux instance. At the base of the tree are high level branches representing objects, such as router and interface. Under each of these branches are further branches. As you navigate through the tree, you gain a more specific context. At the leaves of the tree are actual attributes, represented as key-value pairs. The path through the tree is similar to a filesystem path.

Cumulus Linux installs NVUE by default and enables the NVUE service nvued.

The NVUE CLI has a flat structure as opposed to a modal structure. This means that you can run all commands from the primary prompt instead of only in a specific mode.

Command Syntax

NVUE commands all begin with nv and fall into one of three syntax categories:

• Configuration (nv set and nv unset)
• Monitoring (nv show)
• Configuration management (nv config).

Command Completion

As you enter commands, you can get help with the valid keywords or options using the Tab key. For example, using Tab completion with nv set displays the possible options for the command and returns you to the command prompt to complete the command.

cumulus@switch:~$ nv set <<press Tab>>
acl        evpn       mlag       platform   router     system     
bridge     interface  nve        qos        service    vrf 

cumulus@switch:~$ nv set

Command Question Mark

You can type a question mark (?) after a command to display required information quickly and concisely. When you type ?, NVUE specifies the value type, range, and options with a brief description of each; for example:

cumulus@switch:~$ nv set interface swp1 link state ?
   (down|up)    The state of the interface
cumulus@switch:~$ nv set interface swp1 link mtu ?
   552-9216     interface mtu
cumulus@switch:~$ nv set interface swp1 link speed ?
   (auto|10M|100M|1G|10G|25G|40G|50G|100G|200G|400G)    Link speed

NVUE also indicates if you need to provide specific values for the command:

cumulus@switch:~$ nv set interface swp1 bridge domain ?
   <domain-id>    Bridge domains on this interface

Command Abbreviation

NVUE supports command abbreviation, where you can type a certain number of characters instead of a whole command to speed up CLI interaction. For example, instead of typing nv show interface, you can type nv sh int.

If the command you type is ambiguous, NVUE shows the reason for the ambiguity so that you can correct the shortcut. For example:

cumulus@switch:~$ nv s i 
Ambiguous Command: 
   set interface 
   show interface 

Command Help

As you enter commands, you can get help with command syntax by entering -h or --help at various points within a command entry. For example, to examine the options available for nv set interface, enter nv set interface -h or nv set interface --help.

cumulus@switch:~$ nv set interface -h
Usage:
  nv set interface [options] <interface-id> ...

Description:
  Interfaces

Identifiers:
  <interface-id>    Interface

General Options:
  -h, --help        Show help.

Command List

You can list all the NVUE commands by running nv list-commands. See List All NVUE Commands below.

Command History

At the command prompt, press the Up Arrow and Down Arrow keys to move back and forth through the list of commands you entered. When you find a given command, you can run the command by pressing Enter. Optionally, you can modify the command before you run it.

Command Categories

The NVUE CLI has a flat structure; however, the commands are in three functional categories:

• Configuration
• Monitoring
• Configuration Management

Configuration Commands

The NVUE configuration commands modify switch configuration. You can set and unset configuration options.

The nv set and nv unset commands are in the following categories. Each command group includes subcommands. Use command completion (Tab key) to list the subcommands.

Monitoring Commands

The NVUE monitoring commands show various parts of the network configuration. For example, you can show the complete network configuration or only interface configuration. The monitoring commands are in the following categories. Each command group includes subcommands. Use command completion (Tab key) to list the subcommands.

The following example shows the nv show router commands after pressing the TAB key, then shows the output of the nv show router bgp command.

cumulus@leaf01:mgmt:~$ nv show router <<TAB>>
adaptive-routing  igmp              ospf              pim               vrr               
bgp               nexthop-group     pbr               policy            vrrp 

cumulus@leaf01:mgmt:~$ nv show router bgp
                                operational  applied  pending      description
------------------------------  -----------  -------  -----------  ----------------------------------------------------------------------
enable                                       off      on           Turn the feature 'on' or 'off'.  The default is 'off'.
autonomous-system                                     none         ASN for all VRFs, if a single AS is in use.  If "none", then ASN mu...
graceful-shutdown                                     off          Graceful shutdown enable will initiate the GSHUT community to be an...
policy-update-timer                                   5            Wait time in seconds before processing updates to policies to ensur...
router-id                                             none         BGP router-id for all VRFs, if a common one is used.  If "none", th...
wait-for-install                                      off          bgp waits for routes to be installed into kernel/asic before advert...
convergence-wait
  establish-wait-time                                 0            Maximum time to wait to establish BGP sessions. Any peers which do...
  time                                                0            Time to wait for peers to send end-of-RIB before router performs pa...
graceful-restart
  mode                                                helper-only  Role of router during graceful restart. helper-only, router is in h...
  path-selection-deferral-time                        360          Used by the restarter as an upper-bounds for waiting for peering es...
  restart-time                                        120          Amount of time taken to restart by router. It is advertised to the...
  stale-routes-time                                   360          Specifies an upper-bounds on how long we retain routes from a resta...
cumulus@leaf01:mgmt:~$ 

Additional options are available for the nv show commands. For example, you can choose the configuration you want to show (pending, applied, startup, or operational). You can also turn on colored output, and paginate specific output.

The following example shows pending BGP graceful restart configuration:

cumulus@switch:~$ nv show router bgp graceful-restart --pending
                             pending_20210128_212626_4WSY  description
----------------------------  ----------------------------  ----------------------------------------------------------------------
mode                          helper-only                   Role of router during graceful restart. helper-only, router is in h...
path-selection-deferral-time  360                           Used by the restarter as an upper-bounds for waiting for peeringes...
restart-time                  120                           Amount of time taken to restart by router. It is advertised to the...
stale-routes-time             360                           Specifies an upper-bounds on how long we retain routes from a resta...

Net Show commands

In addition to the nv show commands, Cumulus Linux continues to provide a subset of the NCLU net show commands. Use these commands to get additional views of various parts of your network configuration.

cumulus@leaf01:mgmt:~$ net show 
    bfd            :  Bidirectional forwarding detection
    bgp            :  Border Gateway Protocol
    bridge         :  a layer2 bridge
    clag           :  Multi-Chassis Link Aggregation
    commit         :  apply the commit buffer to the system
    configuration  :  settings, configuration state, etc
    counters       :  net show counters
    debugs         :  Debugs
    dhcp-snoop     :  DHCP snooping for IPv4
    dhcp-snoop6    :  DHCP snooping for IPv6
    dot1x          :  Configure, Enable, Delete or Show IEEE 802.1X EAPOL
    evpn           :  Ethernet VPN
    hostname       :  local hostname
    igmp           :  Internet Group Management Protocol
    interface      :  An interface, such as swp1, swp2, etc.
    ip             :  Internet Protocol version 4/6
    ipv6           :  Internet Protocol version 6
    lldp           :  Link Layer Discovery Protocol
    mpls           :  Multiprotocol Label Switching
    mroute         :  Static unicast routes in MRIB for multicast RPF lookup
    msdp           :  Multicast Source Discovery Protocol
    neighbor       :  A BGP, OSPF, PIM, etc neighbor
    ospf           :  Open Shortest Path First (OSPFv2)
    ospf6          :  Open Shortest Path First (OSPFv3)
    package        :  A Cumulus Linux package name
    pbr            :  Policy Based Routing
    pim            :  Protocol Independent Multicast
    port-mirror    :  port-mirror
    port-security  :  Port security
    ptp            :  Precision Time Protocol
    roce           :  Enable RoCE on all interfaces, default mode is lossless
    rollback       :  revert to a previous configuration state
    route          :  EVPN route information
    route-map      :  Route-map
    snmp-server    :  Configure the SNMP server
    system         :  System
    time           :  Time
    version        :  Version number
    vrf            :  Virtual routing and forwarding
    vrrp           :  Virtual Router Redundancy Protocol

Configuration Management Commands

The NVUE configuration management commands manage and apply configurations.

You can use the NVUE configuration management commands to back up and restore configuration when you upgrade Cumulus Linux on the switch. Refer to Upgrading Cumulus Linux.

List All NVUE Commands

To show the full list of NVUE commands, run nv list-commands. For example:

cumulus@switch:~$ nv list-commands
nv show router
nv show router nexthop-group
nv show router nexthop-group <nexthop-group-id>
nv show router nexthop-group <nexthop-group-id> via
nv show router nexthop-group <nexthop-group-id> via <via-id>
nv show router pbr
nv show router pbr map
nv show router pbr map <pbr-map-id>
nv show router pbr map <pbr-map-id> rule
...

You can show the list of commands for a command grouping. For example, to show the list of interface commands:

cumulus@switch:~$ nv list-commands interface
nv show interface
nv show interface <interface-id>
nv show interface <interface-id> pluggable
nv show interface <interface-id> router
nv show interface <interface-id> router pbr
nv show interface <interface-id> router pbr map
nv show interface <interface-id> router pbr map <pbr-map-id>
nv show interface <interface-id> router ospf
...

Use the Tab key to get help for the command lists you want to see. For example, to show the list of command options available for the interface swp1, run:

cumulus@switch:~$ nv list-commands interface swp1 <<press Tab>>
acl        bridge     ip         lldp       ptp        router     
bond       evpn       link       pluggable  qos        tunnel

NVUE Configuration File

When you save network configuration using NVUE, Cumulus Linux writes the configuration to the /etc/nvue.d/startup.yaml file.

You can edit or replace the contents of the /etc/nvue.d/startup.yaml file. NVUE applies the configuration in the /etc/nvue.d/startup.yaml file during system boot only if the nvue-startup.service is running. If this service is not running, the switch reboots with the same configuration that is running before the reboot.

To start nvue-startup.service:

cumulus@switch:~$ sudo systemctl enable nvue-startup.service
cumulus@switch:~$ sudo systemctl start nvue-startup.service

When you apply a configuration with nv config apply, NVUE also writes to underlying Linux files such as /etc/network/interfaces and /etc/frr/frr.conf. You can view these configuration files; however NVIDIA recommends that you do not manually edit them while using NVUE. If you need to configure certain network settings manually or use automation such as Ansible to configure the switch, see Configure NVUE to Ignore Linux Files below.

Configure NVUE to Ignore Linux Files

You can configure NVUE to ignore certain underlying Linux files when applying configuration changes. For example, if you push certain configuration to the switch using Ansible and Jinja2 file templates or you want to use custom configuration for a particular service such as PTP, you can ensure that NVUE never writes to those configuration files.

The following example configures NVUE to ignore the Linux /etc/ptp4l.conf file when applying configuration changes and saves the configuration so it persists after a reboot.

cumulus@switch:~$ nv set system config apply ignore /etc/ptp4l.conf
cumulus@switch:~$ nv config apply
cumulus@switch:~$ nv config save

Reset NVUE Configuration to Default Values

To reset the NVUE configuration on the switch back to the default values, run the following command:

cumulus@switch:~$ nv config apply empty

Example Configuration Commands

This section provides examples of how to configure a Cumulus Linux switch using NVUE commands.

Configure the System Hostname

The example below shows the NVUE commands required to change the hostname for the switch to leaf01:

cumulus@switch:~$ nv set system hostname leaf01
cumulus@switch:~$ nv config apply

Configure the System DNS Server

The example below shows the NVUE commands required to define the DNS server for the switch:

cumulus@switch:~$ nv set service dns mgmt server 192.168.200.1
cumulus@switch:~$ nv config apply

Configure an Interface

The example below shows the NVUE commands required to bring up swp1.

cumulus@switch:~$ nv set interface swp1
cumulus@switch:~$ nv config apply 

Configure a Bond

The example below shows the NVUE commands required to configure the front panel port interfaces swp1 thru swp4 to be slaves in bond0.

cumulus@switch:~$ nv set interface bond0 bond member swp1-4
cumulus@switch:~$ nv config apply

Configure a Bridge

The example below shows the NVUE commands required to create a VLAN-aware bridge that contains two switch ports (swp1 and swp2) and includes 3 VLANs; tagged VLANs 10 and 20 and an untagged (native) VLAN of 1.

With NVUE, there is a default bridge called br_default, which has no ports assigned to it. The example below configures this default bridge.

cumulus@switch:~$ nv set interface swp1-2 bridge domain br_default
cumulus@switch:~$ nv set bridge domain br_default vlan 10,20
cumulus@switch:~$ nv set bridge domain br_default untagged 1
cumulus@switch:~$ nv config apply

Configure MLAG

The example below shows the NVUE commands required to configure MLAG on leaf01. The commands:

• Place swp1 into bond1 and swp2 into bond2.
• Configure the MLAG ID to 1 for bond1 and to 2 for bond2.
• Add bond1 and bond2 to the default bridge (br_default).
• Create the inter-chassis bond (swp49 and swp50) and the peer link (peerlink)
• Set the peer link IP address to linklocal, the MLAG system MAC address to 44:38:39:BE:EF:AA, and the backup interface to 10.10.10.2.

cumulus@leaf01:~$ nv set interface bond1 bond member swp1
cumulus@leaf01:~$ nv set interface bond2 bond member swp2
cumulus@leaf01:~$ nv set interface bond1 bond mlag id 1
cumulus@leaf01:~$ nv set interface bond2 bond mlag id 2
cumulus@switch:~$ nv set interface bond1-2 bridge domain br_default 
cumulus@leaf01:~$ nv set interface peerlink bond member swp49-50
cumulus@leaf01:~$ nv set mlag mac-address 44:38:39:BE:EF:AA
cumulus@leaf01:~$ nv set mlag backup 10.10.10.2
cumulus@leaf01:~$ nv set mlag peer-ip linklocal
cumulus@leaf01:~$ nv config apply

Configure BGP Unnumbered

The example below shows the NVUE commands required to configure BGP unnumbered on leaf01. The commands:

• Assign the ASN for this BGP node to 65101.
• Set the router ID to 10.10.10.1.
• Distribute routing information to the peer on swp51.
• Originate prefixes 10.10.10.1/32 from this BGP node.

cumulus@leaf01:~$ nv set router bgp autonomous-system 65101
cumulus@leaf01:~$ nv set router bgp router-id 10.10.10.1
cumulus@leaf01:~$ nv set vrf default router bgp neighbor swp51 remote-as external
cumulus@leaf01:~$ nv set vrf default router bgp address-family ipv4-unicast network 10.10.10.1/32
cumulus@leaf01:~$ nv config apply

Example Monitoring Commands

This section provides monitoring command examples.

Show Installed Software

The following example command lists the software installed on the switch:

cumulus@switch:~$ nv show platform software
Installed Software
=====================
                      description                                                     package                version
--------------------- ----------------------------                                    --------------------   ------------
acpi                  displays information on ACPI devices                            acpi                   1.7-1.1                   
acpi-support-base     scripts for handling base ACPI events such as the power button  acpi-support-base      0.142-8
acpid                 Advanced Configuration and Power Interface event daemon         acpid                  1:2.0.31-1
adduser               add and remove users and groups                                 adduser                3.118
apt                   commandline package manager                                     apt                    1.8.2.3
arping                sends IP and/or ARP pings (to the MAC address)                  arping                 2.19-6
arptables             ARP table administration                                        arptables              0.0.4+snapshot20181021-4
atftp                 advanced TFTP client                                            atftp                  0.7.git20120829-3.               
atftpd                advanced TFTP server                                            atftpd                 0.7.git20120829-3.2~deb10u1 
auditd                User space tools for security auditing                          auditd                 1:2.8.4-3              
base-files            Debian base system miscellaneous files                          base-files             10.3+deb10u9                 
base-passwd           Debian base system master password and group files              base-passwd            3.5.46 
bash                  GNU Bourne Again SHell                                          bash                   5.0-4
...

Show Interface Configuration

The following example command shows the running, applied, and pending swp1 interface configuration.

cumulus@leaf01:~$ nv show interface swp1
                         operational  applied  description
-----------------------  -----------  -------  ----------------------------------------------------------------------
type                     swp                   The type of interface
ip
  [address]                                    ipv4 and ipv6 address
link
  mtu                    9216                  interface mtu
  state                  down                  The state of the interface
  stats
    carrier-transitions  3                     Number of times the interface state has transitioned between up and...
    in-bytes             300 Bytes             total number of bytes received on the interface
    in-drops             5                     number of received packets dropped
    in-errors            0                     number of received packets with errors
    in-pkts              5                     total number of packets received on the interface
    out-bytes            0 Bytes               total number of bytes transmitted out of the interface
    out-drops            0                     The number of outbound packets that were chosen to be discarded eve...
    out-errors           0                     The number of outbound packets that could not be transmitted becaus...
    out-pkts             0                     total number of packets transmitted out of the interface
...

Example Configuration Management Commands

This section provides examples of how to use the configuration management commands to apply, save, and detach configurations.

Apply and Save a Configuration

The following example command configures the front panel port interfaces swp1 thru swp4 to be slaves in bond0. The configuration is only in a pending configuration state. The configuration is not applied. NVUE has not yet made any changes to the running configuration.

cumulus@switch:~$ nv set interface bond0 bond member swp1-4

To apply the pending configuration to the running configuration, run the nv config apply command. The configuration does not persist after a reboot.

cumulus@switch:~$ nv config apply

To save the applied configuration to the startup configuration, run the nv config save command. This command overwrites the startup configuration with the applied configuration by writing to the /etc/nvue.d/startup.yaml file. The configuration persists after a reboot.

cumulus@switch:~$ nv config save

Detach a Pending Configuration

The following example configures the IP address of the loopback interface, then detaches the configuration from the current pending configuration. Cumulus Linux saves the detached configuration to a file changeset/cumulus/<date>_<time>_xxxx that includes a timestamp with extra characters to distinguish it from other pending configurations; for example, changeset/cumulus/2021-06-11_18.35.06_FPKP.

cumulus@switch:~$ nv set interface lo ip address 10.10.10.1/32
cumulus@switch:~$ nv config detach

View Differences Between Configurations

To view differences between configurations, run the nv config diff command.

To view differences between two detached pending configurations, run the nv config diff «TAB» command to list all the current detached pending configurations, then run the nv config diff command with the pending configurations you want to diff:

cumulus@switch:~$ nv config diff <<press Tab>>
applied                                     changeset/cumulus/2021-06-11_18.35.06_FPKP
changeset/cumulus/2021-06-11_16.16.41_FPKK  empty
changeset/cumulus/2021-06-11_17.05.12_FPKN  startup

cumulus@switch:~$ nv config diff changeset/cumulus/2021-06-11_18.35.06_FPKP changeset/cumulus/2021-06-11_17.05.12_FPKN

To view differences between a detached pending configuration and the applied configuration:

cumulus@switch:~$ nv config diff changeset/cumulus/2021-06-11_18.35.06_FPKP applied

Replace and Patch a Pending Configuration

The following example replaces the pending configuration with the contents of the YAML configuration file called nv-02/13/2021.yaml located in the /deps directory:

cumulus@switch:~$ nv config replace /deps/nv-02/13/2021.yaml

The following example patches the pending configuration (runs the set or unset commands from the configuration in the nv-02/13/2021.yaml file located in the /deps directory):

cumulus@switch:~$ nv config patch /deps/nv-02/13/2021.yaml

This section discusses how to:

• Set the time zone, and the date and time on the software clock on the switch.
• Configure NTP.
• Configure PTP.

This section shows you how to list all the NVUE commands and see command descriptions.

List All Commands

To see a list of all the NVUE nv show, nv set, nv unset, nv action, and nv config commands, run nv list-commands.

cumulus@leaf01:mgmt:~$ nv list-commands
nv show router
nv show router nexthop-group
nv show router nexthop-group <nexthop-group-id>
nv show router nexthop-group <nexthop-group-id> via
nv show router nexthop-group <nexthop-group-id> via <via-id>
nv show router pbr
nv show router pbr map
nv show router pbr map <pbr-map-id>
nv show router pbr map <pbr-map-id> rule
nv show router pbr map <pbr-map-id> rule <rule-id>
nv show router pbr map <pbr-map-id> rule <rule-id> match
nv show router pbr map <pbr-map-id> rule <rule-id> action
nv show router pbr map <pbr-map-id> rule <rule-id> action nexthop-group
nv show router pbr map <pbr-map-id> rule <rule-id> action nexthop-group <nexthop-group-id>
nv show router policy
nv show router policy community-list
nv show router policy community-list <list-id>
nv show router policy community-list <list-id> rule
nv show router policy community-list <list-id> rule <rule-id>
nv show router policy community-list <list-id> rule <rule-id> community
nv show router policy community-list <list-id> rule <rule-id> community <community-id>
nv show router policy as-path-list
nv show router policy as-path-list <list-id>
nv show router policy as-path-list <list-id> rule
nv show router policy as-path-list <list-id> rule <rule-id>
nv show router policy ext-community-list
nv show router policy ext-community-list <list-id>
nv show router policy ext-community-list <list-id> rule
nv show router policy ext-community-list <list-id> rule <rule-id>
nv show router policy ext-community-list <list-id> rule <rule-id> ext-community
nv show router policy ext-community-list <list-id> rule <rule-id> ext-community rt
nv show router policy ext-community-list <list-id> rule <rule-id> ext-community rt <ext-community-id>
nv show router policy ext-community-list <list-id> rule <rule-id> ext-community soo
nv show router policy ext-community-list <list-id> rule <rule-id> ext-community soo <ext-community-id>
nv show router policy large-community-list
nv show router policy large-community-list <list-id>
nv show router policy large-community-list <list-id> rule
nv show router policy large-community-list <list-id> rule <rule-id>
nv show router policy large-community-list <list-id> rule <rule-id> large-community
nv show router policy large-community-list <list-id> rule <rule-id> large-community <large-community-id>
nv show router policy prefix-list
nv show router policy prefix-list <prefix-list-id>
nv show router policy prefix-list <prefix-list-id> rule
nv show router policy prefix-list <prefix-list-id> rule <rule-id>
nv show router policy prefix-list <prefix-list-id> rule <rule-id> match
nv show router policy prefix-list <prefix-list-id> rule <rule-id> match <match-id>
nv show router policy route-map
nv show router policy route-map <route-map-id>
nv show router policy route-map <route-map-id> rule
nv show router policy route-map <route-map-id> rule <rule-id>
nv show router policy route-map <route-map-id> rule <rule-id> match
nv show router policy route-map <route-map-id> rule <rule-id> set
nv show router policy route-map <route-map-id> rule <rule-id> set as-path-prepend
nv show router policy route-map <route-map-id> rule <rule-id> set community
nv show router policy route-map <route-map-id> rule <rule-id> set community <community-id>
nv show router policy route-map <route-map-id> rule <rule-id> set large-community
nv show router policy route-map <route-map-id> rule <rule-id> set large-community <large-community-id>
nv show router policy route-map <route-map-id> rule <rule-id> set aggregator-as
nv show router policy route-map <route-map-id> rule <rule-id> set aggregator-as <asn-id>
nv show router policy route-map <route-map-id> rule <rule-id> set aggregator-as <asn-id> address
nv show router policy route-map <route-map-id> rule <rule-id> set aggregator-as <asn-id> address <ipv4-address-id>
nv show router policy route-map <route-map-id> rule <rule-id> action
nv show router policy route-map <route-map-id> rule <rule-id> action deny
nv show router policy route-map <route-map-id> rule <rule-id> action permit
nv show router policy route-map <route-map-id> rule <rule-id> action permit exit-policy
nv show router bgp
nv show router bgp graceful-restart
nv show router bgp convergence-wait
nv show router ospf
nv show router ospf timers
nv show router ospf timers lsa
nv show router ospf timers spf
nv show router pim
nv show router pim timers
nv show router igmp
nv show router vrrp
nv show router vrr
nv show router adaptive-routing
nv show platform
nv show platform capabilities
nv show platform hardware
nv show platform hardware component
nv show platform hardware component <component-id>
nv show platform hardware component <component-id> linecard
nv show platform hardware component <component-id> port
nv show platform hardware component <component-id> port <port-id>
nv show platform hardware component <component-id> port <port-id> breakout-mode
nv show platform hardware component <component-id> port <port-id> breakout-mode <mode-id>
nv show platform environment
nv show platform environment fan
nv show platform environment fan <fan-id>
nv show platform environment sensor
nv show platform environment sensor <sensor-id>
nv show platform environment psu
nv show platform environment psu <psu-id>
nv show platform environment led
nv show platform environment led <led-id>
nv show platform software
nv show platform software installed
nv show platform software installed <installed-id>
nv show bridge
nv show bridge domain
nv show bridge domain <domain-id>
nv show bridge domain <domain-id> stp
nv show bridge domain <domain-id> stp state
nv show bridge domain <domain-id> multicast
nv show bridge domain <domain-id> multicast snooping
nv show bridge domain <domain-id> multicast snooping querier
nv show bridge domain <domain-id> vlan
nv show bridge domain <domain-id> vlan <vid>
nv show bridge domain <domain-id> vlan <vid> vni
nv show bridge domain <domain-id> vlan <vid> vni <vni-id>
nv show bridge domain <domain-id> vlan <vid> vni <vni-id> flooding
nv show bridge domain <domain-id> vlan <vid> vni <vni-id> flooding head-end-replication
nv show bridge domain <domain-id> vlan <vid> vni <vni-id> flooding head-end-replication <hrep-id>
nv show bridge domain <domain-id> vlan <vid> ptp
nv show bridge domain <domain-id> vlan <vid> multicast
nv show bridge domain <domain-id> vlan <vid> multicast snooping
nv show bridge domain <domain-id> vlan <vid> multicast snooping querier
nv show bridge domain <domain-id> mac-table
nv show bridge domain <domain-id> mdb
nv show bridge domain <domain-id> router-port
nv show mlag
nv show mlag lacp-conflict
nv show mlag consistency-checker
nv show mlag consistency-checker global
nv show mlag backup
nv show mlag backup <backup-ip>
nv show mlag fdb
nv show mlag fdb local
nv show mlag fdb peer
nv show mlag fdb permanent
nv show mlag mdb
nv show mlag mdb local
nv show mlag mdb peer
nv show mlag multicast-router-port
nv show mlag multicast-router-port local
nv show mlag multicast-router-port peer
nv show mlag vni
nv show mlag vni local
nv show mlag vni peer
nv show mlag lacpdb
nv show mlag lacpdb local
nv show mlag lacpdb peer
nv show mlag neighbor
nv show mlag neighbor dynamic
nv show mlag neighbor permanent
nv show evpn
nv show evpn route-advertise
nv show evpn dad
nv show evpn dad duplicate-action
nv show evpn dad duplicate-action freeze
nv show evpn evi
nv show evpn evi <evi-id>
nv show evpn evi <evi-id> route-advertise
nv show evpn evi <evi-id> route-target
nv show evpn evi <evi-id> route-target export
nv show evpn evi <evi-id> route-target export <rt-id>
nv show evpn evi <evi-id> route-target import
nv show evpn evi <evi-id> route-target import <rt-id>
nv show evpn evi <evi-id> route-target both
nv show evpn evi <evi-id> route-target both <rt-id>
nv show evpn multihoming
nv show evpn multihoming ead-evi-route
nv show evpn multihoming segment
nv show qos
nv show qos roce
nv show qos roce prio-map
nv show qos roce tc-map
nv show qos roce pool-map
nv show qos roce pool
nv show interface
nv show interface <interface-id>
nv show interface <interface-id> pluggable
nv show interface <interface-id> router
nv show interface <interface-id> router pbr
nv show interface <interface-id> router pbr map
nv show interface <interface-id> router pbr map <pbr-map-id>
nv show interface <interface-id> router ospf
nv show interface <interface-id> router ospf timers
nv show interface <interface-id> router ospf authentication
nv show interface <interface-id> router ospf bfd
nv show interface <interface-id> router pim
nv show interface <interface-id> router pim timers
nv show interface <interface-id> router pim bfd
nv show interface <interface-id> router pim address-family
nv show interface <interface-id> router pim address-family ipv4-unicast
nv show interface <interface-id> router pim address-family ipv4-unicast allow-rp
nv show interface <interface-id> router adaptive-routing
nv show interface <interface-id> bond
nv show interface <interface-id> bond member
nv show interface <interface-id> bond member <member-id>
nv show interface <interface-id> bond mlag
nv show interface <interface-id> bond mlag lacp-conflict
nv show interface <interface-id> bond mlag consistency-checker
nv show interface <interface-id> bridge
nv show interface <interface-id> bridge domain
nv show interface <interface-id> bridge domain <domain-id>
nv show interface <interface-id> bridge domain <domain-id> stp
nv show interface <interface-id> bridge domain <domain-id> vlan
nv show interface <interface-id> bridge domain <domain-id> vlan <vid>
nv show interface <interface-id> ip
nv show interface <interface-id> ip address
nv show interface <interface-id> ip address <ip-prefix-id>
nv show interface <interface-id> ip neighbor
nv show interface <interface-id> ip neighbor ipv4
nv show interface <interface-id> ip neighbor ipv4 <neighbor-id>
nv show interface <interface-id> ip neighbor ipv6
nv show interface <interface-id> ip neighbor ipv6 <neighbor-id>
nv show interface <interface-id> ip vrr
nv show interface <interface-id> ip vrr address
nv show interface <interface-id> ip vrr address <ip-prefix-id>
nv show interface <interface-id> ip vrr state
nv show interface <interface-id> ip gateway
nv show interface <interface-id> ip gateway <ip-address-id>
nv show interface <interface-id> ip ipv4
nv show interface <interface-id> ip ipv6
nv show interface <interface-id> ip igmp
nv show interface <interface-id> ip igmp static-group
nv show interface <interface-id> ip igmp static-group <static-group-id>
nv show interface <interface-id> ip vrrp
nv show interface <interface-id> ip vrrp virtual-router
nv show interface <interface-id> ip vrrp virtual-router <virtual-router-id>
nv show interface <interface-id> ip vrrp virtual-router <virtual-router-id> address
nv show interface <interface-id> ip vrrp virtual-router <virtual-router-id> address <ip-address-id>
nv show interface <interface-id> ip neighbor-discovery
nv show interface <interface-id> ip neighbor-discovery rdnss
nv show interface <interface-id> ip neighbor-discovery rdnss <ipv6-address-id>
nv show interface <interface-id> ip neighbor-discovery prefix
nv show interface <interface-id> ip neighbor-discovery prefix <ipv6-prefix-id>
nv show interface <interface-id> ip neighbor-discovery dnssl
nv show interface <interface-id> ip neighbor-discovery dnssl <domain-name-id>
nv show interface <interface-id> ip neighbor-discovery router-advertisement
nv show interface <interface-id> ip neighbor-discovery home-agent
nv show interface <interface-id> lldp
nv show interface <interface-id> lldp neighbor
nv show interface <interface-id> lldp neighbor <neighbor-id>
nv show interface <interface-id> lldp neighbor <neighbor-id> bridge
nv show interface <interface-id> lldp neighbor <neighbor-id> bridge vlan
nv show interface <interface-id> lldp neighbor <neighbor-id> bridge vlan <vid>
nv show interface <interface-id> link
nv show interface <interface-id> link state
nv show interface <interface-id> link dot1x
nv show interface <interface-id> link stats
nv show interface <interface-id> link traffic-engineering
nv show interface <interface-id> link flag
nv show interface <interface-id> qos
nv show interface <interface-id> qos counters
nv show interface <interface-id> qos counters port-stats
nv show interface <interface-id> qos counters port-stats rx-stats
nv show interface <interface-id> qos counters port-stats tx-stats
nv show interface <interface-id> qos counters egress-queue-stats
nv show interface <interface-id> qos counters ingress-buffer-stats
nv show interface <interface-id> qos counters pfc-stats
nv show interface <interface-id> qos roce
nv show interface <interface-id> qos roce counters
nv show interface <interface-id> qos roce status
nv show interface <interface-id> qos roce status pool-map
nv show interface <interface-id> qos roce status prio-map
nv show interface <interface-id> qos roce status tc-map
nv show interface <interface-id> evpn
nv show interface <interface-id> evpn multihoming
nv show interface <interface-id> evpn multihoming segment
nv show interface <interface-id> acl
nv show interface <interface-id> acl <acl-id>
nv show interface <interface-id> acl <acl-id> statistics
nv show interface <interface-id> acl <acl-id> statistics <rule-id>
nv show interface <interface-id> acl <acl-id> inbound
nv show interface <interface-id> acl <acl-id> inbound control-plane
nv show interface <interface-id> acl <acl-id> outbound
nv show interface <interface-id> acl <acl-id> outbound control-plane
nv show interface <interface-id> ptp
nv show interface <interface-id> ptp timers
nv show interface <interface-id> ptp counters
nv show interface <interface-id> tunnel
nv show service
nv show service dns
nv show service dns <vrf-id>
nv show service dns <vrf-id> server
nv show service dns <vrf-id> server <dns-server-id>
nv show service syslog
nv show service syslog <vrf-id>
nv show service syslog <vrf-id> server
nv show service syslog <vrf-id> server <server-id>
nv show service ntp
nv show service ntp <vrf-id>
nv show service ntp <vrf-id> server
nv show service ntp <vrf-id> server <server-id>
nv show service ntp <vrf-id> pool
nv show service ntp <vrf-id> pool <server-id>
nv show service dhcp-relay
nv show service dhcp-relay <vrf-id>
nv show service dhcp-relay <vrf-id> server
nv show service dhcp-relay <vrf-id> server <server-id>
nv show service dhcp-relay <vrf-id> interface
nv show service dhcp-relay <vrf-id> interface <interface-id>
nv show service dhcp-relay <vrf-id> giaddress-interface
nv show service dhcp-relay <vrf-id> giaddress-interface <interface-id>
nv show service dhcp-relay6
nv show service dhcp-relay6 <vrf-id>
nv show service dhcp-relay6 <vrf-id> interface
nv show service dhcp-relay6 <vrf-id> interface upstream
nv show service dhcp-relay6 <vrf-id> interface upstream <interface-id>
nv show service dhcp-relay6 <vrf-id> interface downstream
nv show service dhcp-relay6 <vrf-id> interface downstream <interface-id>
nv show service ptp
nv show service ptp <instance-id>
nv show service ptp <instance-id> acceptable-master
nv show service ptp <instance-id> acceptable-master <clock-id>
nv show service ptp <instance-id> unicast-master
nv show service ptp <instance-id> unicast-master <table-id>
nv show service ptp <instance-id> unicast-master <table-id> address
nv show service ptp <instance-id> unicast-master <table-id> address <ip-mac-address-id>
nv show service ptp <instance-id> profile
nv show service ptp <instance-id> profile <profile-id>
nv show service ptp <instance-id> monitor
nv show service ptp <instance-id> monitor timestamp-log
nv show service ptp <instance-id> monitor violations
nv show service ptp <instance-id> monitor violations log
nv show service ptp <instance-id> monitor violations log acceptable-master
nv show service ptp <instance-id> monitor violations log forced-master
nv show service ptp <instance-id> monitor violations log max-offset
nv show service ptp <instance-id> monitor violations log min-offset
nv show service ptp <instance-id> monitor violations log path-delay
nv show service ptp <instance-id> current
nv show service ptp <instance-id> clock-quality
nv show service ptp <instance-id> parent
nv show service ptp <instance-id> parent grandmaster-clock-quality
nv show service ptp <instance-id> time-properties
nv show service dhcp-server
nv show service dhcp-server <vrf-id>
nv show service dhcp-server <vrf-id> interface
nv show service dhcp-server <vrf-id> interface <interface-id>
nv show service dhcp-server <vrf-id> pool
nv show service dhcp-server <vrf-id> pool <pool-id>
nv show service dhcp-server <vrf-id> pool <pool-id> domain-name-server
nv show service dhcp-server <vrf-id> pool <pool-id> domain-name-server <server-id>
nv show service dhcp-server <vrf-id> pool <pool-id> domain-name
nv show service dhcp-server <vrf-id> pool <pool-id> domain-name <domain-name-id>
nv show service dhcp-server <vrf-id> pool <pool-id> gateway
nv show service dhcp-server <vrf-id> pool <pool-id> gateway <gateway-id>
nv show service dhcp-server <vrf-id> pool <pool-id> range
nv show service dhcp-server <vrf-id> pool <pool-id> range <range-id>
nv show service dhcp-server <vrf-id> domain-name
nv show service dhcp-server <vrf-id> domain-name <domain-name-id>
nv show service dhcp-server <vrf-id> domain-name-server
nv show service dhcp-server <vrf-id> domain-name-server <server-id>
nv show service dhcp-server <vrf-id> static
nv show service dhcp-server <vrf-id> static <static-id>
nv show service dhcp-server6
nv show service dhcp-server6 <vrf-id>
nv show service dhcp-server6 <vrf-id> interface
nv show service dhcp-server6 <vrf-id> interface <interface-id>
nv show service dhcp-server6 <vrf-id> pool
nv show service dhcp-server6 <vrf-id> pool <pool-id>
nv show service dhcp-server6 <vrf-id> pool <pool-id> domain-name-server
nv show service dhcp-server6 <vrf-id> pool <pool-id> domain-name-server <server-id>
nv show service dhcp-server6 <vrf-id> pool <pool-id> domain-name
nv show service dhcp-server6 <vrf-id> pool <pool-id> domain-name <domain-name-id>
nv show service dhcp-server6 <vrf-id> pool <pool-id> range
nv show service dhcp-server6 <vrf-id> pool <pool-id> range <range-id>
nv show service dhcp-server6 <vrf-id> domain-name
nv show service dhcp-server6 <vrf-id> domain-name <domain-name-id>
nv show service dhcp-server6 <vrf-id> domain-name-server
nv show service dhcp-server6 <vrf-id> domain-name-server <server-id>
nv show service dhcp-server6 <vrf-id> static
nv show service dhcp-server6 <vrf-id> static <static-id>
nv show service lldp
nv show system
nv show system control-plane
nv show system control-plane trap
nv show system control-plane trap <trap-id>
nv show system control-plane policer
nv show system control-plane policer <policer-id>
nv show system control-plane policer <policer-id> statistics
nv show system message
nv show system global
nv show system global reserved
nv show system global reserved routing-table
nv show system global reserved routing-table pbr
nv show system global reserved vlan
nv show system global reserved vlan l3-vni-vlan
nv show system ztp
nv show system ztp script
nv show system ztp status
nv show system reboot
nv show system reboot reason
nv show system reboot history
nv show system forwarding
nv show system forwarding lag-hash
nv show system forwarding ecmp-hash
nv show system port-mirror
nv show system port-mirror session
nv show system port-mirror session <session-id>
nv show system port-mirror session <session-id> span
nv show system port-mirror session <session-id> span source-port
nv show system port-mirror session <session-id> span source-port <port-id>
nv show system port-mirror session <session-id> span destination
nv show system port-mirror session <session-id> span destination <port-id>
nv show system port-mirror session <session-id> span truncate
nv show system port-mirror session <session-id> erspan
nv show system port-mirror session <session-id> erspan source-port
nv show system port-mirror session <session-id> erspan source-port <port-id>
nv show system port-mirror session <session-id> erspan destination
nv show system port-mirror session <session-id> erspan destination source-ip
nv show system port-mirror session <session-id> erspan destination source-ip <source-ip>
nv show system port-mirror session <session-id> erspan destination dest-ip
nv show system port-mirror session <session-id> erspan destination dest-ip <dest-ip>
nv show system port-mirror session <session-id> erspan truncate
nv show system config
nv show system config apply
nv show system config apply ignore
nv show system config apply ignore <ignore-id>
nv show system config snippet
nv show vrf
nv show vrf <vrf-id>
nv show vrf <vrf-id> loopback
nv show vrf <vrf-id> loopback ip
nv show vrf <vrf-id> loopback ip address
nv show vrf <vrf-id> loopback ip address <ip-prefix-id>
nv show vrf <vrf-id> evpn
nv show vrf <vrf-id> evpn vni
nv show vrf <vrf-id> evpn vni <vni-id>
nv show vrf <vrf-id> router
nv show vrf <vrf-id> router rib
nv show vrf <vrf-id> router rib <afi>
nv show vrf <vrf-id> router rib <afi> protocol
nv show vrf <vrf-id> router rib <afi> protocol <import-protocol-id>
nv show vrf <vrf-id> router rib <afi> route
nv show vrf <vrf-id> router rib <afi> route <route-id>
nv show vrf <vrf-id> router rib <afi> route <route-id> protocol
nv show vrf <vrf-id> router rib <afi> route <route-id> protocol <protocol-id>
nv show vrf <vrf-id> router rib <afi> route <route-id> protocol <protocol-id> entry-index
nv show vrf <vrf-id> router rib <afi> route <route-id> protocol <protocol-id> entry-index <entry-index>
nv show vrf <vrf-id> router rib <afi> route <route-id> protocol <protocol-id> entry-index <entry-index> flags
nv show vrf <vrf-id> router rib <afi> route <route-id> protocol <protocol-id> entry-index <entry-index> via
nv show vrf <vrf-id> router rib <afi> route <route-id> protocol <protocol-id> entry-index <entry-index> via <via-id>
nv show vrf <vrf-id> router rib <afi> route <route-id> protocol <protocol-id> entry-index <entry-index> via <via-id> flags
nv show vrf <vrf-id> router rib <afi> route <route-id> protocol <protocol-id> entry-index <entry-index> via <via-id> label
nv show vrf <vrf-id> router rib <afi> route <route-id> protocol <protocol-id> entry-index <entry-index> via <via-id> resolved-via
nv show vrf <vrf-id> router rib <afi> route <route-id> protocol <protocol-id> entry-index <entry-index> via <via-id> resolved-via <resolved-via-id>
nv show vrf <vrf-id> router bgp
nv show vrf <vrf-id> router bgp address-family
nv show vrf <vrf-id> router bgp address-family ipv4-unicast
nv show vrf <vrf-id> router bgp address-family ipv4-unicast redistribute
nv show vrf <vrf-id> router bgp address-family ipv4-unicast redistribute static
nv show vrf <vrf-id> router bgp address-family ipv4-unicast redistribute connected
nv show vrf <vrf-id> router bgp address-family ipv4-unicast redistribute kernel
nv show vrf <vrf-id> router bgp address-family ipv4-unicast redistribute ospf
nv show vrf <vrf-id> router bgp address-family ipv4-unicast aggregate-route
nv show vrf <vrf-id> router bgp address-family ipv4-unicast aggregate-route <aggregate-route-id>
nv show vrf <vrf-id> router bgp address-family ipv4-unicast network
nv show vrf <vrf-id> router bgp address-family ipv4-unicast network <static-network-id>
nv show vrf <vrf-id> router bgp address-family ipv4-unicast route-import
nv show vrf <vrf-id> router bgp address-family ipv4-unicast route-import from-vrf
nv show vrf <vrf-id> router bgp address-family ipv4-unicast route-import from-vrf list
nv show vrf <vrf-id> router bgp address-family ipv4-unicast route-import from-vrf list <leak-vrf-id>
nv show vrf <vrf-id> router bgp address-family ipv4-unicast multipaths
nv show vrf <vrf-id> router bgp address-family ipv4-unicast admin-distance
nv show vrf <vrf-id> router bgp address-family ipv4-unicast route-export
nv show vrf <vrf-id> router bgp address-family ipv4-unicast route-export to-evpn
nv show vrf <vrf-id> router bgp address-family ipv4-unicast loc-rib
nv show vrf <vrf-id> router bgp address-family ipv4-unicast loc-rib route
nv show vrf <vrf-id> router bgp address-family ipv4-unicast loc-rib route <route-id>
nv show vrf <vrf-id> router bgp address-family ipv4-unicast loc-rib route <route-id> path
nv show vrf <vrf-id> router bgp address-family ipv4-unicast loc-rib route <route-id> path <path-id>
nv show vrf <vrf-id> router bgp address-family ipv4-unicast loc-rib route <route-id> path <path-id> nexthop
nv show vrf <vrf-id> router bgp address-family ipv4-unicast loc-rib route <route-id> path <path-id> nexthop <nexthop-id>
nv show vrf <vrf-id> router bgp address-family ipv4-unicast loc-rib route <route-id> path <path-id> peer
nv show vrf <vrf-id> router bgp address-family ipv4-unicast loc-rib route <route-id> path <path-id> flags
nv show vrf <vrf-id> router bgp address-family ipv4-unicast loc-rib route <route-id> path <path-id> bestpath
nv show vrf <vrf-id> router bgp address-family ipv4-unicast loc-rib route <route-id> path <path-id> aspath
nv show vrf <vrf-id> router bgp address-family ipv4-unicast loc-rib route <route-id> path <path-id> community
nv show vrf <vrf-id> router bgp address-family ipv4-unicast loc-rib route <route-id> path <path-id> large-community
nv show vrf <vrf-id> router bgp address-family ipv4-unicast loc-rib route <route-id> path <path-id> ext-community
nv show vrf <vrf-id> router bgp address-family l2vpn-evpn
nv show vrf <vrf-id> router bgp address-family ipv6-unicast
nv show vrf <vrf-id> router bgp address-family ipv6-unicast aggregate-route
nv show vrf <vrf-id> router bgp address-family ipv6-unicast aggregate-route <aggregate-route-id>
nv show vrf <vrf-id> router bgp address-family ipv6-unicast network
nv show vrf <vrf-id> router bgp address-family ipv6-unicast network <static-network-id>
nv show vrf <vrf-id> router bgp address-family ipv6-unicast route-import
nv show vrf <vrf-id> router bgp address-family ipv6-unicast route-import from-vrf
nv show vrf <vrf-id> router bgp address-family ipv6-unicast route-import from-vrf list
nv show vrf <vrf-id> router bgp address-family ipv6-unicast multipaths
nv show vrf <vrf-id> router bgp address-family ipv6-unicast admin-distance
nv show vrf <vrf-id> router bgp address-family ipv6-unicast route-export
nv show vrf <vrf-id> router bgp address-family ipv6-unicast route-export to-evpn
nv show vrf <vrf-id> router bgp address-family ipv6-unicast redistribute
nv show vrf <vrf-id> router bgp address-family ipv6-unicast redistribute static
nv show vrf <vrf-id> router bgp address-family ipv6-unicast redistribute connected
nv show vrf <vrf-id> router bgp address-family ipv6-unicast redistribute kernel
nv show vrf <vrf-id> router bgp address-family ipv6-unicast redistribute ospf6
nv show vrf <vrf-id> router bgp address-family ipv6-unicast loc-rib
nv show vrf <vrf-id> router bgp address-family ipv6-unicast loc-rib route
nv show vrf <vrf-id> router bgp address-family ipv6-unicast loc-rib route <route-id>
nv show vrf <vrf-id> router bgp address-family ipv6-unicast loc-rib route <route-id> path
nv show vrf <vrf-id> router bgp address-family ipv6-unicast loc-rib route <route-id> path <path-id>
nv show vrf <vrf-id> router bgp address-family ipv6-unicast loc-rib route <route-id> path <path-id> nexthop
nv show vrf <vrf-id> router bgp address-family ipv6-unicast loc-rib route <route-id> path <path-id> nexthop <nexthop-id>
nv show vrf <vrf-id> router bgp address-family ipv6-unicast loc-rib route <route-id> path <path-id> peer
nv show vrf <vrf-id> router bgp address-family ipv6-unicast loc-rib route <route-id> path <path-id> flags
nv show vrf <vrf-id> router bgp address-family ipv6-unicast loc-rib route <route-id> path <path-id> bestpath
nv show vrf <vrf-id> router bgp address-family ipv6-unicast loc-rib route <route-id> path <path-id> aspath
nv show vrf <vrf-id> router bgp address-family ipv6-unicast loc-rib route <route-id> path <path-id> community
nv show vrf <vrf-id> router bgp address-family ipv6-unicast loc-rib route <route-id> path <path-id> large-community
nv show vrf <vrf-id> router bgp address-family ipv6-unicast loc-rib route <route-id> path <path-id> ext-community
nv show vrf <vrf-id> router bgp path-selection
nv show vrf <vrf-id> router bgp path-selection aspath
nv show vrf <vrf-id> router bgp path-selection med
nv show vrf <vrf-id> router bgp path-selection multipath
nv show vrf <vrf-id> router bgp route-reflection
nv show vrf <vrf-id> router bgp peer-group
nv show vrf <vrf-id> router bgp peer-group <peer-group-id>
nv show vrf <vrf-id> router bgp peer-group <peer-group-id> bfd
nv show vrf <vrf-id> router bgp peer-group <peer-group-id> ttl-security
nv show vrf <vrf-id> router bgp peer-group <peer-group-id> capabilities
nv show vrf <vrf-id> router bgp peer-group <peer-group-id> graceful-restart
nv show vrf <vrf-id> router bgp peer-group <peer-group-id> local-as
nv show vrf <vrf-id> router bgp peer-group <peer-group-id> timers
nv show vrf <vrf-id> router bgp peer-group <peer-group-id> address-family
nv show vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast
nv show vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast community-advertise
nv show vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast attribute-mod
nv show vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast aspath
nv show vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast aspath allow-my-asn
nv show vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast prefix-limits
nv show vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast prefix-limits inbound
nv show vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast default-route-origination
nv show vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast policy
nv show vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast policy inbound
nv show vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast policy outbound
nv show vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast conditional-advertise
nv show vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast
nv show vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast policy
nv show vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast policy inbound
nv show vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast policy outbound
nv show vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast aspath
nv show vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast aspath allow-my-asn
nv show vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast prefix-limits
nv show vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast prefix-limits inbound
nv show vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast default-route-origination
nv show vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast community-advertise
nv show vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast attribute-mod
nv show vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast conditional-advertise
nv show vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn
nv show vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn attribute-mod
nv show vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn aspath
nv show vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn aspath allow-my-asn
nv show vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn policy
nv show vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn policy inbound
nv show vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn policy outbound
nv show vrf <vrf-id> router bgp route-export
nv show vrf <vrf-id> router bgp route-export to-evpn
nv show vrf <vrf-id> router bgp route-export to-evpn route-target
nv show vrf <vrf-id> router bgp route-export to-evpn route-target <rt-id>
nv show vrf <vrf-id> router bgp route-import
nv show vrf <vrf-id> router bgp route-import from-evpn
nv show vrf <vrf-id> router bgp route-import from-evpn route-target
nv show vrf <vrf-id> router bgp route-import from-evpn route-target <rt-id>
nv show vrf <vrf-id> router bgp timers
nv show vrf <vrf-id> router bgp confederation
nv show vrf <vrf-id> router bgp confederation member-as
nv show vrf <vrf-id> router bgp neighbor
nv show vrf <vrf-id> router bgp neighbor <neighbor-id>
nv show vrf <vrf-id> router bgp neighbor <neighbor-id> bfd
nv show vrf <vrf-id> router bgp neighbor <neighbor-id> capabilities
nv show vrf <vrf-id> router bgp neighbor <neighbor-id> local-as
nv show vrf <vrf-id> router bgp neighbor <neighbor-id> graceful-restart
nv show vrf <vrf-id> router bgp neighbor <neighbor-id> ttl-security
nv show vrf <vrf-id> router bgp neighbor <neighbor-id> nexthop
nv show vrf <vrf-id> router bgp neighbor <neighbor-id> message-stats
nv show vrf <vrf-id> router bgp neighbor <neighbor-id> ebgp-policy
nv show vrf <vrf-id> router bgp neighbor <neighbor-id> address-family
nv show vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast
nv show vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast attribute-mod
nv show vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast aspath
nv show vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast aspath allow-my-asn
nv show vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast policy
nv show vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast policy inbound
nv show vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast policy outbound
nv show vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast prefix-limits
nv show vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast prefix-limits inbound
nv show vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast default-route-origination
nv show vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast community-advertise
nv show vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast conditional-advertise
nv show vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast capabilities
nv show vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast graceful-restart
nv show vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast
nv show vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast attribute-mod
nv show vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast aspath
nv show vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast aspath allow-my-asn
nv show vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast prefix-limits
nv show vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast prefix-limits inbound
nv show vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast default-route-origination
nv show vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast policy
nv show vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast policy inbound
nv show vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast policy outbound
nv show vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast community-advertise
nv show vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast conditional-advertise
nv show vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast capabilities
nv show vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast graceful-restart
nv show vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn
nv show vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn attribute-mod
nv show vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn aspath
nv show vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn aspath allow-my-asn
nv show vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn policy
nv show vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn policy inbound
nv show vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn policy outbound
nv show vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn capabilities
nv show vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn graceful-restart
nv show vrf <vrf-id> router bgp neighbor <neighbor-id> timers
nv show vrf <vrf-id> router static
nv show vrf <vrf-id> router static <route-id>
nv show vrf <vrf-id> router static <route-id> distance
nv show vrf <vrf-id> router static <route-id> distance <distance-id>
nv show vrf <vrf-id> router static <route-id> distance <distance-id> via
nv show vrf <vrf-id> router static <route-id> distance <distance-id> via <via-id>
nv show vrf <vrf-id> router static <route-id> distance <distance-id> via <via-id> flag
nv show vrf <vrf-id> router static <route-id> via
nv show vrf <vrf-id> router static <route-id> via <via-id>
nv show vrf <vrf-id> router static <route-id> via <via-id> flag
nv show vrf <vrf-id> router pim
nv show vrf <vrf-id> router pim timers
nv show vrf <vrf-id> router pim ecmp
nv show vrf <vrf-id> router pim msdp-mesh-group
nv show vrf <vrf-id> router pim msdp-mesh-group <msdp-mesh-group-id>
nv show vrf <vrf-id> router pim msdp-mesh-group <msdp-mesh-group-id> member-address
nv show vrf <vrf-id> router pim msdp-mesh-group <msdp-mesh-group-id> member-address <mesh-member-id>
nv show vrf <vrf-id> router pim address-family
nv show vrf <vrf-id> router pim address-family ipv4-unicast
nv show vrf <vrf-id> router pim address-family ipv4-unicast spt-switchover
nv show vrf <vrf-id> router pim address-family ipv4-unicast rp
nv show vrf <vrf-id> router pim address-family ipv4-unicast rp <rp-id>
nv show vrf <vrf-id> router pim address-family ipv4-unicast rp <rp-id> group-range
nv show vrf <vrf-id> router pim address-family ipv4-unicast rp <rp-id> group-range <group-range-id>
nv show vrf <vrf-id> router ospf
nv show vrf <vrf-id> router ospf area
nv show vrf <vrf-id> router ospf area <area-id>
nv show vrf <vrf-id> router ospf area <area-id> filter-list
nv show vrf <vrf-id> router ospf area <area-id> range
nv show vrf <vrf-id> router ospf area <area-id> range <range-id>
nv show vrf <vrf-id> router ospf area <area-id> network
nv show vrf <vrf-id> router ospf area <area-id> network <network-id>
nv show vrf <vrf-id> router ospf default-originate
nv show vrf <vrf-id> router ospf distance
nv show vrf <vrf-id> router ospf max-metric
nv show vrf <vrf-id> router ospf log
nv show vrf <vrf-id> router ospf redistribute
nv show vrf <vrf-id> router ospf redistribute static
nv show vrf <vrf-id> router ospf redistribute connected
nv show vrf <vrf-id> router ospf redistribute kernel
nv show vrf <vrf-id> router ospf redistribute bgp
nv show vrf <vrf-id> router ospf timers
nv show vrf <vrf-id> router ospf timers lsa
nv show vrf <vrf-id> router ospf timers spf
nv show vrf <vrf-id> ptp
nv show nve
nv show nve vxlan
nv show nve vxlan mlag
nv show nve vxlan source
nv show nve vxlan flooding
nv show nve vxlan flooding head-end-replication
nv show nve vxlan flooding head-end-replication <hrep-id>
nv show acl
nv show acl <acl-id>
nv show acl <acl-id> rule
nv show acl <acl-id> rule <rule-id>
nv show acl <acl-id> rule <rule-id> match
nv show acl <acl-id> rule <rule-id> match ip
nv show acl <acl-id> rule <rule-id> match ip source-port
nv show acl <acl-id> rule <rule-id> match ip source-port <ip-port-id>
nv show acl <acl-id> rule <rule-id> match ip dest-port
nv show acl <acl-id> rule <rule-id> match ip dest-port <ip-port-id>
nv show acl <acl-id> rule <rule-id> match ip fragment
nv show acl <acl-id> rule <rule-id> match ip ecn
nv show acl <acl-id> rule <rule-id> match ip ecn flags
nv show acl <acl-id> rule <rule-id> match ip tcp
nv show acl <acl-id> rule <rule-id> match ip tcp flags
nv show acl <acl-id> rule <rule-id> match ip tcp mask
nv show acl <acl-id> rule <rule-id> match mac
nv show acl <acl-id> rule <rule-id> action
nv show acl <acl-id> rule <rule-id> action permit
nv show acl <acl-id> rule <rule-id> action deny
nv show acl <acl-id> rule <rule-id> action log
nv show acl <acl-id> rule <rule-id> action set
nv show acl <acl-id> rule <rule-id> action erspan
nv show acl <acl-id> rule <rule-id> action police
nv set router
nv set router nexthop-group <nexthop-group-id>
nv set router nexthop-group <nexthop-group-id> via <via-id>
nv set router nexthop-group <nexthop-group-id> via <via-id> interface (auto|<interface-name>)
nv set router nexthop-group <nexthop-group-id> via <via-id> vrf (auto|<vrf-name>)
nv set router pbr
nv set router pbr map <pbr-map-id>
nv set router pbr map <pbr-map-id> rule <rule-id>
nv set router pbr map <pbr-map-id> rule <rule-id> match
nv set router pbr map <pbr-map-id> rule <rule-id> match source-ip (<ipv4-prefix>|<ipv6-prefix>)
nv set router pbr map <pbr-map-id> rule <rule-id> match destination-ip (<ipv4-prefix>|<ipv6-prefix>)
nv set router pbr map <pbr-map-id> rule <rule-id> match dscp 0-63
nv set router pbr map <pbr-map-id> rule <rule-id> match ecn 0-3
nv set router pbr map <pbr-map-id> rule <rule-id> action
nv set router pbr map <pbr-map-id> rule <rule-id> action nexthop-group <nexthop-group-id>
nv set router pbr map <pbr-map-id> rule <rule-id> action vrf <vrf-name>
nv set router pbr enable (on|off)
nv set router policy
nv set router policy community-list <list-id>
nv set router policy community-list <list-id> rule <rule-id>
nv set router policy community-list <list-id> rule <rule-id> community <community-id>
nv set router policy community-list <list-id> rule <rule-id> action (permit|deny)
nv set router policy as-path-list <list-id>
nv set router policy as-path-list <list-id> rule <rule-id>
nv set router policy as-path-list <list-id> rule <rule-id> action (permit|deny)
nv set router policy as-path-list <list-id> rule <rule-id> aspath-exp <bgp-regex>
nv set router policy ext-community-list <list-id>
nv set router policy ext-community-list <list-id> rule <rule-id>
nv set router policy ext-community-list <list-id> rule <rule-id> ext-community
nv set router policy ext-community-list <list-id> rule <rule-id> ext-community rt <ext-community-id>
nv set router policy ext-community-list <list-id> rule <rule-id> ext-community soo <ext-community-id>
nv set router policy ext-community-list <list-id> rule <rule-id> action (permit|deny)
nv set router policy large-community-list <list-id>
nv set router policy large-community-list <list-id> rule <rule-id>
nv set router policy large-community-list <list-id> rule <rule-id> large-community <large-community-id>
nv set router policy large-community-list <list-id> rule <rule-id> action (permit|deny)
nv set router policy prefix-list <prefix-list-id>
nv set router policy prefix-list <prefix-list-id> rule <rule-id>
nv set router policy prefix-list <prefix-list-id> rule <rule-id> match <match-id>
nv set router policy prefix-list <prefix-list-id> rule <rule-id> match <match-id> min-prefix-len 0-128
nv set router policy prefix-list <prefix-list-id> rule <rule-id> match <match-id> max-prefix-len 0-128
nv set router policy prefix-list <prefix-list-id> rule <rule-id> action (permit|deny)
nv set router policy prefix-list <prefix-list-id> type (ipv4|ipv6)
nv set router policy route-map <route-map-id>
nv set router policy route-map <route-map-id> rule <rule-id>
nv set router policy route-map <route-map-id> rule <rule-id> match
nv set router policy route-map <route-map-id> rule <rule-id> match ip-prefix-list <instance-name>
nv set router policy route-map <route-map-id> rule <rule-id> match ip-prefix-len 0-128
nv set router policy route-map <route-map-id> rule <rule-id> match ip-nexthop-list <instance-name>
nv set router policy route-map <route-map-id> rule <rule-id> match ip-nexthop-len 0-32
nv set router policy route-map <route-map-id> rule <rule-id> match ip-nexthop (<ipv4>|<ipv6>)
nv set router policy route-map <route-map-id> rule <rule-id> match ip-nexthop-type blackhole
nv set router policy route-map <route-map-id> rule <rule-id> match as-path-list <instance-name>
nv set router policy route-map <route-map-id> rule <rule-id> match community-list <instance-name>
nv set router policy route-map <route-map-id> rule <rule-id> match large-community-list <instance-name>
nv set router policy route-map <route-map-id> rule <rule-id> match metric <value>
nv set router policy route-map <route-map-id> rule <rule-id> match interface (<interface-name>|<vrf-name>)
nv set router policy route-map <route-map-id> rule <rule-id> match tag 1-4294967295
nv set router policy route-map <route-map-id> rule <rule-id> match source-protocol (bgp|connected|kernel|ospf|ospf6|sharp|static)
nv set router policy route-map <route-map-id> rule <rule-id> match origin (egp|igp|incomplete)
nv set router policy route-map <route-map-id> rule <rule-id> match peer (local|<interface-name>|<ipv4>|<ipv6>)
nv set router policy route-map <route-map-id> rule <rule-id> match local-preference 0-4294967295
nv set router policy route-map <route-map-id> rule <rule-id> match evpn-route-type (macip|imet|ip-prefix)
nv set router policy route-map <route-map-id> rule <rule-id> match evpn-vni <value>
nv set router policy route-map <route-map-id> rule <rule-id> match evpn-default-route (on|off)
nv set router policy route-map <route-map-id> rule <rule-id> match source-vrf <vrf-name>
nv set router policy route-map <route-map-id> rule <rule-id> match type (ipv4|ipv6)
nv set router policy route-map <route-map-id> rule <rule-id> set
nv set router policy route-map <route-map-id> rule <rule-id> set as-path-prepend
nv set router policy route-map <route-map-id> rule <rule-id> set as-path-prepend as <asn-range>
nv set router policy route-map <route-map-id> rule <rule-id> set as-path-prepend last-as 1-10
nv set router policy route-map <route-map-id> rule <rule-id> set community <community-id>
nv set router policy route-map <route-map-id> rule <rule-id> set large-community <large-community-id>
nv set router policy route-map <route-map-id> rule <rule-id> set aggregator-as <asn-id>
nv set router policy route-map <route-map-id> rule <rule-id> set aggregator-as <asn-id> address <ipv4-address-id>
nv set router policy route-map <route-map-id> rule <rule-id> set as-path-exclude 1-4294967295
nv set router policy route-map <route-map-id> rule <rule-id> set atomic-aggregate (on|off)
nv set router policy route-map <route-map-id> rule <rule-id> set ext-community-rt <route-distinguisher>
nv set router policy route-map <route-map-id> rule <rule-id> set ext-community-soo <route-distinguisher>
nv set router policy route-map <route-map-id> rule <rule-id> set ext-community-bw (cumulative|multipaths|cumulative-non-transitive|multipaths-non-transitive)
nv set router policy route-map <route-map-id> rule <rule-id> set local-preference 0-4294967295
nv set router policy route-map <route-map-id> rule <rule-id> set weight 0-4294967295
nv set router policy route-map <route-map-id> rule <rule-id> set metric (metric-plus|metric-minus|rtt|rtt-plus|rtt-minus)
nv set router policy route-map <route-map-id> rule <rule-id> set metric-type (type-1|type-2)
nv set router policy route-map <route-map-id> rule <rule-id> set origin (egp|igp|incomplete)
nv set router policy route-map <route-map-id> rule <rule-id> set tag 1-4294967295
nv set router policy route-map <route-map-id> rule <rule-id> set ipv6-nexthop-global <ipv6>
nv set router policy route-map <route-map-id> rule <rule-id> set ipv6-nexthop-local <ipv6>
nv set router policy route-map <route-map-id> rule <rule-id> set ipv6-nexthop-prefer-global (on|off)
nv set router policy route-map <route-map-id> rule <rule-id> set ip-nexthop (unchanged|peer-addr|<ipv4>|<ipv6>)
nv set router policy route-map <route-map-id> rule <rule-id> set source-ip (<ipv4>|<ipv6>)
nv set router policy route-map <route-map-id> rule <rule-id> set community-delete-list (<instance-name>|<integer>)
nv set router policy route-map <route-map-id> rule <rule-id> set large-community-delete-list (<instance-name>|<integer>)
nv set router policy route-map <route-map-id> rule <rule-id> set originator-id <ipv4>
nv set router policy route-map <route-map-id> rule <rule-id> set label-index 0-1048560
nv set router policy route-map <route-map-id> rule <rule-id> set forwarding-address <ipv6>
nv set router policy route-map <route-map-id> rule <rule-id> action
nv set router policy route-map <route-map-id> rule <rule-id> action deny
nv set router policy route-map <route-map-id> rule <rule-id> action permit
nv set router policy route-map <route-map-id> rule <rule-id> action permit exit-policy
nv set router policy route-map <route-map-id> rule <rule-id> action permit exit-policy rule <value>
nv set router policy route-map <route-map-id> rule <rule-id> description none
nv set router bgp
nv set router bgp graceful-restart
nv set router bgp graceful-restart mode (off|helper-only|full)
nv set router bgp graceful-restart restart-time 1-3600
nv set router bgp graceful-restart path-selection-deferral-time 0-3600
nv set router bgp graceful-restart stale-routes-time 1-3600
nv set router bgp convergence-wait
nv set router bgp convergence-wait time 0-3600
nv set router bgp convergence-wait establish-wait-time 0-3600
nv set router bgp enable (on|off)
nv set router bgp autonomous-system (1-4294967295|none|leaf|spine)
nv set router bgp router-id (none|<ipv4>)
nv set router bgp policy-update-timer 0-600
nv set router bgp graceful-shutdown (on|off)
nv set router bgp wait-for-install (on|off)
nv set router ospf
nv set router ospf timers
nv set router ospf timers lsa
nv set router ospf timers lsa min-arrival 0-600000
nv set router ospf timers lsa throttle 0-5000
nv set router ospf timers spf
nv set router ospf timers spf delay 0-600000
nv set router ospf timers spf holdtime 0-600000
nv set router ospf timers spf max-holdtime 0-600000
nv set router ospf timers refresh 10-1800
nv set router ospf enable (on|off)
nv set router ospf router-id (none|<ipv4>)
nv set router pim
nv set router pim timers
nv set router pim timers hello-interval 1-180
nv set router pim timers register-suppress 5-60000
nv set router pim timers join-prune-interval 60-600
nv set router pim timers keep-alive 31-60000
nv set router pim timers rp-keep-alive 31-60000
nv set router pim enable (on|off)
nv set router pim packets 1-100
nv set router igmp
nv set router igmp enable (on|off)
nv set router vrrp
nv set router vrrp enable (on|off)
nv set router vrrp priority 1-254
nv set router vrrp preempt (on|off)
nv set router vrrp advertisement-interval 10-40950
nv set router vrr
nv set router vrr enable (on|off)
nv set router adaptive-routing
nv set router adaptive-routing enable (on|off)
nv set platform
nv set platform hardware
nv set platform hardware component <component-id>
nv set platform hardware component <component-id> linecard
nv set platform hardware component <component-id> linecard provision (16x100GE|4x400GE|8x200GE|NONE)
nv set platform hardware component <component-id> type (switch|linecard)
nv set platform hardware component <component-id> admin-state (enable|disable)
nv set bridge
nv set bridge domain <domain-id>
nv set bridge domain <domain-id> stp
nv set bridge domain <domain-id> stp state (up|down)
nv set bridge domain <domain-id> stp priority 4096-61440
nv set bridge domain <domain-id> multicast
nv set bridge domain <domain-id> multicast snooping
nv set bridge domain <domain-id> multicast snooping querier
nv set bridge domain <domain-id> multicast snooping querier enable (on|off)
nv set bridge domain <domain-id> multicast snooping enable (on|off)
nv set bridge domain <domain-id> vlan <vid>
nv set bridge domain <domain-id> vlan <vid> vni <vni-id>
nv set bridge domain <domain-id> vlan <vid> vni <vni-id> flooding
nv set bridge domain <domain-id> vlan <vid> vni <vni-id> flooding head-end-replication <hrep-id>
nv set bridge domain <domain-id> vlan <vid> vni <vni-id> flooding enable (on|off|auto)
nv set bridge domain <domain-id> vlan <vid> vni <vni-id> flooding multicast-group <ipv4-multicast>
nv set bridge domain <domain-id> vlan <vid> vni <vni-id> mac-learning (on|off|auto)
nv set bridge domain <domain-id> vlan <vid> ptp
nv set bridge domain <domain-id> vlan <vid> ptp enable (on|off)
nv set bridge domain <domain-id> vlan <vid> multicast
nv set bridge domain <domain-id> vlan <vid> multicast snooping
nv set bridge domain <domain-id> vlan <vid> multicast snooping querier
nv set bridge domain <domain-id> vlan <vid> multicast snooping querier source-ip <ipv4>
nv set bridge domain <domain-id> type vlan-aware
nv set bridge domain <domain-id> untagged (1-4094|none)
nv set bridge domain <domain-id> encap 802.1Q
nv set bridge domain <domain-id> mac-address (auto|<mac>)
nv set bridge domain <domain-id> vlan-vni-offset 0-16773120
nv set mlag
nv set mlag lacp-conflict
nv set mlag backup <backup-ip>
nv set mlag backup <backup-ip> vrf <vrf-name>
nv set mlag enable (on|off)
nv set mlag mac-address (auto|<mac>)
nv set mlag peer-ip (linklocal|<ipv4>|<ipv6>)
nv set mlag priority 0-65535
nv set mlag init-delay 0-900
nv set mlag debug (on|off)
nv set evpn
nv set evpn route-advertise
nv set evpn route-advertise nexthop-setting (system-ip-mac|shared-ip-mac)
nv set evpn route-advertise svi-ip (on|off)
nv set evpn route-advertise default-gateway (on|off)
nv set evpn dad
nv set evpn dad duplicate-action
nv set evpn dad duplicate-action freeze
nv set evpn dad duplicate-action freeze duration (30-3600|permanent)
nv set evpn dad enable (on|off)
nv set evpn dad mac-move-threshold 2-1000
nv set evpn dad move-window 2-1800
nv set evpn evi <evi-id>
nv set evpn evi <evi-id> route-advertise
nv set evpn evi <evi-id> route-advertise svi-ip (on|off|auto)
nv set evpn evi <evi-id> route-advertise default-gateway (on|off|auto)
nv set evpn evi <evi-id> route-target
nv set evpn evi <evi-id> route-target export <rt-id>
nv set evpn evi <evi-id> route-target import <rt-id>
nv set evpn evi <evi-id> route-target both <rt-id>
nv set evpn evi <evi-id> rd (auto|<route-distinguisher>)
nv set evpn multihoming
nv set evpn multihoming ead-evi-route
nv set evpn multihoming ead-evi-route rx (on|off)
nv set evpn multihoming ead-evi-route tx (on|off)
nv set evpn multihoming segment
nv set evpn multihoming segment mac-address <mac>
nv set evpn multihoming segment df-preference 1-65535
nv set evpn multihoming enable (on|off)
nv set evpn multihoming mac-holdtime 0-86400
nv set evpn multihoming neighbor-holdtime 0-86400
nv set evpn multihoming startup-delay 0-3600
nv set evpn enable (on|off)
nv set qos
nv set qos roce
nv set qos roce enable (on|off)
nv set qos roce mode (lossy|lossless)
nv set qos roce cable-length 1-100000
nv set interface <interface-id>
nv set interface <interface-id> router
nv set interface <interface-id> router pbr
nv set interface <interface-id> router pbr map <pbr-map-id>
nv set interface <interface-id> router ospf
nv set interface <interface-id> router ospf timers
nv set interface <interface-id> router ospf timers dead-interval (1-65535|minimal)
nv set interface <interface-id> router ospf timers hello-multiplier 1-10
nv set interface <interface-id> router ospf timers hello-interval 1-65535
nv set interface <interface-id> router ospf timers retransmit-interval 1-65535
nv set interface <interface-id> router ospf timers transmit-delay 1-65535
nv set interface <interface-id> router ospf authentication
nv set interface <interface-id> router ospf authentication enable (on|off)
nv set interface <interface-id> router ospf authentication message-digest-key 1-255
nv set interface <interface-id> router ospf authentication md5-key <value>
nv set interface <interface-id> router ospf bfd
nv set interface <interface-id> router ospf bfd enable (on|off)
nv set interface <interface-id> router ospf bfd detect-multiplier 2-255
nv set interface <interface-id> router ospf bfd min-receive-interval 50-60000
nv set interface <interface-id> router ospf bfd min-transmit-interval 50-60000
nv set interface <interface-id> router ospf enable (on|off)
nv set interface <interface-id> router ospf area (0-4294967295|none|<ipv4>)
nv set interface <interface-id> router ospf cost (1-65535|auto)
nv set interface <interface-id> router ospf mtu-ignore (on|off)
nv set interface <interface-id> router ospf network-type (broadcast|non-broadcast|point-to-multipoint|point-to-point)
nv set interface <interface-id> router ospf passive (on|off)
nv set interface <interface-id> router ospf priority 0-255
nv set interface <interface-id> router pim
nv set interface <interface-id> router pim timers
nv set interface <interface-id> router pim timers hello-interval (1-180|auto)
nv set interface <interface-id> router pim bfd
nv set interface <interface-id> router pim bfd enable (on|off)
nv set interface <interface-id> router pim bfd detect-multiplier 2-255
nv set interface <interface-id> router pim bfd min-receive-interval 50-60000
nv set interface <interface-id> router pim bfd min-transmit-interval 50-60000
nv set interface <interface-id> router pim address-family
nv set interface <interface-id> router pim address-family ipv4-unicast
nv set interface <interface-id> router pim address-family ipv4-unicast allow-rp
nv set interface <interface-id> router pim address-family ipv4-unicast allow-rp enable (on|off)
nv set interface <interface-id> router pim address-family ipv4-unicast allow-rp rp-list (none|<instance-name>)
nv set interface <interface-id> router pim address-family ipv4-unicast multicast-boundary-oil (none|<instance-name>)
nv set interface <interface-id> router pim address-family ipv4-unicast use-source (none|<ipv4>)
nv set interface <interface-id> router pim enable (on|off)
nv set interface <interface-id> router pim dr-priority 1-4294967295
nv set interface <interface-id> router pim active-active (on|off)
nv set interface <interface-id> router adaptive-routing
nv set interface <interface-id> router adaptive-routing enable (on|off)
nv set interface <interface-id> router adaptive-routing link-utilization-threshold 1-100
nv set interface <interface-id> bond
nv set interface <interface-id> bond member <member-id>
nv set interface <interface-id> bond mlag
nv set interface <interface-id> bond mlag lacp-conflict
nv set interface <interface-id> bond mlag enable (on|off)
nv set interface <interface-id> bond mlag id (1-65535|auto)
nv set interface <interface-id> bond down-delay 0-65535
nv set interface <interface-id> bond lacp-bypass (on|off)
nv set interface <interface-id> bond lacp-rate (fast|slow)
nv set interface <interface-id> bond mode (lacp|static)
nv set interface <interface-id> bond up-delay 0-65535
nv set interface <interface-id> bridge
nv set interface <interface-id> bridge domain <domain-id>
nv set interface <interface-id> bridge domain <domain-id> stp
nv set interface <interface-id> bridge domain <domain-id> stp bpdu-filter (on|off)
nv set interface <interface-id> bridge domain <domain-id> stp bpdu-guard (on|off)
nv set interface <interface-id> bridge domain <domain-id> stp admin-edge (on|off)
nv set interface <interface-id> bridge domain <domain-id> stp auto-edge (on|off)
nv set interface <interface-id> bridge domain <domain-id> stp network (on|off)
nv set interface <interface-id> bridge domain <domain-id> stp restrrole (on|off)
nv set interface <interface-id> bridge domain <domain-id> vlan <vid>
nv set interface <interface-id> bridge domain <domain-id> learning (on|off)
nv set interface <interface-id> bridge domain <domain-id> untagged (1-4094|none|auto)
nv set interface <interface-id> bridge domain <domain-id> access (1-4094|auto)
nv set interface <interface-id> ip
nv set interface <interface-id> ip address <ip-prefix-id>
nv set interface <interface-id> ip vrr
nv set interface <interface-id> ip vrr address <ip-prefix-id>
nv set interface <interface-id> ip vrr state (up|down)
nv set interface <interface-id> ip vrr enable (on|off)
nv set interface <interface-id> ip vrr mac-id (1-255|none)
nv set interface <interface-id> ip vrr mac-address (auto|<mac>)
nv set interface <interface-id> ip gateway <ip-address-id>
nv set interface <interface-id> ip ipv4
nv set interface <interface-id> ip ipv4 forward (on|off)
nv set interface <interface-id> ip ipv6
nv set interface <interface-id> ip ipv6 enable (on|off)
nv set interface <interface-id> ip ipv6 forward (on|off)
nv set interface <interface-id> ip igmp
nv set interface <interface-id> ip igmp static-group <static-group-id>
nv set interface <interface-id> ip igmp static-group <static-group-id> source-address <ipv4-unicast>
nv set interface <interface-id> ip igmp enable (on|off)
nv set interface <interface-id> ip igmp version (2|3)
nv set interface <interface-id> ip igmp query-interval 1-1800
nv set interface <interface-id> ip igmp query-max-response-time 10-250
nv set interface <interface-id> ip igmp last-member-query-interval 1-255
nv set interface <interface-id> ip vrrp
nv set interface <interface-id> ip vrrp virtual-router <virtual-router-id>
nv set interface <interface-id> ip vrrp virtual-router <virtual-router-id> address <ip-address-id>
nv set interface <interface-id> ip vrrp virtual-router <virtual-router-id> version (2|3)
nv set interface <interface-id> ip vrrp virtual-router <virtual-router-id> priority (1-254|auto)
nv set interface <interface-id> ip vrrp virtual-router <virtual-router-id> preempt (on|off|auto)
nv set interface <interface-id> ip vrrp virtual-router <virtual-router-id> advertisement-interval (10-40950|auto)
nv set interface <interface-id> ip vrrp enable (on|off)
nv set interface <interface-id> ip neighbor-discovery
nv set interface <interface-id> ip neighbor-discovery rdnss <ipv6-address-id>
nv set interface <interface-id> ip neighbor-discovery rdnss <ipv6-address-id> lifetime (0-4294967295|infinite)
nv set interface <interface-id> ip neighbor-discovery prefix <ipv6-prefix-id>
nv set interface <interface-id> ip neighbor-discovery prefix <ipv6-prefix-id> valid-lifetime 0-4294967295
nv set interface <interface-id> ip neighbor-discovery prefix <ipv6-prefix-id> preferred-lifetime 0-4294967295
nv set interface <interface-id> ip neighbor-discovery prefix <ipv6-prefix-id> off-link (on|off)
nv set interface <interface-id> ip neighbor-discovery prefix <ipv6-prefix-id> autoconfig (on|off)
nv set interface <interface-id> ip neighbor-discovery prefix <ipv6-prefix-id> router-address (on|off)
nv set interface <interface-id> ip neighbor-discovery dnssl <domain-name-id>
nv set interface <interface-id> ip neighbor-discovery dnssl <domain-name-id> lifetime (0-4294967295|infinite)
nv set interface <interface-id> ip neighbor-discovery router-advertisement
nv set interface <interface-id> ip neighbor-discovery router-advertisement enable (on|off)
nv set interface <interface-id> ip neighbor-discovery router-advertisement interval 70-1800000
nv set interface <interface-id> ip neighbor-discovery router-advertisement interval-option (on|off)
nv set interface <interface-id> ip neighbor-discovery router-advertisement fast-retransmit (on|off)
nv set interface <interface-id> ip neighbor-discovery router-advertisement lifetime 0-9000
nv set interface <interface-id> ip neighbor-discovery router-advertisement reachable-time 0-3600000
nv set interface <interface-id> ip neighbor-discovery router-advertisement retransmit-time 0-4294967295
nv set interface <interface-id> ip neighbor-discovery router-advertisement managed-config (on|off)
nv set interface <interface-id> ip neighbor-discovery router-advertisement other-config (on|off)
nv set interface <interface-id> ip neighbor-discovery router-advertisement hop-limit 0-255
nv set interface <interface-id> ip neighbor-discovery router-advertisement router-preference (high|medium|low)
nv set interface <interface-id> ip neighbor-discovery home-agent
nv set interface <interface-id> ip neighbor-discovery home-agent lifetime 0-65520
nv set interface <interface-id> ip neighbor-discovery home-agent preference 0-65535
nv set interface <interface-id> ip neighbor-discovery enable (on|off)
nv set interface <interface-id> ip neighbor-discovery mtu 1-65535
nv set interface <interface-id> ip vrf <vrf-name>
nv set interface <interface-id> lldp
nv set interface <interface-id> lldp dcbx-pfc-tlv (on|off)
nv set interface <interface-id> lldp dcbx-ets-config-tlv (on|off)
nv set interface <interface-id> lldp dcbx-ets-recomm-tlv (on|off)
nv set interface <interface-id> link
nv set interface <interface-id> link state (up|down)
nv set interface <interface-id> link dot1x
nv set interface <interface-id> link dot1x mab (on|off)
nv set interface <interface-id> link dot1x parking-vlan (on|off)
nv set interface <interface-id> link auto-negotiate (on|off)
nv set interface <interface-id> link breakout (1x|2x|4x|8x|2x10G|2x25G|2x40G|2x50G|2x100G|2x200G|4x10G|4x25G|4x50G|4x100G|8x50G|disabled|loopback)
nv set interface <interface-id> link duplex (half|full)
nv set interface <interface-id> link speed (auto|10M|100M|1G|10G|25G|40G|50G|100G|200G|400G|800G)
nv set interface <interface-id> link lanes (1|2|4|8)
nv set interface <interface-id> link fec (auto|baser|off|rs|driver-auto)
nv set interface <interface-id> link mtu 552-9216
nv set interface <interface-id> evpn
nv set interface <interface-id> evpn multihoming
nv set interface <interface-id> evpn multihoming segment
nv set interface <interface-id> evpn multihoming segment enable (on|off)
nv set interface <interface-id> evpn multihoming segment local-id 1-16777215
nv set interface <interface-id> evpn multihoming segment identifier <es-identifier>
nv set interface <interface-id> evpn multihoming segment mac-address (auto|<mac>)
nv set interface <interface-id> evpn multihoming segment df-preference (1-65535|auto)
nv set interface <interface-id> evpn multihoming uplink (on|off)
nv set interface <interface-id> acl <acl-id>
nv set interface <interface-id> acl <acl-id> inbound
nv set interface <interface-id> acl <acl-id> inbound control-plane
nv set interface <interface-id> acl <acl-id> outbound
nv set interface <interface-id> acl <acl-id> outbound control-plane
nv set interface <interface-id> ptp
nv set interface <interface-id> ptp timers
nv set interface <interface-id> ptp timers announce-interval -3-4
nv set interface <interface-id> ptp timers sync-interval -7-1
nv set interface <interface-id> ptp timers delay-req-interval -7-6
nv set interface <interface-id> ptp timers announce-timeout 2-10
nv set interface <interface-id> ptp enable (on|off)
nv set interface <interface-id> ptp instance <value>
nv set interface <interface-id> ptp forced-master (on|off)
nv set interface <interface-id> ptp acceptable-master (on|off)
nv set interface <interface-id> ptp delay-mechanism end-to-end
nv set interface <interface-id> ptp transport (ipv4|ipv6|802.3)
nv set interface <interface-id> ptp ttl 1-255
nv set interface <interface-id> ptp mixed-multicast-unicast (on|off)
nv set interface <interface-id> ptp unicast-service-mode (client|server)
nv set interface <interface-id> ptp unicast-request-duration 10-1000
nv set interface <interface-id> ptp unicast-master-table-id <integer>
nv set interface <interface-id> tunnel
nv set interface <interface-id> tunnel source-ip <ipv4>
nv set interface <interface-id> tunnel dest-ip <ipv4>
nv set interface <interface-id> tunnel ttl 1-255
nv set interface <interface-id> tunnel mode gre
nv set interface <interface-id> tunnel interface <interface-name>
nv set interface <interface-id> description <value>
nv set interface <interface-id> type (swp|eth|bond|loopback|svi|sub|peerlink|tunnel)
nv set interface <interface-id> base-interface (none|<interface-name>)
nv set interface <interface-id> vlan 1-4094
nv set service
nv set service dns <vrf-id>
nv set service dns <vrf-id> server <dns-server-id>
nv set service syslog <vrf-id>
nv set service syslog <vrf-id> server <server-id>
nv set service syslog <vrf-id> server <server-id> port 1-32767
nv set service syslog <vrf-id> server <server-id> protocol (tcp|udp)
nv set service ntp <vrf-id>
nv set service ntp <vrf-id> server <server-id>
nv set service ntp <vrf-id> server <server-id> iburst (on|off)
nv set service ntp <vrf-id> pool <server-id>
nv set service ntp <vrf-id> pool <server-id> iburst (on|off)
nv set service ntp <vrf-id> listen <interface-name>
nv set service dhcp-relay <vrf-id>
nv set service dhcp-relay <vrf-id> server <server-id>
nv set service dhcp-relay <vrf-id> interface <interface-id>
nv set service dhcp-relay <vrf-id> giaddress-interface <interface-id>
nv set service dhcp-relay <vrf-id> giaddress-interface <interface-id> address (auto|<ipv4>)
nv set service dhcp-relay <vrf-id> source-ip (auto|giaddress)
nv set service dhcp-relay6 <vrf-id>
nv set service dhcp-relay6 <vrf-id> interface
nv set service dhcp-relay6 <vrf-id> interface upstream <interface-id>
nv set service dhcp-relay6 <vrf-id> interface upstream <interface-id> address <ipv6>
nv set service dhcp-relay6 <vrf-id> interface downstream <interface-id>
nv set service dhcp-relay6 <vrf-id> interface downstream <interface-id> address <ipv6>
nv set service ptp <instance-id>
nv set service ptp <instance-id> acceptable-master <clock-id>
nv set service ptp <instance-id> acceptable-master <clock-id> alt-priority <value>
nv set service ptp <instance-id> unicast-master <table-id>
nv set service ptp <instance-id> unicast-master <table-id> address <ip-mac-address-id>
nv set service ptp <instance-id> unicast-master <table-id> query-interval -3-4
nv set service ptp <instance-id> unicast-master <table-id> peer-address (<ipv4>|<ipv6>)
nv set service ptp <instance-id> profile <profile-id>
nv set service ptp <instance-id> profile <profile-id> profile-type (ieee-1588|itu-g-8275-1)
nv set service ptp <instance-id> profile <profile-id> priority1 <value>
nv set service ptp <instance-id> profile <profile-id> priority2 <value>
nv set service ptp <instance-id> profile <profile-id> local-priority <value>
nv set service ptp <instance-id> profile <profile-id> domain 0-255
nv set service ptp <instance-id> profile <profile-id> delay-mechanism end-to-end
nv set service ptp <instance-id> profile <profile-id> transport (ipv4|ipv6|802.3)
nv set service ptp <instance-id> profile <profile-id> announce-interval -7-7
nv set service ptp <instance-id> profile <profile-id> sync-interval -7-7
nv set service ptp <instance-id> profile <profile-id> delay-req-interval -7-7
nv set service ptp <instance-id> profile <profile-id> announce-timeout 2-255
nv set service ptp <instance-id> monitor
nv set service ptp <instance-id> monitor min-offset-threshold <value>
nv set service ptp <instance-id> monitor max-offset-threshold <value>
nv set service ptp <instance-id> monitor path-delay-threshold <value>
nv set service ptp <instance-id> monitor max-timestamp-entries 400-1000
nv set service ptp <instance-id> monitor max-violation-log-sets 8-128
nv set service ptp <instance-id> monitor max-violation-log-entries 8-128
nv set service ptp <instance-id> monitor violation-log-interval 0-259200
nv set service ptp <instance-id> enable (on|off)
nv set service ptp <instance-id> current-profile <value>
nv set service ptp <instance-id> two-step (on|off)
nv set service ptp <instance-id> priority1 <value>
nv set service ptp <instance-id> priority2 <value>
nv set service ptp <instance-id> domain 0-127
nv set service ptp <instance-id> ip-dscp 0-63
nv set service dhcp-server <vrf-id>
nv set service dhcp-server <vrf-id> interface <interface-id>
nv set service dhcp-server <vrf-id> pool <pool-id>
nv set service dhcp-server <vrf-id> pool <pool-id> domain-name-server <server-id>
nv set service dhcp-server <vrf-id> pool <pool-id> domain-name <domain-name-id>
nv set service dhcp-server <vrf-id> pool <pool-id> domain-name <domain-name-id> domain-name <idn-hostname>
nv set service dhcp-server <vrf-id> pool <pool-id> gateway <gateway-id>
nv set service dhcp-server <vrf-id> pool <pool-id> range <range-id>
nv set service dhcp-server <vrf-id> pool <pool-id> range <range-id> to <ipv4>
nv set service dhcp-server <vrf-id> pool <pool-id> pool-name <value>
nv set service dhcp-server <vrf-id> pool <pool-id> lease-time 180-31536000
nv set service dhcp-server <vrf-id> pool <pool-id> ping-check (on|off)
nv set service dhcp-server <vrf-id> pool <pool-id> default-url <value>
nv set service dhcp-server <vrf-id> pool <pool-id> cumulus-provision-url <value>
nv set service dhcp-server <vrf-id> domain-name <domain-name-id>
nv set service dhcp-server <vrf-id> domain-name <domain-name-id> domain-name <idn-hostname>
nv set service dhcp-server <vrf-id> domain-name-server <server-id>
nv set service dhcp-server <vrf-id> static <static-id>
nv set service dhcp-server <vrf-id> static <static-id> mac-address <mac>
nv set service dhcp-server <vrf-id> static <static-id> ip-address <ipv4>
nv set service dhcp-server <vrf-id> static <static-id> cumulus-provision-url <value>
nv set service dhcp-server6 <vrf-id>
nv set service dhcp-server6 <vrf-id> interface <interface-id>
nv set service dhcp-server6 <vrf-id> pool <pool-id>
nv set service dhcp-server6 <vrf-id> pool <pool-id> domain-name-server <server-id>
nv set service dhcp-server6 <vrf-id> pool <pool-id> domain-name <domain-name-id>
nv set service dhcp-server6 <vrf-id> pool <pool-id> domain-name <domain-name-id> domain-name <idn-hostname>
nv set service dhcp-server6 <vrf-id> pool <pool-id> range <range-id>
nv set service dhcp-server6 <vrf-id> pool <pool-id> range <range-id> to <ipv6>
nv set service dhcp-server6 <vrf-id> pool <pool-id> pool-name <value>
nv set service dhcp-server6 <vrf-id> pool <pool-id> lease-time 180-31536000
nv set service dhcp-server6 <vrf-id> pool <pool-id> ping-check (on|off)
nv set service dhcp-server6 <vrf-id> pool <pool-id> default-url <value>
nv set service dhcp-server6 <vrf-id> pool <pool-id> cumulus-provision-url <value>
nv set service dhcp-server6 <vrf-id> domain-name <domain-name-id>
nv set service dhcp-server6 <vrf-id> domain-name <domain-name-id> domain-name <idn-hostname>
nv set service dhcp-server6 <vrf-id> domain-name-server <server-id>
nv set service dhcp-server6 <vrf-id> static <static-id>
nv set service dhcp-server6 <vrf-id> static <static-id> mac-address <mac>
nv set service dhcp-server6 <vrf-id> static <static-id> ip-address <ipv6>
nv set service dhcp-server6 <vrf-id> static <static-id> cumulus-provision-url <value>
nv set service lldp
nv set service lldp tx-interval 10-300
nv set service lldp tx-hold-multiplier 1-10
nv set service lldp dot1-tlv (on|off)
nv set system
nv set system control-plane
nv set system control-plane trap <trap-id>
nv set system control-plane trap <trap-id> state (on|off)
nv set system control-plane policer <policer-id>
nv set system control-plane policer <policer-id> state (on|off)
nv set system control-plane policer <policer-id> burst 10-10000
nv set system control-plane policer <policer-id> rate 10-50000
nv set system message
nv set system message pre-login <value>
nv set system message post-login <value>
nv set system global
nv set system global reserved
nv set system global reserved routing-table
nv set system global reserved routing-table pbr
nv set system global reserved routing-table pbr begin 10000-4294966272
nv set system global reserved routing-table pbr end 10000-4294966272
nv set system global reserved vlan
nv set system global reserved vlan l3-vni-vlan
nv set system global reserved vlan l3-vni-vlan begin 1-4093
nv set system global reserved vlan l3-vni-vlan end 2-4093
nv set system global system-mac (auto|<mac>)
nv set system global anycast-mac (none|<mac>)
nv set system global anycast-id (1-65535|none)
nv set system global fabric-mac (none|<mac>)
nv set system global fabric-id 1-255
nv set system forwarding
nv set system forwarding lag-hash
nv set system forwarding lag-hash ip-protocol (on|off)
nv set system forwarding lag-hash source-mac (on|off)
nv set system forwarding lag-hash destination-mac (on|off)
nv set system forwarding lag-hash source-ip (on|off)
nv set system forwarding lag-hash destination-ip (on|off)
nv set system forwarding lag-hash source-port (on|off)
nv set system forwarding lag-hash destination-port (on|off)
nv set system forwarding lag-hash ether-type (on|off)
nv set system forwarding lag-hash vlan (on|off)
nv set system forwarding lag-hash gtp-teid (on|off)
nv set system forwarding ecmp-hash
nv set system forwarding ecmp-hash ip-protocol (on|off)
nv set system forwarding ecmp-hash source-ip (on|off)
nv set system forwarding ecmp-hash destination-ip (on|off)
nv set system forwarding ecmp-hash source-port (on|off)
nv set system forwarding ecmp-hash destination-port (on|off)
nv set system forwarding ecmp-hash ipv6-label (on|off)
nv set system forwarding ecmp-hash ingress-interface (on|off)
nv set system forwarding ecmp-hash gtp-teid (on|off)
nv set system forwarding ecmp-hash inner-ip-protocol (on|off)
nv set system forwarding ecmp-hash inner-source-ip (on|off)
nv set system forwarding ecmp-hash inner-destination-ip (on|off)
nv set system forwarding ecmp-hash inner-source-port (on|off)
nv set system forwarding ecmp-hash inner-destination-port (on|off)
nv set system forwarding ecmp-hash inner-ipv6-label (on|off)
nv set system forwarding hash-seed 0-4294967295
nv set system port-mirror
nv set system port-mirror session <session-id>
nv set system port-mirror session <session-id> span
nv set system port-mirror session <session-id> span source-port <port-id>
nv set system port-mirror session <session-id> span destination <port-id>
nv set system port-mirror session <session-id> span truncate
nv set system port-mirror session <session-id> span truncate enable (on|off)
nv set system port-mirror session <session-id> span truncate size (4-4088|none)
nv set system port-mirror session <session-id> span enable (on|off)
nv set system port-mirror session <session-id> span direction (ingress|egress)
nv set system port-mirror session <session-id> erspan
nv set system port-mirror session <session-id> erspan source-port <port-id>
nv set system port-mirror session <session-id> erspan destination
nv set system port-mirror session <session-id> erspan destination source-ip <source-ip>
nv set system port-mirror session <session-id> erspan destination dest-ip <dest-ip>
nv set system port-mirror session <session-id> erspan truncate
nv set system port-mirror session <session-id> erspan truncate enable (on|off)
nv set system port-mirror session <session-id> erspan truncate size (4-4088|none)
nv set system port-mirror session <session-id> erspan enable (on|off)
nv set system port-mirror session <session-id> erspan direction (ingress|egress)
nv set system config
nv set system config apply
nv set system config apply ignore <ignore-id>
nv set system config apply overwrite (all|controlled)
nv set system hostname <idn-hostname>
nv set system timezone (Africa/Abidjan|Africa/Accra|Africa/Addis_Ababa|Africa/Algiers|Africa/Asmara|Africa/Bamako|Africa/Bangui|Africa/Banjul|Africa/Bissau|Africa/Blantyre|Africa/Brazzaville|Africa/Bujumbura|Africa/Cairo|Africa/Casablanca|Africa/Ceuta|Africa/Conakry|Africa/Dakar|Africa/Dar_es_Salaam|Africa/Djibouti|Africa/Douala|Africa/El_Aaiun|Africa/Freetown|Africa/Gaborone|Africa/Harare|Africa/Johannesburg|Africa/Juba|Africa/Kampala|Africa/Khartoum|Africa/Kigali|Africa/Kinshasa|Africa/Lagos|Africa/Libreville|Africa/Lome|Africa/Luanda|Africa/Lubumbashi|Africa/Lusaka|Africa/Malabo|Africa/Maputo|Africa/Maseru|Africa/Mbabane|Africa/Mogadishu|Africa/Monrovia|Africa/Nairobi|Africa/Ndjamena|Africa/Niamey|Africa/Nouakchott|Africa/Ouagadougou|Africa/Porto-Novo|Africa/Sao_Tome|Africa/Timbuktu|Africa/Tripoli|Africa/Tunis|Africa/Windhoek|America/Adak|America/Anchorage|America/Anguilla|America/Antigua|America/Araguaina|America/Argentina/Buenos_Aires|America/Argentina/Catamarca|America/Argentina/ComodRivadavia|America/Argentina/Cordoba|America/Argentina/Jujuy|America/Argentina/La_Rioja|America/Argentina/Mendoza|America/Argentina/Rio_Gallegos|America/Argentina/Salta|America/Argentina/San_Juan|America/Argentina/San_Luis|America/Argentina/Tucuman|America/Argentina/Ushuaia|America/Aruba|America/Asuncion|America/Atikokan|America/Atka|America/Bahia|America/Bahia_Banderas|America/Barbados|America/Belem|America/Belize|America/Blanc-Sablon|America/Boa_Vista|America/Bogota|America/Boise|America/Buenos_Aires|America/Cambridge_Bay|America/Campo_Grande|America/Cancun|America/Caracas|America/Catamarca|America/Cayenne|America/Cayman|America/Chicago|America/Chihuahua|America/Coral_Harbour|America/Cordoba|America/Costa_Rica|America/Creston|America/Cuiaba|America/Curacao|America/Danmarkshavn|America/Dawson|America/Dawson_Creek|America/Denver|America/Detroit|America/Dominica|America/Edmonton|America/Eirunepe|America/El_Salvador|America/Ensenada|America/Fort_Nelson|America/Fort_Wayne|America/Fortaleza|America/Glace_Bay|America/Godthab|America/Goose_Bay|America/Grand_Turk|America/Grenada|America/Guadeloupe|America/Guatemala|America/Guayaquil|America/Guyana|America/Halifax|America/Havana|America/Hermosillo|America/Indiana/Indianapolis|America/Indiana/Knox|America/Indiana/Marengo|America/Indiana/Petersburg|America/Indiana/Tell_City|America/Indiana/Vevay|America/Indiana/Vincennes|America/Indiana/Winamac|America/Indianapolis|America/Inuvik|America/Iqaluit|America/Jamaica|America/Jujuy|America/Juneau|America/Kentucky/Louisville|America/Kentucky/Monticello|America/Knox_IN|America/Kralendijk|America/La_Paz|America/Lima|America/Los_Angeles|America/Louisville|America/Lower_Princes|America/Maceio|America/Managua|America/Manaus|America/Marigot|America/Martinique|America/Matamoros|America/Mazatlan|America/Mendoza|America/Menominee|America/Merida|America/Metlakatla|America/Mexico_City|America/Miquelon|America/Moncton|America/Monterrey|America/Montevideo|America/Montreal|America/Montserrat|America/Nassau|America/New_York|America/Nipigon|America/Nome|America/Noronha|America/North_Dakota/Beulah|America/North_Dakota/Center|America/North_Dakota/New_Salem|America/Ojinaga|America/Panama|America/Pangnirtung|America/Paramaribo|America/Phoenix|America/Port-au-Prince|America/Port_of_Spain|America/Porto_Acre|America/Porto_Velho|America/Puerto_Rico|America/Rainy_River|America/Rankin_Inlet|America/Recife|America/Regina|America/Resolute|America/Rio_Branco|America/Rosario|America/Santa_Isabel|America/Santarem|America/Santiago|America/Santo_Domingo|America/Sao_Paulo|America/Scoresbysund|America/Shiprock|America/Sitka|America/St_Barthelemy|America/St_Johns|America/St_Kitts|America/St_Lucia|America/St_Thomas|America/St_Vincent|America/Swift_Current|America/Tegucigalpa|America/Thule|America/Thunder_Bay|America/Tijuana|America/Toronto|America/Tortola|America/Vancouver|America/Virgin|America/Whitehorse|America/Winnipeg|America/Yakutat|America/Yellowknife|Antarctica/Casey|Antarctica/Davis|Antarctica/DumontDUrville|Antarctica/Macquarie|Antarctica/Mawson|Antarctica/McMurdo|Antarctica/Palmer|Antarctica/Rothera|Antarctica/South_Pole|Antarctica/Syowa|Antarctica/Troll|Antarctica/Vostok|Arctic/Longyearbyen|Asia/Aden|Asia/Almaty|Asia/Amman|Asia/Anadyr|Asia/Aqtau|Asia/Aqtobe|Asia/Ashgabat|Asia/Ashkhabad|Asia/Atyrau|Asia/Baghdad|Asia/Bahrain|Asia/Baku|Asia/Bangkok|Asia/Barnaul|Asia/Beirut|Asia/Bishkek|Asia/Brunei|Asia/Calcutta|Asia/Chita|Asia/Choibalsan|Asia/Chongqing|Asia/Chungking|Asia/Colombo|Asia/Dacca|Asia/Damascus|Asia/Dhaka|Asia/Dili|Asia/Dubai|Asia/Dushanbe|Asia/Famagusta|Asia/Gaza|Asia/Harbin|Asia/Hebron|Asia/Ho_Chi_Minh|Asia/Hong_Kong|Asia/Hovd|Asia/Irkutsk|Asia/Istanbul|Asia/Jakarta|Asia/Jayapura|Asia/Jerusalem|Asia/Kabul|Asia/Kamchatka|Asia/Karachi|Asia/Kashgar|Asia/Kathmandu|Asia/Katmandu|Asia/Khandyga|Asia/Kolkata|Asia/Krasnoyarsk|Asia/Kuala_Lumpur|Asia/Kuching|Asia/Kuwait|Asia/Macao|Asia/Macau|Asia/Magadan|Asia/Makassar|Asia/Manila|Asia/Muscat|Asia/Nicosia|Asia/Novokuznetsk|Asia/Novosibirsk|Asia/Omsk|Asia/Oral|Asia/Phnom_Penh|Asia/Pontianak|Asia/Pyongyang|Asia/Qatar|Asia/Qyzylorda|Asia/Rangoon|Asia/Riyadh|Asia/Saigon|Asia/Sakhalin|Asia/Samarkand|Asia/Seoul|Asia/Shanghai|Asia/Singapore|Asia/Srednekolymsk|Asia/Taipei|Asia/Tashkent|Asia/Tbilisi|Asia/Tehran|Asia/Tel_Aviv|Asia/Thimbu|Asia/Thimphu|Asia/Tokyo|Asia/Tomsk|Asia/Ujung_Pandang|Asia/Ulaanbaatar|Asia/Ulan_Bator|Asia/Urumqi|Asia/Ust-Nera|Asia/Vientiane|Asia/Vladivostok|Asia/Yakutsk|Asia/Yangon|Asia/Yekaterinburg|Asia/Yerevan|Atlantic/Azores|Atlantic/Bermuda|Atlantic/Canary|Atlantic/Cape_Verde|Atlantic/Faeroe|Atlantic/Faroe|Atlantic/Jan_Mayen|Atlantic/Madeira|Atlantic/Reykjavik|Atlantic/South_Georgia|Atlantic/St_Helena|Atlantic/Stanley|Australia/ACT|Australia/Adelaide|Australia/Brisbane|Australia/Broken_Hill|Australia/Canberra|Australia/Currie|Australia/Darwin|Australia/Eucla|Australia/Hobart|Australia/LHI|Australia/Lindeman|Australia/Lord_Howe|Australia/Melbourne|Australia/NSW|Australia/North|Australia/Perth|Australia/Queensland|Australia/South|Australia/Sydney|Australia/Tasmania|Australia/Victoria|Australia/West|Australia/Yancowinna|Brazil/Acre|Brazil/DeNoronha|Brazil/East|Brazil/West|Canada/Atlantic|Canada/Central|Canada/East-Saskatchewan|Canada/Eastern|Canada/Mountain|Canada/Newfoundland|Canada/Pacific|Canada/Saskatchewan|Canada/Yukon|Chile/Continental|Chile/EasterIsland|Etc/GMT|Etc/GMT0|Etc/GMT+0|Etc/GMT+1|Etc/GMT+2|Etc/GMT+3|Etc/GMT+4|Etc/GMT+5|Etc/GMT+6|Etc/GMT+7|Etc/GMT+8|Etc/GMT+9|Etc/GMT+10|Etc/GMT+11|Etc/GMT+12|Etc/GMT-0|Etc/GMT-1|Etc/GMT-2|Etc/GMT-3|Etc/GMT-4|Etc/GMT-5|Etc/GMT-6|Etc/GMT-7|Etc/GMT-8|Etc/GMT-9|Etc/GMT-10|Etc/GMT-11|Etc/GMT-12|Etc/GMT-13|Etc/GMT-14|Etc/Greenwich|Etc/UTC|Etc/Universal|Etc/Zulu|Europe/Amsterdam|Europe/Andorra|Europe/Astrakhan|Europe/Athens|Europe/Belfast|Europe/Belgrade|Europe/Berlin|Europe/Bratislava|Europe/Brussels|Europe/Bucharest|Europe/Budapest|Europe/Busingen|Europe/Chisinau|Europe/Copenhagen|Europe/Dublin|Europe/Gibraltar|Europe/Guernsey|Europe/Helsinki|Europe/Isle_of_Man|Europe/Istanbul|Europe/Jersey|Europe/Kaliningrad|Europe/Kiev|Europe/Kirov|Europe/Lisbon|Europe/Ljubljana|Europe/London|Europe/Luxembourg|Europe/Madrid|Europe/Malta|Europe/Mariehamn|Europe/Minsk|Europe/Monaco|Europe/Moscow|Europe/Nicosia|Europe/Oslo|Europe/Paris|Europe/Podgorica|Europe/Prague|Europe/Riga|Europe/Rome|Europe/Samara|Europe/San_Marino|Europe/Sarajevo|Europe/Saratov|Europe/Simferopol|Europe/Skopje|Europe/Sofia|Europe/Stockholm|Europe/Tallinn|Europe/Tirane|Europe/Tiraspol|Europe/Ulyanovsk|Europe/Uzhgorod|Europe/Vaduz|Europe/Vatican|Europe/Vienna|Europe/Vilnius|Europe/Volgograd|Europe/Warsaw|Europe/Zagreb|Europe/Zaporozhye|Europe/Zurich|Indian/Antananarivo|Indian/Chagos|Indian/Christmas|Indian/Cocos|Indian/Comoro|Indian/Kerguelen|Indian/Mahe|Indian/Maldives|Indian/Mauritius|Indian/Mayotte|Indian/Reunion|Mexico/BajaNorte|Mexico/BajaSur|Mexico/General|Pacific/Apia|Pacific/Auckland|Pacific/Bougainville|Pacific/Chatham|Pacific/Chuuk|Pacific/Easter|Pacific/Efate|Pacific/Enderbury|Pacific/Fakaofo|Pacific/Fiji|Pacific/Funafuti|Pacific/Galapagos|Pacific/Gambier|Pacific/Guadalcanal|Pacific/Guam|Pacific/Honolulu|Pacific/Johnston|Pacific/Kiritimati|Pacific/Kosrae|Pacific/Kwajalein|Pacific/Majuro|Pacific/Marquesas|Pacific/Midway|Pacific/Nauru|Pacific/Niue|Pacific/Norfolk|Pacific/Noumea|Pacific/Pago_Pago|Pacific/Palau|Pacific/Pitcairn|Pacific/Pohnpei|Pacific/Ponape|Pacific/Port_Moresby|Pacific/Rarotonga|Pacific/Saipan|Pacific/Samoa|Pacific/Tahiti|Pacific/Tarawa|Pacific/Tongatapu|Pacific/Truk|Pacific/Wake|Pacific/Wallis|Pacific/Yap|US/Alaska|US/Aleutian|US/Arizona|US/Central|US/East-Indiana|US/Eastern|US/Hawaii|US/Indiana-Starke|US/Michigan|US/Mountain|US/Pacific|US/Pacific-New|US/Samoa)
nv set vrf <vrf-id>
nv set vrf <vrf-id> loopback
nv set vrf <vrf-id> loopback ip
nv set vrf <vrf-id> loopback ip address <ip-prefix-id>
nv set vrf <vrf-id> evpn
nv set vrf <vrf-id> evpn vni <vni-id>
nv set vrf <vrf-id> evpn vni <vni-id> prefix-routes-only (on|off)
nv set vrf <vrf-id> evpn enable (on|off)
nv set vrf <vrf-id> evpn vlan (1-4094|auto)
nv set vrf <vrf-id> router
nv set vrf <vrf-id> router rib <afi>
nv set vrf <vrf-id> router rib <afi> protocol <import-protocol-id>
nv set vrf <vrf-id> router rib <afi> protocol <import-protocol-id> fib-filter (none|<instance-name>)
nv set vrf <vrf-id> router bgp
nv set vrf <vrf-id> router bgp address-family
nv set vrf <vrf-id> router bgp address-family ipv4-unicast
nv set vrf <vrf-id> router bgp address-family ipv4-unicast redistribute
nv set vrf <vrf-id> router bgp address-family ipv4-unicast redistribute static
nv set vrf <vrf-id> router bgp address-family ipv4-unicast redistribute static enable (on|off)
nv set vrf <vrf-id> router bgp address-family ipv4-unicast redistribute static metric (0-4294967295|auto)
nv set vrf <vrf-id> router bgp address-family ipv4-unicast redistribute static route-map (none|<instance-name>)
nv set vrf <vrf-id> router bgp address-family ipv4-unicast redistribute connected
nv set vrf <vrf-id> router bgp address-family ipv4-unicast redistribute connected enable (on|off)
nv set vrf <vrf-id> router bgp address-family ipv4-unicast redistribute connected metric (0-4294967295|auto)
nv set vrf <vrf-id> router bgp address-family ipv4-unicast redistribute connected route-map (none|<instance-name>)
nv set vrf <vrf-id> router bgp address-family ipv4-unicast redistribute kernel
nv set vrf <vrf-id> router bgp address-family ipv4-unicast redistribute kernel enable (on|off)
nv set vrf <vrf-id> router bgp address-family ipv4-unicast redistribute kernel metric (0-4294967295|auto)
nv set vrf <vrf-id> router bgp address-family ipv4-unicast redistribute kernel route-map (none|<instance-name>)
nv set vrf <vrf-id> router bgp address-family ipv4-unicast redistribute ospf
nv set vrf <vrf-id> router bgp address-family ipv4-unicast redistribute ospf enable (on|off)
nv set vrf <vrf-id> router bgp address-family ipv4-unicast redistribute ospf metric (0-4294967295|auto)
nv set vrf <vrf-id> router bgp address-family ipv4-unicast redistribute ospf route-map (none|<instance-name>)
nv set vrf <vrf-id> router bgp address-family ipv4-unicast aggregate-route <aggregate-route-id>
nv set vrf <vrf-id> router bgp address-family ipv4-unicast aggregate-route <aggregate-route-id> summary-only (on|off)
nv set vrf <vrf-id> router bgp address-family ipv4-unicast aggregate-route <aggregate-route-id> as-set (on|off)
nv set vrf <vrf-id> router bgp address-family ipv4-unicast aggregate-route <aggregate-route-id> route-map (none|<instance-name>)
nv set vrf <vrf-id> router bgp address-family ipv4-unicast network <static-network-id>
nv set vrf <vrf-id> router bgp address-family ipv4-unicast network <static-network-id> route-map <instance-name>
nv set vrf <vrf-id> router bgp address-family ipv4-unicast route-import
nv set vrf <vrf-id> router bgp address-family ipv4-unicast route-import from-vrf
nv set vrf <vrf-id> router bgp address-family ipv4-unicast route-import from-vrf list <leak-vrf-id>
nv set vrf <vrf-id> router bgp address-family ipv4-unicast route-import from-vrf enable (on|off)
nv set vrf <vrf-id> router bgp address-family ipv4-unicast route-import from-vrf route-map <instance-name>
nv set vrf <vrf-id> router bgp address-family ipv4-unicast multipaths
nv set vrf <vrf-id> router bgp address-family ipv4-unicast multipaths ebgp 1-128
nv set vrf <vrf-id> router bgp address-family ipv4-unicast multipaths ibgp 1-128
nv set vrf <vrf-id> router bgp address-family ipv4-unicast multipaths compare-cluster-length (on|off)
nv set vrf <vrf-id> router bgp address-family ipv4-unicast admin-distance
nv set vrf <vrf-id> router bgp address-family ipv4-unicast admin-distance external 1-255
nv set vrf <vrf-id> router bgp address-family ipv4-unicast admin-distance internal 1-255
nv set vrf <vrf-id> router bgp address-family ipv4-unicast route-export
nv set vrf <vrf-id> router bgp address-family ipv4-unicast route-export to-evpn
nv set vrf <vrf-id> router bgp address-family ipv4-unicast route-export to-evpn enable (on|off)
nv set vrf <vrf-id> router bgp address-family ipv4-unicast route-export to-evpn route-map (none|<instance-name>)
nv set vrf <vrf-id> router bgp address-family ipv4-unicast route-export to-evpn default-route-origination (on|off)
nv set vrf <vrf-id> router bgp address-family ipv4-unicast rib-filter (none|<instance-name>)
nv set vrf <vrf-id> router bgp address-family ipv4-unicast enable (on|off)
nv set vrf <vrf-id> router bgp address-family l2vpn-evpn
nv set vrf <vrf-id> router bgp address-family l2vpn-evpn enable (on|off)
nv set vrf <vrf-id> router bgp address-family ipv6-unicast
nv set vrf <vrf-id> router bgp address-family ipv6-unicast aggregate-route <aggregate-route-id>
nv set vrf <vrf-id> router bgp address-family ipv6-unicast aggregate-route <aggregate-route-id> summary-only (on|off)
nv set vrf <vrf-id> router bgp address-family ipv6-unicast aggregate-route <aggregate-route-id> as-set (on|off)
nv set vrf <vrf-id> router bgp address-family ipv6-unicast aggregate-route <aggregate-route-id> route-map (none|<instance-name>)
nv set vrf <vrf-id> router bgp address-family ipv6-unicast network <static-network-id>
nv set vrf <vrf-id> router bgp address-family ipv6-unicast network <static-network-id> route-map <instance-name>
nv set vrf <vrf-id> router bgp address-family ipv6-unicast route-import
nv set vrf <vrf-id> router bgp address-family ipv6-unicast route-import from-vrf
nv set vrf <vrf-id> router bgp address-family ipv6-unicast route-import from-vrf list
nv set vrf <vrf-id> router bgp address-family ipv6-unicast route-import from-vrf enable (on|off)
nv set vrf <vrf-id> router bgp address-family ipv6-unicast route-import from-vrf route-map <instance-name>
nv set vrf <vrf-id> router bgp address-family ipv6-unicast multipaths
nv set vrf <vrf-id> router bgp address-family ipv6-unicast multipaths ebgp 1-128
nv set vrf <vrf-id> router bgp address-family ipv6-unicast multipaths ibgp 1-128
nv set vrf <vrf-id> router bgp address-family ipv6-unicast multipaths compare-cluster-length (on|off)
nv set vrf <vrf-id> router bgp address-family ipv6-unicast admin-distance
nv set vrf <vrf-id> router bgp address-family ipv6-unicast admin-distance external 1-255
nv set vrf <vrf-id> router bgp address-family ipv6-unicast admin-distance internal 1-255
nv set vrf <vrf-id> router bgp address-family ipv6-unicast route-export
nv set vrf <vrf-id> router bgp address-family ipv6-unicast route-export to-evpn
nv set vrf <vrf-id> router bgp address-family ipv6-unicast route-export to-evpn enable (on|off)
nv set vrf <vrf-id> router bgp address-family ipv6-unicast route-export to-evpn route-map (none|<instance-name>)
nv set vrf <vrf-id> router bgp address-family ipv6-unicast route-export to-evpn default-route-origination (on|off)
nv set vrf <vrf-id> router bgp address-family ipv6-unicast redistribute
nv set vrf <vrf-id> router bgp address-family ipv6-unicast redistribute static
nv set vrf <vrf-id> router bgp address-family ipv6-unicast redistribute static enable (on|off)
nv set vrf <vrf-id> router bgp address-family ipv6-unicast redistribute static metric (0-4294967295|auto)
nv set vrf <vrf-id> router bgp address-family ipv6-unicast redistribute static route-map (none|<instance-name>)
nv set vrf <vrf-id> router bgp address-family ipv6-unicast redistribute connected
nv set vrf <vrf-id> router bgp address-family ipv6-unicast redistribute connected enable (on|off)
nv set vrf <vrf-id> router bgp address-family ipv6-unicast redistribute connected metric (0-4294967295|auto)
nv set vrf <vrf-id> router bgp address-family ipv6-unicast redistribute connected route-map (none|<instance-name>)
nv set vrf <vrf-id> router bgp address-family ipv6-unicast redistribute kernel
nv set vrf <vrf-id> router bgp address-family ipv6-unicast redistribute kernel enable (on|off)
nv set vrf <vrf-id> router bgp address-family ipv6-unicast redistribute kernel metric (0-4294967295|auto)
nv set vrf <vrf-id> router bgp address-family ipv6-unicast redistribute kernel route-map (none|<instance-name>)
nv set vrf <vrf-id> router bgp address-family ipv6-unicast redistribute ospf6
nv set vrf <vrf-id> router bgp address-family ipv6-unicast redistribute ospf6 enable (on|off)
nv set vrf <vrf-id> router bgp address-family ipv6-unicast redistribute ospf6 metric (0-4294967295|auto)
nv set vrf <vrf-id> router bgp address-family ipv6-unicast redistribute ospf6 route-map (none|<instance-name>)
nv set vrf <vrf-id> router bgp address-family ipv6-unicast rib-filter (none|<instance-name>)
nv set vrf <vrf-id> router bgp address-family ipv6-unicast enable (on|off)
nv set vrf <vrf-id> router bgp path-selection
nv set vrf <vrf-id> router bgp path-selection aspath
nv set vrf <vrf-id> router bgp path-selection aspath compare-lengths (on|off)
nv set vrf <vrf-id> router bgp path-selection aspath compare-confed (on|off)
nv set vrf <vrf-id> router bgp path-selection med
nv set vrf <vrf-id> router bgp path-selection med compare-always (on|off)
nv set vrf <vrf-id> router bgp path-selection med compare-deterministic (on|off)
nv set vrf <vrf-id> router bgp path-selection med compare-confed (on|off)
nv set vrf <vrf-id> router bgp path-selection med missing-as-max (on|off)
nv set vrf <vrf-id> router bgp path-selection multipath
nv set vrf <vrf-id> router bgp path-selection multipath aspath-ignore (on|off)
nv set vrf <vrf-id> router bgp path-selection multipath generate-asset (on|off)
nv set vrf <vrf-id> router bgp path-selection multipath bandwidth (bandwidth|all-paths|skip-missing|default-weight-for-missing|ignore)
nv set vrf <vrf-id> router bgp path-selection routerid-compare (on|off)
nv set vrf <vrf-id> router bgp route-reflection
nv set vrf <vrf-id> router bgp route-reflection enable (on|off)
nv set vrf <vrf-id> router bgp route-reflection cluster-id (0-4294967295|<ipv4>)
nv set vrf <vrf-id> router bgp route-reflection reflect-between-clients (on|off)
nv set vrf <vrf-id> router bgp route-reflection outbound-policy (on|off)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id>
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> bfd
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> bfd enable (on|off)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> bfd detect-multiplier 2-255
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> bfd min-rx-interval 50-60000
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> bfd min-tx-interval 50-60000
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> ttl-security
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> ttl-security enable (on|off)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> ttl-security hops 1-254
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> capabilities
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> capabilities extended-nexthop (on|off|auto)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> capabilities source-address (<interface-name>|<ipv4>|<ipv6>)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> graceful-restart
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> graceful-restart mode (auto|off|helper-only|full)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> local-as
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> local-as enable (on|off)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> local-as asn 1-4294967295
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> local-as prepend (on|off)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> local-as replace (on|off)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> timers
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> timers keepalive (1-65535|none|auto)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> timers hold (3-65535|none|auto)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> timers connection-retry (1-65535|auto)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> timers route-advertisement (1-600|none|auto)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast community-advertise
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast community-advertise regular (on|off)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast community-advertise extended (on|off)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast community-advertise large (on|off)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast attribute-mod
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast attribute-mod aspath (on|off)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast attribute-mod med (on|off)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast attribute-mod nexthop (on|off)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast aspath
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast aspath allow-my-asn
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast aspath allow-my-asn enable (on|off)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast aspath allow-my-asn origin (on|off)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast aspath allow-my-asn occurrences 1-10
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast aspath replace-peer-as (on|off)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast aspath private-as (none|remove|replace)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast prefix-limits
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast prefix-limits inbound
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast prefix-limits inbound maximum (0-4294967295|none)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast prefix-limits inbound warning-threshold 1-100
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast prefix-limits inbound warning-only (on|off)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast prefix-limits inbound reestablish-wait 1-4294967295
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast default-route-origination
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast default-route-origination enable (on|off)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast default-route-origination policy (none|<instance-name>)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast policy
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast policy inbound
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast policy inbound route-map (none|<instance-name>)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast policy inbound prefix-list (none|<instance-name>)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast policy inbound aspath-list none
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast policy outbound
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast policy outbound route-map (none|<instance-name>)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast policy outbound unsuppress-map (none|<instance-name>)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast policy outbound prefix-list (none|<instance-name>)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast policy outbound aspath-list none
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast conditional-advertise
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast conditional-advertise enable (on|off)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast conditional-advertise advertise-map <instance-name>
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast conditional-advertise exist-map <instance-name>
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast conditional-advertise non-exist-map <instance-name>
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast enable (on|off)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast route-reflector-client (on|off)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast route-server-client (on|off)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast soft-reconfiguration (on|off)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast nexthop-setting (auto|self|force)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast add-path-tx (off|all-paths|best-per-as)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast weight 0-65535
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast policy
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast policy inbound
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast policy inbound route-map (none|<instance-name>)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast policy inbound prefix-list (none|<instance-name>)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast policy inbound aspath-list none
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast policy outbound
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast policy outbound route-map (none|<instance-name>)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast policy outbound unsuppress-map (none|<instance-name>)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast policy outbound prefix-list (none|<instance-name>)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast policy outbound aspath-list none
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast aspath
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast aspath allow-my-asn
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast aspath allow-my-asn enable (on|off)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast aspath allow-my-asn origin (on|off)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast aspath allow-my-asn occurrences 1-10
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast aspath replace-peer-as (on|off)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast aspath private-as (none|remove|replace)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast prefix-limits
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast prefix-limits inbound
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast prefix-limits inbound maximum (0-4294967295|none)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast prefix-limits inbound warning-threshold 1-100
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast prefix-limits inbound warning-only (on|off)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast prefix-limits inbound reestablish-wait 1-4294967295
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast default-route-origination
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast default-route-origination enable (on|off)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast default-route-origination policy (none|<instance-name>)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast community-advertise
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast community-advertise regular (on|off)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast community-advertise extended (on|off)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast community-advertise large (on|off)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast attribute-mod
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast attribute-mod aspath (on|off)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast attribute-mod med (on|off)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast attribute-mod nexthop (on|off)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast conditional-advertise
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast conditional-advertise enable (on|off)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast conditional-advertise advertise-map <instance-name>
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast conditional-advertise exist-map <instance-name>
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast conditional-advertise non-exist-map <instance-name>
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast enable (on|off)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast route-reflector-client (on|off)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast route-server-client (on|off)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast soft-reconfiguration (on|off)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast nexthop-setting (auto|self|force)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast add-path-tx (off|all-paths|best-per-as)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast weight 0-65535
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn attribute-mod
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn attribute-mod aspath (on|off)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn attribute-mod med (on|off)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn attribute-mod nexthop (on|off)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn aspath
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn aspath allow-my-asn
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn aspath allow-my-asn enable (on|off)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn aspath allow-my-asn origin (on|off)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn aspath allow-my-asn occurrences 1-10
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn aspath replace-peer-as (on|off)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn aspath private-as (none|remove|replace)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn policy
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn policy inbound
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn policy inbound route-map (none|<instance-name>)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn policy outbound
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn policy outbound route-map (none|<instance-name>)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn policy outbound unsuppress-map (none|<instance-name>)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn enable (on|off)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn route-reflector-client (on|off)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn route-server-client (on|off)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn soft-reconfiguration (on|off)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn nexthop-setting (auto|self|force)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn add-path-tx (off|all-paths|best-per-as)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> password none
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> enforce-first-as (on|off)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> passive-mode (on|off)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> nexthop-connected-check (on|off)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> multihop-ttl (1-255|auto)
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> description none
nv set vrf <vrf-id> router bgp peer-group <peer-group-id> remote-as (1-4294967295|internal|external)
nv set vrf <vrf-id> router bgp route-export
nv set vrf <vrf-id> router bgp route-export to-evpn
nv set vrf <vrf-id> router bgp route-export to-evpn route-target <rt-id>
nv set vrf <vrf-id> router bgp route-import
nv set vrf <vrf-id> router bgp route-import from-evpn
nv set vrf <vrf-id> router bgp route-import from-evpn route-target <rt-id>
nv set vrf <vrf-id> router bgp timers
nv set vrf <vrf-id> router bgp timers keepalive (1-65535|none)
nv set vrf <vrf-id> router bgp timers hold (3-65535|none)
nv set vrf <vrf-id> router bgp timers connection-retry 1-65535
nv set vrf <vrf-id> router bgp timers route-advertisement (1-600|none)
nv set vrf <vrf-id> router bgp timers conditional-advertise (5-240|none)
nv set vrf <vrf-id> router bgp confederation
nv set vrf <vrf-id> router bgp confederation member-as
nv set vrf <vrf-id> router bgp confederation id (1-4294967295|none)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id>
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> bfd
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> bfd enable (on|off)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> bfd detect-multiplier 2-255
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> bfd min-rx-interval 50-60000
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> bfd min-tx-interval 50-60000
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> capabilities
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> capabilities extended-nexthop (on|off|auto)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> capabilities source-address (<interface-name>|<ipv4>|<ipv6>)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> local-as
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> local-as enable (on|off)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> local-as asn 1-4294967295
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> local-as prepend (on|off)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> local-as replace (on|off)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> graceful-restart
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> graceful-restart mode (auto|off|helper-only|full)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> ttl-security
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> ttl-security enable (on|off)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> ttl-security hops 1-254
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast attribute-mod
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast attribute-mod aspath (on|off)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast attribute-mod med (on|off)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast attribute-mod nexthop (on|off)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast aspath
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast aspath allow-my-asn
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast aspath allow-my-asn enable (on|off)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast aspath allow-my-asn origin (on|off)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast aspath allow-my-asn occurrences 1-10
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast aspath replace-peer-as (on|off)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast aspath private-as (none|remove|replace)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast policy
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast policy inbound
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast policy inbound route-map (none|<instance-name>)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast policy inbound prefix-list (none|<instance-name>)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast policy inbound aspath-list none
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast policy outbound
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast policy outbound route-map (none|<instance-name>)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast policy outbound unsuppress-map (none|<instance-name>)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast policy outbound prefix-list (none|<instance-name>)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast policy outbound aspath-list none
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast prefix-limits
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast prefix-limits inbound
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast prefix-limits inbound maximum (0-4294967295|none)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast prefix-limits inbound warning-threshold 1-100
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast prefix-limits inbound warning-only (on|off)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast prefix-limits inbound reestablish-wait 1-4294967295
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast default-route-origination
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast default-route-origination enable (on|off)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast default-route-origination policy (none|<instance-name>)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast community-advertise
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast community-advertise regular (on|off)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast community-advertise extended (on|off)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast community-advertise large (on|off)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast conditional-advertise
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast conditional-advertise enable (on|off)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast conditional-advertise advertise-map <instance-name>
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast conditional-advertise exist-map <instance-name>
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast conditional-advertise non-exist-map <instance-name>
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast enable (on|off)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast route-reflector-client (on|off)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast route-server-client (on|off)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast soft-reconfiguration (on|off)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast nexthop-setting (auto|self|force)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast add-path-tx (off|all-paths|best-per-as)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast weight 0-65535
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast attribute-mod
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast attribute-mod aspath (on|off)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast attribute-mod med (on|off)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast attribute-mod nexthop (on|off)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast aspath
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast aspath allow-my-asn
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast aspath allow-my-asn enable (on|off)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast aspath allow-my-asn origin (on|off)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast aspath allow-my-asn occurrences 1-10
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast aspath replace-peer-as (on|off)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast aspath private-as (none|remove|replace)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast prefix-limits
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast prefix-limits inbound
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast prefix-limits inbound maximum (0-4294967295|none)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast prefix-limits inbound warning-threshold 1-100
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast prefix-limits inbound warning-only (on|off)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast prefix-limits inbound reestablish-wait 1-4294967295
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast default-route-origination
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast default-route-origination enable (on|off)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast default-route-origination policy (none|<instance-name>)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast policy
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast policy inbound
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast policy inbound route-map (none|<instance-name>)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast policy inbound prefix-list (none|<instance-name>)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast policy inbound aspath-list none
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast policy outbound
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast policy outbound route-map (none|<instance-name>)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast policy outbound unsuppress-map (none|<instance-name>)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast policy outbound prefix-list (none|<instance-name>)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast policy outbound aspath-list none
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast community-advertise
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast community-advertise regular (on|off)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast community-advertise extended (on|off)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast community-advertise large (on|off)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast conditional-advertise
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast conditional-advertise enable (on|off)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast conditional-advertise advertise-map <instance-name>
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast conditional-advertise exist-map <instance-name>
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast conditional-advertise non-exist-map <instance-name>
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast enable (on|off)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast route-reflector-client (on|off)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast route-server-client (on|off)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast soft-reconfiguration (on|off)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast nexthop-setting (auto|self|force)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast add-path-tx (off|all-paths|best-per-as)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast weight 0-65535
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn attribute-mod
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn attribute-mod aspath (on|off)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn attribute-mod med (on|off)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn attribute-mod nexthop (on|off)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn aspath
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn aspath allow-my-asn
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn aspath allow-my-asn enable (on|off)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn aspath allow-my-asn origin (on|off)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn aspath allow-my-asn occurrences 1-10
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn aspath replace-peer-as (on|off)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn aspath private-as (none|remove|replace)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn policy
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn policy inbound
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn policy inbound route-map (none|<instance-name>)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn policy outbound
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn policy outbound route-map (none|<instance-name>)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn policy outbound unsuppress-map (none|<instance-name>)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn enable (on|off)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn route-reflector-client (on|off)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn route-server-client (on|off)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn soft-reconfiguration (on|off)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn nexthop-setting (auto|self|force)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn add-path-tx (off|all-paths|best-per-as)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> timers
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> timers keepalive (1-65535|none|auto)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> timers hold (3-65535|none|auto)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> timers connection-retry (1-65535|auto)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> timers route-advertisement (1-600|none|auto)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> password none
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> enforce-first-as (on|off)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> passive-mode (on|off)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> nexthop-connected-check (on|off)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> multihop-ttl (1-255|auto)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> description none
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> enable (on|off)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> type (numbered|unnumbered)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> peer-group (none|<instance-name>)
nv set vrf <vrf-id> router bgp neighbor <neighbor-id> remote-as (1-4294967295|auto|internal|external)
nv set vrf <vrf-id> router bgp enable (on|off)
nv set vrf <vrf-id> router bgp autonomous-system (1-4294967295|auto|leaf|spine)
nv set vrf <vrf-id> router bgp router-id (auto|<ipv4>)
nv set vrf <vrf-id> router bgp rd (none|<route-distinguisher>)
nv set vrf <vrf-id> router bgp dynamic-peer-limit 1-5000
nv set vrf <vrf-id> router static <route-id>
nv set vrf <vrf-id> router static <route-id> distance <distance-id>
nv set vrf <vrf-id> router static <route-id> distance <distance-id> via <via-id>
nv set vrf <vrf-id> router static <route-id> distance <distance-id> via <via-id> flag onlink
nv set vrf <vrf-id> router static <route-id> distance <distance-id> via <via-id> interface (auto|<interface-name>)
nv set vrf <vrf-id> router static <route-id> distance <distance-id> via <via-id> vrf (auto|<vrf-name>)
nv set vrf <vrf-id> router static <route-id> distance <distance-id> via <via-id> type (interface|ipv4-address|ipv6-address|blackhole|reject)
nv set vrf <vrf-id> router static <route-id> distance <distance-id> tag (1-4294967295|none)
nv set vrf <vrf-id> router static <route-id> via <via-id>
nv set vrf <vrf-id> router static <route-id> via <via-id> flag onlink
nv set vrf <vrf-id> router static <route-id> via <via-id> interface (auto|<interface-name>)
nv set vrf <vrf-id> router static <route-id> via <via-id> vrf (auto|<vrf-name>)
nv set vrf <vrf-id> router static <route-id> via <via-id> type (interface|ipv4-address|ipv6-address|blackhole|reject)
nv set vrf <vrf-id> router static <route-id> tag (1-4294967295|none)
nv set vrf <vrf-id> router static <route-id> address-family (ipv4-unicast|ipv6-unicast)
nv set vrf <vrf-id> router pim
nv set vrf <vrf-id> router pim timers
nv set vrf <vrf-id> router pim timers keep-alive (31-60000|auto)
nv set vrf <vrf-id> router pim timers rp-keep-alive (31-60000|auto)
nv set vrf <vrf-id> router pim ecmp
nv set vrf <vrf-id> router pim ecmp enable (on|off)
nv set vrf <vrf-id> router pim ecmp rebalance (on|off)
nv set vrf <vrf-id> router pim msdp-mesh-group <msdp-mesh-group-id>
nv set vrf <vrf-id> router pim msdp-mesh-group <msdp-mesh-group-id> member-address <mesh-member-id>
nv set vrf <vrf-id> router pim msdp-mesh-group <msdp-mesh-group-id> source-address <ipv4>
nv set vrf <vrf-id> router pim address-family
nv set vrf <vrf-id> router pim address-family ipv4-unicast
nv set vrf <vrf-id> router pim address-family ipv4-unicast spt-switchover
nv set vrf <vrf-id> router pim address-family ipv4-unicast spt-switchover action (immediate|infinity)
nv set vrf <vrf-id> router pim address-family ipv4-unicast spt-switchover prefix-list <instance-name>
nv set vrf <vrf-id> router pim address-family ipv4-unicast rp <rp-id>
nv set vrf <vrf-id> router pim address-family ipv4-unicast rp <rp-id> group-range <group-range-id>
nv set vrf <vrf-id> router pim address-family ipv4-unicast rp <rp-id> prefix-list <instance-name>
nv set vrf <vrf-id> router pim address-family ipv4-unicast ssm-prefix-list (none|<instance-name>)
nv set vrf <vrf-id> router pim address-family ipv4-unicast register-accept-list (none|<instance-name>)
nv set vrf <vrf-id> router pim address-family ipv4-unicast send-v6-secondary (on|off)
nv set vrf <vrf-id> router pim enable (on|off)
nv set vrf <vrf-id> router ospf
nv set vrf <vrf-id> router ospf area <area-id>
nv set vrf <vrf-id> router ospf area <area-id> filter-list
nv set vrf <vrf-id> router ospf area <area-id> filter-list in (none|<instance-name>)
nv set vrf <vrf-id> router ospf area <area-id> filter-list out (none|<instance-name>)
nv set vrf <vrf-id> router ospf area <area-id> range <range-id>
nv set vrf <vrf-id> router ospf area <area-id> range <range-id> suppress (on|off)
nv set vrf <vrf-id> router ospf area <area-id> range <range-id> cost (0-16777215|auto)
nv set vrf <vrf-id> router ospf area <area-id> network <network-id>
nv set vrf <vrf-id> router ospf area <area-id> type (normal|stub|totally-stub|nssa|totally-nssa)
nv set vrf <vrf-id> router ospf area <area-id> default-lsa-cost 0-16777215
nv set vrf <vrf-id> router ospf default-originate
nv set vrf <vrf-id> router ospf default-originate enable (on|off)
nv set vrf <vrf-id> router ospf default-originate metric (0-16777214|none)
nv set vrf <vrf-id> router ospf default-originate metric-type 1-2
nv set vrf <vrf-id> router ospf default-originate route-map (none|<instance-name>)
nv set vrf <vrf-id> router ospf default-originate always (on|off)
nv set vrf <vrf-id> router ospf distance
nv set vrf <vrf-id> router ospf distance external (1-255|none)
nv set vrf <vrf-id> router ospf distance inter-area (1-255|none)
nv set vrf <vrf-id> router ospf distance intra-area (1-255|none)
nv set vrf <vrf-id> router ospf max-metric
nv set vrf <vrf-id> router ospf max-metric administrative (on|off)
nv set vrf <vrf-id> router ospf max-metric on-shutdown (5-100|none)
nv set vrf <vrf-id> router ospf max-metric on-startup (5-86400|none)
nv set vrf <vrf-id> router ospf log
nv set vrf <vrf-id> router ospf log adjacency-changes (on|off|detail)
nv set vrf <vrf-id> router ospf redistribute
nv set vrf <vrf-id> router ospf redistribute static
nv set vrf <vrf-id> router ospf redistribute static enable (on|off)
nv set vrf <vrf-id> router ospf redistribute static metric (0-16777214|none)
nv set vrf <vrf-id> router ospf redistribute static metric-type 1-2
nv set vrf <vrf-id> router ospf redistribute static route-map (none|<instance-name>)
nv set vrf <vrf-id> router ospf redistribute connected
nv set vrf <vrf-id> router ospf redistribute connected enable (on|off)
nv set vrf <vrf-id> router ospf redistribute connected metric (0-16777214|none)
nv set vrf <vrf-id> router ospf redistribute connected metric-type 1-2
nv set vrf <vrf-id> router ospf redistribute connected route-map (none|<instance-name>)
nv set vrf <vrf-id> router ospf redistribute kernel
nv set vrf <vrf-id> router ospf redistribute kernel enable (on|off)
nv set vrf <vrf-id> router ospf redistribute kernel metric (0-16777214|none)
nv set vrf <vrf-id> router ospf redistribute kernel metric-type 1-2
nv set vrf <vrf-id> router ospf redistribute kernel route-map (none|<instance-name>)
nv set vrf <vrf-id> router ospf redistribute bgp
nv set vrf <vrf-id> router ospf redistribute bgp enable (on|off)
nv set vrf <vrf-id> router ospf redistribute bgp metric (0-16777214|none)
nv set vrf <vrf-id> router ospf redistribute bgp metric-type 1-2
nv set vrf <vrf-id> router ospf redistribute bgp route-map (none|<instance-name>)
nv set vrf <vrf-id> router ospf timers
nv set vrf <vrf-id> router ospf timers lsa
nv set vrf <vrf-id> router ospf timers lsa min-arrival (0-600000|auto)
nv set vrf <vrf-id> router ospf timers lsa throttle (0-5000|auto)
nv set vrf <vrf-id> router ospf timers spf
nv set vrf <vrf-id> router ospf timers spf delay (0-600000|auto)
nv set vrf <vrf-id> router ospf timers spf holdtime (0-600000|auto)
nv set vrf <vrf-id> router ospf timers spf max-holdtime (0-600000|auto)
nv set vrf <vrf-id> router ospf timers refresh (10-1800|auto)
nv set vrf <vrf-id> router ospf enable (on|off)
nv set vrf <vrf-id> router ospf reference-bandwidth 1-4294967
nv set vrf <vrf-id> router ospf rfc1583-compatible (on|off)
nv set vrf <vrf-id> router ospf router-id (auto|<ipv4>)
nv set vrf <vrf-id> ptp
nv set vrf <vrf-id> ptp enable (on|off)
nv set vrf <vrf-id> table auto
nv set nve
nv set nve vxlan
nv set nve vxlan mlag
nv set nve vxlan mlag shared-address (none|<ipv4-unicast>)
nv set nve vxlan source
nv set nve vxlan source address (auto|<ipv4>)
nv set nve vxlan flooding
nv set nve vxlan flooding head-end-replication <hrep-id>
nv set nve vxlan flooding enable (on|off)
nv set nve vxlan flooding multicast-group <ipv4-multicast>
nv set nve vxlan enable (on|off)
nv set nve vxlan mac-learning (on|off)
nv set nve vxlan port 1024-65535
nv set nve vxlan arp-nd-suppress (on|off)
nv set nve vxlan mtu 552-9216
nv set acl <acl-id>
nv set acl <acl-id> rule <rule-id>
nv set acl <acl-id> rule <rule-id> match
nv set acl <acl-id> rule <rule-id> match ip
nv set acl <acl-id> rule <rule-id> match ip source-port <ip-port-id>
nv set acl <acl-id> rule <rule-id> match ip dest-port <ip-port-id>
nv set acl <acl-id> rule <rule-id> match ip fragment
nv set acl <acl-id> rule <rule-id> match ip ecn
nv set acl <acl-id> rule <rule-id> match ip ecn flags (tcp-cwr|tcp-ece)
nv set acl <acl-id> rule <rule-id> match ip ecn ip-ect 0-3
nv set acl <acl-id> rule <rule-id> match ip tcp
nv set acl <acl-id> rule <rule-id> match ip tcp flags (syn|ack|fin|rst|urg|psh|all|none)
nv set acl <acl-id> rule <rule-id> match ip tcp mask (syn|ack|fin|rst|urg|psh|all|none)
nv set acl <acl-id> rule <rule-id> match ip tcp state established
nv set acl <acl-id> rule <rule-id> match ip source-ip (ANY|<ipv4>|<ipv6>|<ipv4-prefix>|<ipv6-prefix>|<ipv4-netmask>|<ipv6-netmask>)
nv set acl <acl-id> rule <rule-id> match ip dest-ip (ANY|<ipv4>|<ipv6>|<ipv4-prefix>|<ipv6-prefix>|<ipv4-netmask>|<ipv6-netmask>)
nv set acl <acl-id> rule <rule-id> match ip protocol (0-255|tcp|udp|ospf|pim|icmp|icmpv6|igmp)
nv set acl <acl-id> rule <rule-id> match ip dscp (0-64|ANY|af11|af12|af13|af21|af22|af23|af31|af32|af33|af41|af42|af43|cs1|cs2|cs3|cs4|cs5|cs6|cs7|be|ef)
nv set acl <acl-id> rule <rule-id> match ip icmp-type (0-255|echo-reply|echo-request|time-exceeded|dest-unreachable|port-unreachable)
nv set acl <acl-id> rule <rule-id> match ip icmpv6-type (0-255|router-solicitation|router-advertisement|neighbor-solicitation|neighbor-advertisement)
nv set acl <acl-id> rule <rule-id> match mac
nv set acl <acl-id> rule <rule-id> match mac source-mac (ANY|bpdu|cdp|cisco-pvst|lacp|lldp|<mac>)
nv set acl <acl-id> rule <rule-id> match mac source-mac-mask <mac>
nv set acl <acl-id> rule <rule-id> match mac dest-mac (ANY|bpdu|cdp|cisco-pvst|lacp|lldp|<mac>)
nv set acl <acl-id> rule <rule-id> match mac dest-mac-mask <mac>
nv set acl <acl-id> rule <rule-id> match mac protocol (0-255|ANY|arp|ipv4|ipv6)
nv set acl <acl-id> rule <rule-id> match mac vlan 1-4094
nv set acl <acl-id> rule <rule-id> action
nv set acl <acl-id> rule <rule-id> action permit
nv set acl <acl-id> rule <rule-id> action deny
nv set acl <acl-id> rule <rule-id> action log
nv set acl <acl-id> rule <rule-id> action set
nv set acl <acl-id> rule <rule-id> action set dscp (0-64|ANY|af11|af12|af13|af21|af22|af23|af31|af32|af33|af41|af42|af43|cs1|cs2|cs3|cs4|cs5|cs6|cs7|be|ef)
nv set acl <acl-id> rule <rule-id> action set class 0-7
nv set acl <acl-id> rule <rule-id> action set cos 0-7
nv set acl <acl-id> rule <rule-id> action erspan
nv set acl <acl-id> rule <rule-id> action erspan source-ip (<ipv4>|<ipv6>)
nv set acl <acl-id> rule <rule-id> action erspan dest-ip (<ipv4>|<ipv6>)
nv set acl <acl-id> rule <rule-id> action erspan ttl 1-255
nv set acl <acl-id> rule <rule-id> action police
nv set acl <acl-id> rule <rule-id> action police mode (packet|bps|kbps|mbps|gbps)
nv set acl <acl-id> rule <rule-id> action police burst 1-2147483647
nv set acl <acl-id> rule <rule-id> action police rate 1-2147483647
nv set acl <acl-id> rule <rule-id> action span <interface-name>
nv set acl <acl-id> type (ipv4|ipv6|mac)
nv unset router
nv unset router nexthop-group
nv unset router nexthop-group <nexthop-group-id>
nv unset router nexthop-group <nexthop-group-id> via
nv unset router nexthop-group <nexthop-group-id> via <via-id>
nv unset router nexthop-group <nexthop-group-id> via <via-id> interface
nv unset router nexthop-group <nexthop-group-id> via <via-id> vrf
nv unset router pbr
nv unset router pbr map
nv unset router pbr map <pbr-map-id>
nv unset router pbr map <pbr-map-id> rule
nv unset router pbr map <pbr-map-id> rule <rule-id>
nv unset router pbr map <pbr-map-id> rule <rule-id> match
nv unset router pbr map <pbr-map-id> rule <rule-id> match source-ip
nv unset router pbr map <pbr-map-id> rule <rule-id> match destination-ip
nv unset router pbr map <pbr-map-id> rule <rule-id> match dscp
nv unset router pbr map <pbr-map-id> rule <rule-id> match ecn
nv unset router pbr map <pbr-map-id> rule <rule-id> action
nv unset router pbr map <pbr-map-id> rule <rule-id> action nexthop-group
nv unset router pbr map <pbr-map-id> rule <rule-id> action nexthop-group <nexthop-group-id>
nv unset router pbr map <pbr-map-id> rule <rule-id> action vrf
nv unset router pbr enable
nv unset router policy
nv unset router policy community-list
nv unset router policy community-list <list-id>
nv unset router policy community-list <list-id> rule
nv unset router policy community-list <list-id> rule <rule-id>
nv unset router policy community-list <list-id> rule <rule-id> community
nv unset router policy community-list <list-id> rule <rule-id> community <community-id>
nv unset router policy community-list <list-id> rule <rule-id> action
nv unset router policy as-path-list
nv unset router policy as-path-list <list-id>
nv unset router policy as-path-list <list-id> rule
nv unset router policy as-path-list <list-id> rule <rule-id>
nv unset router policy as-path-list <list-id> rule <rule-id> action
nv unset router policy as-path-list <list-id> rule <rule-id> aspath-exp
nv unset router policy ext-community-list
nv unset router policy ext-community-list <list-id>
nv unset router policy ext-community-list <list-id> rule
nv unset router policy ext-community-list <list-id> rule <rule-id>
nv unset router policy ext-community-list <list-id> rule <rule-id> ext-community
nv unset router policy ext-community-list <list-id> rule <rule-id> ext-community rt
nv unset router policy ext-community-list <list-id> rule <rule-id> ext-community rt <ext-community-id>
nv unset router policy ext-community-list <list-id> rule <rule-id> ext-community soo
nv unset router policy ext-community-list <list-id> rule <rule-id> ext-community soo <ext-community-id>
nv unset router policy ext-community-list <list-id> rule <rule-id> action
nv unset router policy large-community-list
nv unset router policy large-community-list <list-id>
nv unset router policy large-community-list <list-id> rule
nv unset router policy large-community-list <list-id> rule <rule-id>
nv unset router policy large-community-list <list-id> rule <rule-id> large-community
nv unset router policy large-community-list <list-id> rule <rule-id> large-community <large-community-id>
nv unset router policy large-community-list <list-id> rule <rule-id> action
nv unset router policy prefix-list
nv unset router policy prefix-list <prefix-list-id>
nv unset router policy prefix-list <prefix-list-id> rule
nv unset router policy prefix-list <prefix-list-id> rule <rule-id>
nv unset router policy prefix-list <prefix-list-id> rule <rule-id> match
nv unset router policy prefix-list <prefix-list-id> rule <rule-id> match <match-id>
nv unset router policy prefix-list <prefix-list-id> rule <rule-id> match <match-id> min-prefix-len
nv unset router policy prefix-list <prefix-list-id> rule <rule-id> match <match-id> max-prefix-len
nv unset router policy prefix-list <prefix-list-id> rule <rule-id> action
nv unset router policy prefix-list <prefix-list-id> type
nv unset router policy route-map
nv unset router policy route-map <route-map-id>
nv unset router policy route-map <route-map-id> rule
nv unset router policy route-map <route-map-id> rule <rule-id>
nv unset router policy route-map <route-map-id> rule <rule-id> match
nv unset router policy route-map <route-map-id> rule <rule-id> match ip-prefix-list
nv unset router policy route-map <route-map-id> rule <rule-id> match ip-prefix-len
nv unset router policy route-map <route-map-id> rule <rule-id> match ip-nexthop-list
nv unset router policy route-map <route-map-id> rule <rule-id> match ip-nexthop-len
nv unset router policy route-map <route-map-id> rule <rule-id> match ip-nexthop
nv unset router policy route-map <route-map-id> rule <rule-id> match ip-nexthop-type
nv unset router policy route-map <route-map-id> rule <rule-id> match as-path-list
nv unset router policy route-map <route-map-id> rule <rule-id> match community-list
nv unset router policy route-map <route-map-id> rule <rule-id> match large-community-list
nv unset router policy route-map <route-map-id> rule <rule-id> match metric
nv unset router policy route-map <route-map-id> rule <rule-id> match interface
nv unset router policy route-map <route-map-id> rule <rule-id> match tag
nv unset router policy route-map <route-map-id> rule <rule-id> match source-protocol
nv unset router policy route-map <route-map-id> rule <rule-id> match origin
nv unset router policy route-map <route-map-id> rule <rule-id> match peer
nv unset router policy route-map <route-map-id> rule <rule-id> match local-preference
nv unset router policy route-map <route-map-id> rule <rule-id> match evpn-route-type
nv unset router policy route-map <route-map-id> rule <rule-id> match evpn-vni
nv unset router policy route-map <route-map-id> rule <rule-id> match evpn-default-route
nv unset router policy route-map <route-map-id> rule <rule-id> match source-vrf
nv unset router policy route-map <route-map-id> rule <rule-id> match type
nv unset router policy route-map <route-map-id> rule <rule-id> set
nv unset router policy route-map <route-map-id> rule <rule-id> set as-path-prepend
nv unset router policy route-map <route-map-id> rule <rule-id> set as-path-prepend as
nv unset router policy route-map <route-map-id> rule <rule-id> set as-path-prepend last-as
nv unset router policy route-map <route-map-id> rule <rule-id> set community
nv unset router policy route-map <route-map-id> rule <rule-id> set community <community-id>
nv unset router policy route-map <route-map-id> rule <rule-id> set large-community
nv unset router policy route-map <route-map-id> rule <rule-id> set large-community <large-community-id>
nv unset router policy route-map <route-map-id> rule <rule-id> set aggregator-as
nv unset router policy route-map <route-map-id> rule <rule-id> set aggregator-as <asn-id>
nv unset router policy route-map <route-map-id> rule <rule-id> set aggregator-as <asn-id> address
nv unset router policy route-map <route-map-id> rule <rule-id> set aggregator-as <asn-id> address <ipv4-address-id>
nv unset router policy route-map <route-map-id> rule <rule-id> set as-path-exclude
nv unset router policy route-map <route-map-id> rule <rule-id> set atomic-aggregate
nv unset router policy route-map <route-map-id> rule <rule-id> set ext-community-rt
nv unset router policy route-map <route-map-id> rule <rule-id> set ext-community-soo
nv unset router policy route-map <route-map-id> rule <rule-id> set ext-community-bw
nv unset router policy route-map <route-map-id> rule <rule-id> set local-preference
nv unset router policy route-map <route-map-id> rule <rule-id> set weight
nv unset router policy route-map <route-map-id> rule <rule-id> set metric
nv unset router policy route-map <route-map-id> rule <rule-id> set metric-type
nv unset router policy route-map <route-map-id> rule <rule-id> set origin
nv unset router policy route-map <route-map-id> rule <rule-id> set tag
nv unset router policy route-map <route-map-id> rule <rule-id> set ipv6-nexthop-global
nv unset router policy route-map <route-map-id> rule <rule-id> set ipv6-nexthop-local
nv unset router policy route-map <route-map-id> rule <rule-id> set ipv6-nexthop-prefer-global
nv unset router policy route-map <route-map-id> rule <rule-id> set ip-nexthop
nv unset router policy route-map <route-map-id> rule <rule-id> set source-ip
nv unset router policy route-map <route-map-id> rule <rule-id> set community-delete-list
nv unset router policy route-map <route-map-id> rule <rule-id> set large-community-delete-list
nv unset router policy route-map <route-map-id> rule <rule-id> set originator-id
nv unset router policy route-map <route-map-id> rule <rule-id> set label-index
nv unset router policy route-map <route-map-id> rule <rule-id> set forwarding-address
nv unset router policy route-map <route-map-id> rule <rule-id> action
nv unset router policy route-map <route-map-id> rule <rule-id> action deny
nv unset router policy route-map <route-map-id> rule <rule-id> action permit
nv unset router policy route-map <route-map-id> rule <rule-id> action permit exit-policy
nv unset router policy route-map <route-map-id> rule <rule-id> action permit exit-policy rule
nv unset router policy route-map <route-map-id> rule <rule-id> description
nv unset router bgp
nv unset router bgp graceful-restart
nv unset router bgp graceful-restart mode
nv unset router bgp graceful-restart restart-time
nv unset router bgp graceful-restart path-selection-deferral-time
nv unset router bgp graceful-restart stale-routes-time
nv unset router bgp convergence-wait
nv unset router bgp convergence-wait time
nv unset router bgp convergence-wait establish-wait-time
nv unset router bgp enable
nv unset router bgp autonomous-system
nv unset router bgp router-id
nv unset router bgp policy-update-timer
nv unset router bgp graceful-shutdown
nv unset router bgp wait-for-install
nv unset router ospf
nv unset router ospf timers
nv unset router ospf timers lsa
nv unset router ospf timers lsa min-arrival
nv unset router ospf timers lsa throttle
nv unset router ospf timers spf
nv unset router ospf timers spf delay
nv unset router ospf timers spf holdtime
nv unset router ospf timers spf max-holdtime
nv unset router ospf timers refresh
nv unset router ospf enable
nv unset router ospf router-id
nv unset router pim
nv unset router pim timers
nv unset router pim timers hello-interval
nv unset router pim timers register-suppress
nv unset router pim timers join-prune-interval
nv unset router pim timers keep-alive
nv unset router pim timers rp-keep-alive
nv unset router pim enable
nv unset router pim packets
nv unset router igmp
nv unset router igmp enable
nv unset router vrrp
nv unset router vrrp enable
nv unset router vrrp priority
nv unset router vrrp preempt
nv unset router vrrp advertisement-interval
nv unset router vrr
nv unset router vrr enable
nv unset router adaptive-routing
nv unset router adaptive-routing enable
nv unset platform
nv unset platform hardware
nv unset platform hardware component
nv unset platform hardware component <component-id>
nv unset platform hardware component <component-id> linecard
nv unset platform hardware component <component-id> linecard provision
nv unset platform hardware component <component-id> type
nv unset platform hardware component <component-id> admin-state
nv unset bridge
nv unset bridge domain
nv unset bridge domain <domain-id>
nv unset bridge domain <domain-id> stp
nv unset bridge domain <domain-id> stp state
nv unset bridge domain <domain-id> stp priority
nv unset bridge domain <domain-id> multicast
nv unset bridge domain <domain-id> multicast snooping
nv unset bridge domain <domain-id> multicast snooping querier
nv unset bridge domain <domain-id> multicast snooping querier enable
nv unset bridge domain <domain-id> multicast snooping enable
nv unset bridge domain <domain-id> vlan
nv unset bridge domain <domain-id> vlan <vid>
nv unset bridge domain <domain-id> vlan <vid> vni
nv unset bridge domain <domain-id> vlan <vid> vni <vni-id>
nv unset bridge domain <domain-id> vlan <vid> vni <vni-id> flooding
nv unset bridge domain <domain-id> vlan <vid> vni <vni-id> flooding head-end-replication
nv unset bridge domain <domain-id> vlan <vid> vni <vni-id> flooding head-end-replication <hrep-id>
nv unset bridge domain <domain-id> vlan <vid> vni <vni-id> flooding enable
nv unset bridge domain <domain-id> vlan <vid> vni <vni-id> flooding multicast-group
nv unset bridge domain <domain-id> vlan <vid> vni <vni-id> mac-learning
nv unset bridge domain <domain-id> vlan <vid> ptp
nv unset bridge domain <domain-id> vlan <vid> ptp enable
nv unset bridge domain <domain-id> vlan <vid> multicast
nv unset bridge domain <domain-id> vlan <vid> multicast snooping
nv unset bridge domain <domain-id> vlan <vid> multicast snooping querier
nv unset bridge domain <domain-id> vlan <vid> multicast snooping querier source-ip
nv unset bridge domain <domain-id> type
nv unset bridge domain <domain-id> untagged
nv unset bridge domain <domain-id> encap
nv unset bridge domain <domain-id> mac-address
nv unset bridge domain <domain-id> vlan-vni-offset
nv unset mlag
nv unset mlag lacp-conflict
nv unset mlag backup
nv unset mlag backup <backup-ip>
nv unset mlag backup <backup-ip> vrf
nv unset mlag enable
nv unset mlag mac-address
nv unset mlag peer-ip
nv unset mlag priority
nv unset mlag init-delay
nv unset mlag debug
nv unset evpn
nv unset evpn route-advertise
nv unset evpn route-advertise nexthop-setting
nv unset evpn route-advertise svi-ip
nv unset evpn route-advertise default-gateway
nv unset evpn dad
nv unset evpn dad duplicate-action
nv unset evpn dad duplicate-action freeze
nv unset evpn dad duplicate-action freeze duration
nv unset evpn dad enable
nv unset evpn dad mac-move-threshold
nv unset evpn dad move-window
nv unset evpn evi
nv unset evpn evi <evi-id>
nv unset evpn evi <evi-id> route-advertise
nv unset evpn evi <evi-id> route-advertise svi-ip
nv unset evpn evi <evi-id> route-advertise default-gateway
nv unset evpn evi <evi-id> route-target
nv unset evpn evi <evi-id> route-target export
nv unset evpn evi <evi-id> route-target export <rt-id>
nv unset evpn evi <evi-id> route-target import
nv unset evpn evi <evi-id> route-target import <rt-id>
nv unset evpn evi <evi-id> route-target both
nv unset evpn evi <evi-id> route-target both <rt-id>
nv unset evpn evi <evi-id> rd
nv unset evpn multihoming
nv unset evpn multihoming ead-evi-route
nv unset evpn multihoming ead-evi-route rx
nv unset evpn multihoming ead-evi-route tx
nv unset evpn multihoming segment
nv unset evpn multihoming segment mac-address
nv unset evpn multihoming segment df-preference
nv unset evpn multihoming enable
nv unset evpn multihoming mac-holdtime
nv unset evpn multihoming neighbor-holdtime
nv unset evpn multihoming startup-delay
nv unset evpn enable
nv unset qos
nv unset qos roce
nv unset qos roce enable
nv unset qos roce mode
nv unset qos roce cable-length
nv unset interface
nv unset interface <interface-id>
nv unset interface <interface-id> router
nv unset interface <interface-id> router pbr
nv unset interface <interface-id> router pbr map
nv unset interface <interface-id> router pbr map <pbr-map-id>
nv unset interface <interface-id> router ospf
nv unset interface <interface-id> router ospf timers
nv unset interface <interface-id> router ospf timers dead-interval
nv unset interface <interface-id> router ospf timers hello-multiplier
nv unset interface <interface-id> router ospf timers hello-interval
nv unset interface <interface-id> router ospf timers retransmit-interval
nv unset interface <interface-id> router ospf timers transmit-delay
nv unset interface <interface-id> router ospf authentication
nv unset interface <interface-id> router ospf authentication enable
nv unset interface <interface-id> router ospf authentication message-digest-key
nv unset interface <interface-id> router ospf authentication md5-key
nv unset interface <interface-id> router ospf bfd
nv unset interface <interface-id> router ospf bfd enable
nv unset interface <interface-id> router ospf bfd detect-multiplier
nv unset interface <interface-id> router ospf bfd min-receive-interval
nv unset interface <interface-id> router ospf bfd min-transmit-interval
nv unset interface <interface-id> router ospf enable
nv unset interface <interface-id> router ospf area
nv unset interface <interface-id> router ospf cost
nv unset interface <interface-id> router ospf mtu-ignore
nv unset interface <interface-id> router ospf network-type
nv unset interface <interface-id> router ospf passive
nv unset interface <interface-id> router ospf priority
nv unset interface <interface-id> router pim
nv unset interface <interface-id> router pim timers
nv unset interface <interface-id> router pim timers hello-interval
nv unset interface <interface-id> router pim bfd
nv unset interface <interface-id> router pim bfd enable
nv unset interface <interface-id> router pim bfd detect-multiplier
nv unset interface <interface-id> router pim bfd min-receive-interval
nv unset interface <interface-id> router pim bfd min-transmit-interval
nv unset interface <interface-id> router pim address-family
nv unset interface <interface-id> router pim address-family ipv4-unicast
nv unset interface <interface-id> router pim address-family ipv4-unicast allow-rp
nv unset interface <interface-id> router pim address-family ipv4-unicast allow-rp enable
nv unset interface <interface-id> router pim address-family ipv4-unicast allow-rp rp-list
nv unset interface <interface-id> router pim address-family ipv4-unicast multicast-boundary-oil
nv unset interface <interface-id> router pim address-family ipv4-unicast use-source
nv unset interface <interface-id> router pim enable
nv unset interface <interface-id> router pim dr-priority
nv unset interface <interface-id> router pim active-active
nv unset interface <interface-id> router adaptive-routing
nv unset interface <interface-id> router adaptive-routing enable
nv unset interface <interface-id> router adaptive-routing link-utilization-threshold
nv unset interface <interface-id> bond
nv unset interface <interface-id> bond member
nv unset interface <interface-id> bond member <member-id>
nv unset interface <interface-id> bond mlag
nv unset interface <interface-id> bond mlag lacp-conflict
nv unset interface <interface-id> bond mlag enable
nv unset interface <interface-id> bond mlag id
nv unset interface <interface-id> bond down-delay
nv unset interface <interface-id> bond lacp-bypass
nv unset interface <interface-id> bond lacp-rate
nv unset interface <interface-id> bond mode
nv unset interface <interface-id> bond up-delay
nv unset interface <interface-id> bridge
nv unset interface <interface-id> bridge domain
nv unset interface <interface-id> bridge domain <domain-id>
nv unset interface <interface-id> bridge domain <domain-id> stp
nv unset interface <interface-id> bridge domain <domain-id> stp bpdu-filter
nv unset interface <interface-id> bridge domain <domain-id> stp bpdu-guard
nv unset interface <interface-id> bridge domain <domain-id> stp admin-edge
nv unset interface <interface-id> bridge domain <domain-id> stp auto-edge
nv unset interface <interface-id> bridge domain <domain-id> stp network
nv unset interface <interface-id> bridge domain <domain-id> stp restrrole
nv unset interface <interface-id> bridge domain <domain-id> vlan
nv unset interface <interface-id> bridge domain <domain-id> vlan <vid>
nv unset interface <interface-id> bridge domain <domain-id> learning
nv unset interface <interface-id> bridge domain <domain-id> untagged
nv unset interface <interface-id> bridge domain <domain-id> access
nv unset interface <interface-id> ip
nv unset interface <interface-id> ip address
nv unset interface <interface-id> ip address <ip-prefix-id>
nv unset interface <interface-id> ip vrr
nv unset interface <interface-id> ip vrr address
nv unset interface <interface-id> ip vrr address <ip-prefix-id>
nv unset interface <interface-id> ip vrr state
nv unset interface <interface-id> ip vrr enable
nv unset interface <interface-id> ip vrr mac-id
nv unset interface <interface-id> ip vrr mac-address
nv unset interface <interface-id> ip gateway
nv unset interface <interface-id> ip gateway <ip-address-id>
nv unset interface <interface-id> ip ipv4
nv unset interface <interface-id> ip ipv4 forward
nv unset interface <interface-id> ip ipv6
nv unset interface <interface-id> ip ipv6 enable
nv unset interface <interface-id> ip ipv6 forward
nv unset interface <interface-id> ip igmp
nv unset interface <interface-id> ip igmp static-group
nv unset interface <interface-id> ip igmp static-group <static-group-id>
nv unset interface <interface-id> ip igmp static-group <static-group-id> source-address
nv unset interface <interface-id> ip igmp enable
nv unset interface <interface-id> ip igmp version
nv unset interface <interface-id> ip igmp query-interval
nv unset interface <interface-id> ip igmp query-max-response-time
nv unset interface <interface-id> ip igmp last-member-query-interval
nv unset interface <interface-id> ip vrrp
nv unset interface <interface-id> ip vrrp virtual-router
nv unset interface <interface-id> ip vrrp virtual-router <virtual-router-id>
nv unset interface <interface-id> ip vrrp virtual-router <virtual-router-id> address
nv unset interface <interface-id> ip vrrp virtual-router <virtual-router-id> address <ip-address-id>
nv unset interface <interface-id> ip vrrp virtual-router <virtual-router-id> version
nv unset interface <interface-id> ip vrrp virtual-router <virtual-router-id> priority
nv unset interface <interface-id> ip vrrp virtual-router <virtual-router-id> preempt
nv unset interface <interface-id> ip vrrp virtual-router <virtual-router-id> advertisement-interval
nv unset interface <interface-id> ip vrrp enable
nv unset interface <interface-id> ip neighbor-discovery
nv unset interface <interface-id> ip neighbor-discovery rdnss
nv unset interface <interface-id> ip neighbor-discovery rdnss <ipv6-address-id>
nv unset interface <interface-id> ip neighbor-discovery rdnss <ipv6-address-id> lifetime
nv unset interface <interface-id> ip neighbor-discovery prefix
nv unset interface <interface-id> ip neighbor-discovery prefix <ipv6-prefix-id>
nv unset interface <interface-id> ip neighbor-discovery prefix <ipv6-prefix-id> valid-lifetime
nv unset interface <interface-id> ip neighbor-discovery prefix <ipv6-prefix-id> preferred-lifetime
nv unset interface <interface-id> ip neighbor-discovery prefix <ipv6-prefix-id> off-link
nv unset interface <interface-id> ip neighbor-discovery prefix <ipv6-prefix-id> autoconfig
nv unset interface <interface-id> ip neighbor-discovery prefix <ipv6-prefix-id> router-address
nv unset interface <interface-id> ip neighbor-discovery dnssl
nv unset interface <interface-id> ip neighbor-discovery dnssl <domain-name-id>
nv unset interface <interface-id> ip neighbor-discovery dnssl <domain-name-id> lifetime
nv unset interface <interface-id> ip neighbor-discovery router-advertisement
nv unset interface <interface-id> ip neighbor-discovery router-advertisement enable
nv unset interface <interface-id> ip neighbor-discovery router-advertisement interval
nv unset interface <interface-id> ip neighbor-discovery router-advertisement interval-option
nv unset interface <interface-id> ip neighbor-discovery router-advertisement fast-retransmit
nv unset interface <interface-id> ip neighbor-discovery router-advertisement lifetime
nv unset interface <interface-id> ip neighbor-discovery router-advertisement reachable-time
nv unset interface <interface-id> ip neighbor-discovery router-advertisement retransmit-time
nv unset interface <interface-id> ip neighbor-discovery router-advertisement managed-config
nv unset interface <interface-id> ip neighbor-discovery router-advertisement other-config
nv unset interface <interface-id> ip neighbor-discovery router-advertisement hop-limit
nv unset interface <interface-id> ip neighbor-discovery router-advertisement router-preference
nv unset interface <interface-id> ip neighbor-discovery home-agent
nv unset interface <interface-id> ip neighbor-discovery home-agent lifetime
nv unset interface <interface-id> ip neighbor-discovery home-agent preference
nv unset interface <interface-id> ip neighbor-discovery enable
nv unset interface <interface-id> ip neighbor-discovery mtu
nv unset interface <interface-id> ip vrf
nv unset interface <interface-id> lldp
nv unset interface <interface-id> lldp dcbx-pfc-tlv
nv unset interface <interface-id> lldp dcbx-ets-config-tlv
nv unset interface <interface-id> lldp dcbx-ets-recomm-tlv
nv unset interface <interface-id> link
nv unset interface <interface-id> link state
nv unset interface <interface-id> link dot1x
nv unset interface <interface-id> link dot1x mab
nv unset interface <interface-id> link dot1x parking-vlan
nv unset interface <interface-id> link auto-negotiate
nv unset interface <interface-id> link breakout
nv unset interface <interface-id> link duplex
nv unset interface <interface-id> link speed
nv unset interface <interface-id> link lanes
nv unset interface <interface-id> link fec
nv unset interface <interface-id> link mtu
nv unset interface <interface-id> evpn
nv unset interface <interface-id> evpn multihoming
nv unset interface <interface-id> evpn multihoming segment
nv unset interface <interface-id> evpn multihoming segment enable
nv unset interface <interface-id> evpn multihoming segment local-id
nv unset interface <interface-id> evpn multihoming segment identifier
nv unset interface <interface-id> evpn multihoming segment mac-address
nv unset interface <interface-id> evpn multihoming segment df-preference
nv unset interface <interface-id> evpn multihoming uplink
nv unset interface <interface-id> acl
nv unset interface <interface-id> acl <acl-id>
nv unset interface <interface-id> acl <acl-id> inbound
nv unset interface <interface-id> acl <acl-id> inbound control-plane
nv unset interface <interface-id> acl <acl-id> outbound
nv unset interface <interface-id> acl <acl-id> outbound control-plane
nv unset interface <interface-id> ptp
nv unset interface <interface-id> ptp timers
nv unset interface <interface-id> ptp timers announce-interval
nv unset interface <interface-id> ptp timers sync-interval
nv unset interface <interface-id> ptp timers delay-req-interval
nv unset interface <interface-id> ptp timers announce-timeout
nv unset interface <interface-id> ptp enable
nv unset interface <interface-id> ptp instance
nv unset interface <interface-id> ptp forced-master
nv unset interface <interface-id> ptp acceptable-master
nv unset interface <interface-id> ptp delay-mechanism
nv unset interface <interface-id> ptp transport
nv unset interface <interface-id> ptp ttl
nv unset interface <interface-id> ptp mixed-multicast-unicast
nv unset interface <interface-id> ptp unicast-service-mode
nv unset interface <interface-id> ptp unicast-request-duration
nv unset interface <interface-id> ptp unicast-master-table-id
nv unset interface <interface-id> tunnel
nv unset interface <interface-id> tunnel source-ip
nv unset interface <interface-id> tunnel dest-ip
nv unset interface <interface-id> tunnel ttl
nv unset interface <interface-id> tunnel mode
nv unset interface <interface-id> tunnel interface
nv unset interface <interface-id> description
nv unset interface <interface-id> type
nv unset interface <interface-id> base-interface
nv unset interface <interface-id> vlan
nv unset service
nv unset service dns
nv unset service dns <vrf-id>
nv unset service dns <vrf-id> server
nv unset service dns <vrf-id> server <dns-server-id>
nv unset service syslog
nv unset service syslog <vrf-id>
nv unset service syslog <vrf-id> server
nv unset service syslog <vrf-id> server <server-id>
nv unset service syslog <vrf-id> server <server-id> port
nv unset service syslog <vrf-id> server <server-id> protocol
nv unset service ntp
nv unset service ntp <vrf-id>
nv unset service ntp <vrf-id> server
nv unset service ntp <vrf-id> server <server-id>
nv unset service ntp <vrf-id> server <server-id> iburst
nv unset service ntp <vrf-id> pool
nv unset service ntp <vrf-id> pool <server-id>
nv unset service ntp <vrf-id> pool <server-id> iburst
nv unset service ntp <vrf-id> listen
nv unset service dhcp-relay
nv unset service dhcp-relay <vrf-id>
nv unset service dhcp-relay <vrf-id> server
nv unset service dhcp-relay <vrf-id> server <server-id>
nv unset service dhcp-relay <vrf-id> interface
nv unset service dhcp-relay <vrf-id> interface <interface-id>
nv unset service dhcp-relay <vrf-id> giaddress-interface
nv unset service dhcp-relay <vrf-id> giaddress-interface <interface-id>
nv unset service dhcp-relay <vrf-id> giaddress-interface <interface-id> address
nv unset service dhcp-relay <vrf-id> source-ip
nv unset service dhcp-relay6
nv unset service dhcp-relay6 <vrf-id>
nv unset service dhcp-relay6 <vrf-id> interface
nv unset service dhcp-relay6 <vrf-id> interface upstream
nv unset service dhcp-relay6 <vrf-id> interface upstream <interface-id>
nv unset service dhcp-relay6 <vrf-id> interface upstream <interface-id> address
nv unset service dhcp-relay6 <vrf-id> interface downstream
nv unset service dhcp-relay6 <vrf-id> interface downstream <interface-id>
nv unset service dhcp-relay6 <vrf-id> interface downstream <interface-id> address
nv unset service ptp
nv unset service ptp <instance-id>
nv unset service ptp <instance-id> acceptable-master
nv unset service ptp <instance-id> acceptable-master <clock-id>
nv unset service ptp <instance-id> acceptable-master <clock-id> alt-priority
nv unset service ptp <instance-id> unicast-master
nv unset service ptp <instance-id> unicast-master <table-id>
nv unset service ptp <instance-id> unicast-master <table-id> address
nv unset service ptp <instance-id> unicast-master <table-id> address <ip-mac-address-id>
nv unset service ptp <instance-id> unicast-master <table-id> query-interval
nv unset service ptp <instance-id> unicast-master <table-id> peer-address
nv unset service ptp <instance-id> profile
nv unset service ptp <instance-id> profile <profile-id>
nv unset service ptp <instance-id> profile <profile-id> profile-type
nv unset service ptp <instance-id> profile <profile-id> priority1
nv unset service ptp <instance-id> profile <profile-id> priority2
nv unset service ptp <instance-id> profile <profile-id> local-priority
nv unset service ptp <instance-id> profile <profile-id> domain
nv unset service ptp <instance-id> profile <profile-id> delay-mechanism
nv unset service ptp <instance-id> profile <profile-id> transport
nv unset service ptp <instance-id> profile <profile-id> announce-interval
nv unset service ptp <instance-id> profile <profile-id> sync-interval
nv unset service ptp <instance-id> profile <profile-id> delay-req-interval
nv unset service ptp <instance-id> profile <profile-id> announce-timeout
nv unset service ptp <instance-id> monitor
nv unset service ptp <instance-id> monitor min-offset-threshold
nv unset service ptp <instance-id> monitor max-offset-threshold
nv unset service ptp <instance-id> monitor path-delay-threshold
nv unset service ptp <instance-id> monitor max-timestamp-entries
nv unset service ptp <instance-id> monitor max-violation-log-sets
nv unset service ptp <instance-id> monitor max-violation-log-entries
nv unset service ptp <instance-id> monitor violation-log-interval
nv unset service ptp <instance-id> enable
nv unset service ptp <instance-id> current-profile
nv unset service ptp <instance-id> two-step
nv unset service ptp <instance-id> priority1
nv unset service ptp <instance-id> priority2
nv unset service ptp <instance-id> domain
nv unset service ptp <instance-id> ip-dscp
nv unset service dhcp-server
nv unset service dhcp-server <vrf-id>
nv unset service dhcp-server <vrf-id> interface
nv unset service dhcp-server <vrf-id> interface <interface-id>
nv unset service dhcp-server <vrf-id> pool
nv unset service dhcp-server <vrf-id> pool <pool-id>
nv unset service dhcp-server <vrf-id> pool <pool-id> domain-name-server
nv unset service dhcp-server <vrf-id> pool <pool-id> domain-name-server <server-id>
nv unset service dhcp-server <vrf-id> pool <pool-id> domain-name
nv unset service dhcp-server <vrf-id> pool <pool-id> domain-name <domain-name-id>
nv unset service dhcp-server <vrf-id> pool <pool-id> domain-name <domain-name-id> domain-name
nv unset service dhcp-server <vrf-id> pool <pool-id> gateway
nv unset service dhcp-server <vrf-id> pool <pool-id> gateway <gateway-id>
nv unset service dhcp-server <vrf-id> pool <pool-id> range
nv unset service dhcp-server <vrf-id> pool <pool-id> range <range-id>
nv unset service dhcp-server <vrf-id> pool <pool-id> range <range-id> to
nv unset service dhcp-server <vrf-id> pool <pool-id> pool-name
nv unset service dhcp-server <vrf-id> pool <pool-id> lease-time
nv unset service dhcp-server <vrf-id> pool <pool-id> ping-check
nv unset service dhcp-server <vrf-id> pool <pool-id> default-url
nv unset service dhcp-server <vrf-id> pool <pool-id> cumulus-provision-url
nv unset service dhcp-server <vrf-id> domain-name
nv unset service dhcp-server <vrf-id> domain-name <domain-name-id>
nv unset service dhcp-server <vrf-id> domain-name <domain-name-id> domain-name
nv unset service dhcp-server <vrf-id> domain-name-server
nv unset service dhcp-server <vrf-id> domain-name-server <server-id>
nv unset service dhcp-server <vrf-id> static
nv unset service dhcp-server <vrf-id> static <static-id>
nv unset service dhcp-server <vrf-id> static <static-id> mac-address
nv unset service dhcp-server <vrf-id> static <static-id> ip-address
nv unset service dhcp-server <vrf-id> static <static-id> cumulus-provision-url
nv unset service dhcp-server6
nv unset service dhcp-server6 <vrf-id>
nv unset service dhcp-server6 <vrf-id> interface
nv unset service dhcp-server6 <vrf-id> interface <interface-id>
nv unset service dhcp-server6 <vrf-id> pool
nv unset service dhcp-server6 <vrf-id> pool <pool-id>
nv unset service dhcp-server6 <vrf-id> pool <pool-id> domain-name-server
nv unset service dhcp-server6 <vrf-id> pool <pool-id> domain-name-server <server-id>
nv unset service dhcp-server6 <vrf-id> pool <pool-id> domain-name
nv unset service dhcp-server6 <vrf-id> pool <pool-id> domain-name <domain-name-id>
nv unset service dhcp-server6 <vrf-id> pool <pool-id> domain-name <domain-name-id> domain-name
nv unset service dhcp-server6 <vrf-id> pool <pool-id> range
nv unset service dhcp-server6 <vrf-id> pool <pool-id> range <range-id>
nv unset service dhcp-server6 <vrf-id> pool <pool-id> range <range-id> to
nv unset service dhcp-server6 <vrf-id> pool <pool-id> pool-name
nv unset service dhcp-server6 <vrf-id> pool <pool-id> lease-time
nv unset service dhcp-server6 <vrf-id> pool <pool-id> ping-check
nv unset service dhcp-server6 <vrf-id> pool <pool-id> default-url
nv unset service dhcp-server6 <vrf-id> pool <pool-id> cumulus-provision-url
nv unset service dhcp-server6 <vrf-id> domain-name
nv unset service dhcp-server6 <vrf-id> domain-name <domain-name-id>
nv unset service dhcp-server6 <vrf-id> domain-name <domain-name-id> domain-name
nv unset service dhcp-server6 <vrf-id> domain-name-server
nv unset service dhcp-server6 <vrf-id> domain-name-server <server-id>
nv unset service dhcp-server6 <vrf-id> static
nv unset service dhcp-server6 <vrf-id> static <static-id>
nv unset service dhcp-server6 <vrf-id> static <static-id> mac-address
nv unset service dhcp-server6 <vrf-id> static <static-id> ip-address
nv unset service dhcp-server6 <vrf-id> static <static-id> cumulus-provision-url
nv unset service lldp
nv unset service lldp tx-interval
nv unset service lldp tx-hold-multiplier
nv unset service lldp dot1-tlv
nv unset system
nv unset system control-plane
nv unset system control-plane trap
nv unset system control-plane trap <trap-id>
nv unset system control-plane trap <trap-id> state
nv unset system control-plane policer
nv unset system control-plane policer <policer-id>
nv unset system control-plane policer <policer-id> state
nv unset system control-plane policer <policer-id> burst
nv unset system control-plane policer <policer-id> rate
nv unset system message
nv unset system message pre-login
nv unset system message post-login
nv unset system global
nv unset system global reserved
nv unset system global reserved routing-table
nv unset system global reserved routing-table pbr
nv unset system global reserved routing-table pbr begin
nv unset system global reserved routing-table pbr end
nv unset system global reserved vlan
nv unset system global reserved vlan l3-vni-vlan
nv unset system global reserved vlan l3-vni-vlan begin
nv unset system global reserved vlan l3-vni-vlan end
nv unset system global system-mac
nv unset system global anycast-mac
nv unset system global anycast-id
nv unset system global fabric-mac
nv unset system global fabric-id
nv unset system forwarding
nv unset system forwarding lag-hash
nv unset system forwarding lag-hash ip-protocol
nv unset system forwarding lag-hash source-mac
nv unset system forwarding lag-hash destination-mac
nv unset system forwarding lag-hash source-ip
nv unset system forwarding lag-hash destination-ip
nv unset system forwarding lag-hash source-port
nv unset system forwarding lag-hash destination-port
nv unset system forwarding lag-hash ether-type
nv unset system forwarding lag-hash vlan
nv unset system forwarding lag-hash gtp-teid
nv unset system forwarding ecmp-hash
nv unset system forwarding ecmp-hash ip-protocol
nv unset system forwarding ecmp-hash source-ip
nv unset system forwarding ecmp-hash destination-ip
nv unset system forwarding ecmp-hash source-port
nv unset system forwarding ecmp-hash destination-port
nv unset system forwarding ecmp-hash ipv6-label
nv unset system forwarding ecmp-hash ingress-interface
nv unset system forwarding ecmp-hash gtp-teid
nv unset system forwarding ecmp-hash inner-ip-protocol
nv unset system forwarding ecmp-hash inner-source-ip
nv unset system forwarding ecmp-hash inner-destination-ip
nv unset system forwarding ecmp-hash inner-source-port
nv unset system forwarding ecmp-hash inner-destination-port
nv unset system forwarding ecmp-hash inner-ipv6-label
nv unset system forwarding hash-seed
nv unset system port-mirror
nv unset system port-mirror session
nv unset system port-mirror session <session-id>
nv unset system port-mirror session <session-id> span
nv unset system port-mirror session <session-id> span source-port
nv unset system port-mirror session <session-id> span source-port <port-id>
nv unset system port-mirror session <session-id> span destination
nv unset system port-mirror session <session-id> span destination <port-id>
nv unset system port-mirror session <session-id> span truncate
nv unset system port-mirror session <session-id> span truncate enable
nv unset system port-mirror session <session-id> span truncate size
nv unset system port-mirror session <session-id> span enable
nv unset system port-mirror session <session-id> span direction
nv unset system port-mirror session <session-id> erspan
nv unset system port-mirror session <session-id> erspan source-port
nv unset system port-mirror session <session-id> erspan source-port <port-id>
nv unset system port-mirror session <session-id> erspan destination
nv unset system port-mirror session <session-id> erspan destination source-ip
nv unset system port-mirror session <session-id> erspan destination source-ip <source-ip>
nv unset system port-mirror session <session-id> erspan destination dest-ip
nv unset system port-mirror session <session-id> erspan destination dest-ip <dest-ip>
nv unset system port-mirror session <session-id> erspan truncate
nv unset system port-mirror session <session-id> erspan truncate enable
nv unset system port-mirror session <session-id> erspan truncate size
nv unset system port-mirror session <session-id> erspan enable
nv unset system port-mirror session <session-id> erspan direction
nv unset system config
nv unset system config apply
nv unset system config apply ignore
nv unset system config apply ignore <ignore-id>
nv unset system config apply overwrite
nv unset system hostname
nv unset system timezone
nv unset vrf
nv unset vrf <vrf-id>
nv unset vrf <vrf-id> loopback
nv unset vrf <vrf-id> loopback ip
nv unset vrf <vrf-id> loopback ip address
nv unset vrf <vrf-id> loopback ip address <ip-prefix-id>
nv unset vrf <vrf-id> evpn
nv unset vrf <vrf-id> evpn vni
nv unset vrf <vrf-id> evpn vni <vni-id>
nv unset vrf <vrf-id> evpn vni <vni-id> prefix-routes-only
nv unset vrf <vrf-id> evpn enable
nv unset vrf <vrf-id> evpn vlan
nv unset vrf <vrf-id> router
nv unset vrf <vrf-id> router rib
nv unset vrf <vrf-id> router rib <afi>
nv unset vrf <vrf-id> router rib <afi> protocol
nv unset vrf <vrf-id> router rib <afi> protocol <import-protocol-id>
nv unset vrf <vrf-id> router rib <afi> protocol <import-protocol-id> fib-filter
nv unset vrf <vrf-id> router bgp
nv unset vrf <vrf-id> router bgp address-family
nv unset vrf <vrf-id> router bgp address-family ipv4-unicast
nv unset vrf <vrf-id> router bgp address-family ipv4-unicast redistribute
nv unset vrf <vrf-id> router bgp address-family ipv4-unicast redistribute static
nv unset vrf <vrf-id> router bgp address-family ipv4-unicast redistribute static enable
nv unset vrf <vrf-id> router bgp address-family ipv4-unicast redistribute static metric
nv unset vrf <vrf-id> router bgp address-family ipv4-unicast redistribute static route-map
nv unset vrf <vrf-id> router bgp address-family ipv4-unicast redistribute connected
nv unset vrf <vrf-id> router bgp address-family ipv4-unicast redistribute connected enable
nv unset vrf <vrf-id> router bgp address-family ipv4-unicast redistribute connected metric
nv unset vrf <vrf-id> router bgp address-family ipv4-unicast redistribute connected route-map
nv unset vrf <vrf-id> router bgp address-family ipv4-unicast redistribute kernel
nv unset vrf <vrf-id> router bgp address-family ipv4-unicast redistribute kernel enable
nv unset vrf <vrf-id> router bgp address-family ipv4-unicast redistribute kernel metric
nv unset vrf <vrf-id> router bgp address-family ipv4-unicast redistribute kernel route-map
nv unset vrf <vrf-id> router bgp address-family ipv4-unicast redistribute ospf
nv unset vrf <vrf-id> router bgp address-family ipv4-unicast redistribute ospf enable
nv unset vrf <vrf-id> router bgp address-family ipv4-unicast redistribute ospf metric
nv unset vrf <vrf-id> router bgp address-family ipv4-unicast redistribute ospf route-map
nv unset vrf <vrf-id> router bgp address-family ipv4-unicast aggregate-route
nv unset vrf <vrf-id> router bgp address-family ipv4-unicast aggregate-route <aggregate-route-id>
nv unset vrf <vrf-id> router bgp address-family ipv4-unicast aggregate-route <aggregate-route-id> summary-only
nv unset vrf <vrf-id> router bgp address-family ipv4-unicast aggregate-route <aggregate-route-id> as-set
nv unset vrf <vrf-id> router bgp address-family ipv4-unicast aggregate-route <aggregate-route-id> route-map
nv unset vrf <vrf-id> router bgp address-family ipv4-unicast network
nv unset vrf <vrf-id> router bgp address-family ipv4-unicast network <static-network-id>
nv unset vrf <vrf-id> router bgp address-family ipv4-unicast network <static-network-id> route-map
nv unset vrf <vrf-id> router bgp address-family ipv4-unicast route-import
nv unset vrf <vrf-id> router bgp address-family ipv4-unicast route-import from-vrf
nv unset vrf <vrf-id> router bgp address-family ipv4-unicast route-import from-vrf list
nv unset vrf <vrf-id> router bgp address-family ipv4-unicast route-import from-vrf list <leak-vrf-id>
nv unset vrf <vrf-id> router bgp address-family ipv4-unicast route-import from-vrf enable
nv unset vrf <vrf-id> router bgp address-family ipv4-unicast route-import from-vrf route-map
nv unset vrf <vrf-id> router bgp address-family ipv4-unicast multipaths
nv unset vrf <vrf-id> router bgp address-family ipv4-unicast multipaths ebgp
nv unset vrf <vrf-id> router bgp address-family ipv4-unicast multipaths ibgp
nv unset vrf <vrf-id> router bgp address-family ipv4-unicast multipaths compare-cluster-length
nv unset vrf <vrf-id> router bgp address-family ipv4-unicast admin-distance
nv unset vrf <vrf-id> router bgp address-family ipv4-unicast admin-distance external
nv unset vrf <vrf-id> router bgp address-family ipv4-unicast admin-distance internal
nv unset vrf <vrf-id> router bgp address-family ipv4-unicast route-export
nv unset vrf <vrf-id> router bgp address-family ipv4-unicast route-export to-evpn
nv unset vrf <vrf-id> router bgp address-family ipv4-unicast route-export to-evpn enable
nv unset vrf <vrf-id> router bgp address-family ipv4-unicast route-export to-evpn route-map
nv unset vrf <vrf-id> router bgp address-family ipv4-unicast route-export to-evpn default-route-origination
nv unset vrf <vrf-id> router bgp address-family ipv4-unicast rib-filter
nv unset vrf <vrf-id> router bgp address-family ipv4-unicast enable
nv unset vrf <vrf-id> router bgp address-family l2vpn-evpn
nv unset vrf <vrf-id> router bgp address-family l2vpn-evpn enable
nv unset vrf <vrf-id> router bgp address-family ipv6-unicast
nv unset vrf <vrf-id> router bgp address-family ipv6-unicast aggregate-route
nv unset vrf <vrf-id> router bgp address-family ipv6-unicast aggregate-route <aggregate-route-id>
nv unset vrf <vrf-id> router bgp address-family ipv6-unicast aggregate-route <aggregate-route-id> summary-only
nv unset vrf <vrf-id> router bgp address-family ipv6-unicast aggregate-route <aggregate-route-id> as-set
nv unset vrf <vrf-id> router bgp address-family ipv6-unicast aggregate-route <aggregate-route-id> route-map
nv unset vrf <vrf-id> router bgp address-family ipv6-unicast network
nv unset vrf <vrf-id> router bgp address-family ipv6-unicast network <static-network-id>
nv unset vrf <vrf-id> router bgp address-family ipv6-unicast network <static-network-id> route-map
nv unset vrf <vrf-id> router bgp address-family ipv6-unicast route-import
nv unset vrf <vrf-id> router bgp address-family ipv6-unicast route-import from-vrf
nv unset vrf <vrf-id> router bgp address-family ipv6-unicast route-import from-vrf list
nv unset vrf <vrf-id> router bgp address-family ipv6-unicast route-import from-vrf enable
nv unset vrf <vrf-id> router bgp address-family ipv6-unicast route-import from-vrf route-map
nv unset vrf <vrf-id> router bgp address-family ipv6-unicast multipaths
nv unset vrf <vrf-id> router bgp address-family ipv6-unicast multipaths ebgp
nv unset vrf <vrf-id> router bgp address-family ipv6-unicast multipaths ibgp
nv unset vrf <vrf-id> router bgp address-family ipv6-unicast multipaths compare-cluster-length
nv unset vrf <vrf-id> router bgp address-family ipv6-unicast admin-distance
nv unset vrf <vrf-id> router bgp address-family ipv6-unicast admin-distance external
nv unset vrf <vrf-id> router bgp address-family ipv6-unicast admin-distance internal
nv unset vrf <vrf-id> router bgp address-family ipv6-unicast route-export
nv unset vrf <vrf-id> router bgp address-family ipv6-unicast route-export to-evpn
nv unset vrf <vrf-id> router bgp address-family ipv6-unicast route-export to-evpn enable
nv unset vrf <vrf-id> router bgp address-family ipv6-unicast route-export to-evpn route-map
nv unset vrf <vrf-id> router bgp address-family ipv6-unicast route-export to-evpn default-route-origination
nv unset vrf <vrf-id> router bgp address-family ipv6-unicast redistribute
nv unset vrf <vrf-id> router bgp address-family ipv6-unicast redistribute static
nv unset vrf <vrf-id> router bgp address-family ipv6-unicast redistribute static enable
nv unset vrf <vrf-id> router bgp address-family ipv6-unicast redistribute static metric
nv unset vrf <vrf-id> router bgp address-family ipv6-unicast redistribute static route-map
nv unset vrf <vrf-id> router bgp address-family ipv6-unicast redistribute connected
nv unset vrf <vrf-id> router bgp address-family ipv6-unicast redistribute connected enable
nv unset vrf <vrf-id> router bgp address-family ipv6-unicast redistribute connected metric
nv unset vrf <vrf-id> router bgp address-family ipv6-unicast redistribute connected route-map
nv unset vrf <vrf-id> router bgp address-family ipv6-unicast redistribute kernel
nv unset vrf <vrf-id> router bgp address-family ipv6-unicast redistribute kernel enable
nv unset vrf <vrf-id> router bgp address-family ipv6-unicast redistribute kernel metric
nv unset vrf <vrf-id> router bgp address-family ipv6-unicast redistribute kernel route-map
nv unset vrf <vrf-id> router bgp address-family ipv6-unicast redistribute ospf6
nv unset vrf <vrf-id> router bgp address-family ipv6-unicast redistribute ospf6 enable
nv unset vrf <vrf-id> router bgp address-family ipv6-unicast redistribute ospf6 metric
nv unset vrf <vrf-id> router bgp address-family ipv6-unicast redistribute ospf6 route-map
nv unset vrf <vrf-id> router bgp address-family ipv6-unicast rib-filter
nv unset vrf <vrf-id> router bgp address-family ipv6-unicast enable
nv unset vrf <vrf-id> router bgp path-selection
nv unset vrf <vrf-id> router bgp path-selection aspath
nv unset vrf <vrf-id> router bgp path-selection aspath compare-lengths
nv unset vrf <vrf-id> router bgp path-selection aspath compare-confed
nv unset vrf <vrf-id> router bgp path-selection med
nv unset vrf <vrf-id> router bgp path-selection med compare-always
nv unset vrf <vrf-id> router bgp path-selection med compare-deterministic
nv unset vrf <vrf-id> router bgp path-selection med compare-confed
nv unset vrf <vrf-id> router bgp path-selection med missing-as-max
nv unset vrf <vrf-id> router bgp path-selection multipath
nv unset vrf <vrf-id> router bgp path-selection multipath aspath-ignore
nv unset vrf <vrf-id> router bgp path-selection multipath generate-asset
nv unset vrf <vrf-id> router bgp path-selection multipath bandwidth
nv unset vrf <vrf-id> router bgp path-selection routerid-compare
nv unset vrf <vrf-id> router bgp route-reflection
nv unset vrf <vrf-id> router bgp route-reflection enable
nv unset vrf <vrf-id> router bgp route-reflection cluster-id
nv unset vrf <vrf-id> router bgp route-reflection reflect-between-clients
nv unset vrf <vrf-id> router bgp route-reflection outbound-policy
nv unset vrf <vrf-id> router bgp peer-group
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id>
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> bfd
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> bfd enable
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> bfd detect-multiplier
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> bfd min-rx-interval
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> bfd min-tx-interval
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> ttl-security
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> ttl-security enable
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> ttl-security hops
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> capabilities
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> capabilities extended-nexthop
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> capabilities source-address
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> graceful-restart
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> graceful-restart mode
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> local-as
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> local-as enable
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> local-as asn
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> local-as prepend
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> local-as replace
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> timers
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> timers keepalive
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> timers hold
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> timers connection-retry
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> timers route-advertisement
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast community-advertise
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast community-advertise regular
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast community-advertise extended
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast community-advertise large
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast attribute-mod
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast attribute-mod aspath
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast attribute-mod med
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast attribute-mod nexthop
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast aspath
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast aspath allow-my-asn
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast aspath allow-my-asn enable
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast aspath allow-my-asn origin
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast aspath allow-my-asn occurrences
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast aspath replace-peer-as
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast aspath private-as
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast prefix-limits
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast prefix-limits inbound
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast prefix-limits inbound maximum
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast prefix-limits inbound warning-threshold
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast prefix-limits inbound warning-only
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast prefix-limits inbound reestablish-wait
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast default-route-origination
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast default-route-origination enable
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast default-route-origination policy
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast policy
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast policy inbound
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast policy inbound route-map
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast policy inbound prefix-list
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast policy inbound aspath-list
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast policy outbound
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast policy outbound route-map
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast policy outbound unsuppress-map
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast policy outbound prefix-list
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast policy outbound aspath-list
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast conditional-advertise
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast conditional-advertise enable
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast conditional-advertise advertise-map
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast conditional-advertise exist-map
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast conditional-advertise non-exist-map
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast enable
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast route-reflector-client
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast route-server-client
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast soft-reconfiguration
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast nexthop-setting
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast add-path-tx
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv4-unicast weight
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast policy
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast policy inbound
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast policy inbound route-map
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast policy inbound prefix-list
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast policy inbound aspath-list
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast policy outbound
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast policy outbound route-map
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast policy outbound unsuppress-map
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast policy outbound prefix-list
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast policy outbound aspath-list
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast aspath
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast aspath allow-my-asn
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast aspath allow-my-asn enable
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast aspath allow-my-asn origin
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast aspath allow-my-asn occurrences
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast aspath replace-peer-as
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast aspath private-as
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast prefix-limits
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast prefix-limits inbound
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast prefix-limits inbound maximum
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast prefix-limits inbound warning-threshold
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast prefix-limits inbound warning-only
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast prefix-limits inbound reestablish-wait
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast default-route-origination
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast default-route-origination enable
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast default-route-origination policy
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast community-advertise
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast community-advertise regular
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast community-advertise extended
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast community-advertise large
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast attribute-mod
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast attribute-mod aspath
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast attribute-mod med
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast attribute-mod nexthop
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast conditional-advertise
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast conditional-advertise enable
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast conditional-advertise advertise-map
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast conditional-advertise exist-map
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast conditional-advertise non-exist-map
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast enable
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast route-reflector-client
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast route-server-client
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast soft-reconfiguration
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast nexthop-setting
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast add-path-tx
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family ipv6-unicast weight
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn attribute-mod
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn attribute-mod aspath
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn attribute-mod med
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn attribute-mod nexthop
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn aspath
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn aspath allow-my-asn
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn aspath allow-my-asn enable
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn aspath allow-my-asn origin
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn aspath allow-my-asn occurrences
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn aspath replace-peer-as
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn aspath private-as
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn policy
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn policy inbound
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn policy inbound route-map
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn policy outbound
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn policy outbound route-map
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn policy outbound unsuppress-map
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn enable
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn route-reflector-client
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn route-server-client
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn soft-reconfiguration
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn nexthop-setting
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> address-family l2vpn-evpn add-path-tx
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> password
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> enforce-first-as
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> passive-mode
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> nexthop-connected-check
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> multihop-ttl
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> description
nv unset vrf <vrf-id> router bgp peer-group <peer-group-id> remote-as
nv unset vrf <vrf-id> router bgp route-export
nv unset vrf <vrf-id> router bgp route-export to-evpn
nv unset vrf <vrf-id> router bgp route-export to-evpn route-target
nv unset vrf <vrf-id> router bgp route-export to-evpn route-target <rt-id>
nv unset vrf <vrf-id> router bgp route-import
nv unset vrf <vrf-id> router bgp route-import from-evpn
nv unset vrf <vrf-id> router bgp route-import from-evpn route-target
nv unset vrf <vrf-id> router bgp route-import from-evpn route-target <rt-id>
nv unset vrf <vrf-id> router bgp timers
nv unset vrf <vrf-id> router bgp timers keepalive
nv unset vrf <vrf-id> router bgp timers hold
nv unset vrf <vrf-id> router bgp timers connection-retry
nv unset vrf <vrf-id> router bgp timers route-advertisement
nv unset vrf <vrf-id> router bgp timers conditional-advertise
nv unset vrf <vrf-id> router bgp confederation
nv unset vrf <vrf-id> router bgp confederation member-as
nv unset vrf <vrf-id> router bgp confederation id
nv unset vrf <vrf-id> router bgp neighbor
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id>
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> bfd
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> bfd enable
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> bfd detect-multiplier
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> bfd min-rx-interval
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> bfd min-tx-interval
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> capabilities
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> capabilities extended-nexthop
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> capabilities source-address
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> local-as
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> local-as enable
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> local-as asn
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> local-as prepend
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> local-as replace
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> graceful-restart
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> graceful-restart mode
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> ttl-security
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> ttl-security enable
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> ttl-security hops
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast attribute-mod
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast attribute-mod aspath
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast attribute-mod med
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast attribute-mod nexthop
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast aspath
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast aspath allow-my-asn
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast aspath allow-my-asn enable
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast aspath allow-my-asn origin
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast aspath allow-my-asn occurrences
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast aspath replace-peer-as
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast aspath private-as
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast policy
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast policy inbound
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast policy inbound route-map
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast policy inbound prefix-list
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast policy inbound aspath-list
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast policy outbound
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast policy outbound route-map
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast policy outbound unsuppress-map
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast policy outbound prefix-list
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast policy outbound aspath-list
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast prefix-limits
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast prefix-limits inbound
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast prefix-limits inbound maximum
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast prefix-limits inbound warning-threshold
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast prefix-limits inbound warning-only
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast prefix-limits inbound reestablish-wait
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast default-route-origination
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast default-route-origination enable
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast default-route-origination policy
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast community-advertise
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast community-advertise regular
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast community-advertise extended
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast community-advertise large
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast conditional-advertise
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast conditional-advertise enable
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast conditional-advertise advertise-map
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast conditional-advertise exist-map
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast conditional-advertise non-exist-map
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast enable
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast route-reflector-client
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast route-server-client
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast soft-reconfiguration
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast nexthop-setting
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast add-path-tx
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv4-unicast weight
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast attribute-mod
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast attribute-mod aspath
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast attribute-mod med
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast attribute-mod nexthop
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast aspath
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast aspath allow-my-asn
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast aspath allow-my-asn enable
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast aspath allow-my-asn origin
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast aspath allow-my-asn occurrences
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast aspath replace-peer-as
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast aspath private-as
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast prefix-limits
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast prefix-limits inbound
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast prefix-limits inbound maximum
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast prefix-limits inbound warning-threshold
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast prefix-limits inbound warning-only
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast prefix-limits inbound reestablish-wait
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast default-route-origination
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast default-route-origination enable
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast default-route-origination policy
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast policy
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast policy inbound
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast policy inbound route-map
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast policy inbound prefix-list
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast policy inbound aspath-list
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast policy outbound
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast policy outbound route-map
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast policy outbound unsuppress-map
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast policy outbound prefix-list
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast policy outbound aspath-list
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast community-advertise
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast community-advertise regular
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast community-advertise extended
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast community-advertise large
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast conditional-advertise
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast conditional-advertise enable
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast conditional-advertise advertise-map
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast conditional-advertise exist-map
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast conditional-advertise non-exist-map
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast enable
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast route-reflector-client
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast route-server-client
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast soft-reconfiguration
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast nexthop-setting
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast add-path-tx
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family ipv6-unicast weight
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn attribute-mod
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn attribute-mod aspath
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn attribute-mod med
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn attribute-mod nexthop
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn aspath
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn aspath allow-my-asn
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn aspath allow-my-asn enable
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn aspath allow-my-asn origin
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn aspath allow-my-asn occurrences
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn aspath replace-peer-as
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn aspath private-as
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn policy
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn policy inbound
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn policy inbound route-map
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn policy outbound
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn policy outbound route-map
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn policy outbound unsuppress-map
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn enable
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn route-reflector-client
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn route-server-client
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn soft-reconfiguration
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn nexthop-setting
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> address-family l2vpn-evpn add-path-tx
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> timers
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> timers keepalive
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> timers hold
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> timers connection-retry
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> timers route-advertisement
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> password
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> enforce-first-as
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> passive-mode
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> nexthop-connected-check
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> multihop-ttl
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> description
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> enable
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> type
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> peer-group
nv unset vrf <vrf-id> router bgp neighbor <neighbor-id> remote-as
nv unset vrf <vrf-id> router bgp enable
nv unset vrf <vrf-id> router bgp autonomous-system
nv unset vrf <vrf-id> router bgp router-id
nv unset vrf <vrf-id> router bgp rd
nv unset vrf <vrf-id> router bgp dynamic-peer-limit
nv unset vrf <vrf-id> router static
nv unset vrf <vrf-id> router static <route-id>
nv unset vrf <vrf-id> router static <route-id> distance
nv unset vrf <vrf-id> router static <route-id> distance <distance-id>
nv unset vrf <vrf-id> router static <route-id> distance <distance-id> via
nv unset vrf <vrf-id> router static <route-id> distance <distance-id> via <via-id>
nv unset vrf <vrf-id> router static <route-id> distance <distance-id> via <via-id> flag
nv unset vrf <vrf-id> router static <route-id> distance <distance-id> via <via-id> interface
nv unset vrf <vrf-id> router static <route-id> distance <distance-id> via <via-id> vrf
nv unset vrf <vrf-id> router static <route-id> distance <distance-id> via <via-id> type
nv unset vrf <vrf-id> router static <route-id> distance <distance-id> tag
nv unset vrf <vrf-id> router static <route-id> via
nv unset vrf <vrf-id> router static <route-id> via <via-id>
nv unset vrf <vrf-id> router static <route-id> via <via-id> flag
nv unset vrf <vrf-id> router static <route-id> via <via-id> interface
nv unset vrf <vrf-id> router static <route-id> via <via-id> vrf
nv unset vrf <vrf-id> router static <route-id> via <via-id> type
nv unset vrf <vrf-id> router static <route-id> tag
nv unset vrf <vrf-id> router static <route-id> address-family
nv unset vrf <vrf-id> router pim
nv unset vrf <vrf-id> router pim timers
nv unset vrf <vrf-id> router pim timers keep-alive
nv unset vrf <vrf-id> router pim timers rp-keep-alive
nv unset vrf <vrf-id> router pim ecmp
nv unset vrf <vrf-id> router pim ecmp enable
nv unset vrf <vrf-id> router pim ecmp rebalance
nv unset vrf <vrf-id> router pim msdp-mesh-group
nv unset vrf <vrf-id> router pim msdp-mesh-group <msdp-mesh-group-id>
nv unset vrf <vrf-id> router pim msdp-mesh-group <msdp-mesh-group-id> member-address
nv unset vrf <vrf-id> router pim msdp-mesh-group <msdp-mesh-group-id> member-address <mesh-member-id>
nv unset vrf <vrf-id> router pim msdp-mesh-group <msdp-mesh-group-id> source-address
nv unset vrf <vrf-id> router pim address-family
nv unset vrf <vrf-id> router pim address-family ipv4-unicast
nv unset vrf <vrf-id> router pim address-family ipv4-unicast spt-switchover
nv unset vrf <vrf-id> router pim address-family ipv4-unicast spt-switchover action
nv unset vrf <vrf-id> router pim address-family ipv4-unicast spt-switchover prefix-list
nv unset vrf <vrf-id> router pim address-family ipv4-unicast rp
nv unset vrf <vrf-id> router pim address-family ipv4-unicast rp <rp-id>
nv unset vrf <vrf-id> router pim address-family ipv4-unicast rp <rp-id> group-range
nv unset vrf <vrf-id> router pim address-family ipv4-unicast rp <rp-id> group-range <group-range-id>
nv unset vrf <vrf-id> router pim address-family ipv4-unicast rp <rp-id> prefix-list
nv unset vrf <vrf-id> router pim address-family ipv4-unicast ssm-prefix-list
nv unset vrf <vrf-id> router pim address-family ipv4-unicast register-accept-list
nv unset vrf <vrf-id> router pim address-family ipv4-unicast send-v6-secondary
nv unset vrf <vrf-id> router pim enable
nv unset vrf <vrf-id> router ospf
nv unset vrf <vrf-id> router ospf area
nv unset vrf <vrf-id> router ospf area <area-id>
nv unset vrf <vrf-id> router ospf area <area-id> filter-list
nv unset vrf <vrf-id> router ospf area <area-id> filter-list in
nv unset vrf <vrf-id> router ospf area <area-id> filter-list out
nv unset vrf <vrf-id> router ospf area <area-id> range
nv unset vrf <vrf-id> router ospf area <area-id> range <range-id>
nv unset vrf <vrf-id> router ospf area <area-id> range <range-id> suppress
nv unset vrf <vrf-id> router ospf area <area-id> range <range-id> cost
nv unset vrf <vrf-id> router ospf area <area-id> network
nv unset vrf <vrf-id> router ospf area <area-id> network <network-id>
nv unset vrf <vrf-id> router ospf area <area-id> type
nv unset vrf <vrf-id> router ospf area <area-id> default-lsa-cost
nv unset vrf <vrf-id> router ospf default-originate
nv unset vrf <vrf-id> router ospf default-originate enable
nv unset vrf <vrf-id> router ospf default-originate metric
nv unset vrf <vrf-id> router ospf default-originate metric-type
nv unset vrf <vrf-id> router ospf default-originate route-map
nv unset vrf <vrf-id> router ospf default-originate always
nv unset vrf <vrf-id> router ospf distance
nv unset vrf <vrf-id> router ospf distance external
nv unset vrf <vrf-id> router ospf distance inter-area
nv unset vrf <vrf-id> router ospf distance intra-area
nv unset vrf <vrf-id> router ospf max-metric
nv unset vrf <vrf-id> router ospf max-metric administrative
nv unset vrf <vrf-id> router ospf max-metric on-shutdown
nv unset vrf <vrf-id> router ospf max-metric on-startup
nv unset vrf <vrf-id> router ospf log
nv unset vrf <vrf-id> router ospf log adjacency-changes
nv unset vrf <vrf-id> router ospf redistribute
nv unset vrf <vrf-id> router ospf redistribute static
nv unset vrf <vrf-id> router ospf redistribute static enable
nv unset vrf <vrf-id> router ospf redistribute static metric
nv unset vrf <vrf-id> router ospf redistribute static metric-type
nv unset vrf <vrf-id> router ospf redistribute static route-map
nv unset vrf <vrf-id> router ospf redistribute connected
nv unset vrf <vrf-id> router ospf redistribute connected enable
nv unset vrf <vrf-id> router ospf redistribute connected metric
nv unset vrf <vrf-id> router ospf redistribute connected metric-type
nv unset vrf <vrf-id> router ospf redistribute connected route-map
nv unset vrf <vrf-id> router ospf redistribute kernel
nv unset vrf <vrf-id> router ospf redistribute kernel enable
nv unset vrf <vrf-id> router ospf redistribute kernel metric
nv unset vrf <vrf-id> router ospf redistribute kernel metric-type
nv unset vrf <vrf-id> router ospf redistribute kernel route-map
nv unset vrf <vrf-id> router ospf redistribute bgp
nv unset vrf <vrf-id> router ospf redistribute bgp enable
nv unset vrf <vrf-id> router ospf redistribute bgp metric
nv unset vrf <vrf-id> router ospf redistribute bgp metric-type
nv unset vrf <vrf-id> router ospf redistribute bgp route-map
nv unset vrf <vrf-id> router ospf timers
nv unset vrf <vrf-id> router ospf timers lsa
nv unset vrf <vrf-id> router ospf timers lsa min-arrival
nv unset vrf <vrf-id> router ospf timers lsa throttle
nv unset vrf <vrf-id> router ospf timers spf
nv unset vrf <vrf-id> router ospf timers spf delay
nv unset vrf <vrf-id> router ospf timers spf holdtime
nv unset vrf <vrf-id> router ospf timers spf max-holdtime
nv unset vrf <vrf-id> router ospf timers refresh
nv unset vrf <vrf-id> router ospf enable
nv unset vrf <vrf-id> router ospf reference-bandwidth
nv unset vrf <vrf-id> router ospf rfc1583-compatible
nv unset vrf <vrf-id> router ospf router-id
nv unset vrf <vrf-id> ptp
nv unset vrf <vrf-id> ptp enable
nv unset vrf <vrf-id> table
nv unset nve
nv unset nve vxlan
nv unset nve vxlan mlag
nv unset nve vxlan mlag shared-address
nv unset nve vxlan source
nv unset nve vxlan source address
nv unset nve vxlan flooding
nv unset nve vxlan flooding head-end-replication
nv unset nve vxlan flooding head-end-replication <hrep-id>
nv unset nve vxlan flooding enable
nv unset nve vxlan flooding multicast-group
nv unset nve vxlan enable
nv unset nve vxlan mac-learning
nv unset nve vxlan port
nv unset nve vxlan arp-nd-suppress
nv unset nve vxlan mtu
nv unset acl
nv unset acl <acl-id>
nv unset acl <acl-id> rule
nv unset acl <acl-id> rule <rule-id>
nv unset acl <acl-id> rule <rule-id> match
nv unset acl <acl-id> rule <rule-id> match ip
nv unset acl <acl-id> rule <rule-id> match ip source-port
nv unset acl <acl-id> rule <rule-id> match ip source-port <ip-port-id>
nv unset acl <acl-id> rule <rule-id> match ip dest-port
nv unset acl <acl-id> rule <rule-id> match ip dest-port <ip-port-id>
nv unset acl <acl-id> rule <rule-id> match ip fragment
nv unset acl <acl-id> rule <rule-id> match ip ecn
nv unset acl <acl-id> rule <rule-id> match ip ecn flags
nv unset acl <acl-id> rule <rule-id> match ip ecn ip-ect
nv unset acl <acl-id> rule <rule-id> match ip tcp
nv unset acl <acl-id> rule <rule-id> match ip tcp flags
nv unset acl <acl-id> rule <rule-id> match ip tcp mask
nv unset acl <acl-id> rule <rule-id> match ip tcp state
nv unset acl <acl-id> rule <rule-id> match ip source-ip
nv unset acl <acl-id> rule <rule-id> match ip dest-ip
nv unset acl <acl-id> rule <rule-id> match ip protocol
nv unset acl <acl-id> rule <rule-id> match ip dscp
nv unset acl <acl-id> rule <rule-id> match ip icmp-type
nv unset acl <acl-id> rule <rule-id> match ip icmpv6-type
nv unset acl <acl-id> rule <rule-id> match mac
nv unset acl <acl-id> rule <rule-id> match mac source-mac
nv unset acl <acl-id> rule <rule-id> match mac source-mac-mask
nv unset acl <acl-id> rule <rule-id> match mac dest-mac
nv unset acl <acl-id> rule <rule-id> match mac dest-mac-mask
nv unset acl <acl-id> rule <rule-id> match mac protocol
nv unset acl <acl-id> rule <rule-id> match mac vlan
nv unset acl <acl-id> rule <rule-id> action
nv unset acl <acl-id> rule <rule-id> action permit
nv unset acl <acl-id> rule <rule-id> action deny
nv unset acl <acl-id> rule <rule-id> action log
nv unset acl <acl-id> rule <rule-id> action set
nv unset acl <acl-id> rule <rule-id> action set dscp
nv unset acl <acl-id> rule <rule-id> action set class
nv unset acl <acl-id> rule <rule-id> action set cos
nv unset acl <acl-id> rule <rule-id> action erspan
nv unset acl <acl-id> rule <rule-id> action erspan source-ip
nv unset acl <acl-id> rule <rule-id> action erspan dest-ip
nv unset acl <acl-id> rule <rule-id> action erspan ttl
nv unset acl <acl-id> rule <rule-id> action police
nv unset acl <acl-id> rule <rule-id> action police mode
nv unset acl <acl-id> rule <rule-id> action police burst
nv unset acl <acl-id> rule <rule-id> action police rate
nv unset acl <acl-id> rule <rule-id> action span
nv unset acl <acl-id> type
nv action
nv config apply [<revision>]
nv config save
nv config replace <cue-file>
nv config detach
nv config diff [<revision>] [<revision>]
nv config show
nv config patch <cue-file>
nv config history [<revision>]

Show a Command Description

To see a description for a command, type the command with -h at the end:

cumulus@leaf01:mgmt:~$ nv set mlag backup -h
usage: 
  nv [options] set mlag backup <backup-ip>

Description:
  backup                Set of MLAG backups

Identifiers:
  <backup-ip>           Backup IP of MLAG peer (ipv4 | ipv6)

Output Options:
  -o <format>, --output <format>
                        Supported formats: json, yaml, auto, constable, end-table, commands (default:auto)
  --color (on|off|auto)
                        Toggle coloring of output (default: auto)
  --paginate (on|off|auto)
                        Whether to send output to a pager (default: off)

General Options:
  -h, --help            Show help.

When you use -h, replace any variables in the command with a value. For example, for the nv set vrf <vrf-id> router pim command, type nv set vrf default router pim -h:

cumulus@leaf01:mgmt:~$ nv set vrf default router pim -h
usage: 
  nv [options] set vrf <vrf-id> router pim [address-family ...]
  nv [options] set vrf <vrf-id> router pim [ecmp ...]
  nv [options] set vrf <vrf-id> router pim [enable ...]
  nv [options] set vrf <vrf-id> router pim [msdp-mesh-group ...]
  nv [options] set vrf <vrf-id> router pim [timers ...]

Description:
  pim                   PIM VRF configuration.

Identifiers:
  <vrf-id>              VRF (vrf-name)

Attributes:
  address-family        Address family specific configuration
  ecmp                  Choose all available ECMP paths for a particular RPF.  If 'off', the first nexthop found will be used.  This is the default.
  enable                Turn the feature 'on' or 'off'.  The default is 'off'.
  msdp-mesh-group       To connect multiple PIM-SM multicast domains using RPs.
  timers                Timers

Output Options:
  -o <format>, --output <format>
                        Supported formats: json, yaml, auto, constable, end-table, commands (default:auto)
  --color (on|off|auto)
                        Toggle coloring of output (default: auto)
  --paginate (on|off|auto)
                        Whether to send output to a pager (default: off)

General Options:
  -h, --help            Show help.

NVUE supports both snippets and flexible snippets:

• Use snippets to add configuration to either the /etc/frr/frr.conf or /etc/network/interfaces file.
• Use flexible snippets to manage any other text file on the system.

Snippets

Use snippets if you configure Cumulus Linux with NVUE commands, then want to configure a feature that does not yet support the NVUE Object Model. You create a snippet in yaml format, then add the configuration to either the /etc/frr/frr.conf or /etc/network/interfaces file with the nv config patch command.

/etc/frr/frr.conf Snippets

Example 1: Top Level Configuration

NVUE does not support configuring BGP to peer across the default route. The following example configures BGP to peer across the default route from the default VRF:

1. Create a .yaml file with the following snippet:

   cumulus@switch:~$ sudo nano bgp_snippet.yaml
   - set:
       system:
         config:
           snippet:
             frr.conf: |
               ip nht resolve-via-default
   

2. Run the following command to patch the configuration:

   cumulus@switch:~$ nv config patch bgp_snippet.yaml
   

3. Run the nv config apply command to apply the configuration:

   cumulus@switch:~$ nv config apply
   

4. Verify that the configuration exists at the end of the /etc/frr/frr.conf file:

   cumulus@switch:~$ sudo cat /etc/frr/frr.conf
   ...
   ! end of router ospf block
   !---- CUE snippets ----
   ip nht resolve-via-default
   

Example 2: Nested Configuration

NVUE does not support configuring EVPN route targets using auto derived values from RFC 8365. The following example configures BGP to enable RFC 8365 derived router targets:

1. Create a .yaml file with the following snippet:

   cumulus@switch:~$ sudo nano bgp_snippet.yaml
   - set:
       system:
         config:
           snippet:
             frr.conf: |
               router bgp 65517 vrf default
                 address-family l2vpn evpn
                   autort rfc8365-compatible
   

Make sure to use spaces not tabs; the parser expects spaces in yaml format.

2. Run the following command to patch the configuration:

   cumulus@switch:~$ nv config patch bgp_snippet.yaml
   

3. Run the nv config apply command to apply the configuration:

   cumulus@switch:~$ nv config apply
   

4. Verify that the configuration exists at the end of the /etc/frr/frr.conf file:

   cumulus@switch:~$ sudo cat /etc/frr/frr.conf
   ...
   ! end of router bgp 65517 vrf default
   !---- CUE snippets ----
   router bgp 65517 vrf default
   address-family l2vpn evpn
   autort rfc8365-compatible
   

The snippets for FRR write content to the /etc/frr/frr.conf file. When you apply the configuration and snippet with the nv config apply command, the FRR service goes through and reads in the /etc/frr/frr.conf file.

Example 3: EVPN Multihoming FRR Debugging

NVUE does not support configuring FRR debugging for EVPN multihoming. The following example configures FRR debugging:

1. Create a .yaml file and add the following traditional snippet:

   cumulus@switch:~$ sudo nano mh_debug_snippet.yaml
   - set:
       system:
         config:
           snippet:
             frr.conf: |
               debug bgp evpn mh es
               debug bgp evpn mh route
               debug bgp zebra
               debug zebra evpn mh es
               debug zebra evpn mh mac
               debug zebra evpn mh neigh
               debug zebra evpn mh nh
               debug zebra vxlan
   

2. Run the following command to patch the configuration:

   cumulus@switch:~$ nv config patch mh_debug_snippet.yaml
   

3. Run the nv config apply command to apply the configuration:

   cumulus@switch:~$ nv config apply
   

4. Verify that the configuration exists in the /etc/frr/frr.conf file:

   cumulus@switch:~$ sudo cat /etc/frr/frr.conf
   ...
   !---- NVUE snippets ----
   debug bgp evpn mh es
   debug bgp evpn mh route
   debug bgp zebra
   debug zebra evpn mh es
   debug zebra evpn mh mac
   debug zebra evpn mh neigh
   debug zebra evpn mh nh
   debug zebra vxlan
   

The traditional snippets for FRR write content to the /etc/frr/frr.conf file. When you apply the configuration and snippet with the nv config apply command, the FRR service goes through and reads in the /etc/frr/frr.conf file.

/etc/network/interfaces Snippets

MLAG Timers Example

NVUE supports configuring only one of the MLAG service timeouts (initDelay). The following example configures the MLAG peer timeout to 400 seconds:

1. Create a .yaml file and add the following snippet:

   cumulus@switch:~$ sudo nano mlag_snippet.yaml
   - set:
       system:
         config:
           snippet:
             ifupdown2_eni:
               peerlink.4094: |
                 clagd-args --peerTimeout 400
   

2. Run the following command to patch the configuration:

   cumulus@switch:~$ nv config patch mlag_snippet.yaml
   

3. Run the nv config apply command to apply the configuration:

   cumulus@switch:~$ nv config apply
   

4. Verify that the configuration exists in the peerlink.4094 stanza of the /etc/network/interfaces file:

   cumulus@switch:~$ sudo cat /etc/network/interfaces
   ...
   auto peerlink.4094
   iface peerlink.4094
    clagd-args --peerTimeout 400
    clagd-peer-ip linklocal
    clagd-backup-ip 10.10.10.2
    clagd-sys-mac 44:38:39:BE:EF:AA
    clagd-args --initDelay 180
   ...
   

Traditional Bridge Example

NVUE does not support configuring traditional bridges. The following example configures a traditional bridge called br0 with the IP address 11.0.0.10/24. swp1, swp2 are members of the bridge.

1. Create a .yaml file and add the following snippet:

   cumulus@switch:~$ sudo nano bridge_snippet.yaml
   - set:
       system:
        config:
          snippet:
            ifupdown2_eni:
              eni_stanzas: |
                auto br0
                iface br0
                  address 11.0.0.10/24
                  bridge-ports swp1 swp2
                  bridge-vlan-aware no
   

2. Run the following command to patch the configuration:

   cumulus@switch:~$ nv config patch bridge_snippet.yaml
   

3. Run the nv config apply command to apply the configuration:

   cumulus@switch:~$ nv config apply
   

4. Verify that the configuration exists at the end of the /etc/network/interfaces file:

   cumulus@switch:~$ sudo cat /etc/network/interfaces
   ...
   auto br0
   iface br0
     address 11.0.0.10/24
     bridge-ports swp1 swp2
     bridge-vlan-aware no
   

Flexible Snippets

Flexible snippets are an extension of regular snippets that let you manage any text file on the system. You can add content to an existing text file or create a new text file and add content.

Flexible snippets do not support:

• Binary files.
• Symbolic links.
• More than 1MB of content.
• More than one flexible snippet in the same destination file.

To create flexible snippets:

1. Create a file in yaml format and add each flexible snippet you want to apply in the format shown below.

   cumulus@leaf01:mgmt:~$ sudo nano <filename>.yaml>
   - set:
       system:
        config:
          snippet:
            <snippet-name>:
              file: "<filename>"
              permissions: "<umask-permissions>"
              content: |
                # This is my content
              services:
                 <name>:
                   service: <service-name>
                   action: <action>
   

   NVUE appends the flexible snippet at the end of an existing file. If the file does not exist, NVUE creates the file and adds the content.

   • You can only set the umast permissions to a new file that you create. Adding the permissions: line is optional. The default umask persmissions are 644.
   • You can add a service with an action, such as start, restart, or stop. Adding the services: lines is optional. The SNMP example below restarts the snmpd service.

TACACS+ Client Example

The following example creates a snippet called tacacs-config in a file called tacacs.yaml. The snippet adds the server 192.168.0.30 and the shared secret tacacskey to the /etc/tacplus_servers file.

1. Create the tacacs.yaml snippet:

   cumulus@leaf01:mgmt:~$ sudo nano tacacs.yaml
   - set:
       system:
        config:
          snippet:
            tacacs-config:
              file: "/etc/tacplus_servers"
              content: |
                secret=tacacskey
                server=192.168.0.30
   

2. Run the following command to patch the configuration:

   cumulus@switch:~$ nv config patch tacacs.yaml
   

3. Run the nv config apply command to apply the configuration:

   cumulus@switch:~$ nv config apply
   

NVUE appends the snippet at the end of the /etc/tacplus_servers file.

SNMP Example

The following example creates a snippet called snmp-config in a file called snmp.yaml. The snippet adds content to the /etc/snmp/snmpd.conf file to:

• Configure the switch to listen on any interface on the management VRF.
• Create a read-only community.
• Restart the snmpd service.

1. Create the snmp.yaml snippet:

   cumulus@leaf01:mgmt:~$ sudo nano snmp.yaml
   - set:
       system:
         config:
           snippet:
             snmp-config:
               file: /etc/snmp/snmpd.conf
               content: |
                 # Listen on any interface on MGMT VRF
                 agentaddress udp:@mgmt:161
                 # Create a Read-Only Community
                 rocommunity cumuluspassword default
               services:
                 snmp:
                   service: snmpd
                   action: restart
   

2. Run the following command to patch the configuration:

   cumulus@switch:~$ nv config patch snmp.yaml
   

3. Run the nv config apply command to apply the configuration:

   cumulus@switch:~$ nv config apply
   

NVUE appends the snippet at the end of the /etc/snmp/snmpd.conf file.

Remove a Snippet

To remove a traditional or flexible snippet, edit the snippet’s .yaml file to change set to unset, then patch and apply the configuration. Alternatively, you can use the REST API DELETE and PATCH methods.

The following example removes the MLAG timer traditional snippet created above to configure the MLAG peer timeout:

1. Edit the mlag_snippet.yaml file to change set to unset:

   cumulus@switch:~$ sudo nano mlag_snippet.yaml
   - unset:
       system:
         config:
           snippet:
             ifupdown2_eni:
   

2. Run the following command to patch the configuration:

   cumulus@switch:~$ nv config patch mlag_snippet.yaml
   

3. Run the nv config apply command to apply the configuration:

   cumulus@switch:~$ nv config apply
   

4. Verify that the peer timeout parameter no longer exists in the peerlink.4094 stanza of the /etc/network/interfaces file:

   cumulus@switch:~$ sudo cat /etc/network/interfaces
   ...
   auto peerlink.4094
   iface peerlink.4094
    clagd-peer-ip linklocal
    clagd-backup-ip 10.10.10.2
    clagd-sys-mac 44:38:39:BE:EF:AA
    clagd-args --initDelay 180
   ...
   

This section discusses how to set the time zone, and how to set the date and time on the software clock on the switch. To configure NTP, see Network Time Protocol - NTP. To configure PTP, see Precision Time Protocol - PTP.

Setting the time zone, and the date and time on the software clock requires root privileges; use sudo.

Set the Time Zone

You can use one of these methods to set the time zone on the switch:

• Run NVUE commands.
• Use the guided wizard.
• Edit the /etc/timezone file.

cumulus@switch:~$ nv set system timezone US/Eastern
cumulus@switch:~$ nv config apply

Set the Date and Time

The switch contains a battery backed hardware clock that maintains the time while the switch powers off and between reboots. When the switch is running, the Cumulus Linux operating system maintains its own software clock.

During boot up, the switch copies the time from the hardware clock to the operating system software clock. The software clock takes care of all the timekeeping. During system shutdown, the switch copies the software clock back to the battery backed hardware clock.

You can set the date and time on the software clock with the date command. First, determine your current time zone:

cumulus@switch:~$ date +%Z

To set the software clock according to the configured time zone:

cumulus@switch:~$ sudo date -s "Tue Jan 26 00:37:13 2021"

You can write the current value of the software clock to the hardware clock using the hwclock command:

cumulus@switch:~$ sudo hwclock -w

See man hwclock(8) for more information.

Related Information

• Debian System Administrator's Manual - Time

This section provides information about using the NVUE API.

Access the API

To access the NVUE API, run these commands on the switch:

cumulus@switch:~$ sudo ln -s /etc/nginx/sites-{available,enabled}/nvue.conf
cumulus@switch:~$ sudo sed -i 's/listen localhost:8765 ssl;/listen \[::\]:8765 ipv6only=off ssl;/g' /etc/nginx/sites-available/nvue.conf
cumulus@switch:~$ sudo systemctl restart nginx

Access the NVUE REST API from a Front Panel Port

To access the NVUE REST API from a front panel port (swp) on the switch:

1. Ensure that the nvue.conf file is present in the /etc/nginx/sites-enabled directory.

   Either copy the packaged template file nvue.conf from the /etc/nginx/sites-available directory to the /etc/nginx/sites-enabled directory or create a symbolic link.

2. Edit the nvue.conf file and add the listen directive with the IPv4 or IPv6 address of the swp interface you want to use.

   The default nvue.conf file includes a single listen localhost:8765 ssl; entry. Add an entry for each swp interface with its IP address. Make sure to use an accessible HTTP (TCP) port (subject to any ACL/firewall rules). For information on the NGINX listen directive, see the NGINX documentation.

3. Restart the nginx service:

   cumulus@switch:~$ sudo systemctl reload-or-restart nginx
   

Transport Layer Security

Cumulus Linux contains a self-signed certificate and private key used server-side in this application so that it works out of the box; however, NVIDIA recommends you use your own certificates and keys. Certificates must be in PEM format.

For step by step documentation on generating self-signed certificates and keys, and installing them on the switch, refer to the Ubuntu Certificates and Security documentation.

After installing the certificates and keys, edit the /etc/nginx/sites-available/nvue.conf file to set the ssl_certificate and the ssl_certificate_key values to your keys, then restart NGINX with the sudo systemctl restart nginx command.

Run cURL Commands

You can run the cURL commands from the command line. Use the username and password for the switch. For example:

cumulus@switch:~$ curl  -u 'cumulus:CumulusLinux!' --insecure https://127.0.0.1:8765/nvue_v1/interface
{
  "eth0": {
    "ip": {
      "address": {
        "192.168.200.12/24": {}
      }
    },
    "link": {
      "mtu": 1500,
      "state": {
        "up": {}
      },
      "stats": {
        "carrier-transitions": 2,
        "in-bytes": 184151,
        "in-drops": 0,
        "in-errors": 0,
        "in-pkts": 2371,
        "out-bytes": 117506,
        "out-drops": 0,
        "out-errors": 0,
        "out-pkts": 762
      }
...

Set a Configuration Change

To make a configuration change with the NVUE API:

1. Create a new revision ID using a POST:

   cumulus@switch:~$ curl -u 'cumulus:cumulus' --insecure -X POST https://127.0.0.1:8765/nvue_v1/revision
   {
     "changeset/cumulus/2021-11-02_16.09.18_5Z1K": {
       "state": "pending",
       "transition": {
         "issue": {},
         "progress": ""
       }
     }
   }
   
   

   To allow the cumulus user access to the NVUE API, you must change the default password for the cumulus user.

2. Record the revision ID. In the above example, the revision ID is "changeset/cumulus/2021-11-02_16.09.18_5Z1K"

3. Make the change using a PATCH and link it to the revision ID:

   cumulus@switch:~$ curl -u 'cumulus:cumulus' -d '{"99.99.99.99/32": {}}' -H 'Content-Type: application/json' --insecure -X PATCH https://127.0.0.1:8765/nvue_v1/interface/lo/ip/address?rev=changeset/cumulus/2021-11-02_16.09.18_5Z1K
   {
     "99.99.99.99/32": {}
   }
   

4. Apply the changes using a PATCH to the revision changeset. You must use the full key value for the revision and replace /​ with %2F​ in the list:

   cumulus@switch:~$ curl -u 'cumulus:cumulus' -d '{"state":"apply"}' -H 'Content-Type:application/json' --insecure -X PATCH https://127.0.0.1:8765/nvue_v1/revision/changeset%2Fcumulus%2F2021-11-02_16.09.18_5Z1K
   {
     "state": "apply",
     "transition": {
       "issue": {},
       "progress": ""
     }
   }
   

5. Review the status of the apply and the configuration:

   cumulus@switch:~$ curl -u 'cumulus:cumulus' --insecure https://127.0.0.1:8765/nvue_v1/revision/changeset%2Fcumulus%2F2021-11-02_16.09.18_5Z1K
   {
     "state": "applied",
     "transition": {
       "issue": {},
       "progress": ""
     }
   }
   

   cumulus@switch:~$ curl -u 'cumulus:cumulus' --insecure https://127.0.0.1:8765/nvue_v1/interface/lo/ip/address
   {
     "127.0.0.1/8": {},
     "99.99.99.99/32": {},
     "::1/128": {}
   }
   

Unset a Configuration Change

To unset a change, use the null value to the key. For example, to delete vlan100 from a switch, use the following syntax:

cumulus@switch:~$ curl -u 'cumulus:cumulus' -d '{"vlan100":null}' -H 'Content-Type: application/json' --insecure -X PATCH https://127.0.0.1:8765/nvue_v1/interface?rev=changeset/cumulus/2021-11-29_11.46.23_6C7T

When you unset a change, you must still use the PATCH action. The value indicates removal of the entry. The data is {"vlan100":null} with the PATCH action.

Troubleshoot Configuration Changes

When a configuration change fails, you see an error in the change request.

Configuration Fails Because of a Dependency

If you stage a configuration but it fails because of a dependency, the failure shows the reason. In the following example, the change fails because the BGP router ID is not set:

cumulus@switch:~$ curl -u 'cumulus:cumulus' --insecure https://127.0.0.1:8765/nvue_v1/revision/changeset%2Fcumulus%2F2021-11-02_13.57.25_5Z1H
{
  "state": "invalid",
  "transition": {
    "issue": {
      "0": {
        "code": "config_invalid",
        "data": {
          "location": "router.bgp.enable",
          "reason": "BGP requires router-id to be set globally or in the VRF.\n"
        },
        "message": "Config invalid at router.bgp.enable: BGP requires router-id to be set globally or in the VRF.\n",
        "severity": "error"
      }
    },
    "progress": "Invalid config"
  }
}

The staged configuration is missing router-id:

cumulus@switch:~$ curl -u 'cumulus:cumulus' --insecure https://127.0.0.1:8765/nvue_v1/vrf/default/router/bgp?rev=changeset%2Fcumulus%2F2021-11-02_13.57.25_5Z1H
{
  "autonomous-system": 65999,
  "enable": "on"
}

Configuration Apply Fails with Warnings

In some cases, such as the first push with NVUE or if you change a file manually instead of using NVUE, you see a warning prompt and the apply fails:

cumulus@switch:~$ curl -u 'cumulus:cumulus' --insecure -X GET https://127.0.0.1:8765/nvue_v1/revision/changeset%2Fcumulus%2F2021-11-02_16.09.18_5Z1K
{
  "changeset/cumulus/2021-11-02_16.09.18_5Z1K": {
    "state": "ays_fail",
    "transition": {
      "issue": {
        "0": {
          "code": "client_timeout",
          "data": {},
          "message": "Timeout while waiting for client response",
          "severity": "error"
        }
      },
      "progress": "Aborted apply after warnings"
    }
  }

To resolve this issue, include "auto-prompt":{"ays": "ays_yes"} to the configuration apply:

cumulus@switch:~$ curl -u 'cumulus:cumulus' -d '{"state":"apply","auto-prompt":{"ays": "ays_yes"}}' -H 'Content-Type:application/json' --insecure -X PATCH https://127.0.0.1:8765/nvue_v1/revision/changeset%2Fcumulus%2F2021-11-02_16.09.18_5Z1K

NVUE REST API Documentation

For information about using the NVUE REST API, refer to the NVUE API documentation.

Related Information

• NGINX documentation
• Ubuntu Certificates and Security documentation

The ntpd daemon running on the switch implements the NTP protocol. It synchronizes the system time with time servers in the /etc/ntp.conf file. The ntpd daemon starts at boot by default.

Configure NTP Servers

The default NTP configuration includes the following servers, which are in the /etc/ntp.conf file:

• server 0.cumulusnetworks.pool.ntp.org iburst
• server 1.cumulusnetworks.pool.ntp.org iburst
• server 2.cumulusnetworks.pool.ntp.org iburst
• server 3.cumulusnetworks.pool.ntp.org iburst

To add the NTP servers you want to use, run the following commands. Include the iburst option to increase the sync speed.

cumulus@switch:~$ nv set service ntp default server 4.cumulusnetworks.pool.ntp.org iburst on
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo nano /etc/ntp.conf
# pool.ntp.org maps to about 1000 low-stratum NTP servers.  Your server will
# pick a different set every time it starts up.  Please consider joining the
# pool: <http://www.pool.ntp.org/join.html>
server 0.cumulusnetworks.pool.ntp.org iburst
server 1.cumulusnetworks.pool.ntp.org iburst
server 2.cumulusnetworks.pool.ntp.org iburst
server 3.cumulusnetworks.pool.ntp.org iburst
server 4.cumulusnetworks.pool.ntp.org iburst

To verify that ntpd is running on the system:

cumulus@switch:~$ ps -ef | grep ntp
ntp       4074     1  0 Jun20 ?        00:00:33 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 101:102

To check the NTP peer status:

cumulus@switch:~$ nv show service ntp default server

cumulus@switch:~$ ntpq -p
      remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
+ec2-34-225-6-20 129.6.15.30      2 u   73 1024  377   70.414   -2.414   4.110
+lax1.m-d.net    132.163.96.1     2 u   69 1024  377   11.676    0.155   2.736
*69.195.159.158  199.102.46.72    2 u  133 1024  377   48.047   -0.457   1.856
-2.time.dbsinet. 198.60.22.240    2 u 1057 1024  377   63.973    2.182   2.692

The following example commands remove some of the default NTP servers:

cumulus@switch:~$ nv unset service ntp default server 0.cumulusnetworks.pool.ntp.org
cumulus@switch:~$ nv unset service ntp default server 1.cumulusnetworks.pool.ntp.org
cumulus@switch:~$ nv unset service ntp default server 2.cumulusnetworks.pool.ntp.org
cumulus@switch:~$ nv unset service ntp default server 3.cumulusnetworks.pool.ntp.org
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo nano /etc/ntp.conf
...
# pool.ntp.org maps to about 1000 low-stratum NTP servers.  Your server will
# pick a different set every time it starts up.  Please consider joining the
# pool: <http://www.pool.ntp.org/join.html>
server 4.cumulusnetworks.pool.ntp.org iburst
...

Specify the NTP Source Interface

By default, the source interface that NTP uses is eth0. The following example command configures the NTP source interface to be swp10.

cumulus@switch:~$ nv set service ntp default listen swp10
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo nano /etc/ntp.conf
...
# Specify interfaces
interface listen swp10
...

Use NTP in a DHCP Environment

You can use DHCP to specify your NTP servers. Ensure that the DHCP-generated configuration file /run/ntp.conf.dhcp exists. The /etc/dhcp/dhclient-exit-hooks.d/ntp script generates this file, which is a copy of the default /etc/ntp.conf file with a modified server list from the DHCP server. If this file does not exist and you plan on using DHCP in the future, you can copy your current /etc/ntp.conf file to the location of the DHCP file.

To use DHCP to specify your NTP servers, run the sudo -E systemctl edit ntp.service command and add the ExecStart= line:

cumulus@switch:~$ sudo -E systemctl edit ntp.service
[Service]
ExecStart=
ExecStart=/usr/sbin/ntpd -n -u ntp:ntp -g -c /run/ntp.conf.dhcp

To validate that your configuration, run these commands:

cumulus@switch:~$ sudo systemctl restart ntp
cumulus@switch:~$ sudo systemctl status -n0 ntp.service

If the state is not Active, or the alternate configuration file does not appear in the ntp command line, it is likely that you made a configuration mistake. Correct the mistake and rerun the commands above to verify.

Configure NTP with Authorization Keys

For added security, you can configure NTP to use authorization keys.

Configure the NTP Server

1. Create a .keys file, such as /etc/ntp.keys. Specify a key identifier (a number between 1 and 65535), an encryption method (M for MD5), and the password. The following provides an example:

   #
   # PLEASE DO NOT USE THE DEFAULT VALUES HERE.
   #
   #65535  M  akey
   #1      M  pass
   
   1  M  CumulusLinux!
   

2. In the /etc/ntp.conf file, add a pointer to the /etc/ntp.keys file you created above and specify the key identifier. For example:

   keys /etc/ntp/ntp.keys
   trustedkey 1
   controlkey 1
   requestkey 1
   

3. Restart NTP with the sudo systemctl restart ntp command.

Configure the NTP Client

The NTP client is the Cumulus Linux switch.

1. Create the same .keys file you created on the NTP server (/etc/ntp.keys). For example:

   cumulus@switch:~$  sudo nano /etc/ntp.keys
   #
   # DO NOT USE THE DEFAULT VALUES HERE.
   #
   #65535  M  akey
   #1      M  pass
   
   1  M  CumulusLinux!
   

2. Edit the /etc/ntp.conf file to specify the server you want to use, the key identifier, and a pointer to the /etc/ntp.keys file you created in step 1. For example:

   cumulus@switch:~$ sudo nano /etc/ntp.conf
   ...
   # You do need to talk to an NTP server or two (or three).
   #pool ntp.your-provider.example
   # OR
   #server ntp.your-provider.example
   
   # pool.ntp.org maps to about 1000 low-stratum NTP servers.  Your server will
   # pick a different set every time it starts up.  Please consider joining the
   # pool: <http://www.pool.ntp.org/join.html>
   #server 0.cumulusnetworks.pool.ntp.org iburst
   #server 1.cumulusnetworks.pool.ntp.org iburst
   #server 2.cumulusnetworks.pool.ntp.org iburst
   #server 3.cumulusnetworks.pool.ntp.org iburst
   server 10.50.23.121 key 1
   
   #keys
   keys /etc/ntp.keys
   trustedkey 1
   controlkey 1
   requestkey 1
   ...
   

3. Restart NTP in the active VRF (default or management). For example:

   cumulus@switch:~$ systemctl restart ntp@mgmt.service
   

4. Wait a few minutes, then run the ntpq -c as command to verify the configuration:

   cumulus@switch:~$ ntpq -c as
   
   ind assid status  conf reach auth condition  last_event cnt
   ===========================================================
     1 40828  f014   yes   yes   ok     reject   reachable  1
   

   After a successful authorization, you see the following command output:

   cumulus@switch:~$ ntpq -c as
   
   ind assid status  conf reach auth condition  last_event cnt
   ===========================================================
     1 40828  f61a   yes   yes   ok   sys.peer    sys_peer  1
   

Considerations

NTP in Cumulus Linux uses the /usr/share/zoneinfo/leap-seconds.list file, which expires periodically and results in generated log messages about the expiration. When the file expires, update it from https://www.ietf.org/timezones/data/leap-seconds.list or upgrade the tzdata package to the newest version.

Related Information

• NTP website
• Wikipedia - Network Time Protocol
• Debian wiki - NTP

Cumulus Linux supports IEEE 1588-2008 Precision Timing Protocol (PTPv2), which defines the algorithm and method for synchronizing clocks of various devices across packet-based networks, including Ethernet switches and IP routers.

PTP is capable of sub-microsecond accuracy. The clocks are in a master-slave hierarchy, where the slaves synchronize to their masters, which can be slaves to their own masters. The best master clock (BMC) algorithm, which runs on every clock, creates and updates the hierarchy automatically. The grandmaster clock is the top-level master. To provide a high-degree of accuracy, a Global Positioning System (GPS) time source typically synchronizes the grandmaster clock.

In the following example:

• Boundary clock 2 receives time from Master 1 (the grandmaster) on a PTP slave port, sets its clock and passes the time down from the PTP master port to Boundary clock 1.
• Boundary clock 1 receives the time on a PTP slave port, sets its clock and passes the time down the hierarchy through the PTP master ports to the hosts that receive the time.

Cumulus Linux and PTP

PTP in Cumulus Linux uses the linuxptp package that includes the following programs:

• ptp4l provides the PTP protocol and state machines
• phc2sys provides PTP Hardware Clock and System Clock synchronization
• timemaster provides System Clock and PTP synchronization

Cumulus Linux supports:

• PTP boundary clock mode only (the switch provides timing to downstream servers; it is a slave to a higher-level clock and a master to downstream clocks).
• Both IPv4 and IPv6 UDP PTP encapsulation.
• Only a single PTP domain per network.
• PTP on layer 3 interfaces, layer 3 bonds, trunk ports, and switch ports belonging to a VLAN.
• Multicast, unicast, and mixed message mode.
• End-to-End delay mechanism. Cumulus Linux does not support Peer-to-Peer.
• Two-step clock correction mode, where PTP notes the time when the packet goes out of the port and sends the time in a separate (follow-up) message. Cumulus Linux does not support one-step mode.
• Hardware time stamping for PTP packets. This allows PTP to avoid inaccuracies caused by message transfer delays and improves the accuracy of time synchronization.

Basic Configuration

Basic PTP configuration requires you:

• Enable PTP on the switch.
• Configure PTP on at least one interface; this can be a layer 3 routed port, switch port, or trunk port. You do not need to specify which is a master interface and which is a slave interface; the PTP Best Master Clock Algorithm (BMCA) determines the master and slave.

The basic configuration shown below uses the default PTP settings:

• The clock mode is Boundary. This is the only clock mode that Cumulus Linux supports.

• The PTP clock domain is 0.
• PTP Priority1 and Priority2 are both 128.
• The DSCP is 46 for both general and event messages.
• The PPT interface transport mode is IPv4.
• Announce messages from any master are accepted.
• The PTP Interface Message Mode is multicast.
• The delay mechanism is End-to-End (E2E).
• The hardware packet time stamping mode is two-step.

To configure optional settings, such as the PTP profile, domain, priority, and DSCP, the PTP interface transport mode and timers, and PTP monitoring, see the Optional Configuration sections below.

cumulus@switch:~$ nv set service ptp 1 enable on
cumulus@switch:~$ nv set interface swp1 ip address 10.0.0.9/32
cumulus@switch:~$ nv set interface swp2 ip address 10.0.0.10/32
cumulus@switch:~$ nv set interface swp1 ptp enable on
cumulus@switch:~$ nv set interface swp2 ptp enable on
cumulus@switch:~$ nv config apply

cumulus@switch:~$ nv set service ptp 1 enable on
cumulus@switch:~$ nv set bridge domain br_default
cumulus@switch:~$ nv set bridge domain br_default type vlan-aware
cumulus@switch:~$ nv set bridge domain br_default vlan 10-30
cumulus@switch:~$ nv set bridge domain br_default vlan 10 ptp enable on
cumulus@switch:~$ nv set interface vlan10 type svi
cumulus@switch:~$ nv set interface vlan10 ip address 10.1.10.2/24
cumulus@switch:~$ nv set interface vlan10 ptp enable on
cumulus@switch:~$ nv set interface swp1 bridge domain br_default
cumulus@switch:~$ nv set interface swp1 bridge domain br_default vlan 10
cumulus@switch:~$ nv set interface swp1 ptp enable on
cumulus@switch:~$ nv config apply

cumulus@switch:~$ nv set service ptp 1 enable on
cumulus@switch:~$ nv set bridge domain br_default
cumulus@switch:~$ nv set bridge domain br_default type vlan-aware
cumulus@switch:~$ nv set bridge domain br_default vlan 10-30
cumulus@switch:~$ nv set bridge domain br_default vlan 10 ptp enable on
cumulus@switch:~$ nv set interface vlan10 type svi
cumulus@switch:~$ nv set interface vlan10 ip address 10.1.10.2/24
cumulus@switch:~$ nv set interface swp2 bridge domain br_default
cumulus@switch:~$ nv set interface swp2 bridge domain br_default access 10
cumulus@switch:~$ nv set interface swp2 ptp enable on
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo nano /etc/ptp4l.conf
...
[global]
#
# Default Data Set
#
slaveOnly               0
priority1               128
priority2               128
domainNumber            0

twoStepFlag             1
dscp_event              46
dscp_general            46

offset_from_master_min_threshold   -50
offset_from_master_max_threshold   50
mean_path_delay_threshold          200

#
# Run time options
#
logging_level           6
path_trace_enabled      0
use_syslog              1
verbose                 0
summary_interval        0

#
# servo parameters
#
pi_proportional_const          0.000000
pi_integral_const              0.000000
pi_proportional_scale          0.700000
pi_proportional_exponent       -0.300000
pi_proportional_norm_max       0.700000
pi_integral_scale              0.300000
pi_integral_exponent           0.400000
pi_integral_norm_max           0.300000
step_threshold                 0.000002
first_step_threshold           0.000020
max_frequency                  900000000
sanity_freq_limit              0

#
# Default interface options
#
time_stamping                  software


# Interfaces in which ptp should be enabled
# these interfaces should be routed ports
# if an interface does not have an ip address
# the ptp4l will not work as expected.

[swp1]
udp_ttl                 1
masterOnly              0
delay_mechanism         E2E
network_transport       UDPv4

[swp2]
udp_ttl                 1
masterOnly              0
delay_mechanism         E2E
network_transport       UDPv4

[swp1]
l2_mode                 trunk
vlan_intf               vlan10
src_ip                  10.1.10.2
udp_ttl                 1
masterOnly              0
delay_mechanism         E2E
network_transport       UDPv4

[swp2]
l2_mode                 access
vlan_intf               vlan10
src_ip                  10.1.10.2
udp_ttl                 1
masterOnly              0
delay_mechanism         E2E
network_transport       UDPv4

Optional Global PTP Configuration

PTP Profiles

PTP profiles are a standardized set of configurations and rules intended to meet the requirements of a specific application. Profiles define required, allowed, and restricted PTP options, network restrictions, and performance requirements.

Cumulus Linux supports the following predefined profiles:

• IEEE 1588 is the profile specified in the IEEE 1588 standard. This profile addresses some common applications and does not have any network restrictions.
• ITU 8275.1 is the PTP profile for use in telecom networks that require phase or time-of-day synchronization. Each device in the network must participate in the PTP protocol.

The following table shows the default parameter values for the predefined profiles.

The switch has a predefined default profile of each profile type, one for IEEE1588 and one for ITU8275.1. You can configure the switch to use a predefined profile or you can create a custom profile. You can change the profile settings of the predfined profiles, such as the announce rate, sync rate, domain, priority, transport, and so on. These changes conform to the ranges and allowed values of the profile type. You can also configure these parameters for individual PTP interfaces. When you configure parameters for an individual interface, the configuration takes precedence over the profile configuration. The interface is not part of the profile.

To set a predefined profile:

cumulus@switch:~$ nv set service ptp 1 current-profile default-itu-8275-1
cumulus@switch:~$ nv config apply

cumulus@switch:~$ nv set service ptp 1 current-profile default-1588
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo nano /etc/ptp4l.conf
...
[global]
#
# Default Data Set
#
slaveOnly                      0
priority1                      128
priority2                      128
domainNumber                   24

twoStepFlag                    1
dscp_event                     46
dscp_general                   46
dataset_comparison             G.8275.x
G.8275.defaultDS.localPriority 128
G.8275.portDS.localPriority    128
ptp_dst_mac                    01:80:C2:00:00:0E
network_transport              L2

#
# Port Data Set
#
logAnnounceInterval            -3
logSyncInterval                -4
logMinDelayReqInterval         -4
announceReceiptTimeout         3
delay_mechanism                E2E

offset_from_master_min_threshold   -50
offset_from_master_max_threshold   50
mean_path_delay_threshold          200

#
# Run time options
#
logging_level                  6
path_trace_enabled             0
use_syslog                     1
verbose                        0
summary_interval               0

#
# servo parameters
#
pi_proportional_const          0.000000
pi_integral_const              0.000000
pi_proportional_scale          0.700000
pi_proportional_exponent       -0.300000
pi_proportional_norm_max       0.700000
pi_integral_scale              0.300000
pi_integral_exponent           0.400000
pi_integral_norm_max           0.300000
step_threshold                 0.000002
first_step_threshold           0.000020
max_frequency                  900000000
sanity_freq_limit              0

#
# Default interface options
#
time_stamping                  software


# Interfaces in which ptp should be enabled
# these interfaces should be routed ports
# if an interface does not have an ip address
# the ptp4l will not work as expected.

[swp1]
udp_ttl                 1
masterOnly              0
delay_mechanism         E2E
network_transport       L2

[swp2]
udp_ttl                 1
masterOnly              0
delay_mechanism         E2E
network_transport       L2

cumulus@switch:~$ sudo systemctl restart ptp4l.service

cumulus@switch:~$ sudo nano /etc/ptp4l.conf
[global]
#
# Default Data Set
#
slaveOnly                      0
priority1                      128
priority2                      128
domainNumber                   0

twoStepFlag                    1
dscp_event                     46
dscp_general                   46
dataset_comparison             ieee1588

#
# Port Data Set
#
logAnnounceInterval            1
logSyncInterval                0
logMinDelayReqInterval         0
announceReceiptTimeout         3
delay_mechanism                E2E

offset_from_master_min_threshold   -50
offset_from_master_max_threshold   50
mean_path_delay_threshold          200

#
# Run time options
#
logging_level                  6
path_trace_enabled             0
use_syslog                     1
verbose                        0
summary_interval               0

#
# servo parameters
#
pi_proportional_const          0.000000
pi_integral_const              0.000000
pi_proportional_scale          0.700000
pi_proportional_exponent       -0.300000
pi_proportional_norm_max       0.700000
pi_integral_scale              0.300000
pi_integral_exponent           0.400000
pi_integral_norm_max           0.300000
step_threshold                 0.000002
first_step_threshold           0.000020
max_frequency                  900000000
sanity_freq_limit              0

#
# Default interface options
#
time_stamping                  software


# Interfaces in which ptp should be enabled
# these interfaces should be routed ports
# if an interface does not have an ip address
# the ptp4l will not work as expected.

[swp1]
udp_ttl                 1
masterOnly              0
delay_mechanism         E2E
network_transport       UDPv4

[swp2]
udp_ttl                 1
masterOnly              0
delay_mechanism         E2E
network_transport       UDPv4

cumulus@switch:~$ sudo systemctl restart ptp4l.service

To create a custom profile:

cumulus@switch:~$  nv set service ptp 1 profile CUSTOM1 
cumulus@switch:~$  nv set service ptp 1 profile CUSTOM1 profile-type itu-g-8275-1  
cumulus@switch:~$  nv set service ptp 1 profile CUSTOM1 domain 19
cumulus@switch:~$  nv set service ptp 1 profile CUSTOM1 announce-timeout 5
cumulus@switch:~$  nv set service ptp 1 current-profile CUSTOM1
cumulus@switch:~$  nv config apply

cumulus@switch:~$ sudo nano /etc/ptp4l.conf
[global]
#
# Default Data Set
#
slaveOnly                      0
priority1                      128
priority2                      128
domainNumber                   3

twoStepFlag                    1
dscp_event                     46
dscp_general                   46
dataset_comparison             G.8275.x
G.8275.defaultDS.localPriority 128
G.8275.portDS.localPriority    128
ptp_dst_mac                    01:80:C2:00:00:0E
network_transport              L2

#
# Port Data Set
#
logAnnounceInterval            5
logSyncInterval                -4
logMinDelayReqInterval         -4
announceReceiptTimeout         3
delay_mechanism                E2E

offset_from_master_min_threshold   -50
offset_from_master_max_threshold   50
mean_path_delay_threshold          200

#
# Run time options
#
logging_level                  6
path_trace_enabled             0
use_syslog                     1
verbose                        0
summary_interval               0

#
# servo parameters
#
pi_proportional_const          0.000000
pi_integral_const              0.000000
pi_proportional_scale          0.700000
pi_proportional_exponent       -0.300000
pi_proportional_norm_max       0.700000
pi_integral_scale              0.300000
pi_integral_exponent           0.400000
pi_integral_norm_max           0.300000
step_threshold                 0.000002
first_step_threshold           0.000020
max_frequency                  900000000
sanity_freq_limit              0

#
# Default interface options
#
time_stamping                  software


# Interfaces in which ptp should be enabled
# these interfaces should be routed ports
# if an interface does not have an ip address
# the ptp4l will not work as expected.

[swp1]
udp_ttl                 1
masterOnly              0
delay_mechanism         E2E
network_transport       L2

[swp2]
udp_ttl                 1
masterOnly              0
delay_mechanism         E2E
network_transport       L2

cumulus@switch:~$ sudo systemctl restart ptp4l.service

To show the current PTP profile setting, run the nv show service ptp <ptp-instance> command:

cumulus@switch:~$ nv show service ptp 1
                             operational  applied             description
---------------------------  -----------  ------------------  --------------------------------------------------------------------
enable                       on           on                  Turn the feature 'on' or 'off'.  The default is 'off'.
current-profile                           default-itu-8275-1  Current PTP profile index
domain                       24           0                   Domain number of the current syntonization
ip-dscp                      46           46                  Sets the Diffserv code point for all PTP packets originated locally.
priority1                    128          128                 Priority1 attribute of the local clock
priority2                    128          128                 Priority2 attribute of the local clock
two-step                     on           on                  Determines if the Clock is a 2 step clock
...

To show the settings for a profile, run the nv show service ptp 1 profile <profile-name> command:

cumulus@switch:~$ nv show service ptp 1 profile CUSTOM1
                    operational  applied         description
------------------  -----------  ------------    ----------------------------------------------------------------------
announce-interval                -3              Mean time interval between successive Announce messages.  It's spec...
announce-timeout                 5               The number of announceIntervals that have to pass without receipt o...
delay-mechanism                  end-to-end      Mode in which PTP message is transmitted.
delay-req-interval               -4              The minimum permitted mean time interval between successive Delay R...
domain                           19              Domain number of the current syntonization
local-priority                   128             Local priority attribute of the local clock
priority1                        128             Priority1 attribute of the local clock
priority2                        128             Priority2 attribute of the local clock
profile-type                     itu-g-8275-1    The profile type
sync-interval                    -4              The mean SyncInterval for multicast messages.  It's specified as a...
transport                        802.3           Transport method for the PTP messages.

Clock Domains

PTP domains allow different independent timing systems to be present in the same network without confusing each other. A PTP domain is a network or a portion of a network within which all the clocks synchronize. Every PTP message contains a domain number. A PTP instance works in only one domain and ignores messages that contain a different domain number.

You can specify multiple PTP clock domains. PTP isolates each domain from other domains so that each domain is a different PTP network. You can specify a number between 0 and 127.

The following example commands configure domain 3:

cumulus@switch:~$ nv set service ptp 1 domain 3
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo nano /etc/ptp4l.conf
[global]
#
# Default Data Set
#
slaveOnly               0
priority1               128
priority2               128
domainNumber            3
...

cumulus@switch:~$ sudo systemctl restart ptp4l.service

PTP Priority

The BMC selects the PTP master according to the criteria in the following order:

1. Priority 1
2. Clock class
3. Clock accuracy
4. Clock variance
5. Priority 2
6. Port ID

Use the PTP priority to select the best master clock. You can set priority 1 and 2:

• Priority 1 overrides the clock class and quality selection criteria to select the best master clock.
• Priority 2 identifies primary and backup clocks among identical redundant Grandmasters.

The range for both priority1 and priority2 is between 0 and 255. The default priority is 128. For the boundary clock, use a number above 128. The lower priority applies first.

The following example commands set priority 1 and priority 2 to 200:

cumulus@switch:~$ nv set service ptp 1 priority1 200
cumulus@switch:~$ nv set service ptp 1 priority2 200
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo nano /etc/ptp4l.conf
[global]
#
# Default Data Set
#
slaveOnly               0
priority1               200
priority2               200
domainNumber            3
...

cumulus@switch:~$ sudo systemctl restart ptp4l.service

DSCP

You can configure the DiffServ code point (DSCP) value for all PTP IPv4 packets originated locally. You can set a value between 0 and 63.

cumulus@switch:~$ nv set service ptp 1 ip-dscp 22
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo nano /etc/ptp4l.conf
[global]
#
# Default Data Set
#
slaveOnly               0
priority1               200
priority2               200
domainNumber            3

twoStepFlag             1
dscp_event              22
dscp_general            22
...

cumulus@switch:~$ sudo systemctl restart ptp4l.service

Optional PTP Interface Configuration

Transport Mode

By default, Cumulus Linux encapsulates PTP messages in UDP/IPV4 frames. To encapsulate PTP messages on an interface in UDP/IPV6 frames:

cumulus@switch:~$ nv set interface swp1 ptp transport ipv6
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo nano /etc/ptp4l.conf
...
# Default interface options
#
time_stamping           hardware

# Interfaces in which ptp should be enabled
# these interfaces should be routed ports
# if an interface does not have an ip address
# the ptp4l will not work as expected.

[swp1]
udp_ttl                 1
masterOnly              0
delay_mechanism         E2E
network_transport       UDPv6

[swp2]
udp_ttl                 1
masterOnly              0
delay_mechanism         E2E
network_transport       UDPv6
...

cumulus@switch:~$ sudo systemctl restart ptp4l.service

Forced Master Mode

By default, PTP ports are in auto mode, where the BMC algorithm determines the state of the port.

You can configure Forced Master mode on a PTP port so that it is always in a master state and the BMC algorithm does not run for this port. This port ignores any Announce messages it receives.

cumulus@switch:~$ nv set interface swp1 ptp forced-master on
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo nano /etc/ptp4l.conf
...
# Default interface options
#
time_stamping           hardware

# Interfaces in which ptp should be enabled
# these interfaces should be routed ports
# if an interface does not have an ip address
# the ptp4l will not work as expected.

[swp1]
udp_ttl                 1
masterOnly              1
delay_mechanism         E2E
network_transport       UDPv4
...

cumulus@switch:~$ sudo systemctl restart ptp4l.service

Message Mode

Cumulus Linux supports the following PTP message modes:

• Multicast, where the ports subscribe to two multicast addresses, one for event messages with timestamps and the other for general messages without timestamps. The Sync message that the master sends is a multicast message; all slave ports receive this message because the slaves need the time from the master. The slave ports in turn generate a Delay Request to the master. This is a multicast message that the intended master for the message and other slave ports receive. Similarly, all slave ports in addition to the intended slave port receive the master’s Delay Response. The slave ports receiving the unintended Delay Requests and Responses need to drop the packets. This can affect network bandwidth if there are hundreds of slave ports.
• Unicast, where you configure the port as a unicast client or server. The client sends out requests for Announce, Sync and Delay Response from its list of servers (masters) from the Unicast Master Table. The servers respond, then start sending Announce Messages. The client uses the Announce Messages to run the BMCA and to choose the best master. You typically use PTP unicast when multicast is not an option in the network or when sending multicast traffic to unintended devices is not desirable.
• Mixed, where Sync and Announce messages are multicast messages but Delay Request and Response messages are unicast. This avoids the issue seen in multicast message mode where every slave port sees Delay Requests and Responses from every other slave port.

Multicast and Mixed Mode

Multicast mode is the default setting; when you enable PTP on an interface, the message mode is multicast.

To change the message mode to mixed on swp1:

cumulus@switch:~$ nv set interface swp1 ptp mixed-multicast-unicast on
cumulus@switch:~$ nv config apply

cumulus@switch:~$ nv set interface swp1 ptp mixed-multicast-unicast off
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo nano /etc/ptp4l.conf
...
# Default interface options
#
time_stamping           hardware

# Interfaces in which ptp should be enabled
# these interfaces should be routed ports
# if an interface does not have an ip address
# the ptp4l will not work as expected.

[swp1]
Hybrid_e2e              1
...

cumulus@switch:~$ sudo systemctl restart ptp4l.service

Unicast Mode

You can configure a PTP interface on the switch to be a unicast client or a unicast server. Unicast mode reduces the amount of bandwidth consumed.

To configure a PTP interface to be the unicast client:

• Configure the unicast master table. You must configure at least one unicast master table on the switch. If you configure more than one unicast master table, each table must have a unique ID.
  • Set the unicast table ID; a unique ID that identifies the unicast master table.
  • Set the unicast master address. You can set more than one unicast master address, which can be an IPv4, IPv6, or MAC address.
  • Set the IP address for peer delay requests. You can set an IPv4 or IPv6 address.
  • Optional: Set the unicast master query interval, which is the mean interval between requests for announce messages. Specify this value as a power of two in seconds. You can specify a value between -3 and 4. The default value is -0 (2 power).
• On the PTP interface:
  • Set the table ID of the unicast master table you want to use.
  • Set the unicast service mode to client.
  • Optional: Set the unicast request duration; the service time in seconds requested during discovery. The default value is 300 seconds.

The following example commands configure a unicast master table with ID 1. The commands set the unicast master address and the peer address to 10.10.10.1, the query interval to 4, the unicast service mode to client, and the unicast request duration to 20 in the unicast master table.

cumulus@switch:~$ nv set service ptp 1 unicast-master 1 address 10.10.10.1
cumulus@switch:~$ nv set service ptp 1 unicast-master 1 peer-address 10.10.10.1
cumulus@switch:~$ nv set service ptp 1 unicast-master 1 query-interval 4
cumulus@switch:~$ nv set interface swp1 ptp unicast-master-table-id 1
cumulus@switch:~$ nv set interface swp1 ptp unicast-service-mode client
cumulus@switch:~$ nv set interface swp1 ptp unicast-request-duration 20
cumulus@switch:~$ nv config apply

To configure a PTP interface to be the unicast server:

• Set the unicast table ID and the unicast master address. You can set more than one unicast master address, which can be an IPv4, IPv6, or MAC address.
  • Set the IP address for peer delay requests. You can set an IPv4 or IPv6 address.
  • Optional: Set the unicast master query interval, which is the mean interval between requests for announce messages. Specify this value as a power of two in seconds. You can specify a value between -3 and 4. The default value is -0 (2 power).
• On the PTP interface:
  • Set the table index of the unicast master table you want to use.
  • Set the unicast service mode to server.

The following example commands set the unicast table ID to 1, the unicast master address and the peer address to 10.10.10.1, the query interval to 4, and the unicast service mode to server.

cumulus@switch:~$ nv set service ptp 1 unicast-master 1 address 10.10.10.1
cumulus@switch:~$ nv set service ptp 1 unicast-master 1 peer-address 10.10.10.1
cumulus@switch:~$ nv set service ptp 1 unicast-master 1 query-interval 4
cumulus@switch:~$ nv set interface swp1 ptp unicast-master-table-id 1
cumulus@switch:~$ nv set interface swp1 ptp unicast-service-mode server
cumulus@switch:~$ nv config apply

Show Unicast Master Information

To show the unicast master table configuration on the switch, run the nv show service ptp <instance-id> unicast-master <table-id> command:

cumulus@switch:~$ nv show service ptp 1 unicast-master 1
                operational  applied     description
--------------  -----------  ----------  ----------------------------------------------------------------------
peer-address                 10.10.10.1  IP address for Peer Delay request
query-interval               4           Mean interval between requests for Announce messages. It is specifi...
[address]                    10.10.10.1  ipv4, ipv6 or mac address

To show information about a specific unicast master, run the nv show service ptp <instance-id> unicast-master <table-id> address <ip-mac-address-id> command:

cumulus@switch:~$ nv show service ptp 1 unicast-master 1 address 10.10.10.1

TTL for a PTP Message

To restrict the number of hops a PTP message can travel, set the TTL on the PTP interface. You can set a value between 1 and 255.

cumulus@switch:~$ nv set interface swp1 ptp ttl 20
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo nano /etc/ptp4l.conf
...
# Default interface options
#
time_stamping           hardware

# Interfaces in which ptp should be enabled
# these interfaces should be routed ports
# if an interface does not have an ip address
# the ptp4l will not work as expected.

[swp1]
udp_ttl                 20
masterOnly              1
delay_mechanism         E2E
network_transport       UDPv4
...

cumulus@switch:~$ sudo systemctl restart ptp4l.service

PTP Interface Timers

You can set the following timers for PTP messages.

• To set the timers with NVUE, run the nv set interface <interface> ptp timers <timer> <value> command.
• To set the timers with Linux commands, edit the /etc/ptp4l.conf file and set the timers in the Default interface options section.

cumulus@switch:~$ nv set interface swp1 ptp timers announce-interval -1
cumulus@switch:~$ nv config apply

cumulus@switch:~$ nv set interface swp1 ptp timers sync-interval -5
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo nano /etc/ptp4l.conf
...
# Default interface options
#
time_stamping           hardware

# Interfaces in which ptp should be enabled
# these interfaces should be routed ports
# if an interface does not have an ip address
# the ptp4l will not work as expected.

[swp1]
logAnnounceInterval     -1
logSyncInterval         -5
udp_ttl                 20
masterOnly              1
delay_mechanism         E2E
network_transport       UDPv4
...

cumulus@switch:~$ sudo systemctl restart ptp4l.service

Acceptable Master Table

The acceptable master table option is a security feature that prevents a rogue player from pretending to be the Grandmaster to take over the PTP network. To use this feature, you configure the clock IDs of known Grandmasters in the acceptable master table and set the acceptable master table option on a PTP port. The BMC algorithm checks if the Grandmaster received on the Announce message is in this table before proceeding with the master selection. Cumulus Linux disables this option by default on PTP ports.

The following example command adds the Grandmaster clock ID 24:8a:07:ff:fe:f4:16:06 to the acceptable master table and enable the PTP acceptable master table option for swp1:

cumulus@switch:~$ nv set service ptp 1 acceptable-master 24:8a:07:ff:fe:f4:16:06
cumulus@switch:~$ nv config apply

cumulus@switch:~$ nv set service ptp 1 acceptable-master 24:8a:07:ff:fe:f4:16:06 alt-priority 2

cumulus@switch:~$ nv set interface swp1 ptp acceptable-master on
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo nano /etc/ptp4l.conf
...
#
# Default interface options
#
time_stamping           hardware


[acceptable_master_table]
maxTableSize 16
acceptable_master_clockIdentity 248a07.fffe.f41606
...

cumulus@switch:~$ sudo nano /etc/ptp4l.conf
...
#
# Default interface options
#
time_stamping           hardware


[acceptable_master_table]
maxTableSize 16
acceptable_master_clockIdentity 248a07.fffe.f41606 2

...
# Default interface options
#
time_stamping           hardware

# Interfaces in which ptp should be enabled
# these interfaces should be routed ports
# if an interface does not have an ip address
# the ptp4l will not work as expected.

[swp1]
udp_ttl                 20
masterOnly              1
delay_mechanism         E2E
network_transport       UDPv4
acceptable_master       on
...

cumulus@switch:~$ sudo systemctl restart ptp4l.service

Optional Monitor Configuration

Cumulus Linux monitors clock correction and path delay against thresholds, and generates counters that show in the nv show interface swp5 ptp command output and log messages when PTP reaches the thresholds. You can configure the following monitor settings:

cumulus@switch:~$ nv set service ptp 1 monitor path-delay-threshold 300
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo nano /etc/ptp4l.conf
...
global]
#
# Default Data Set
#
slaveOnly               0
priority1               128
priority2               128
domainNumber            0

twoStepFlag             1
dscp_event              46
dscp_general            46

offset_from_master_min_threshold   -50
offset_from_master_max_threshold   50
mean_path_delay_threshold          300
...

Delete PTP Configuration

To delete PTP configuration, delete the PTP master and slave interfaces. The following example commands delete the PTP interfaces swp1, swp2, and swp3.

cumulus@switch:~$ nv unset interface swp1 ptp
cumulus@switch:~$ nv unset interface swp2 ptp
cumulus@switch:~$ nv unset interface swp3 ptp
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo nano /etc/ptp4l.conf
...
# Default interface options
#
time_stamping           hardware

# Interfaces in which ptp should be enabled
# these interfaces should be routed ports
# if an interface does not have an ip address
# the ptp4l will not work as expected.

cumulus@switch:~$ sudo systemctl restart ptp4l.service

To disable PTP on the switch and stop the ptp4l and phc2sys processes:

cumulus@switch:~$ nv set service ptp 1 enable off
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo systemctl stop ptp4l.service phc2sys.service
cumulus@switch:~$ sudo systemctl disable ptp4l.service phc2sys.service

Troubleshooting

PTP Configuration and Status

To show a summary of the PTP configuration on the switch, run the nv show service ptp <instance> command:

cumulus@switch:~$ nv show service ptp 1

---------------------------  -----------  -------  --------------------------------------------------------------------
enable                       on           on       Turn the feature 'on' or 'off'.  The default is 'off'.
domain                       0            0        Domain number of the current syntonization
ip-dscp                      46           46       Sets the Diffserv code point for all PTP packets originated locally.
priority1                    128          128      Priority1 attribute of the local clock
priority2                    128          128      Priority2 attribute of the local clock
two-step                     on           on       Determines if the Clock is a 2 step clock
monitor
  max-offset-threshold       50           50       Maximum offset threshold in nano seconds
  max-timestamp-entries                   400      Maximum timestamp entries allowed
  max-violation-log-entries               8        Maximum violation log entries per set
  max-violation-log-sets                  8        Maximum violation logs sets allowed
  min-offset-threshold       -50          -50      Minimum offset threshold in nano seconds
  path-delay-threshold       200          200      Path delay threshold in nano seconds
  violation-log-interval                  0        violation log intervals in seconds
...

You can drill down with the following nv show service ptp <instance> commands:

• nv show service ptp <instance> acceptable-master shows a collection of acceptable masters.
• nv show service ptp <instance> monitor shows PTP monitor configuration.
• nv show service ptp <instance> current shows the local states learned during PTP message exchange.
• nv show service ptp <instance> clock-quality shows the clock quality status.
• nv show service ptp <instance> parent shows the local states learned during PTP message exchange.
• nv show service ptp <instance> time-properties shows the clock time attributes.

To check configuration and counters for a PTP interface, run the nv show interface <interface> ptp command:

cumulus@leaf03:mgmt:~$ nv show interface swp1 ptp
                           operational  applied     description
-------------------------  -----------  ----------  ----------------------------------------------------------------------
enable                                  on          Turn the feature 'on' or 'off'.  The default is 'off'.
acceptable-master                       off         Determines if acceptable master check is enabled for this interface.
delay-mechanism            end-to-end   end-to-end  Mode in which PTP message is transmitted.
forced-master              off          off         Configures PTP interfaces to forced master state.
instance                                1           PTP instance number.
message-mode                            multicast   Mode in which PTP delay message is transmitted.
transport                  ipv4         ipv4        Transport method for the PTP messages.
ttl                        1            1           Maximum number of hops the PTP messages can make before it gets dro...
timers
  announce-interval        0            0           Mean time interval between successive Announce messages.  It's spec...
  announce-timeout         3            3           The number of announceIntervals that have to pass without receipt o...
  delay-req-interval       -3           -3          The minimum permitted mean time interval between successive Delay R...
  sync-interval            -3           -3          The mean SyncInterval for multicast messages.  It's specified as a...
peer-mean-path-delay       0                        An estimate of the current one-way propagation delay on the link wh...
port-state                 master                   State of the port
protocol-version           2                        The PTP version in use on the port
counters
  rx-announce              0                        Number of Announce messages received
  rx-delay-req             0                        Number of Delay Request messages received
  rx-delay-resp            0                        Number of Delay response messages received
  rx-delay-resp-follow-up  0                        Number of Delay response follow upmessages received
  rx-follow-up             0                        Number of Follow up messages received
  rx-management            0                        Number of Management messages received
  rx-peer-delay-req        0                        Number of Peer Delay Request messages received
  rx-peer-delay-resp       0                        Number of Peer Delay Response messages received
  rx-signaling             0                        Number of singnaling messages received
  rx-sync                  0                        Number of Sync messages received
  tx-announce              2639                     Number of Announce messages transmitted
  tx-delay-req             0                        Number of Delay Request messages transmitted
  tx-delay-resp            0                        Number of Delay response messages transmitted
  tx-delay-resp-follow-up  0                        Number of Delay response follow upmessages transmitted
  tx-follow-up             21099                    Number of Follow up messages transmitted
  tx-management            0                        Number of Management messages transmitted
  tx-peer-delay-req        0                        Number of Peer Delay Request messages transmitted
  tx-peer-delay-resp       0                        Number of Peer Delay Response messages transmitted
  tx-signaling             0                        Number of singnaling messages transmitted
  tx-sync                  21099                    Number of Sync messages transmitted

To view PTP status information, including the delta in nanoseconds from the master clock:

cumulus@switch:~$ sudo pmc -u -b 0 'GET TIME_STATUS_NP'
sending: GET TIME_STATUS_NP
    7cfe90.fffe.f56dfc-0 seq 0 RESPONSE MANAGEMENT TIME_STATUS_NP
        master_offset              12610
        ingress_time               1525717806521177336
        cumulativeScaledRateOffset +0.000000000
        scaledLastGmPhaseChange    0
        gmTimeBaseIndicator        0
        lastGmPhaseChange          0x0000'0000000000000000.0000
        gmPresent                  true
        gmIdentity                 000200.fffe.000005
    000200.fffe.000005-1 seq 0 RESPONSE MANAGEMENT TIME_STATUS_NP
        master_offset              0
        ingress_time               0
        cumulativeScaledRateOffset +0.000000000
        scaledLastGmPhaseChange    0
        gmTimeBaseIndicator        0
        lastGmPhaseChange          0x0000'0000000000000000.0000
        gmPresent                  false
        gmIdentity                 000200.fffe.000005
    000200.fffe.000006-1 seq 0 RESPONSE MANAGEMENT TIME_STATUS_NP
        master_offset              5544033534
        ingress_time               1525717812106811842
        cumulativeScaledRateOffset +0.000000000
        scaledLastGmPhaseChange    0
        gmTimeBaseIndicator        0
        lastGmPhaseChange          0x0000'0000000000000000.0000
        gmPresent                  true
        gmIdentity                 000200.fffe.000005

PTP Violations

You can check PTP violations:

• To show the collection of violation logs, run the nv show service ptp <instance> monitor timestamp-log command.
• To show PTP violations, run the nv show service ptp <instance> monitor violations command.

The following example shows that there are no violations:

cumulus@switch:~$ nv show service ptp 1 monitor violations
                  operational  applied  description
----------------  -----------  -------  -----------------------------------------------
last-max-offset                         Time at which last max offest violation occurred
last-min-offset                         Time at which last min offest violation occurred
last-path-delay                         Time at which last path delay violation occurred
max-offset-count  0                     Number of maximum offset violations
min-offset-count  0                     Number of min offset violations
path-delay-count  0                     Number of Path delay violations

PTP Show Commands

• To see the list of NVUE show commands for PTP, run the nv list-commands service ptp command.
• To show the list of show commands for a PTP interface, run the nv list-commands interface command, then scroll to see PTP.

cumulus@switch:~$ nv list-commands service ptp
nv show service ptp
nv show service ptp <instance-id>
nv show service ptp <instance-id> acceptable-master
nv show service ptp <instance-id> acceptable-master <clock-id>
nv show service ptp <instance-id> monitor
nv show service ptp <instance-id> monitor timestamp-log
nv show service ptp <instance-id> monitor violations
nv show service ptp <instance-id> monitor violations log
nv show service ptp <instance-id> monitor violations log acceptable-master
nv show service ptp <instance-id> monitor violations log forced-master
nv show service ptp <instance-id> monitor violations log max-offset
nv show service ptp <instance-id> monitor violations log min-offset
nv show service ptp <instance-id> monitor violations log path-delay
nv show service ptp <instance-id> current
nv show service ptp <instance-id> clock-quality
nv show service ptp <instance-id> parent
nv show service ptp <instance-id> parent grandmaster-clock-quality
nv show service ptp <instance-id> time-properties
...

cumulus@switch:~$ nv list-commands interface
...
nv show interface <interface-id> ptp
nv show interface <interface-id> ptp timers
nv show interface <interface-id> ptp counters
...

Example Configuration

In the following example, the boundary clock on the switch receives time from Master 1 (the grandmaster) on PTP slave port swp1, sets its clock and passes the time down through PTP master ports swp2, swp3, and swp4 to the hosts that receive the time.

The following example configuration assumes that you have already configured the layer 3 routed interfaces (swp1, swp2, swp3, and swp4) you want to use for PTP.

cumulus@switch:~$ nv set service ptp 1 enable on
cumulus@switch:~$ nv set service ptp 1 priority2 254
cumulus@switch:~$ nv set service ptp 1 priority1 254
cumulus@switch:~$ nv set service ptp 1 domain 3
cumulus@switch:~$ nv set interface swp1 ptp enable on
cumulus@switch:~$ nv set interface swp2 ptp enable on
cumulus@switch:~$ nv set interface swp3 ptp enable on
cumulus@switch:~$ nv set interface swp4 ptp enable on
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo cat /etc/nvue.d/startup.yaml
- set:
    interface:
      lo:
        ip:
          address:
            10.10.10.1/32: {}
        type: loopback
      swp1:
        ptp:
          enable: on
        type: swp
      swp2:
        ptp:
          enable: on
        type: swp
      swp3:
        ptp:
          enable: on
        type: swp
      swp4:
        ptp:
          enable: on
        type: swp
    service:
      ptp:
        '1':
          domain: 3
          enable: on
          priority1: 254
          priority2: 254

cumulus@switch:~$ sudo cat /etc/ptp4l.conf
...
[global]
#
# Default Data Set
#
slaveOnly               0
priority1               254
priority2               254
domainNumber            3

twoStepFlag             1
dscp_event              46
dscp_general            46

offset_from_master_min_threshold   -50
offset_from_master_max_threshold   50
mean_path_delay_threshold          200

#
# Run time options
#
logging_level           6
path_trace_enabled      0
use_syslog              1
verbose                 0
summary_interval        0

#
# Default interface options
#
time_stamping           hardware

# Interfaces in which ptp should be enabled
# these interfaces should be routed ports
# if an interface does not have an ip address
# the ptp4l will not work as expected.

[swp1]
logAnnounceInterval     0
logSyncInterval         -3
logMinDelayReqInterval  -3
announceReceiptTimeout  3
udp_ttl                 1
masterOnly              0
delay_mechanism         E2E
network_transport       UDPv4

[swp2]
logAnnounceInterval     0
logSyncInterval         -3
logMinDelayReqInterval  -3
announceReceiptTimeout  3
udp_ttl                 1
masterOnly              0
delay_mechanism         E2E
network_transport       UDPv4

[swp3]
logAnnounceInterval     0
logSyncInterval         -3
logMinDelayReqInterval  -3
announceReceiptTimeout  3
udp_ttl                 1
masterOnly              0
delay_mechanism         E2E
network_transport       UDPv4

[swp4]
logAnnounceInterval     0
logSyncInterval         -3
logMinDelayReqInterval  -3
announceReceiptTimeout  3
udp_ttl                 1
masterOnly              0
delay_mechanism         E2E
network_transport       UDPv4

Considerations

Spanning Tree and PTP

PTP frames are affected by STP filtering; events, such as an STP topology change (where ports temporarily go into the blocking state), can cause interruptions to PTP communications.

If you configure PTP on bridge ports, NVIDIA recommends that the bridge ports are spanning tree edge ports or in a bridge domain where spanning tree is disabled.

This section describes how to set up user accounts and ssh for remote access, and configure LDAP authentication, TACACS+, and RADIUS AAA.

You can generate authentication keys to access a Cumulus Linux switch securely with the ssh-keygen component of the Secure Shell (SSH) protocol. Cumulus Linux uses the OpenSSH package to provide this functionality. This section describes how to generate an SSH key pair.

Generate an SSH Key Pair

1. To generate the SSH key pair, run the ssh-keygen command and follow the prompts:

   To configure the system without a password, do not enter a passphrase when prompted in the following step.

   cumulus@leaf01:~$ ssh-keygen
   Generating public/private rsa key pair.
   Enter file in which to save the key (/home/cumulus/.ssh/id_rsa):
   Enter passphrase (empty for no passphrase):
   Enter same passphrase again:
   Your identification has been saved in /home/cumulus/.ssh/id_rsa.
   Your public key has been saved in /home/cumulus/.ssh/id_rsa.pub.
   The key fingerprint is:
   5a:b4:16:a0:f9:14:6b:51:f6:f6:c0:76:1a:35:2b:bb cumulus@leaf04
   The key's randomart image is:
   +---[RSA 2048]----+
   |      +.o   o    |
   |     o * o . o   |
   |    o + o O o    |
   |     + . = O     |
   |      . S o .    |
   |       +   .     |
   |      .   E      |
   |                 |
   |                 |
   +-----------------+
   

2. To copy the generated public key to the desired location, run the ssh-copy-id command and follow the prompts:

   cumulus@leaf01:~$ ssh-copy-id -i /home/cumulus/.ssh/id_rsa.pub cumulus@leaf02
   The authenticity of host 'leaf02 (192.168.0.11)' can't be established.
   ECDSA key fingerprint is b1:ce:b7:6a:20:f4:06:3a:09:3c:d9:42:de:99:66:6e.
   Are you sure you want to continue connecting (yes/no)? yes
   /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
   /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
   cumulus@leaf01's password:
   
   Number of key(s) added: 1
   

   ssh-copy-id does not work if the username on the remote switch is different from the username on the local switch. To work around this issue, use the scp command instead:

   cumulus@leaf01:~$ scp .ssh/id_rsa.pub cumulus@leaf02:.ssh/authorized_keys
   Enter passphrase for key '/home/cumulus/.ssh/id_rsa':
   id_rsa.pub
   

3. Connect to the remote switch to confirm that the authentication keys are in place:

   cumulus@leaf01:~$ ssh cumulus@leaf02
   
   Welcome to Cumulus VX (TM)
   
   Cumulus VX (TM) is a community supported virtual appliance designed for
   experiencing, testing and prototyping the latest technology.
   For any questions or technical support, visit our community site at:
   http://community.cumulusnetworks.com
   
   The registered trademark Linux (R) is used pursuant to a sublicense from LMI,
   the exclusive licensee of Linus Torvalds, owner of the mark on a world-wide basis.
   Last login: Thu Sep 29 16:56:54 2016
   

By default, Cumulus Linux has two user accounts: cumulus and root.

The cumulus account:

• Uses the default password cumulus. You must change the default password when you log into Cumulus Linux for the first time.
• Is a user account in the sudo group with sudo privileges.
• Can log in to the system through all the usual channels, such as console and SSH.
• Along with the cumulus group, has both show and edit rights for NVUE.

The root account:

• Has the default password disabled by default
• Has the standard Linux root user access to everything on the switch
• The disabled password prevents you from using SSH, telnet, FTP, and so on, to log in to the switch.

You can add additional user accounts as needed. Like the cumulus account, these accounts must use sudo to execute privileged commands; be sure to include them in the sudo group. For example:

cumulus@switch:~$ sudo adduser NEWUSERNAME sudo

To access the switch without a password, you need to boot into a single shell/user mode.

NVUE Command Access

You can control local user account access to NVUE commands by changing the Linux group membership for a user. Use the following groups to set the permissions for local user accounts. To add users to these groups, use the useradd(8) or usermod(8) commands:

Enable Remote Access for the root User

The root user does not have a password and cannot log into a switch using SSH. This default account behavior is consistent with Debian. To connect to a switch using the root account, you can do one of the following:

• Generate an SSH key
• Set a password

Generate an SSH Key for the root Account

1. In a terminal on your host system (not the switch), check to see if a key already exists:

   root@host:~# ls -al ~/.ssh/
   

   The name of the key is similar to id_dsa.pub, id_rsa.pub, or id_ecdsa.pub.

2. If a key does not exist, generate a new one by first creating the RSA key pair:

   root@host:~# ssh-keygen -t rsa
   

3. At the prompt, enter a file in which to save the key (/root/.ssh/id_rsa). Press Enter to use the home directory of the root user or provide a different destination.

4. At the prompt, enter a passphrase (empty for no passphrase). This is optional but it does provide an extra layer of security.

5. The public key is now located in /root/.ssh/id_rsa.pub. The private key (identification) is now located in /root/.ssh/id_rsa.

6. Copy the public key to the switch. SSH to the switch as the cumulus user, then run:

   cumulus@switch:~$ sudo mkdir -p /root/.ssh
   cumulus@switch:~$ echo <SSH public key string> | sudo tee -a /root/.ssh/authorized_keys
   

Set the root User Password

1. Run the following command:

   cumulus@switch:~$ sudo passwd root
   

2. Change the PermitRootLogin setting in the /etc/ssh/sshd_config file from without-password to yes.

   cumulus@switch:~$ sudo nano /etc/ssh/sshd_config
   ...
   # Authentication:
   LoginGraceTime 120
   PermitRootLogin yes
   StrictModes yes
   ...  
   

3. Restart the ssh service:

   cumulus@switch:~$ sudo systemctl reload ssh.service
   

By default, Cumulus Linux has two user accounts: root and cumulus. The cumulus account is a normal user and is in the group sudo.

You can add more user accounts as needed. Like the cumulus account, these accounts must use sudo to execute privileged commands.

sudo Basics

sudo allows you to execute a command as superuser or another user as specified by the security policy.

The default security policy is sudoers, which you configure in the /etc/sudoers file. Use /etc/sudoers.d/ to add to the default sudoers policy.

By default, users in the sudo group can use sudo to execute privileged commands. To add users to the sudo group, use the useradd(8) or usermod(8) command. To see which users belong to the sudo group, see /etc/group (man group(5)).

You can run any command as sudo, including su. You must enter a password.

The example below shows how to use sudo as a non-privileged user cumulus to bring up an interface:

cumulus@switch:~$ ip link show dev swp1
3: swp1: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master br0 state DOWN mode DEFAULT qlen 500
link/ether 44:38:39:00:27:9f brd ff:ff:ff:ff:ff:ff

cumulus@switch:~$ ip link set dev swp1 up
RTNETLINK answers: Operation not permitted

cumulus@switch:~$ sudo ip link set dev swp1 up
Password:

umulus@switch:~$ ip link show dev swp1
3: swp1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP mode DEFAULT qlen 500
link/ether 44:38:39:00:27:9f brd ff:ff:ff:ff:ff:ff

sudoers Examples

The following examples show how you grant as few privileges as necessary to a user or group of users to allow them to perform the required task. Each example uses the system group noc; groups include the prefix %.

When an unprivileged user runs a command, the command must include the sudo prefix.

Related Information

• Debian wiki - sudo

Cumulus Linux uses Pluggable Authentication Modules (PAM) and Name Service Switch (NSS) for user authentication. NSS enables PAM to use LDAP to provide user authentication, group mapping, and information for other services on the system.

• NSS specifies the order of the information sources that resolve names for each service. Using NSS with authentication and authorization provides the order and location for user lookup and group mapping on the system.
• PAM handles the interaction between the user and the system, providing login handling, session setup, authentication of users, and authorization of user actions.

Install libnss-ldapd

The libldap-2.4-2 and libldap-common LDAP packages are already installed on the Cumulus Linux image; however you need to install these additional packages to use LDAP authentication:

• libnss-ldapd
• libpam-ldapd
• ldap-utils

To install the additional packages, run the following command:

cumulus@switch:~$ sudo apt-get install libnss-ldapd libpam-ldapd ldap-utils nslcd

You can also install these packages even if the switch does not connect to the internet, as they are in the cumulus-local-apt-archive repository that is embedded in the Cumulus Linux image.

Follow the interactive prompts to specify the LDAP URI, search base distinguished name (DN), and services that must have LDAP lookups enabled. You need to select at least the passwd, group, and shadow services (press space to select a service). When done, select OK. This creates a basic LDAP configuration using anonymous bind and initiates user search under the base DN specified.

After the dialog closes, the install process prints information similar to the following:

/etc/nsswitch.conf: enable LDAP lookups for group
/etc/nsswitch.conf: enable LDAP lookups for passwd
/etc/nsswitch.conf: enable LDAP lookups for shadow

After the installation is complete, the name service caching daemon (nslcd) runs. This service handles all the LDAP protocol interactions and caches information that returns from the LDAP server. nslcd appends ldap to the /etc/nsswitch.conf file, as well as the secondary information source for passwd, group, and shadow. nslcd references the local files (/etc/passwd, /etc/groups and /etc/shadow) first, as specified by the compat source.

passwd: compat ldap
group: compat ldap
shadow: compat ldap

Entering incorrect information during the installation process produces configuration errors. You can correct the information after installation by editing certain configuration files.

• Edit the /etc/nslcd.conf file to update the LDAP URI and search base DN (see Update the nslcd.conf File, below).
• Edit the /etc/nssswitch.conf file to update the service selections.

Restart nvued.service and nginx.service after editing the files.

cumulus@switch:~$ sudo systemctl restart nvued.service
cumulus@switch:~$ sudo systemctl restart nginx.service

Update the nslcd.conf File

After installation, update the main configuration file (/etc/nslcd.conf) to accommodate the expected LDAP server settings.

This section documents some of the more important options that relate to security and queries. For details on all the available configuration options, read the nslcd.conf man page.

Connection

The LDAP client starts a session by connecting to the LDAP server on TCP and UDP port 389 or on port 636 for LDAPS. Depending on the configuration, this connection establishes without authentication (anonymous bind); otherwise, the client must provide a bind user and password. The variables you use to define the connection to the LDAP server are the URI and bind credentials.

The URI is mandatory and specifies the LDAP server location using the FQDN or IP address. The URI also designates whether to use ldap:// for clear text transport, or ldaps:// for SSL/TLS encrypted transport. You can also specify an alternate port in the URI. In production environments, use the LDAPS protocol so that all communications are secure.

After the connection to the server is complete, the BIND operation authenticates the session. The BIND credentials are optional; if you do not specify the credentials, the switch assumes an anonymous bind. Configure authenticated (Simple) BIND by specifying the user (binddn) and password (bindpw) in the configuration. Another option is to use SASL (Simple Authentication and Security Layer) BIND, which provides authentication services using other mechanisms, like Kerberos. Contact your LDAP server administrator for this information as it depends on the configuration of the LDAP server and the credentials for the client device.

# The location at which the LDAP server(s) should be reachable.
uri ldaps://ldap.example.com
# The DN to bind with for normal lookups.
binddn cn=CLswitch,ou=infra,dc=example,dc=com
bindpw CuMuLuS

Search Function

When an LDAP client requests information about a resource, it must connect and bind to the server. Then, it performs one or more resource queries depending on the lookup. All search queries to the LDAP server use the configured search base, filter, and the desired entry (uid=myuser). If the LDAP directory is large, this search takes a long time. Define a more specific search base for the common maps (passwd and group).

# The search base that will be used for all queries.
base dc=example,dc=com
# Mapped search bases to speed up common queries.
base passwd ou=people,dc=example,dc=com
base group ou=groups,dc=example,dc=com

Search Filters

To limit the search scope when authenticating users, use search filters to specify criteria when searching for objects within the directory. The default filters applied are:

filter passwd (objectClass=posixAccount)
filter group (objectClass=posixGroup)

Attribute Mapping

The map configuration allows you to override the attributes pushed from LDAP. To override an attribute for a given map, specify the attribute name and the new value. This is useful to ensure that the shell is bash and the home directory is /home/cumulus:

map    passwd homeDirectory "/home/cumulus"
map    passwd shell "/bin/bash"

Create Home Directory on Login

If you want to use unique home directories, run the sudo pam-auth-update command and select Create home directory on login in the PAM configuration dialog (press the space bar to select the option). Select OK, then press Enter to save the update and close the dialog.

cumulus@switch:~$ sudo pam-auth-update

The home directory for any user that logs in (using LDAP or not) populates with the standard dotfiles from /etc/skel.

nslcd[5816]: unable to dlopen /usr/lib/x86_64-linux-gnu/sasl2/libsasldb.so: libdb-5.3.so: cannot open
shared object file: No such file or directory

Example Configuration

Here is an example configuration using Cumulus Linux.

# /etc/nslcd.conf
# nslcd configuration file. See nslcd.conf(5)
# for details.

# The user and group nslcd should run as.
uid nslcd
gid nslcd

# The location at which the LDAP server(s) should be reachable.
uri ldaps://myadserver.rtp.example.test

# The search base that will be used for all queries.
base ou=support,dc=rtp,dc=example,dc=test

# The LDAP protocol version to use.
#ldap_version 3

# The DN to bind with for normal lookups.
# defconf-set-selections doesn't seem to set this. so have to manually set this.
binddn CN=cumulus admin,CN=Users,DC=rtp,DC=example,DC=test
bindpw 1Q2w3e4r!

# The DN used for password modifications by root.
#rootpwmoddn cn=admin,dc=example,dc=com

# SSL options
#ssl off (default)
# Not good does not prevent man in the middle attacks
#tls_reqcert demand(default)
tls_cacertfile /etc/ssl/certs/rtp-example-ca.crt

# The search scope.
#scope sub

# Add nested group support
# Supported in nslcd 0.9 and higher.
# default wheezy install of nslcd supports on 0.8. wheezy-backports has 0.9
nss_nested_groups yes

# Mappings for Active Directory
# (replace the SIDs in the objectSid mappings with the value for your domain)
# "dsquery * -filter (samaccountname=testuser1) -attr ObjectSID" where cn == 'testuser1'
pagesize 1000
referrals off
idle_timelimit 1000

# Do not allow uids lower than 100 to login (aka Administrator)
# not needed as pam already has this support
# nss_min_uid 1000

# This filter says to get all users who are part of the cumuluslnxadm group. Supports nested groups.
# Example, mary is part of the snrnetworkadm group which is part of cumuluslnxadm group
# Ref: http://msdn.microsoft.com/en-us/library/aa746475%28VS.85%29.aspx (LDAP_MATCHING_RULE_IN_CHAIN)
filter passwd (&(Objectclass=user)(!(objectClass=computer))(memberOf:1.2.840.113556.1.4.1941:=cn=cumuluslnxadm,ou=groups,ou=support,dc=rtp,dc=example,dc=test))
map    passwd uid           sAMAccountName
map    passwd uidNumber     objectSid:S-1-5-21-1391733952-3059161487-1245441232
map    passwd gidNumber     objectSid:S-1-5-21-1391733952-3059161487-1245441232
map    passwd homeDirectory "/home/$sAMAccountName"
map    passwd gecos         displayName
map    passwd loginShell    "/bin/bash"

# Filter for any AD group or user in the baseDN. the reason for filtering for the
# user to make sure group listing for user files don't say '<user> <gid>'. instead will say '<user> <user>'
# So for cosmetic reasons..nothing more.
filter group (&(|(objectClass=group)(Objectclass=user))(!(objectClass=computer)))
map    group gidNumber     objectSid:S-1-5-21-1391733952-3059161487-1245441232
map    group cn            sAMAccountName

Configure LDAP Authorization

Linux uses the sudo command to allow non-administrator users (such as the default cumulus user account) to perform privileged operations. To control the users that can use sudo, define a series of rules in the /etc/sudoers file and files in the /etc/sudoers.d/ directory. The rules apply to groups but you can also define specific users. You can add sudo rules using the group names from LDAP. For example, if a group of users are in the group netadmin, you can add a rule to give those users sudo privileges. Refer to the sudoers manual (man sudoers) for a complete usage description. The following shows an example in the /etc/sudoers file:

# The basic structure of a user specification is "who where = (as_whom) what ".
%sudo ALL=(ALL:ALL) ALL
%netadmin ALL=(ALL:ALL) ALL

Active Directory Configuration

Active Directory (AD) is a fully featured LDAP-based NIS server create by Microsoft. It offers unique features that classic OpenLDAP servers do not have. AD can be more complicated to configure on the client and each version works a little differently with Linux-based LDAP clients. Some more advanced configuration examples, from testing LDAP clients on Cumulus Linux with Active Directory (AD/LDAP), are available in the knowledge base.

LDAP Verification Tools

The LDAP client daemon retrieves and caches password and group information from LDAP. To verify the LDAP interaction, use these command-line tools to trigger an LDAP query from the device.

Identify a User with the id Command

The id command performs a username lookup by following the lookup information sources in NSS for the passwd service. This returns the user ID, group ID and the group list retrieved from the information source. In the following example, the user cumulus is locally defined in /etc/passwd, and myuser is on LDAP. The NSS configuration has the passwd map configured with the sources compat ldap:

cumulus@switch:~$ id cumulus
uid=1000(cumulus) gid=1000(cumulus) groups=1000(cumulus),24(cdrom),25(floppy),27(sudo),29(audio),30(dip),44(video),46(plugdev)
cumulus@switch:~$ id myuser
uid=1230(myuser) gid=3000(Development) groups=3000(Development),500(Employees),27(sudo)

getent

The getent command retrieves all records found with NSS for a given map. It can also retrieve a specific entry under that map. You can perform tests with the passwd, group, shadow, or any other map in the /etc/nsswitch.conf file. The output from this command formats according to the map requested. For the passwd service, the structure of the output is the same as the entries in /etc/passwd. The group map outputs the same structure as /etc/group.

In this example, looking up a specific user in the passwd map, the user cumulus is locally defined in /etc/passwd, and myuser is only in LDAP.

cumulus@switch:~$ getent passwd cumulus
cumulus:x:1000:1000::/home/cumulus:/bin/bash
cumulus@switch:~$ getent passwd myuser
myuser:x:1230:3000:My Test User:/home/myuser:/bin/bash

In the next example, looking up a specific group in the group service, the group cumulus is locally defined in /etc/groups, and netadmin is on LDAP.

cumulus@switch:~$ getent group cumulus
cumulus:x:1000:
cumulus@switch:~$ getent group netadmin
netadmin:*:502:larry,moe,curly,shemp

Running the command getent passwd or getent group without a specific request returns all local and LDAP entries for the passwd and group maps.

LDAP search

The ldapsearch command performs LDAP operations directly on the LDAP server. This does not interact with NSS. This command displays the information that the LDAP daemon process receives back from the server. The command has several options. The simplest option uses anonymous bind to the host and specifies the search DN and the attribute to look up.

cumulus@switch:~$ ldapsearch -H ldap://ldap.example.com -b dc=example,dc=com -x uid=myuser

# extended LDIF
#
# LDAPv3
# base <dc=example,dc=com> with scope subtree
# filter: uid=myuser
# requesting: ALL
#
# myuser, people, example.com
dn: uid=myuser,ou=people,dc=example,dc=com
cn: My User
displayName: My User
gecos: myuser
gidNumber: 3000
givenName: My
homeDirectory: /home/myuser
initials: MU
loginShell: /bin/bash
mail: myuser@example.com
objectClass: inetOrgPerson
objectClass: posixAccount
objectClass: shadowAccount
objectClass: top
shadowExpire: -1
shadowFlag: 0
shadowMax: 999999
shadowMin: 8
shadowWarning: 7
sn: User
uid: myuser
uidNumber: 1234

# search result
search: 2
result: 0 Success

# numResponses: 2
# numEntries: 1

LDAP Browsers

The GUI LDAP clients are free tools that show the structure of the LDAP database graphically.

• Apache Directory Studio
• LDAPManager

Troubleshooting

nslcd Debug Mode

When setting up LDAP authentication for the first time, turn off the nslcd service using the systemctl stop nslcd.service command (or the systemctl stop nslcd@mgmt.service if you are running the service in a management VRF) and run it in debug mode. Debug mode works whether you are using LDAP over SSL (port 636) or an unencrypted LDAP connection (port 389).

cumulus@switch:~$ sudo systemctl stop nslcd.service
cumulus@switch:~$ sudo nslcd -d

After you enable debug mode, run the following command to test LDAP queries:

cumulus@switch:~$ getent passwd

If you configure LDAP correctly, the following messages appear after you run the getent command:

nslcd: DEBUG: accept() failed (ignored): Resource temporarily unavailable
nslcd: [8e1f29] DEBUG: connection from pid=11766 uid=0 gid=0
nslcd: [8e1f29] <passwd(all)> DEBUG: myldap_search(base="dc=example,dc=com", filter="(objectClass=posixAccount)")
nslcd: [8e1f29] <passwd(all)> DEBUG: ldap_result(): uid=myuser,ou=people,dc=example,dc=com
nslcd: [8e1f29] <passwd(all)> DEBUG: ldap_result(): ... 152 more results
nslcd: [8e1f29] <passwd(all)> DEBUG: ldap_result(): end of results (162 total)

In the example output above, <passwd(all)> shows a query of the entire directory structure.

You can query a specific user with the following command:

cumulus@switch:~$ getent passwd myuser

You can replace myuser with any username on the switch. The following debug output indicates that user myuser exists:

nslcd: DEBUG: add_uri(ldap://10.50.21.101)
nslcd: version 0.8.10 starting
nslcd: DEBUG: unlink() of /var/run/nslcd/socket failed (ignored): No such file or directory
nslcd: DEBUG: setgroups(0,NULL) done
nslcd: DEBUG: setgid(110) done
nslcd: DEBUG: setuid(107) done
nslcd: accepting connections
nslcd: DEBUG: accept() failed (ignored): Resource temporarily unavailable
nslcd: [8b4567] DEBUG: connection from pid=11369 uid=0 gid=0
nslcd: [8b4567] <passwd="myuser"> DEBUG: myldap_search(base="dc=cumulusnetworks,dc=com", filter="(&(objectClass=posixAccount)(uid=myuser))")
nslcd: [8b4567] <passwd="myuser"> DEBUG: ldap_initialize(ldap://<ip_address>)
nslcd: [8b4567] <passwd="myuser"> DEBUG: ldap_set_rebind_proc()
nslcd: [8b4567] <passwd="myuser"> DEBUG: ldap_set_option(LDAP_OPT_PROTOCOL_VERSION,3)
nslcd: [8b4567] <passwd="myuser"> DEBUG: ldap_set_option(LDAP_OPT_DEREF,0)
nslcd: [8b4567] <passwd="myuser"> DEBUG: ldap_set_option(LDAP_OPT_TIMELIMIT,0)
nslcd: [8b4567] <passwd="myuser"> DEBUG: ldap_set_option(LDAP_OPT_TIMEOUT,0)
nslcd: [8b4567] <passwd="myuser"> DEBUG: ldap_set_option(LDAP_OPT_NETWORK_TIMEOUT,0)
nslcd: [8b4567] <passwd="myuser"> DEBUG: ldap_set_option(LDAP_OPT_REFERRALS,LDAP_OPT_ON)
nslcd: [8b4567] <passwd="myuser"> DEBUG: ldap_set_option(LDAP_OPT_RESTART,LDAP_OPT_ON)
nslcd: [8b4567] <passwd="myuser"> DEBUG: ldap_simple_bind_s(NULL,NULL) (uri="ldap://<ip_address>")
nslcd: [8b4567] <passwd="myuser"> DEBUG: ldap_result(): end of results (0 total)

Common Problems

SSL/TLS

• The FQDN of the LDAP server URI does not match the FQDN in the CA-signed server certificate.
• nslcd cannot read the SSL certificate and reports a Permission denied error in the debug during server connection negotiation. Check the permission on each directory in the path of the root SSL certificate. Ensure that it is readable by the nslcd user.

NSCD

• If the nscd cache daemon is also enabled and you make some changes to the user from LDAP, you can clear the cache using the following commands:

  nscd --invalidate = passwd
  nscd --invalidate = group
  

• The nscd package works with nslcd to cache name entries returned from the LDAP server. This sometimes causes authentication failures. To work around these issues, disable nscd, restart the nslcd service, then retry authentication:

  cumulus@switch:~$ sudo nscd -K
  cumulus@switch:~$ sudo systemctl restart nslcd.service
  

cumulus@switch:~$ sudo nscd -K
cumulus@switch:~$ sudo systemctl restart nslcd@mgmt.service

LDAP

• The search filter returns incorrect results. Check for typos in the search filter. Use ldapsearch to test your filter.

• Optionally, configure the basic LDAP connection and search parameters in /etc/ldap/ldap.conf.

  # ldapsearch -D 'cn=CLadmin' -w 'CuMuLuS' "(&(ObjectClass=inetOrgUser)(uid=myuser))"
  

Related Information

• Debian - configuring LDAP authentication
• Debian - configuring PAM to use LDAP
• GitHub - Arthur de Jong nslcd.conf file
• Debian backports

Cumulus Linux implements TACACS+ client AAA (Accounting, Authentication, and Authorization) in a transparent way with minimal configuration. The client implements the TACACS+ protocol as described in this IETF document. There is no need to create accounts or directories on the switch. Accounting records go to all configured TACACS+ servers by default. Using per-command authorization requires additional setup on the switch.

Supported Features

• Authentication using PAM; includes login, ssh, sudo and su
• Runs over the eth0 management interface
• Ability to run in the management VRF
• TACACS+ privilege 15 users can run any command with sudo using the /etc/sudoers.d/tacplus file that the libtacplus-map1 package installs
• Up to seven TACACS+ servers

Install the TACACS+ Client Packages

You can install the TACACS+ packages even if the switch is not connected to the internet; the packages are in the cumulus-local-apt-archive repository in the Cumulus Linux image.

To install all required packages, run these commands:

cumulus@switch:~$ sudo -E apt-get update
cumulus@switch:~$ sudo -E apt-get install tacplus-client

Configure the TACACS+ Client

cumulus@switch:~$ sudo systemctl restart nvued.service
cumulus@switch:~$ sudo systemctl restart nginx.service

After installing TACACS+, edit the /etc/tacplus_servers file to add at least one server and one shared secret (key). You can specify the server and secret parameters in any order anywhere in the file. Whitespace (spaces or tabs) are not allowed. For example, if your TACACS+ server IP address is 192.168.0.30 and your shared secret is tacacskey, add these parameters to the /etc/tacplus_servers file:

secret=tacacskey
server=192.168.0.30

Cumulus Linux supports a maximum of seven TACACS+ servers. To specify multiple servers, add one per line to the /etc/tacplus_servers file.

Connections establish in the order that this file lists. In most cases, you do not need to change any other parameters. You can add parameters the packages use to this file, which affects all the TACACS+ client software. For example, the timeout value for NSS lookups is 5 seconds by default in the /etc/tacplus_nss.conf file, whereas the timeout value for other packages is 10 seconds in the /etc/tacplus_servers file. The timeout value applies to each connection to the TACACS+ servers. (If you configure authorization per command, the timeout occurs for each command.) There are several connections to the server per login attempt from PAM, as well as two or more through NSS. Therefore, with the default timeout values, a TACACS+ server that is not reachable can delay logins by a minute or more for each unreachable server. If you must list unreachable TACACS+ servers, place them at the end of the server list and consider reducing the timeout values.

When you add or remove TACACS+ servers, you must restart auditd (with the systemctl restart auditd command) or you must send a signal (with killall -HUP audisp-tacplus) before audisp-tacplus rereads the configuration to see the changed server list.

You can also configure the IP address used as the source IP address when communicating with the TACACS+ server. See TACACS Configuration Parameters below for the full list of TACACS+ parameters.

Following is the complete list of the TACACS+ client configuration files, and their use.

TACACS+ Authentication (login)

PAM modules and an updated version of the libpam-tacplus package configure authentication initially. When you install the package, the pam-auth-update command updates the PAM configuration in /etc/pam.d. If you make changes to your PAM configuration, you need to integrate these changes. If you also use LDAP with the libpam-ldap package, you need to edit the PAM configuration with the LDAP and TACACS ordering you prefer. The libpam-tacplus package ignore rules and the values in success=2 require adjustments to ignore LDAP rules.

The TACACS+ privilege attribute priv_lvl determines the privilege level for the user that the TACACS+ server returns during the user authorization exchange. The client accepts the attribute in either the mandatory or optional forms and also accepts priv-lvl as the attribute name. The attribute value must be a numeric string in the range 0 to 15, with 15 the most privileged level.

TACACS+ Client Sequencing

Due to SSH and login processing mechanisms, Cumulus Linux needs to know the following at the beginning of the AAA sequence:

• Whether the user is a valid TACACS+ user
• The user privilege level

For non-local users (users not in the local password file) you need to send a TACACS+ authorization request as the first communication with the TACACS+ server, before authentication and before the user logging in requests a password.

You need to configure certain TACACS+ servers to allow authorization requests before authentication. Contact your TACACS+ server vendor for information.

Local Fallback Authentication

You can configure the switch to allow local fallback authentication for a user when the TACACS servers are unreachable, do not include the user for authentication, or have the user in the exclude user list.

To allow local fallback authentication for a user, add a local privileged user account on the switch with the same username as a TACACS user. A local user is always active even when the TACACS service is not running.

To configure local fallback authentication:

1. Edit the /etc/nsswitch.conf file to remove the keyword tacplus from the line starting with passwd. (You need to add the keyword back in step 3.)

   The following example shows the /etc/nsswitch.conf file with no tacplus keyword in the line starting with passwd.

   cumulus@switch:~$ sudo nano /etc/nsswitch.conf
   #
   # Example configuration of GNU Name Service Switch functionality.
   # If you have the `glibc-doc-reference' and `info' packages installed, try:
   # `info libc "Name Service Switch"' for information about this file.
   
   passwd:         files
   group:          tacplus files
   shadow:         files
   gshadow:        files
   ...
   

2. To enable the local privileged user to run sudo and NVUE commands, run the adduser commands shown below. In the example commands, the TACACS account name is tacadmin.

   The first adduser command prompts for information and a password. You can skip most of the requested information by pressing ENTER.

   cumulus@switch:~$ sudo adduser --ingroup tacacs tacadmin
   cumulus@switch:~$ sudo adduser tacadmin nvset
   cumulus@switch:~$ sudo adduser tacadmin nvapply
   cumulus@switch:~$ sudo adduser tacadmin sudo
   

3. Edit the /etc/nsswitch.conf file to add the keyword tacplus back to the line starting with passwd (the keyword you removed in the first step).

    cumulus@switch:~$ sudo nano /etc/nsswitch.conf
    #
    # Example configuration of GNU Name Service Switch functionality.
    # If you have the `glibc-doc-reference' and `info' packages installed, try:
    # `info libc "Name Service Switch"' for information about this file.
    passwd:         tacplus files
    group:          tacplus files
    shadow:         files
    gshadow:        files
    ...
   

4. Restart the nvued service and the nginx service with the following commands:

   cumulus@switch:~$ sudo systemctl restart nvued.service
   cumulus@switch:~$ sudo systemctl restart nginx.service
   

TACACS+ Accounting

TACACS+ accounting uses the audisp module, with an additional plugin for auditd/audisp. The plugin maps the auid in the accounting record to a TACACS login, which it bases on the auid and sessionid. The audisp module requires libnss_tacplus and uses the libtacplus_map.so library interfaces as part of the modified libpam_tacplus package.

Communication with the TACACS+ servers occurs with the libsimple-tacact1 library, through dlopen(). A maximum of 240 bytes of command name and arguments send in the accounting record, due to the TACACS+ field length limitation of 255 bytes.

Configure the IP address and encryption key of the server in the /etc/tacplus_servers file. Minimal configuration to auditd and audisp is necessary to enable the audit records needed for accounting. These records install as part of the package.

audisp-tacplus installs the audit rules for command accounting. When you configure a management VRF, you must add the vrf parameter and signal the audisp-tacplus process to reread the configuration. The example below shows that the management VRF name is mgmt. You can add the vrf parameter to either the /etc/tacplus_servers file or the /etc/audisp/audisp-tac_plus.conf file.

vrf=mgmt

After editing the configuration file, send the HUP signal killall -HUP audisp-tacplus to notify the accounting process to reread the file.

For more information, refer to the audisp.8 and auditd.8 man pages.

TACACS+ Per-command Authorization

The tacplus-auth command handles authorization for each command. To make this an enforced authorization, change the TACACS+ login to use a restricted shell, with a very limited executable search path. Otherwise, the user can bypass the authorization. The tacplus-restrict utility simplifies setting up the restricted environment. The example below initializes the environment for the tacacs0 user account. This is the account for TACACS+ users at privilege level 0.

tacuser0@switch:~$ sudo tacplus-restrict -i -u tacacs0 -a command1 command2 command3

The following table provides the command options:

For example, if you want to allow the user to be able to run the nv and ip commands (if authorized by the TACACS+ server):

cumulus@switch:~$ sudo tacplus-restrict -i -u tacacs0 -a ip nv

After running this command, examine the tacacs0 directory::

cumulus@switch:~$ sudo ls -lR ~tacacs0
total 12
lrwxrwxrwx 1 root root 22 Nov 21 22:07 ip -> /usr/sbin/tacplus-auth
lrwxrwxrwx 1 root root 22 Nov 21 22:07 nv -> /usr/sbin/tacplus-auth

Other than shell built-ins, privilege level 0 TACACS users can only run the ip and nv commands.

If add commands with the -a option by mistake, you can remove them. The example below removes the nv command:

cumulus@switch:~$ sudo rm ~tacacs0/bin/nv

You can remove all commands:

cumulus@switch:~$ sudo rm ~tacacs0/bin/*

For more information on tacplus-auth and tacplus-restrict, run the man command.

cumulus@switch:~$ man tacplus-auth tacplus-restrict

NSS Plugin

With pam_tacplus, TACACS+ authenticated users can log in without a local account on the system using the NSS plugin that comes with the tacplus_nss package. The plugin uses the mapped tacplus information if the user is not in the local password file, provides the getpwnam() and getpwuid()entry points, and uses the TACACS+ authentication functions.

The plugin asks the TACACS+ server if it knows the user, and then for relevant attributes to determine the privilege level of the user. When you install the libnss_tacplus package, nsswitch.conf changes to set tacplus as the first lookup method for passwd. If you change the order, lookups return the local accounts, such as tacacs0

If TACACS+ server does not find the user, it uses the libtacplus.so exported functions to do a mapped lookup. The privilege level appends to tacacs and the lookup searches for the name in the local password file. For example, privilege level 15 searches for the tacacs15 user. If the TACACS+ server finds the user, it adds information for the user in the password structure.

If the TACACS+ server does not find the user, it decrements the privilege level and checks again until it reaches privilege level 0 (user tacacs0). This allows you to use only the two local users tacacs0 and tacacs15, for minimal configuration.

TACACS Configuration Parameters

The recognized configuration options are the same as the libpam_tacplus command line arguments; however, Cumulus Linux does not support all pam_tacplus options. For a description of the configuration parameters, refer to the tacplus_servers.5 man page, which is part of the libpam-tacplus package.

The table below describes the configuration options available:

Remove the TACACS+ Client Packages

To remove all the TACACS+ client packages, use the following commands:

cumulus@switch:~$ sudo -E apt-get remove tacplus-client
cumulus@switch:~$ sudo -E apt-get autoremove

To remove the TACACS+ client configuration files as well as the packages (recommended), use this command:

cumulus@switch:~$ sudo -E apt-get autoremove --purge

Troubleshooting

Basic Server Connectivity or NSS Issues

You can use the getent command to determine if you configured TACACS+ correctly and if the local password is in the configuration files. In the example commands below, the cumulus user represents the local user, while cumulusTAC represents the TACACS user.

To look up the username within all NSS methods:

cumulus@switch:~$ sudo getent passwd cumulusTAC
cumulusTAC:x:1016:1001:TACACS+ mapped user at privilege level 15,,,:/home/tacacs15:/bin/bash

To look up the user within the local database only:

cumulus@switch:~$ sudo getent -s compat passwd cumulus
cumulus:x:1000:1000:cumulus,,,:/home/cumulus:/bin/bash

To look up the user within the TACACS+ database only:

cumulus@switch:~$ sudo getent -s tacplus passwd cumulusTAC
cumulusTAC:x:1016:1001:TACACS+ mapped user at privilege level 15,,,:/home/tacacs15:/bin/bash

If TACACS is not working correctly, debug the following configuration files by adding the debug=1 parameter to one or more of these files:

• /etc/tacplus_servers
• /etc/tacplus_nss.conf

All log messages are in /var/log/syslog.

Incorrect Shared Key

The TACACS client on the switch and the TACACS server must have the same shared secret key. If this key is incorrect, the following message prints to syslog:

2017-09-05T19:57:00.356520+00:00 leaf01 sshd[3176]: nss_tacplus: TACACS+ server 192.168.0.254:49 read failed with protocol error (incorrect shared secret?) user cumulus

Issues with Per-command Authorization

To debug TACACS user command authorization, have the TACACS+ user enter the following command at a shell prompt, then try the command again:

tacuser0@switch:~$ export TACACSAUTHDEBUG=1

When you enable debugging, the command authorization conversation with the TACACS+ server shows additional information.

To disable debugging:

tacuser0@switch:~$ export -n TACACSAUTHDEBUG

Debug Issues with Accounting Records

If you add or delete TACACS+ servers from the configuration files, make sure you notify the audisp plugin with this command:

cumulus@switch:~$ sudo killall -HUP audisp-tacplus

If accounting records do not send, add debug=1 to the /etc/audisp/audisp-tac_plus.conf file, then run the command above to notify the plugin. Ask the TACACS+ user to run a command and examine the end of /var/log/syslog for messages from the plugin. You can also check the auditing log file /var/log/audit audit.log to be sure the auditing records exist. If the auditing records do not exist, restart the audit daemon with:

cumulus@switch:~$ sudo systemctl restart auditd.service

TACACS Component Software Descriptions

Cumulus Linux uses the following packages for TACACS.

Considerations

TACACS+ Client Is only Supported through the Management Interface

The TACACS+ client is only supported through the management interface on the switch: eth0, eth1, or the VRF management interface. The TACACS+ client is not supported through bonds, switch virtual interfaces (SVIs), or switch port interfaces (swp).

Multiple TACACS+ Users

If two or more TACACS+ users log in simultaneously with the same privilege level, while the accounting records are correct, a lookup on either name matches both users, while a UID lookup only returns the user that logs in first.

This means that any processes that either user runs apply to both, and all files either user creates apply to the first name matched. This is similar to adding two local users to the password file with the same UID and GID, and is an inherent limitation of using the UID for the base user from the password file.

To work around this issue, you can use the switch audit log or the TACACS server accounting logs to determine which processes and files each user creates.

• For commands that do not execute other commands (for example, changes to configurations in an editor or actions with tools like clagctl and vtysh), there is no additional accounting.
• Per-command authorization is at the most basic level (Cumulus Linux uses standard Linux user permissions for the local TACACS users and only privilege level 15 users can run sudo commands by default).

The Linux auditd system does not always generate audit events for processes when terminated with a signal (with the kill system call or internal errors such as SIGSEGV). As a result, processes that exit on a signal that you do not handle, generate a STOP accounting record.

Issues with deluser Command

TACACS+ and other non-local users that run the deluser command with the --remove-home option see the following error:

tacuser0@switch: deluser --remove-home USERNAME
userdel: cannot remove entry 'USERNAME' from /etc/passwd
/usr/sbin/deluser: `/usr/sbin/userdel USERNAME' returned error code 1. Exiting

The command does remove the home directory. The user can still log in on that account but does not have a valid home directory. This is a known upstream issue with the deluser command for all non-local users.

Only use the --remove-home option with the user_homedir=1 configuration command.

Both TACACS+ and RADIUS AAA Clients

When you install both the TACACS+ and the RADIUS AAA client, Cumulus Linux does not attempt RADIUS login. As a workaround, do not install both the TACACS+ and the RADIUS AAA client on the same switch.

Various add-on packages enable RADIUS users to log in to Cumulus Linux switches in a transparent way with minimal configuration. There is no need to create accounts or directories on the switch. Authentication uses PAM and includes login, ssh, sudo and su.

Install the RADIUS Packages

You can install the RADIUS packages even if the switch is not connected to the internet, as they are in the cumulus-local-apt-archive repository, which is embedded in the Cumulus Linux image.

To install the RADIUS packages:

cumulus@switch:~$ sudo apt-get update
cumulus@switch:~$ sudo apt-get install libnss-mapuser libpam-radius-auth

After installation is complete, either reboot the switch or run the sudo systemctl restart nvued command.

The libpam-radius-auth package supplied with the Cumulus Linux RADIUS client is a newer version than the one in Debian Buster. This package contains support for IPv6, the src_ip option described below, as well as bug fixes and minor features. The package also includes VRF support, provides man pages describing the PAM and RADIUS configuration, and sets the SUDO_PROMPT environment variable to the login name for RADIUS mapping support.

The libnss-mapuser package is specific to Cumulus Linux and supports the getgrent, getgrnam and getgrgid library interfaces. These interfaces add logged in RADIUS users to the group member list for groups that contain the mapped_user (radius_user) if the RADIUS account does not have privileges, and add privileged RADIUS users to the group member list for groups that contain the mapped_priv_user (radius_priv_user) during the group lookups.

During package installation:

• The PAM configuration updates automatically using pam-auth-update (8), and the NSS configuration file /etc/nsswitch.conf adds the mapuser and mapuid plugins. If you remove or purge the packages, these files remove the configuration for these plugins.
• The radius_shell package installs the /sbin/radius_shell and setcap cap_setuid program for the login shell for RADIUS accounts. The package adjusts the UID when needed, then runs the bash shell with the same arguments. When installed, the package changes the shell of the RADIUS accounts to /sbin//radius_shell, and to /bin/shell if you remove the package. You need this package to enable privileged RADIUS users. You do not need this package for regular RADIUS clients.
• The nvshow group includes the radius_user account, the nvset and nvapply groups and sudo groups include the radius_priv_user account. This change enables all RADUS logins to run NVUE nv show commands and all privileged RADIUS users to also run nv set, nv unset, and nv apply commands, and to use sudo.

Configure the RADIUS Client

To configure the RADIUS client, edit the /etc/pam_radius_auth.conf file.

cumulus@switch:~$ sudo systemctl restart nvued.service
cumulus@switch:~$ sudo systemctl restart nginx.service

1. Add the hostname or IP address of at least one RADIUS server (such as a freeradius server on Linux), and the shared secret used to authenticate and encrypt communication with each server.

   You must be able to resolve the hostname of the switch to an IP address. If for some reason you cannot find the hostname in DNS, you can add the hostname to the /etc/hosts file manually. However, this can cause problems because DHCP assigns the IP address, which can change at any time.

   Multiple server configuration lines are verified in the order listed. Other than memory, there is no limit to the number of RADIUS servers you can use.

   The server port number or name is optional. The system looks up the port in the /etc/services file. However, you can override the ports in the /etc/pam_radius_auth.conf file.

2. If the server is slow or latencies are high, change the timeout setting. The setting defaults to 3 seconds.

3. If you want to use a specific interface to reach the RADIUS server, specify the src_ip option. You can specify the hostname of the interface, an IPv4, or an IPv6 address. If you specify the src_ip option, you must also specify the timeout option.

4. Set the vrf-name field. This is typically set to mgmt if you are using a management VRF. You cannot specify more than one VRF.

The configuration file includes the mapped_priv_user field that sets the account used for privileged RADIUS users and the priv-lvl field that sets the minimum value for the privilege level to be a privileged login (the default value is 15). If you edit these fields, make sure the values match those set in the /etc/nss_mapuser.conf file.

The following example provides a sample /etc/pam_radius_auth.conf file configuration:

mapped_priv_user   radius_priv_user
# server[:port]    shared_secret   timeout (secs)  src_ip
192.168.0.254      secretkey
other-server       othersecret     3               192.168.1.10
# when mgmt vrf is in use
vrf-name mgmt

As an optional step, you can set PAM configuration keywords by editing the /usr/share/pam-configs/radius file. After you edit the file, you must run the pam-auth-update --package command. The pam_radius_auth (8) man page describes the PAM configuration keywords.

Service-Type = Administrative-User,
Cisco-AVPair = "shell:roles=network-administrator",
Cisco-AVPair += "shell:priv-lvl=15"

Enable Login without Local Accounts

LDAP is not commonly used with switches and adding accounts locally is cumbersome, Cumulus Linux includes a mapping capability with the libnss-mapuser package.

Mapping uses two NSS (Name Service Switch) plugins, one for account name, and one for UID lookup. The installation process configures these accounts automatically in the /etc/nsswitch.conf file and removes them when you delete the package. See the nss_mapuser (8) man page for the full description of this plugin.

A username is mapped at login to a fixed account specified in the configuration file, with the fields of the fixed account used as a template for the user that is logging in.

For example, if you look up the name dave and the fixed account in the configuration file is radius\_user, and that entry in /etc/passwd is:

radius_user:x:1017:1002:radius user:/home/radius_user:/bin/bash

then the matching line that returns when you run getent passwd dave is:

cumulus@switch:~$ getent passwd dave
dave:x:1017:1002:dave mapped user:/home/dave:/bin/bash

The login process creates the home directory /home/dave if it does not already exist and populates it with the standard skeleton files by the mkhomedir_helper command.

The configuration file /etc/nss_mapuser.conf configures the plugins. The file includes the mapped account name, which is radius_user by default. You can change the mapped account name by editing the file. The nss_mapuser (5) man page describes the configuration file.

A flat file mapping derives from the session number assigned during login, which persists across su and sudo. Cumulus Linux removes the mapping at logout.

Local Fallback Authentication

If a site wants to allow local fallback authentication for a user when none of the RADIUS servers are reachable, you can add a privileged user account as a local account on the switch. The local account must have the same unique identifier as the privileged user and the shell must be the same.

To configure local fallback authentication:

1. Add a local privileged user account. For example, if the radius_priv_user account in the /etc/passwd file is radius_priv_user:x:1002:1001::/home/radius_priv_user:/sbin/radius_shell, run the following command to add a local privileged user account named johnadmin:

   cumulus@switch:~$ sudo useradd -u 1002 -g 1001 -o -s /sbin/radius_shell johnadmin
   

2. To enable the local privileged user to run sudo and NVUE commands, run the following commands:

   cumulus@switch:~$ sudo adduser johnadmin nvset
   cumulus@switch:~$ sudo adduser johnadmin nvapply
   cumulus@switch:~$ sudo adduser johnadmin sudo
   cumulus@switch:~$ sudo systemctl restart nvued
   

3. Edit the /etc/passwd file to move the local user line before to the radius_priv_user line:

   cumulus@switch:~$ sudo vi /etc/passwd
   ...
   johnadmin:x:1002:1001::/home/johnadmin:/sbin/radius_shell
   radius_priv_user:x:1002:1001::/home/radius_priv_user:/sbin/radius_shell
   

4. To set the local password for the local user, run the following command:

   cumulus@switch:~$ sudo passwd johnadmin
   

Verify RADIUS Client Configuration

To verify that you configured the RADIUS client correctly, log in as a non-privileged user and run a nv set interface command.

In this example, the ops user is not a privileged RADIUS user so the ops user cannot add an interface.

ops@leaf01:~$ nv set interface swp1
ERROR: User ops does not have permission to make networking changes.

In this example, the admin user is a privileged RADIUS user (with privilege level 15) so is able to add interface swp1.

admin@leaf01:~$ nv set interface swp1
admin@leaf01:~$ nv apply

Remove RADIUS Client Packages

Remove the RADIUS packages with the following command:

cumulus@switch:~$ sudo apt-get remove libnss-mapuser libpam-radius-auth

When you remove the packages, Cumulus Linux deletes the plugins from the /etc/nsswitch.conf file and from the PAM files.

To remove all configuration files for these packages, run:

cumulus@switch:~$ sudo apt-get purge libnss-mapuser libpam-radius-auth

To remove the home directories of the RADIUS users, first get the list by running:

cumulus@switch:~$ sudo ls -l /home | grep radius

For all users listed, except the radius_user, run this command to remove the home directories:

cumulus@switch:~$ sudo deluser --remove-home USERNAME

where USERNAME is the account name (the home directory relative portion). This command gives the following warning because the user is not listed in the /etc/passwd file.

userdel: cannot remove entry 'USERNAME' from /etc/passwd
/usr/sbin/deluser: `/usr/sbin/userdel USERNAME' returned error code 1. Exiting.

After you remove all the RADIUS users, run the command to remove the fixed account. If there are changes to the account in the /etc/nss_mapuser.conf file, use that account name instead of radius_user.

cumulus@switch:~$ sudo deluser --remove-home radius_user
cumulus@switch:~$ sudo deluser --remove-home radius_priv_user
cumulus@switch:~$ sudo delgroup radius_users

Considerations

• If two or more RADIUS users log in simultaneously, a UID lookup only returns the user that logs in first. Any process that either user runs applies to both, and all files that either user creates apply to the first name matched. This process is similar to adding two local users to the password file with the same UID and GID, and is an inherent limitation of using the UID for the fixed user from the password file. The current algorithm returns the first name matching the UID from the mapping file, which is either the first or second user that logs in.

• When you install both the TACACS+ and the RADIUS AAA client, Cumulus Linux does not attempt the RADIUS login. As a workaround, do not install both the TACACS+ and the RADIUS AAA client on the same switch.

• When the RADIUS server is reachable outside of the management VRF, such as in the default VRF, you might see the following error message when you try to run sudo:

  2008-10-31T07:06:36.191359+00:00 SW01 sudo: pam_radius_auth(sudo:auth): Bind for server 10.1.1.25 failed: Cannot assign requested address
  2008-10-31T07:06:36.192307+00:00 sw01 sudo: pam_radius_auth(sudo:auth): No valid server found in configuration file /etc/pam_radius_auth.conf
  

  The error occurs because sudo tries to authenticate to the RADIUS server through the management VRF. Before you run sudo, you must set the shell to the correct VRF:

  cumulus@switch:~$ vrf exec default bash
  cumulus@switch:~$ sudo 
  

Netfilter is the packet filtering framework in Cumulus Linux and other Linux distributions. You can use several different tools to configure ACLs in Cumulus Linux:

• iptables, ip6tables, and ebtables are Linux userspace tools you use to administer filtering rules for IPv4 packets, IPv6 packets, and Ethernet frames (layer 2 using MAC addresses).
• cl-acltool is a Cumulus Linux-specific userspace tool you use to administer filtering rules and configure default ACLs. cl-acltool operates on various configuration files and uses iptables, ip6tables, and ebtables to install rules into the kernel. In addition, cl-acltool programs rules in hardware for switch port interfaces, which iptables, ip6tables and ebtables cannot do on their own.
• NVUE is a Cumulus Linux-specific userspace tool you can use to configure custom ACLs.

Traffic Rules

Chains

Netfilter describes the way that the Linux kernel classifies and controls packets to, from, and across the switch. Netfilter does not require a separate software daemon to run; it is part of the Linux kernel. Netfilter asserts policies at layer 2, 3 and 4 of the OSI model by inspecting packet and frame headers according to a list of rules. The iptables, ip6tables, and ebtables userspace applications provide syntax you use to define rules.

The rules inspect or operate on packets at several points (chains) in the life of the packet through the system:

• PREROUTING touches packets before the switch routes them.
• INPUT touches packets after the switch determines that the packets are for the local system but before the control plane software receives them.
• FORWARD touches transit traffic as it moves through the switch.
• OUTPUT touches packets from the control plane software before they leave the switch.
• POSTROUTING touches packets immediately before they leave the switch but after a routing decision.

Tables

When you build rules to affect the flow of traffic, tables can access the individual chains. Linux provides three tables by default:

• Filter classifies traffic or filters traffic
• NAT applies Network Address Translation rules
• Mangle alters packets as they move through the switch

Each table has a set of default chains that modify or inspect packets at different points of the path through the switch. Chains contain the individual rules to influence traffic.

Rules

Rules classify the traffic you want to control. You apply rules to chains, which attach to tables.

Rules have several different components:

• Table: The first argument is the table.
• Chain: The second argument is the chain. Each table supports several different chains. See Tables above.
• Matches: The third argument is the match. You can specify multiple matches in a single rule. However, the more matches you use in a rule, the more memory the rule consumes.
• Jump: The jump specifies the target of the rule; what action to take if the packet matches the rule. If you omit this option in a rule, matching the rule has no effect on the packet, but the counters on the rule increment.
• Targets: The target is a user-defined chain (other than the one this rule is in), one of the special built-in targets that decides the fate of the packet immediately (like DROP), or an extended target. See Supported Rule Types below for different target examples.

How Rules Parse and Apply

The switch reads all the rules from each chain from iptables, ip6tables, and ebtables and enters them in order into either the filter table or the mangle table. The switch reads the rules from the kernel in the following order:

• IPv6 (ip6tables)
• IPv4 (iptables)
• ebtables

When you combine and put rules into one table, the order determines the relative priority of the rules; iptables and ip6tables have the highest precedence and ebtables has the lowest.

The Linux packet forwarding construct is an overlay for how the silicon underneath processes packets. Be aware of the following:

• The switch silicon reorders rules when switchd writes to the ASIC, whereas traditional iptables execute the list of rules in order.

• All rules, except for POLICE and SETCLASS rules, are terminating; after a rule matches, the action occurs and no more rules process.

• When processing traffic, rules affecting the FORWARD chain that specify an ingress interface process before rules that match on an egress interface. As a workaround, rules that only affect the egress interface can have an ingress interface wildcard (only swp+ and bond+) that matches any interface you apply so that you can maintain order of operations with other input interface rules. For example, with the following rules:

  -A FORWARD -i swp1 -j ACCEPT
  -A FORWARD -o swp1 -j ACCEPT   <-- This rule processes LAST (because of egress interface matching)
  -A FORWARD -i swp2 -j DROP
  

  If you modify the rules like this, they process in order:

  -A FORWARD -i swp1 -j ACCEPT
  -A FORWARD -i swp+ -o $PORTA -j ACCEPT   <-- These rules are performed in order (because of wildcard match on the ingress interface)
  -A FORWARD -i swp2 -j DROP
  

• When using rules that do a mangle and a filter lookup for a packet, Cumulus Linux processes them in parallel and combines the action.

• If there is no ingress interface or egress interface match, Cumulus Linux installs FORWARD chain rules in ingress by default.

• When using the OUTPUT chain, you must assign rules to the source. For example, if you assign a rule to the switch port in the direction of traffic but the source is a bridge (VLAN), the rule does not affect the traffic and you must apply it to the bridge.

• If you need to apply a rule to all transit traffic, use the FORWARD chain, not the OUTPUT chain.

• The switch puts ebtable rules into either the IPv4 or IPv6 memory space depending on whether the rule uses IPv4 or IPv6 to make a decision. The switch only puts layer 2 rules that match the MAC address into the IPv4 memory space.

Rule Placement in Memory

INPUT and ingress (FORWARD -i) rules occupy the same memory space. A rule counts as ingress if you set the -i option. If you set both input and output options (-i and -o), the switch considers the rule as ingress and occupies that memory space. For example:

-A FORWARD -i swp1 -o swp2 -s 10.0.14.2 -d 10.0.15.8 -p tcp -j ACCEPT

-A FORWARD,INPUT -i swp1 -o swp2 -s 10.0.14.2 -d 10.0.15.8 -p tcp -j ACCEPT
error: line 2 : output interface specified with INPUT chain error processing rule '-A FORWARD,INPUT -i swp1 -o swp2 -s 10.0.14.2 -d 10.0.15.8 -p tcp -j ACCEPT'

Nonatomic Update Mode and Atomic Update Mode

Cumulus Linux enables atomic update mode by default. However, this mode limits the number of ACL rules that you can configure.

To increase the number of configurable ACL rules, configure the switch to operate in nonatomic mode.

Instead of reserving 50% of your TCAM space for atomic updates, incremental update uses the available free space to write the new TCAM rules and swap over to the new rules after this is complete. Cumulus Linux then deletes the old rules and frees up the original TCAM space. If there is insufficient free space to complete this task, the original nonatomic update runs, which interrupts traffic.

You can enable nonatomic updates for switchd, which offer better scaling because all TCAM resources actively impact traffic. With atomic updates, half of the hardware resources are on standby and do not actively impact traffic.

Incremental nonatomic updates are table based, so they do not interrupt network traffic when you install new rules. The rules map to the following tables and update in this order:

• mirror (ingress only)
• ipv4-mac (can be both ingress and egress)
• ipv6 (ingress only)

The incremental nonatomic update operation follows this order:

1. Updates are incremental, one table at a time without stopping traffic.
2. Cumulus Linux checks if the rules in a table are different from installation time; if a table does not have any changes, it does not reinstall the rules.
3. If there are changes in a table, the new rules populate in new groups or slices in hardware, then that table switches over to the new groups or slices.
4. Finally, old resources for that table free up. This process repeats for each of the tables listed above.
5. If there are insufficient resources to hold both the new rule set and old rule set, Cumulus Linux tries the regular nonatomic mode, which interrupts network traffic.
6. If the regular nonatomic update fails, Cumulus Linux reverts back to the previous rules.

To always start switchd with nonatomic updates:

1. Edit /etc/cumulus/switchd.conf.

2. Add the following line to the file:

   acl.non_atomic_update_mode = TRUE
   

3. Restart switchd:

   cumulus@switch:~$ sudo systemctl restart switchd.service

   Restarting the switchd service causes all network ports to reset, interrupting network services, in addition to resetting the switch hardware configuration.

Use iptables, ip6tables, and ebtables Directly

Do not use iptables, ip6tables, ebtables directly; installed rules only apply to the Linux kernel and Cumulus Linux does not hardware accelerate. When you run cl-acltool -i, Cumulus Linux resets all rules and deletes anything that is not in /etc/cumulus/acl/policy.conf.

For example, the following rule appears to work:

cumulus@switch:~$ sudo iptables -A INPUT -p icmp --icmp-type echo-request -j DROP

The cl-acltool -L command shows the rule:

cumulus@switch:~$ sudo cl-acltool -L ip
-------------------------------
Listing rules of type iptables:
-------------------------------

TABLE filter :
Chain INPUT (policy ACCEPT 72 packets, 5236 bytes)
pkts bytes target prot opt in out source destination
0 0 DROP icmp -- any any anywhere anywhere icmp echo-request

However, Cumulus Linux does not synchronize the rule to hardware. Running cl-acltool -i or reboot removes the rule without replacing it. To ensure that Cumulus Linux hardware accelerates all rules that can be in hardware, add them to /etc/cumulus/acl/policy.conf and install them with the cl-acltool -i command.

Estimate the Number of Rules

To estimate the number of rules you can create from an ACL entry, first determine if that entry is an ingress or an egress. Then, determine if it is an IPv4-mac or IPv6 type rule. This determines the slice to which the rule belongs. Use the following to determine how many entries the switch uses for each type.

By default, each entry occupies one double wide entry, except if the entry is one of the following:

• An entry with multiple comma-separated input interfaces splits into one rule for each input interface. For example, this entry splits into two rules:

  -A FORWARD -i swp1s0,swp1s1 -p icmp -j ACCEPT
  

• An entry with multiple comma-separated output interfaces splits into one rule for each output interface. This entry splits into two rules:

  -A FORWARD -i swp+ -o swp1s0,swp1s1 -p icmp -j ACCEPT
  

• An entry with both input and output comma-separated interfaces splits into one rule for each combination of input and output interface This entry splits into four rules:

  -A FORWARD -i swp1s0,swp1s1 -o swp1s2,swp1s3 -p icmp -j ACCEPT
  

• An entry with multiple layer 4 port ranges splits into one rule for each range. For example, this entry splits into two rules:

  -A FORWARD -i swp+ -p tcp -m multiport --dports 1050:1051,1055:1056 -j ACCEPT
  

  You can only use port ranges for ingress rules.

Match on VLAN IDs on Layer 2 Interfaces

You can match on VLAN IDs on layer 2 interfaces for ingress rules. The following example matches on a VLAN and DSCP class, and sets the internal class of the packet. For extended matching on IP fields, combine this rule with ingress iptable rules.

[ebtables]
-A FORWARD -p 802_1Q --vlan-id 100 -j mark --mark-set 102

[iptables]
-A FORWARD -i swp31 -m mark --mark 102 -m dscp --dscp-class CS1 -j SETCLASS --class 2

Install and Manage ACL Rules with NVUE

Instead of crafting a rule by hand, then installing it with cl-acltool, you can use NVUE commands. Cumulus Linux converts the commands to the /etc/cumulus/acl/policy.d/50_nvue.rules file. The rules you create with NVUE are independent of the default files /etc/cumulus/acl/policy.d/00control_plane.rules and 99control_plane_catch_all.rules.

Consider the following iptables rule:

-t mangle -A PREROUTING -i swp1 -s 10.0.14.2/32 -d 10.0.15.8/32 -p tcp -j ACCEPT

To create this rule with NVUE, follow the steps below. NVUE adds all options in the rule automatically.

1. Set the rule type, the matching protocol, source IP address and port, destination IP address and port, and the action. You must provide a name for the rule (EXAMPLE1 in the commands below):

   cumulus@switch:~$ nv set acl EXAMPLE1 type ipv4
   cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 match ip protocol tcp
   cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 match ip source-ip 10.0.14.2/32
   cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 match ip source-port ANY
   cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 match ip dest-ip 10.0.15.8/32
   cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 match ip dest-port ANY
   cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 action permit
   

2. Apply the rule to an inbound or outbound interface with the nv set interface <interface> acl command.

   • For rules affecting the -t mangle -A PREROUTING chain (-A FORWARD in previous releases), apply the rule to an inbound or outbound interface: For example:

   cumulus@switch:~$ nv set interface swp1 acl EXAMPLE1 inbound
   cumulus@switch:~$ nv config apply
   

   • For rules affecting the INPUT or OUPUT chain (-A INPUT or -A OUTPUT), apply the rule to a control plane interface. For example:

   cumulus@switch:~$ nv set interface swp1 acl EXAMPLE1 inbound control-plane
   cumulus@switch:~$ nv config apply
   

To see all installed rules, examine the /etc/cumulus/acl/policy.d/50_nvue.rules file:

cumulus@switch:~$ sudo cat /etc/cumulus/acl/policy.d/50_nvue.rules
[iptables]

## ACL EXAMPLE1 in dir inbound on interface swp1 ##
-t mangle -A PREROUTING -i swp1 -s 10.0.14.2/32 -d 10.0.15.8/32 -p tcp -j ACCEPT
...

To remove this rule, run the nv unset acl <acl-name> and nv unset interface <interface> acl <acl-name> commands. These commands delete the rule from the /etc/cumulus/acl/policy.d/50_nvue.rules file.

cumulus@switch:~$ nv unset acl EXAMPLE1
cumulus@switch:~$ nv unset interface swp1 acl EXAMPLE1
cumulus@switch:~$ nv config apply

To show ACL statistics per interface, such as the total number of bytes that match the ACL rule, run the nv show interface <interface-id> acl <acl-id> statistics or nv show interface <interface-id> acl <acl-id> statistics <rule-id> command.

To see the list of all NVUE ACL commands, run the nv list-commands acl command.

Install and Manage ACL Rules with cl-acltool

You can manage Cumulus Linux ACLs with cl-acltool. Rules write first to the iptables chains, as described above, and then synchronize to hardware through switchd.

To examine the current state of chains and list all installed rules, run:

cumulus@switch:~$ sudo cl-acltool -L all
 -------------------------------
Listing rules of type iptables:
-------------------------------

TABLE filter :
Chain INPUT (policy ACCEPT 90 packets, 14456 bytes)
pkts bytes target prot opt in out source destination
0 0 DROP all -- swp+ any 240.0.0.0/5 anywhere
0 0 DROP all -- swp+ any loopback/8 anywhere
0 0 DROP all -- swp+ any base-address.mcast.net/8 anywhere
0 0 DROP all -- swp+ any 255.255.255.255 anywhere ...

To list installed rules using native iptables, ip6tables and ebtables, use the -L option with the respective commands:

cumulus@switch:~$ sudo iptables -L
cumulus@switch:~$ sudo ip6tables -L
cumulus@switch:~$ sudo ebtables -L

To remove all installed rules, run:

cumulus@switch:~$ sudo cl-acltool -F all

To remove only the IPv4 iptables rules, run:

cumulus@switch:~$ sudo cl-acltool -F ip

If the install fails, ACL rules in the kernel and hardware roll back to the previous state. You also see errors from programming rules in the kernel or ASIC.

Install Packet Filtering (ACL) Rules

cl-acltool takes access control list (ACL) rule input in files. Each ACL policy file includes iptables, ip6tables and ebtables categories under the tags [iptables], [ip6tables] and [ebtables]. You must assign each rule in an ACL policy to one of the rule categories.

See man cl-acltool(5) for ACL rule details. For iptables rule syntax, see man iptables(8). For ip6tables rule syntax, see man ip6tables(8). For ebtables rule syntax, see man ebtables(8).

See man cl-acltool(5) and man cl-acltool(8) for more details on using cl-acltool.

Here is an example ACL policy file:

[iptables]
-A INPUT -i swp1 -p tcp --dport 80 -j ACCEPT
-A FORWARD -i swp1 -p tcp --dport 80 -j ACCEPT

[ip6tables]
-A INPUT -i swp1 -p tcp --dport 80 -j ACCEPT
-A FORWARD -i swp1 -p tcp --dport 80 -j ACCEPT

[ebtables]
-A INPUT -p IPv4 -j ACCEPT
-A FORWARD -p IPv4 -j ACCEPT

You can use wildcards or variables to specify chain and interface lists.

INGRESS = swp+
INPUT_PORT_CHAIN = INPUT,FORWARD

[iptables]
-A $INPUT_PORT_CHAIN -i $INGRESS -p tcp --dport 80 -j ACCEPT

[ip6tables]
-A $INPUT_PORT_CHAIN -i $INGRESS -p tcp --dport 80 -j ACCEPT

[ebtables]
-A INPUT -p IPv4 -j ACCEPT

You can write ACL rules for the system into multiple files under the default /etc/cumulus/acl/policy.d/ directory. The ordering of rules during installation follows the sort order of the files according to their file names.

Use multiple files to stack rules. The example below shows two rule files that separate rules for management and datapath traffic:

cumulus@switch:~$ ls /etc/cumulus/acl/policy.d/
00sample_mgmt.rules 01sample_datapath.rules
cumulus@switch:~$ cat /etc/cumulus/acl/policy.d/00sample_mgmt.rules

INGRESS_INTF = swp+
INGRESS_CHAIN = INPUT

[iptables]
# protect the switch management
-A $INGRESS_CHAIN -i $INGRESS_INTF -s 10.0.14.2 -d 10.0.15.8 -p tcp -j ACCEPT
-A $INGRESS_CHAIN -i $INGRESS_INTF -s 10.0.11.2 -d 10.0.12.8 -p tcp -j ACCEPT
-A $INGRESS_CHAIN -i $INGRESS_INTF -d 10.0.16.8 -p udp -j DROP

cumulus@switch:~$ cat /etc/cumulus/acl/policy.d/01sample_datapath.rules
INGRESS_INTF = swp+
INGRESS_CHAIN = INPUT, FORWARD

[iptables]
-A $INGRESS_CHAIN -i $INGRESS_INTF -s 192.0.2.5 -p icmp -j ACCEPT
-A $INGRESS_CHAIN -i $INGRESS_INTF -s 192.0.2.6 -d 192.0.2.4 -j DROP
-A $INGRESS_CHAIN -i $INGRESS_INTF -s 192.0.2.2 -d 192.0.2.8 -j DROP

Install all ACL policies under a directory:

cumulus@switch:~$ sudo cl-acltool -i -P ./rules
Reading files under rules
Reading rule file ./rules/01_http_rules.txt ...
Processing rules in file ./rules/01_http_rules.txt ...
Installing acl policy ...
Done.

Apply all rules and policies included in /etc/cumulus/acl/policy.conf:

cumulus@switch:~$ sudo cl-acltool -i

Specify the Policy Files to Install

By default, Cumulus Linux installs any .rules file you configure in /etc/cumulus/acl/policy.d/. To add other policy files to an ACL, you need to include them in /etc/cumulus/acl/policy.conf. For example, for Cumulus Linux to install a rule in a policy file called 01_new.datapathacl, add include /etc/cumulus/acl/policy.d/01_new.rules to policy.conf:

cumulus@switch:~$ sudo nano /etc/cumulus/acl/policy.conf

#
# This file is a master file for acl policy file inclusion
#
# Note: This is not a file where you list acl rules.
#
# This file can contain:
# - include lines with acl policy files
#   example:
#     include <filepath>
#
# see manpage cl-acltool(5) and cl-acltool(8) for how to write policy files 
#

include /etc/cumulus/acl/policy.d/01_new.datapathacl

Hardware Limitations on Number of Rules

The maximum number of rules that the hardware process depends on:

• The mix of IPv4 and IPv6 rules; Cumulus Linux does not support the maximum number of rules for both IPv4 and IPv6 simultaneously.
• The number of default rules that Cumulus Linux provides.
• Whether the rules apply on ingress or egress.
• Whether the rules are in atomic or nonatomic mode; Cumulus Linux uses nonatomic mode rules when you enable nonatomic updates (see above).

If you exceed the maximum number of rules for a particular table, cl-acltool -i generates the following error:

error: hw sync failed (sync_acl hardware installation failed) Rolling back .. failed.

In the table below, the default rules count toward the limits listed. The raw limits below assume only one ingress and one egress table are present.

The NVIDIA Spectrum ASIC has one common TCAM for both ingress and egress, which you can use for other non-ACL-related resources. However, the number of supported rules varies with the TCAM profile for the switch.

Supported Rule Types

The iptables/ip6tables/ebtables construct tries to layer the Linux implementation on top of the underlying hardware but they are not always directly compatible. Here are the supported rules for chains in iptables, ip6tables and ebtables.

To learn more about any of the options shown in the tables below, run iptables -h [name of option]. The same help syntax works for options for ip6tables and ebtables.

root@leaf1# ebtables -h tricolorpolice
...
tricolorpolice option:
--set-color-mode STRING setting the mode in blind or aware
--set-cir INT setting committed information rate in kbits per second
--set-cbs INT setting committed burst size in kbyte
--set-pir INT setting peak information rate in kbits per second
--set-ebs INT setting excess burst size in kbyte
--set-conform-action-dscp INT setting dscp value if the action is accept for conforming packets
--set-exceed-action-dscp INT setting dscp value if the action is accept for exceeding packets
--set-violate-action STRING setting the action (accept/drop) for violating packets
--set-violate-action-dscp INT setting dscp value if the action is accept for violating packets
Supported chains for the filter table:
INPUT FORWARD OUTPUT

iptables and ip6tables Rule Support

ebtables Rule Support

Other Unsupported Rules

• Rules that have no matches and accept all packets in a chain are currently ignored.
• Chain default rules (that are ACCEPT) are also ignored.

Considerations

Splitting rules across the ingress TCAM and the egress TCAM causes the ingress IPv6 part of the rule to match packets going to all destinations, which can interfere with the regular expected linear rule match in a sequence. For example:

A higher rule can prevent a lower rule from matching:

Rule 1: -A FORWARD -o vlan100 -p icmp6 -j ACCEPT

Rule 2: -A FORWARD -o vlan101 -p icmp6 -s 01::02 -j ACCEPT

Rule 1 matches all icmp6 packets from to all out interfaces in the ingress TCAM.

This prevents rule 2 from matching, which is more specific but with a different out interface. Make sure to put more specific matches above more general matches even if the output interfaces are different.

When you have two rules with the same output interface, the lower rule might match depending on the presence of the previous rules.

Rule 1: -A FORWARD -o vlan100 -p icmp6 -j ACCEPT

Rule 2: -A FORWARD -o vlan101 -s 00::01 -j DROP

Rule 3: -A FORWARD -o vlan101 -p icmp6 -j ACCEPT

Rule 3 still matches for an icmp6 packet with sip 00:01 going out of vlan101. Rule 1 interferes with the normal function of rule 2 and/or rule 3.

When you have two adjacent rules with the same match and different output interfaces, such as:

Rule 1: -A FORWARD -o vlan100 -p icmp6 -j ACCEPT

Rule 2: -A FORWARD -o vlan101 -p icmp6 -j DROP

Rule 2 never matches on ingress. Both rules share the same mark.

Common Examples

Data Plane Policers

You can configure quality of service for traffic on the data plane. By using QoS policers, you can rate limit traffic so incoming packets get dropped if they exceed specified thresholds.

cumulus@switch:~$ nv set acl example1 type ipv4
cumulus@switch:~$ nv set acl example1 rule 10 action police
cumulus@switch:~$ nv set acl example1 rule 10 action police mode packet
cumulus@switch:~$ nv set acl example1 rule 10 action police burst 200
cumulus@switch:~$ nv set acl example1 rule 10 action police rate 400
cumulus@switch:~$ nv set interface swp1 acl example1 inbound
cumulus@switch:~$ nv config apply

-t mangle -A PREROUTING -i swp1  -j POLICE --set-mode pkt --set-rate 400 --set-burst 200

Control Plane Policers

You can configure quality of service for traffic on the control plane and rate limit traffic so incoming packets drop if they exceed certain thresholds in the following ways:

• Run NVUE commands.
• Edit the /etc/cumulus/control-plane/policers.conf file.

cumulus@switch:~$ nv set system control-plane policer pim-ospf-rip rate 400
cumulus@switch:~$ nv set system control-plane policer pim-ospf-rip burst 400
cumulus@switch:~$ nv set system control-plane policer pim-ospf-rip state on
cumulus@switch:~$ nv set system control-plane policer igmp rate 400
cumulus@switch:~$ nv set system control-plane policer igmp burst 200
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo nano /etc/cumulus/control-plane/policers.conf
...
copp.pim_ospf_rip.enable = TRUE
copp.pim_ospf_rip.rate = 400
copp.pim_ospf_rip.burst = 400
...
copp.igmp.enable = TRUE
copp.igmp.rate = 400
copp.igmp.burst = 200
...

cumulus@switch:~$ switchdctl --load /etc/cumulus/control-plane/policers.conf

To show the control plane police configuration and statistics, run the NVUE nv show system control-plane policer --view=statistics command.

cumulus@leaf01:mgmt:~$ sudo cat /etc/cumulus/control-plane/policers.conf
copp.arp.enable = TRUE
copp.arp.rate = 800
copp.arp.burst = 800

copp.bfd.enable = TRUE
copp.bfd.rate = 2000
copp.bfd.burst = 2000

copp.pim_ospf_rip.enable = TRUE
copp.pim_ospf_rip.rate = 2000
copp.pim_ospf_rip.burst = 2000

copp.bgp.enable = TRUE
copp.bgp.rate = 2000
copp.bgp.burst = 2000

copp.clag.enable = TRUE
copp.clag.rate = 2000
copp.clag.burst = 2000

copp.icmp_def.enable = TRUE
copp.icmp_def.rate = 100
copp.icmp_def.burst = 40

copp.dhcp_ptp.enable = TRUE
copp.dhcp_ptp.rate = 2000
copp.dhcp_ptp.burst = 2000

copp.igmp.enable = TRUE
copp.igmp.rate = 1000
copp.igmp.burst = 1000

copp.ssh.enable = TRUE
copp.ssh.rate = 1000
copp.ssh.burst = 1000

copp.icmp6_neigh.enable = TRUE
copp.icmp6_neigh.rate = 500
copp.icmp6_neigh.burst = 500

copp.icmp6_def_mld.enable = TRUE
copp.icmp6_def_mld.rate = 300
copp.icmp6_def_mld.burst = 100

copp.lacp.enable = TRUE
copp.lacp.rate = 2000
copp.lacp.burst = 2000

copp.lldp.enable = TRUE
copp.lldp.rate = 200
copp.lldp.burst = 200

copp.rpvst.enable = TRUE
copp.rpvst.rate = 2000
copp.rpvst.burst = 2000

copp.eapol.enable = TRUE
copp.eapol.rate = 2000
copp.eapol.burst = 2000

copp.ip2me.enable = TRUE
copp.ip2me.rate = 1000
copp.ip2me.burst = 1000

copp.acl_log.enable = TRUE
copp.acl_log.rate = 100
copp.acl_log.burst = 100

copp.nat.enable = TRUE
copp.nat.rate = 200
copp.nat.burst = 200

copp.stp.enable = TRUE
copp.stp.rate = 2000
copp.stp.burst = 2000

copp.l3_local.enable = TRUE
copp.l3_local.rate = 400
copp.l3_local.burst = 100

copp.span_cpu.enable = TRUE
copp.span_cpu.rate = 100
copp.span_cpu.burst = 100

copp.catch_all.enable = TRUE
copp.catch_all.rate = 100
copp.catch_all.burst = 100

Set DSCP on Transit Traffic

The examples here use the mangle table to modify the packet as it transits the switch. DSCP is in decimal notation in the examples below.

[iptables]

#Set SSH as high priority traffic.
-t mangle -A PREROUTING -i swp+ -p tcp -m multiport --dports 22 -j SETQOS --set-dscp 46

#Set everything coming in swp1 as AF13
-t mangle -A PREROUTING -i swp1  -j SETQOS --set-dscp 14

#Set Packets destined for 10.0.100.27 as best effort
-t mangle -A PREROUTING -i swp+ -d 10.0.100.27/32 -j SETQOS --set-dscp 0

#Example using a range of ports for TCP traffic
-t mangle -A PREROUTING -i swp+ -s 10.0.0.17/32 -d 10.0.100.27/32 -p tcp -m multiport --sports 10000:20000 -m multiport --dports 10000:20000 -j SETQOS --set-dscp 34

cumulus@switch:~$ sudo cl-acltool -i

cumulus@switch:~$ nv set acl EXAMPLE1 type ipv4
cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 match ip protocol tcp
cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 match ip dest-port 22
cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 action set dscp 46
cumulus@switch:~$ nv set interface swp1-48 acl EXAMPLE1 inbound
cumulus@switch:~$ nv config apply

cumulus@switch:~$ nv set acl EXAMPLE1 type ipv4
cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 action set dscp 14
cumulus@switch:~$ nv set interface swp1 acl EXAMPLE1 inbound
cumulus@switch:~$ nv config apply

cumulus@switch:~$ nv set acl EXAMPLE1 type ipv4
cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 match ip dest-ip 10.0.100.27/32
cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 action set dscp 0
cumulus@switch:~$ nv set interface swp1-48 acl EXAMPLE1 inbound
cumulus@switch:~$ nv config apply

cumulus@switch:~$ nv set acl EXAMPLE1 type ipv4
cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 match ip protocol tcp
cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 match ip source-ip 10.0.0.17/32
cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 match ip source-port 10000:20000
cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 match ip dest-ip 10.0.100.27/32
cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 match ip dest-port 10000:20000
cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 action set dscp 34
cumulus@switch:~$ nv set interface swp1-48 acl EXAMPLE1 inbound
cumulus@switch:~$ nv config apply

Filter Specific TCP Flags

The example rule below drops ingress IPv4 TCP packets when you set the SYN bit and reset the RST, ACK, and FIN bits. The rule applies inbound on interface swp1. After configuring this rule, you cannot establish new TCP sessions that originate from ingress port swp1. You can establish TCP sessions that originate from any other port.

-t mangle -A PREROUTING -i swp1 -p tcp --tcp-flags  ACK,SYN,FIN,RST SYN -j DROP

cumulus@switch:~$ sudo cl-acltool -i

cumulus@switch:~$ nv set acl EXAMPLE1 type ipv4
cumulus@switch:~$ nv set acl EXAMPLE1 rule 20 match ip protocol tcp
cumulus@switch:~$ nv set acl EXAMPLE1 rule 20 match ip tcp flags syn
cumulus@switch:~$ nv set acl EXAMPLE1 rule 20 match ip tcp mask rst
cumulus@switch:~$ nv set acl EXAMPLE1 rule 20 match ip tcp mask syn
cumulus@switch:~$ nv set acl EXAMPLE1 rule 20 match ip tcp mask fin
cumulus@switch:~$ nv set acl EXAMPLE1 rule 20 match ip tcp mask ack
cumulus@switch:~$ nv set acl EXAMPLE1 rule 20 action deny 
cumulus@switch:~$ nv set interface swp1 acl EXAMPLE1 inbound
cumulus@switch:~$ nv config apply

Control Who Can SSH into the Switch

Run the following commands to control who can SSH into the switch. In the following example, 10.10.10.1/32 is the interface IP address (or loopback IP address) of the switch and 10.255.4.0/24 can SSH into the switch.

-A INPUT -i swp+ -s 10.255.4.0/24 -d 10.10.10.1/32 -j ACCEPT
-A INPUT -i swp+ -d 10.10.10.1/32 -j DROP

cumulus@switch:~$ sudo cl-acltool -i

cumulus@switch:~$ nv set acl example2 type ipv4
cumulus@switch:~$ nv set acl example2 rule 10 match ip source-ip 10.255.4.0/24 
cumulus@switch:~$ nv set acl example2 rule 10 match ip dest-ip 10.10.10.1/32
cumulus@switch:~$ nv set acl example2 rule 10 action permit
cumulus@switch:~$ nv set acl example2 rule 20 match ip source-ip ANY 
cumulus@switch:~$ nv set acl example2 rule 20 match ip dest-ip 10.10.10.1/32
cumulus@switch:~$ nv set acl example2 rule 20 action deny
cumulus@switch:~$ nv set interface swp1-48 acl example2 inbound control-plane
cumulus@switch:~$ nv config apply

Match on ECN Bits in the TCP IP Header

ECN allows end-to-end notification of network congestion without dropping packets. You can add ECN rules to match on the ECE, CWR, and ECT flags in the TCP IPv4 header.

By default, ECN rules match a packet with the bit set. You can reverse the match by using an explanation point (!).

Match on the ECE Bit

After an endpoint receives a packet with the CE bit set by a router, it sets the ECE bit in the returning ACK packet to notify the other endpoint that it needs to slow down.

To match on the ECE bit:

cumulus@switch:~$ sudo nano /etc/cumulus/acl/policy.d/30-tcp-flags.rules
[iptables]
-t mangle -A PREROUTING -i swp1 -p tcp -m ecn  --ecn-tcp-ece  -j ACCEPT

cumulus@switch:~$ sudo cl-acltool -i

cumulus@switch:~$ nv set acl example2 type ipv4
cumulus@switch:~$ nv set acl example2 rule 10 match ip protocol tcp
cumulus@switch:~$ nv set acl example2 rule 10 match ip ecn flags tcp-ece
cumulus@switch:~$ nv set acl example2 rule 10 action permit
cumulus@switch:~$ nv set interface swp1 acl example2 inbound
cumulus@switch:~$ nv config apply

Match on the CWR Bit

The CWR bit notifies the other endpoint of the connection that it received and reacted to an ECE.

To match on the CWR bit:

cumulus@switch:~$ sudo nano /etc/cumulus/acl/policy.d/30-tcp-flags.rules
[iptables]
-t mangle -A PREROUTING -i swp1 -p tcp -m ecn  --ecn-tcp-cwr  -j ACCEPT

cumulus@switch:~$ sudo cl-acltool -i

cumulus@switch:~$ nv set acl example2 type ipv4
cumulus@switch:~$ nv set acl example2 rule 10 match ip protocol tcp
cumulus@switch:~$ nv set acl example2 rule 10 match ip ecn flags tcp-cwr
cumulus@switch:~$ nv set acl example2 rule 10 action permit
cumulus@switch:~$ nv set interface swp1 acl example2 inbound
cumulus@switch:~$ nv config apply

Match on the ECT Bit

The ECT codepoints negotiate if the connection is ECN capable by setting one of the two bits to 1. Routers also use the ECT bit to indicate that they are experiencing congestion by setting both the ECT codepoints to 1.

To match on the ECT bit:

cumulus@switch:~$ sudo nano /etc/cumulus/acl/policy.d/30-tcp-flags.rules
[iptables]
-t mangle -A PREROUTING -i swp1 -p tcp -m ecn  --ecn-ip-ect 1 -j ACCEPT

cumulus@switch:~$ sudo cl-acltool -i

cumulus@switch:~$ nv set acl example2 type ipv4
cumulus@switch:~$ nv set acl example2 rule 10 match ip protocol tcp
cumulus@switch:~$ nv set acl example2 rule 10 match ip ecn ip-ect 1
cumulus@switch:~$ nv set acl example2 rule 10 action permit
cumulus@switch:~$ nv set interface swp1 acl example2 inbound
cumulus@switch:~$ nv config apply

Example Configuration

The following example demonstrates how Cumulus Linux applies several different rules.

Egress Rule

The following rule blocks any TCP traffic with destination port 200 going through leaf01 to server01 (rule 1 in the diagram above).

[iptables]
-t mangle -A POSTROUTING -o swp1 -p tcp -m multiport --dports 200 -j DROP

cumulus@switch:~$ nv set acl EXAMPLE1 type ipv4
cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 match ip protocol tcp
cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 match ip dest-port 200
cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 action deny
cumulus@switch:~$ nv set interface swp1 acl EXAMPLE1 outbound
cumulus@switch:~$ nv config apply

Ingress Rule

The following rule blocks any UDP traffic with source port 200 going from server01 through leaf01 (rule 2 in the diagram above).

[iptables] 
-t mangle -A PREROUTING -i swp1 -p udp -m multiport --sports 200 -j DROP

cumulus@switch:~$ nv set acl EXAMPLE1 type ipv4
cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 match ip protocol udp
cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 match ip source-port 200
cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 action deny
cumulus@switch:~$ nv set interface swp1 acl EXAMPLE1 inbound
cumulus@switch:~$ nv config apply

Input Rule

The following rule blocks any UDP traffic with source port 200 and destination port 50 going from server02 to the leaf02 control plane (rule 3 in the diagram above).

[iptables] 
-A INPUT -i swp2 -p udp -m multiport --dports 50 -j DROP

cumulus@switch:~$ nv set acl EXAMPLE1 type ipv4
cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 match ip protocol udp
cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 match ip dest-port 50
cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 action deny
cumulus@switch:~$ nv set interface swp2 acl EXAMPLE1 inbound control-plane
cumulus@switch:~$ nv config apply

Output Rule

The following rule blocks any TCP traffic with source port 123 and destination port 123 going from leaf02 to server02 (rule 4 in the diagram above).

[iptables] 
-A OUTPUT -o swp2 -p tcp -m multiport --sports 123 -m multiport --dports 123 -j DROP

cumulus@switch:~$ nv set acl EXAMPLE1 type ipv4
cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 match ip protocol tcp
cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 match ip source-port 123
cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 match ip dest-port 123
cumulus@switch:~$ nv set acl EXAMPLE1 rule 10 action deny
cumulus@switch:~$ nv set interface swp2 acl EXAMPLE1 outbound control-plane
cumulus@switch:~$ nv config apply

Layer 2 Rules (ebtables)

The following rule blocks any traffic with source MAC address 00:00:00:00:00:12 and destination MAC address 08:9e:01:ce:e2:04 going from any switch port egress or ingress.

[ebtables]
-A FORWARD -s 00:00:00:00:00:12 -d 08:9e:01:ce:e2:04 -j DROP

cumulus@switch:~$ nv set acl EXAMPLE type mac
cumulus@switch:~$ nv set acl EXAMPLE rule 10 match mac source-mac 00:00:00:00:00:12
cumulus@switch:~$ nv set acl EXAMPLE rule 10 match mac dest-mac 08:9e:01:ce:e2:04
cumulus@switch:~$ nv set acl EXAMPLE rule 10 action deny
cumulus@switch:~$ nv set interface swp1-48 acl EXAMPLE inbound
cumulus@switch:~$ nv config apply

Considerations

Not All Rules Supported

Cumulus Linux does not support all iptables, ip6tables, or ebtables rules. Refer to Supported Rules for specific rule support.

ACL Log Policer Limits Traffic

To protect the CPU from overloading, Cumulus Linux limits traffic copied to the CPU to 1 packet per second by an ACL Log Policer.

Bridge Traffic Limitations

Bridge traffic that matches LOG ACTION rules do not log to syslog; the kernel and hardware identify packets using different information.

You Cannot Forward Log Actions

You cannot forward logged packets. The hardware cannot both forward a packet and send the packet to the control plane (or kernel) for logging. A log action must also have a drop action.

SPAN Sessions that Reference an Outgoing Interface

SPAN sessions that reference an outgoing interface create mirrored packets based on the ingress interface before the routing/switching decision. See SPAN Sessions that Reference an Outgoing Interface and Use the CPU Port as the SPAN Destination in the Network Troubleshooting section.

iptables Interactions with cl-acltool

Because Cumulus Linux is a Linux operating system, you can use the iptables commands. However, consider using cl-acltool instead for the following reasons:

• Without using cl-acltool, rules do not install into hardware.
• Running cl-acltool -i (the installation command) resets all rules and deletes anything that is not in the /etc/cumulus/acl/policy.conf file.

For example, running the following command works:

cumulus@switch:~$ sudo iptables -A INPUT -p icmp --icmp-type echo-request -j DROP

The rules appear when you run cl-acltool -L:

cumulus@switch:~$ sudo cl-acltool -L ip
-------------------------------
Listing rules of type iptables:
-------------------------------
TABLE filter :
Chain INPUT (policy ACCEPT 72 packets, 5236 bytes)
pkts bytes target  prot opt in   out   source    destination

0     0 DROP    icmp --  any  any   anywhere  anywhere      icmp echo-request

However, running cl-acltool -i or reboot removes them. To ensure that Cumulus Linux can hardware accelerate all rules that can be in hardware, place them in the /etc/cumulus/acl/policy.conf file, then run cl-acltool -i.

Where to Assign Rules

• If you assign a switch port to a bond, you must assign any egress rules to the bond.
• When using the OUTPUT chain, you must assign rules to the source. For example, if you assign a rule to the switch port in the direction of traffic but the source is a bridge (VLAN), the rule does not affect the traffic and you must apply the rule to the bridge.
• If you need to apply a rule to all transit traffic, use the FORWARD chain, not the OUTPUT chain.

ACL Rule Installation Failure

After an ACL rule installation failure, you see a generic error message like the following:

cumulus@switch:$ sudo cl-acltool -i -p 00control_plane.rules
Using user provided rule file 00control_plane.rules
Reading rule file 00control_plane.rules ...
Processing rules in file 00control_plane.rules ...
error: hw sync failed (sync_acl hardware installation failed)
Installing acl policy... Rolling back ..
failed.

ACLs Do not Match when the Output Port on the ACL is a Subinterface

The ACL does not match on packets when you configure a subinterface as the output port. The ACL matches on packets only if the primary port is as an output port. If a subinterface is an output or egress port, the packets match correctly.

For example:

-A FORWARD -o swp49s1.100 -j ACCEPT

Egress ACL Matching on Bonds

Cumulus Linux does not support ACL rules that match on an outbound bond interface. For example, you cannot create the following rule:

[iptables]
-A FORWARD -o <bond_intf> -j DROP

To work around this issue, duplicate the ACL rule on each physical port of the bond. For example:

[iptables]
-A FORWARD -o <bond-member-port-1> -j DROP
-A FORWARD -o <bond-member-port-2> -j DROP

SSH Traffic to the Management VRF

To allow SSH traffic to the management VRF, use -i mgmt, not -i eth0. For example:

-A INPUT -i mgmt -s 10.0.14.2/32 -p tcp --dport ssh -j ACCEPT

INPUT Chain Rules and swp+

In INPUT chain rules, the -i swp+ match works only if the destination of the packet is towards a layer 3 swp interface; the match does not work if the packet terminates at an SVI interface (for example, vlan10). To allow traffic towards specific SVIs, use rules without any interface match or rules with individual -i <SVI> matches.

Related Information

• Netfilter website
• Netfilter.org packet filtering how-to

Services (also known as daemons) and processes are at the heart of how a Linux system functions. Most of the time, a service takes care of itself; you just enable and start it, then let it run. However, because a Cumulus Linux switch is a Linux system, you can dig deeper if you like. Services can start multiple processes as they run. Services are important to monitor on a Cumulus Linux switch.

You manage services in Cumulus Linux in the following ways:

• Identify all active or stopped services
• Identify boot time state of a specific service
• Disable or enable a specific service
• Identify active listener ports

systemd and the systemctl Command

You manage services using systemd with the systemctl command. You run the systemctl command with any service on the switch to start, stop, restart, reload, enable, disable, reenable, or get the status of the service.

cumulus@switch:~$ sudo systemctl start | stop | restart | status | reload | enable | disable | reenable SERVICENAME.service

For example to restart networking, run the command:

cumulus@switch:~$ sudo systemctl restart networking.service

To show all running services, use the systemctl status command. For example:

cumulus@switch:~$ sudo systemctl status
● switch
    State: running
      Jobs: 0 queued
    Failed: 0 units
    Since: Thu 2019-01-10 00:19:34 UTC; 23h ago
    CGroup: /
            ├─init.scope
            │ └─1 /sbin/init
            └─system.slice
              ├─haveged.service
              │ └─234 /usr/sbin/haveged --Foreground --verbose=1 -w 1024
              ├─sysmonitor.service
              │ ├─  658 /bin/bash /usr/lib/cumulus/sysmonitor
              │ └─26543 sleep 60
              ├─systemd-udevd.service
              │ └─218 /lib/systemd/systemd-udevd
              ├─system-ntp.slice
              │ └─ntp@mgmt.service
              │   └─vrf
              │     └─mgmt
              │       └─12108 /usr/sbin/ntpd -n -u ntp:ntp -g
              ├─cron.service
              │ └─274 /usr/sbin/cron -f -L 38
              ├─system-serial\x2dgetty.slice
              │ └─serial-getty@ttyS0.service
              │   └─745 /sbin/agetty -o -p -- \u --keep-baud 115200,38400,9600 ttyS0 vt220
              ├─nginx.service
              │ ├─332 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
              │ └─333 nginx: worker process
              ├─auditd.service
              │ └─235 /sbin/auditd
              ├─rasdaemon.service
              │ └─275 /usr/sbin/rasdaemon -f -r
              ├─clagd.service
              │ └─11443 /usr/bin/python /usr/sbin/clagd --daemon 169.254.1.2 peerlink.4094 44:39:39:ff:40:9
              --priority 100 --vxlanAnycas
              ├─switchd.service
              │ └─430 /usr/sbin/switchd -vx
              ...

systemctl Commands

systemctl has commands that perform a specific operation on a given service:

• status returns the status of the specified service.
• start starts the service.
• stop stops the service.
• restart stops, then starts the service, all the while maintaining state. If there are dependent services or services that mark the restarted service as Required, the other services also restart. For example, running systemctl restart frr.service restarts any of the routing protocol services that you enable and that are running, such as bgpd or ospfd.
• reload reloads the configuration for the service.
• enable enables the service to start when the system boots, but does not start it unless you use the systemctl start SERVICENAME.service command or reboot the switch.
• disable disables the service, but does not stop it unless you use the systemctl stop SERVICENAME.service command or reboot the switch. You can start or stop a disabled service.
• reenable disables, then enables a service. Run this command so that any new Wants or WantedBy lines create the symlinks necessary for ordering. This has no side effects on other services.

You do not need to interact with the services directly using these commands. If a critical service crashes or encounters an error, systemd restarts it automatically. systemd is the caretaker of services in modern Linux systems and responsible for starting all the necessary services at boot time.

Ensure a Service Starts after Multiple Restarts

By default, systemd tries to restart a particular service only a certain number of times within a given interval before the service fails to start. The settings StartLimitInterval (which defaults to 10 seconds) and StartBurstLimit (which defaults to 5 attempts) are in the service script; however, certain services override these defaults, sometimes with much longer times. For example, switchd.service sets StartLimitInterval=10m and StartBurstLimit=3; therefore, if you restart switchd more than three times in ten minutes, it does not start.

When the restart fails for this reason, you see a message similar to the following:

Job for switchd.service failed. See 'systemctl status switchd.service' and 'journalctl -xn' for details.

systemctl status switchd.service shows output similar to:

Active: failed (Result: start-limit) since Thu 2016-04-07 21:55:14 UTC; 15s ago

To clear this error, run systemctl reset-failed switchd.service. If you know you are going to restart frequently (multiple times within the StartLimitInterval), you can run the same command before you issue the restart request. This also applies to stop followed by start.

Keep systemd Services from Hanging after Starting

If you start, restart, or reload a systemd service that you can start from another systemd service, you must use the --no-block option with systemctl.

Identify Active Listener Ports for IPv4 and IPv6

You can identify the active listener ports under both IPv4 and IPv6 using the netstat command:

cumulus@switch:~$ netstat -nlp --inet --inet6
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:53              0.0.0.0:*               LISTEN      444/dnsmasq
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      874/sshd
tcp6       0      0 :::53                   :::*                    LISTEN      444/dnsmasq
tcp6       0      0 :::22                   :::*                    LISTEN      874/sshd
udp        0      0 0.0.0.0:28450           0.0.0.0:*                           839/dhclient
udp        0      0 0.0.0.0:53              0.0.0.0:*                           444/dnsmasq
udp        0      0 0.0.0.0:68              0.0.0.0:*                           839/dhclient
udp        0      0 192.168.0.42:123        0.0.0.0:*                           907/ntpd
udp        0      0 127.0.0.1:123           0.0.0.0:*                           907/ntpd
udp        0      0 0.0.0.0:123             0.0.0.0:*                           907/ntpd
udp        0      0 0.0.0.0:4784            0.0.0.0:*                           909/ptmd
udp        0      0 0.0.0.0:3784            0.0.0.0:*                           909/ptmd
udp        0      0 0.0.0.0:3785            0.0.0.0:*                           909/ptmd
udp6       0      0 :::58352                :::*                                839/dhclient
udp6       0      0 :::53                   :::*                                444/dnsmasq
udp6       0      0 fe80::a200:ff:fe00::123 :::*                                907/ntpd
udp6       0      0 ::1:123                 :::*                                907/ntpd
udp6       0      0 :::123                  :::*                                907/ntpd
udp6       0      0 :::4784                 :::*                                909/ptmd
udp6       0      0 :::3784                 :::*                                909/ptmd

Identify Active or Stopped Services

To see active or stopped services, run the cl-service-summary command:

cumulus@switch:~$ cl-service-summary
Service cron               enabled    active
Service ssh                enabled    active
Service syslog             enabled    active
Service asic-monitor       enabled    inactive
Service clagd              enabled    inactive
Service cumulus-poe                   inactive
Service lldpd              enabled    active
Service mstpd              enabled    active
Service neighmgrd          enabled    active
Service nvued              enabled    active
Service netq-agent         enabled    active
Service ntp                enabled    active
Service ptmd               enabled    active
Service pwmd               enabled    active
Service smond              enabled    active
Service switchd            enabled    active
Service sysmonitor         enabled    active
Service rdnbrd             disabled   inactive
Service frr                enabled    inactive
...

You can also run the systemctl list-unit-files --type service command to list all services on the switch and to see their status:

cumulus@switch:~$ systemctl list-unit-files --type service
UNIT FILE                              STATE
aclinit.service                        enabled
acltool.service                        enabled
acpid.service                          disabled
asic-monitor.service                   enabled
auditd.service                         enabled
autovt@.service                        disabled
bmcd.service                           disabled
bootlog.service                        enabled
bootlogd.service                       masked  
bootlogs.service                       masked  
bootmisc.service                       masked  
checkfs.service                        masked  
checkroot-bootclean.service            masked  
checkroot.service                      masked
clagd.service                          enabled
console-getty.service                  disabled
console-shell.service                  disabled
container-getty@.service               static  
cron.service                           enabled
cryptdisks-early.service               masked  
cryptdisks.service                     masked  
cumulus-aclcheck.service               static  
cumulus-core.service                   static  
cumulus-fastfailover.service           enabled
cumulus-firstboot.service              disabled
cumulus-platform.service               enabled  
...

Identify Essential Services

To identify which services must run when the switch boots:

cumulus@switch:~$ systemctl list-dependencies --before basic.target

To identify which services you need for networking:

cumulus@switch:~$ systemctl list-dependencies --after network.target
 ●  ├─switchd.service
 ●  ├─wd_keepalive.service
 ●  └─network-pre.target

To identify the services needed for a multi-user environment, run:

cumulus@switch:~$ systemctl list-dependencies --before multi-user.target

 ●  ├─bootlog.service
 ●  ├─systemd-readahead-done.service
 ●  ├─systemd-readahead-done.timer
 ●  ├─systemd-update-utmp-runlevel.service
 ●  └─graphical.target
 ●  └─systemd-update-utmp-runlevel.service

Important Services

The following table lists the most important services in Cumulus Linux.

The switchd service enables the switch to communicate with Cumulus Linux and all the applications running on Cumulus Linux.

Configure switchd Parameters

You can control certain options associated with the switchd process. For example, you can set polling intervals, optimize ACL hardware resources for better utilization, configure log message levels, set the internal VLAN range, and configure VXLAN encapsulation and decapsulation.

To configure the switchd parameters, edit the /etc/cumulus/switchd.conf file. Change the setting and uncomment the line if needed. The switchd.conf file contains comments with a description for each setting.

The following example shows the first few lines of the /etc/cumulus/switchd.conf file.

cumulus@switch:~$ sudo nano /etc/cumulus/switchd.conf
#
# /etc/cumulus/switchd.conf - switchd configuration file
#

# Statistic poll interval (in msec)
#stats.poll_interval = 2000

# Buffer utilization poll interval (in msec), 0 means disable
#buf_util.poll_interval = 0

# Buffer utilization measurement interval (in mins)
#buf_util.measure_interval = 0

# Optimize ACL HW resources for better utilization
#acl.optimize_hw = FALSE

# Enable Flow based mirroring.
#acl.flow_based_mirroring = TRUE

# Enable non atomic acl update
acl.non_atomic_update_mode = FALSE

# Send ARPs for next hops
#arp.next_hops = TRUE

# Kernel routing table ID, range 1 - 2^31, default 254
#route.table = 254
...

The following table describes the /etc/cumulus/switchd.conf file parameters and indicates if you need to restart switchd with the sudo systemctl restart switchd.service command or reload switchd with the sudo systemctl reload switchd.service command for changes to take effect when you update the setting.

You configure global HTTP and HTTPS proxies in the /etc/profile.d/ directory of Cumulus Linux. Set the http_proxy and https_proxy variables to configure the switch with the address of the proxy server you want to use to get URLs on the command line. This is useful for programs such as apt, apt-get, curl and wget, which can all use this proxy.

1. In a terminal, create a new file in the /etc/profile.d/ directory.

   cumulus@switch:~$ sudo nano /etc/profile.d/proxy.sh
   

2. Add a line to the file to configure either an HTTP or an HTTPS proxy, or both:

   • HTTP proxy:

     http_proxy=http://myproxy.domain.com:8080
     export http_proxy
     

   • HTTPS proxy:

     https_proxy=https://myproxy.domain.com:8080
     export https_proxy
     

3. Create a file in the /etc/apt/apt.conf.d directory and add the following lines to the file to get the HTTP and HTTPS proxies. The example below uses http_proxy as the file name:

   cumulus@switch:~$ sudo nano /etc/apt/apt.conf.d/http_proxy
   Acquire::http::Proxy "http://myproxy.domain.com:8080";
   Acquire::https::Proxy "https://myproxy.domain.com:8080";
   

4. Add the proxy addresses to the /etc/wgetrc file, then uncomment the http_proxy and https_proxy lines, if necessary:

   cumulus@switch:~$ sudo nano /etc/wgetrc
   ...
   https_proxy = https://myproxy.domain.com:8080
   http_proxy = http://myproxy.domain.com:8080
   ...
   

5. To execute the /etc/profile.d/proxy.sh file in the current environment, run the source command:

   cumulus@switch:~$ source /etc/profile.d/proxy.sh
   

Use the echo command to confirm the configuration:

• HTTP proxy:

  cumulus@switch:~$ echo $http_proxy
  http://myproxy.domain.com:8080
  

• HTTPS proxy:

  cumulus@switch:~$ echo $https_proxy
  https://myproxy.domain.com:8080
  

Related Information

Set up an apt package cache

Use ISSU to upgrade and troubleshoot an active switch with minimal disruption to the network.

ISSU includes the following modes:

• Restart
• Upgrade
• Maintenance mode
• Maintenance ports

Restart Mode

You can restart the switch in one of the following modes.

• cold restarts the system and resets all the hardware devices on the switch (including the switching ASIC).

• fast restarts the system more efficiently with minimal impact to traffic by reloading the kernel and software stack without a hard reset of the hardware. During a fast restart, the system decouples from the network to the extent possible using existing protocol extensions before recovering to the operational mode of the system. The restart process maintains the forwarding entries of the switching ASIC and the data plane is not affected. Traffic outage is much lower in this mode as there is a momentary interruption after reboot, while the system reinitializes.

• warm restarts the system with no interruption to traffic for existing route entries. Warm mode diverts traffic from itself and restarts the system without a hardware reset of the switch ASIC. While this process does not affect the data plane, the control plane is absent during restart and is unable to process routing updates. However, if no alternate paths exist, the switch continues forwarding with the existing entries with no interruptions.

  When you restart the switch in warm mode, BGP only performs a graceful restart if the BGP graceful restart option is set to full. To set BGP graceful restart to full, run the nv set router bgp graceful-restart mode full command, then apply the configuration with nv config apply. For more information about BGP graceful restart, refer to Optional BGP Configuration.

The following command restarts the system in cold mode:

cumulus@switch:~$ sudo csmgrctl -c

The following command restarts the system in fast mode:

cumulus@switch:~$ sudo csmgrctl -f

The following command restarts the system in warm mode.

cumulus@switch:~$ sudo csmgrctl -w

Upgrade Mode

Upgrade mode updates all the components and services on the switch to the latest Cumulus Linux minor release without impacting traffic. After upgrade is complete, you must restart the switch with either a cold or fast restart.

Upgrade mode includes the following options:

• all runs apt-get upgrade to upgrade all the system components to the latest release without affecting traffic flow. You must restart the system after the upgrade completes with one of the restart modes.
• dry-run provides information on the components you want to upgrade.

The following command upgrades all the system components:

cumulus@switch:~$ sudo csmgrctl -u

The following command provides information on the components you want to upgrade:

cumulus@switch:~$ sudo csmgrctl -d

Maintenance Mode

Maintenance mode globally manages the BGP and MLAG control plane.

• When you enable maintenance mode, BGP and MLAG shut down gracefully.
• When you disable maintenance mode, BGP and MLAG are enabled based on the individual parameter settings.

To enable maintenance mode:

cumulus@switch:~$ nv action enable system maintenance mode
Action executing ...
System maintenance mode has been enabled successfully
 Current System Mode: Maintenance, cold  
 Maintenance mode since Thu Jun 13 23:59:47 2024 (Duration: 00:00:00)
 Ports shutdown for Maintenance
 frr             : Maintenance, cold, down, up time: 29:06:27
 switchd         : Maintenance, cold, down, up time: 29:06:31
 System Services : Maintenance, cold, down, up time: 29:07:00

Action succeeded

cumulus@switch:~$ sudo csmgrctl -m1

To disable maintenance mode:

cumulus@switch:~$ nv action disable system maintenance mode
Action executing ...
System maintenance mode has been disabled successfully
 Current System Mode: cold  
 frr             : cold, up, up time: 12:57:48 (1 restart)
 switchd         : cold, up, up time: 13:12:13
 System Services : cold, up, up time: 13:12:32
Action succeeded

cumulus@switch:~$ sudo csmgrctl -m0

To show maintenance mode status either run the NVUE nv show system maintenance command or the Linux sudo csmgrctl -s command:

cumulus@switch:~$ nv show system maintenance 
       operational
-----  -----------
mode   enabled   
ports  disabled 

cumulus@switch:~$ sudo csmgrctl -s
Current System Mode: cold  
 frr             : cold, up, up time: 00:14:51 (2 restarts)
 clagd           : cold, up, up time: 00:14:47
 switchd         : cold, up, up time: 01:09:48
 System Services : cold, up, up time: 01:10:07

Maintenance Ports

Maintenance ports globally disables or enables all configured ports.

• When you enable maintenance ports, swp interfaces follow individual admin states.
• When you disable maintenance ports, swp interfaces are globally admin down, overriding the admin state in the configuration.

To enable maintenance ports:

cumulus@switch:~$ nv action enable system maintenance ports
Action executing ...
System maintenance ports has been enabled successfully
 Current System Mode: cold  
 frr             : cold, up, up time: 28:54:36
 switchd         : cold, up, up time: 28:54:40
 System Services : cold, up, up time: 28:55:09

Action succeeded

cumulus@switch:~$ sudo csmgrctl -p0

To disable maintenance ports:

cumulus@switch:~$ nv action disable system maintenance ports
Action executing ...
System maintenance ports has been disabled successfully
 Current System Mode: cold  
 Ports shutdown for Maintenance
 frr             : cold, up, up time: 28:55:49
 switchd         : cold, up, up time: 28:55:53
 System Services : cold, up, up time: 28:56:22

Action succeeded

cumulus@switch:~$ sudo csmgrctl -p1

To see the status of maintenance ports, run the NVUE nv show system maintenance command:

cumulus@switch:~$ nv show system maintenance 
       operational
-----  -----------
mode   enabled   
ports  disabled 

This section discusses the following layer 1 and switch port configuration:

• DHCP
• Interface configuration management
• QoS
• Port Security
• Prescriptive Topology Manager (PTM)

Cumulus Linux uses ifupdown2 to manage network interfaces, which is a new implementation of the Debian network interface manager ifupdown.

Bring an Interface Up or Down

An interface status can be in an:

• Administrative state, where you configure the interface to be up or down.
• Operational state, which reflects the current operational status of an interface.

cumulus@switch:~$ nv set interface swp1
cumulus@switch:~$ nv config apply

cumulus@switch:~$ nv set interface swp1 link state down
cumulus@switch:~$ nv config apply

cumulus@switch:~$ nv set interface swp1 link state up
cumulus@switch:~$ nv config apply

cumulus@switch:~$ nv unset interface swp1
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo nano /etc/network/interfaces
auto lo
iface lo inet loopback
    address 10.10.10.1/32
auto mgmt
iface mgmt
    address 127.0.0.1/8
    address ::1/128
    vrf-table auto
auto eth0
iface eth0 inet dhcp
    ip-forward off
    ip6-forward off
    vrf mgmt
auto swp1
iface swp1
...

auto swp1
iface swp1
 link-down yes

For additional information on interface administrative state and physical state, refer to this knowledge base article.

Interface Classes

ifupdown2 enables you to group interfaces into separate classes. A class is a user-defined label that groups interfaces that share a common function (such as uplink, downlink or compute). You specify classes in the /etc/network/interfaces file.

The most common class is auto, which you configure like this:

auto swp1
iface swp1

You can add other classes using the allow prefix. For example, if you have multiple interfaces used for uplinks, you can define a class called uplinks:

auto swp1
allow-uplink swp1
iface swp1 inet static
    address 10.1.1.1/31

auto swp2
allow-uplink swp2
iface swp2 inet static
    address 10.1.1.3/31

This allows you to perform operations on only these interfaces using the --allow=uplinks option. You can still use the -a options because these interfaces are also in the auto class:

cumulus@switch:~$ sudo ifup --allow=uplinks
cumulus@switch:~$ sudo ifreload -a

If you are using Management VRF, you can use the special interface class called mgmt and put the management interface into that class. The management VRF must have an IPv6 address in addition to an IPv4 address to work correctly.

allow-mgmt eth0
iface eth0 inet dhcp
    vrf mgmt

allow-mgmt mgmt
iface mgmt
    address 127.0.0.1/8
    address ::1/128
    vrf-table auto

All ifupdown2 commands (ifup, ifdown, ifquery, ifreload) can take a class. Include the --allow=<class> option when you run the command. For example, to reload the configuration for the management interface described above, run:

cumulus@switch:~$ sudo ifreload --allow=mgmt

Use the -a option to bring up or down all interfaces with the common auto class in the /etc/network/interfaces file.

To administratively bring up all interfaces marked auto, run:

cumulus@switch:~$ sudo ifup -a

To administratively bring down all interfaces marked auto, run:

cumulus@switch:~$ sudo ifdown -a

To reload all network interfaces marked auto, use the ifreload command. This command is equivalent to running ifdown then ifup; however, ifreload skips unchanged configurations:

cumulus@switch:~$ sudo ifreload -a

cumulus@switch:~$ sudo bash -c "ifreload -s -a && ifreload -a"

For more information, see the individual man pages for ifup(8), ifdown(8), ifreload(8).

Loopback Interface

Cumulus Linux has a preconfigured loopback interface. When the switch boots up, the loopback interface called lo is up and assigned an IP address of 127.0.0.1.

To configure an IP address for the loopback interface:

cumulus@switch:~$ nv set interface lo ip address 10.10.10.1
cumulus@switch:~$ nv config apply

auto lo
iface lo inet loopback
    address 10.10.10.1

Subinterfaces

On Linux, an interface is a network device that can be either physical, (for example, swp1) or virtual (for example, vlan100). A VLAN subinterface is a VLAN device on an interface, and the VLAN ID appends to the parent interface using dot (.) VLAN notation. For example, a VLAN with ID 100 that is a subinterface of swp1 is swp1.100. The dot VLAN notation for a VLAN device name is a standard way to specify a VLAN device on Linux.

A VLAN subinterface only receives traffic tagged for that VLAN; therefore, swp1.100 only receives packets that have a VLAN 100 tag on switch port swp1. Any packets that transmit from swp1.100 have a VLAN 100 tag.

The following example configures a routed subinterface on swp1 in VLAN 100:

cumulus@switch:~$ nv set interface swp1.100 ip address 192.168.100.1/24
cumulus@switch:~$ nv config apply

cumulus@switch:~$ sudo nano /etc/network/interfaces
...
auto swp1.100
iface swp1.100
 address 192.168.100.1/24

cumulus@switch:~$ sudo ifreload -a

Interface IP Addresses

You can specify both IPv4 and IPv6 addresses for the same interface.

For IPv6 addresses:

• You can create or modify the IP address for an interface using either :: or 0:0:0 notation. For example, both 2620:149:43:c109:0:0:0:5 and 2001:DB8::1/126 are valid.
• Cumulus Linux assigns the IPv6 address with all zeroes in the interface identifier (2001:DB8::/126) for each subnet; connected hosts cannot use this address.

The following example commands configure three IP addresses for swp1; two IPv4 addresses and one IPv6 address.

cumulus@switch:~$ nv set interface swp1 ip address 10.0.0.1/30
cumulus@switch:~$ nv set interface swp1 ip address 10.0.0.2/30
cumulus@switch:~$ nv set interface swp1 ip address 2001:DB8::1/126
cumulus@switch:~$ nv config apply

auto swp1
iface swp1
    address 10.0.0.1/30
    address 10.0.0.2/30
    address 2001:DB8::1/126

auto lo
iface lo inet loopback

cumulus@switch:~$ sudo ip addr add 10.0.0.1/30 dev swp1
cumulus@switch:~$ sudo ip addr add 2001:DB8::1/126 dev swp1

cumulus@switch:~$ sudo ip addr del 10.0.0.1/30 dev swp1
cumulus@switch:~$ sudo ip addr del 2001:DB8::1/126 dev swp1

Interface Descriptions

You can add a description (alias) to an interface.

Interface descriptions also appear in the SNMP OID IF-MIB::ifAlias

The following example commands create the description hypervisor_port_1 for swp1:

cumulus@switch:~$ nv set interface swp1 description hypervisor_port_1
cumulus@switch:~$ nv config apply

cumulus@switch:~# sudo nano /etc/network/interfaces

auto swp1
iface swp1
    alias swp1 hypervisor_port_1

Interface Commands

You can specify user commands for an interface that run at pre-up, up, post-up, pre-down, down, and post-down.

You can add any valid command in the sequence to bring an interface up or down; however, limit the scope to network-related commands associated with the particular interface. For example, it does not make sense to install a Debian package on ifup of swp1, even though it is technically possible. See man interfaces for more details.

The following examples adds a command to an interface to enable proxy ARP:

cumulus@switch:~# sudo nano /etc/network/interfaces
auto swp1
iface swp1
    address 12.0.0.1/30
    post-up echo 1 > /proc/sys/net/ipv4/conf/swp1/proxy_arp

auto bridge.100
iface bridge.100
    post-up systemctl --no-block restart dhcrelay.service

Source Interface File Snippets

Sourcing interface files helps organize and manage the /etc/network/interfaces file. For example:

cumulus@switch:~$ sudo cat /etc/network/interfaces
# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet dhcp

source /etc/network/interfaces.d/bond0

The contents of the sourced file used above are:

cumulus@switch:~$ sudo cat /etc/network/interfaces.d/bond0
auto bond0
iface bond0
    address 14.0.0.9/30
    address 2001:ded:beef:2::1/64
    bond-slaves swp25 swp26

Port Ranges

To specify port ranges in commands:

cumulus@switch:~$ nv set interface swp1-4,6,10-12 bridge domain br_default
cumulus@switch:~$ nv config apply

auto br0
iface br0
    bridge-ports glob swp1-6.100

auto br1
iface br1
    bridge-ports glob swp7-9.100  swp11.100 glob swp15-18.100

Link Flap Protection

Cumulus Linux enables link flap detection by default. Link flap detection triggers when there are five link flaps within ten seconds, at which point the interface goes into a protodown state and shows linkflap as the reason. The switchd service also shows a log message similar to the following:

2023-02-10T17:53:21.264621+00:00 cumulus switchd[10109]: sync_port.c:2263 ERR swp2 link flapped more than 3 times in the last 60 seconds, setting protodown

To show interfaces with the protodown flag, run the Linux ip link command:

cumulus@switch:~$ ip link
37: swp2: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 9178 qdisc pfifo_fast master bond131 state DOWN mode DEFAULT group default qlen 1000
  link/ether 1c:34:da:ba:bb:2a brd ff:ff:ff:ff:ff:ff protodown on protodown_reason <linkflap>

Clear the Interface Protodown State and Reason

The ifdown and ifup commands do not clear the protodown state. You must clear the protodown state and the reason manually using the sudo ip link set <interface> protodown_reason linkflap off and sudo ip link set <interface> protodown off commands.

cumulus@switch:~$ sudo ip link set swp2 protodown_reason linkflap off
cumulus@switch:~$ sudo ip link set swp2 protodown off

After a few seconds the port state returns to UP. Run the ip link show <interface> command to verify that the interface is no longer in a protodown state and that the reason is cleared:

cumulus@switch:~$ ip link show swp2
37: swp2: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 9178 qdisc pfifo_fast master bond131 state UP mode DEFAULT group default qlen 1000
  link/ether 1c:34:da:ba:bb:2a brd ff:ff:ff:ff:ff:ff

Change Link Flap Protection Settings

You can change link flap protection settings in the /etc/cumulus/switchd.conf file:

• To change the duration during which a link must flap the number of times set in the link flap threshold before link flap protection triggers, change the link_flap_window setting.
• To change the number of times the link must flap within the link flap window before link flap protection triggers, change the link_flap_threshold setting.
• To disable link flap protection, set the link_flap_window and link_flap_threshold parameters to 0 (zero).

After you change the link flap settings, you must restart switchd with the sudo systemctl restart switchd.service command.

Mako Templates

ifupdown2 supports Mako-style templates. The Mako template engine processes the interfaces file before parsing.

Use the template to declare cookie-cutter bridges and to declare addresses in the interfaces file:

%for i in [1,12]:
auto swp${i}
iface swp${i}
    address 10.20.${i}.3/24

To comment out content in Mako templates, use double hash marks (##). For example:

## % for i in range(1, 4):
## auto swp${i}
## iface swp${i}
## % endfor
##

For more Mako template examples, refer to this knowledge base article.

ifupdown Scripts

Unlike the traditional ifupdown system, ifupdown2 does not run scripts installed in /etc/network/*/ automatically to configure network interfaces.

To enable or disable ifupdown2 scripting, edit the addon_scripts_support line in the /etc/network/ifupdown2/ifupdown2.conf file. 1 enables scripting and 2 disables scripting. For example:

cumulus@switch:~$ sudo nano /etc/network/ifupdown2/ifupdown2.conf
# Support executing of ifupdown style scripts.
# Note that by default python addon modules override scripts with the same name
addon_scripts_support=1

ifupdown2 sets the following environment variables when executing commands:

• $IFACE represents the physical name of the interface; for example, br0 or vxlan42. The name comes from the /etc/network/interfaces file.
• $LOGICAL represents the logical name (configuration name) of the interface.
• $METHOD represents the address method; for example, loopback, DHCP, DHCP6, manual, static, and so on.
• $ADDRFAM represents the address families associated with the interface in a comma-separated list; for example, "inet,inet6".

Troubleshooting

To see the link and administrative state of an interface:

cumulus@switch:~$ nv show interface swp1 link state

cumulus@switch:~$ ip link show dev swp1
3: swp1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 500
    link/ether 44:38:39:00:03:c1 brd ff:ff:ff:ff:ff:ff

To show the assigned IP address on an interface:

cumulus@switch:~$ nv show interface swp1 ip address

cumulus@switch:~$ ip addr show swp1
3: swp1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 500
    link/ether 44:38:39:00:03:c1 brd ff:ff:ff:ff:ff:ff
    inet 192.0.2.1/30 scope global swp1
    inet 192.0.2.2/30 scope global swp1
    inet6 2001:DB8::1/126 scope global tentative
        valid_lft forever preferred_lft forever

To show the description (alias) for an interface:

cumulus@switch$ nv show interface swp1

cumulus@switch$ ip link show swp1
3: swp1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN mode DEFAULT qlen 500
    link/ether aa:aa:aa:aa:aa:bc brd ff:ff:ff:ff:ff:ff
    alias hypervisor_port_1

Considerations

Even though ifupdown2 supports the inclusion of multiple iface stanzas for the same interface, use a single iface stanza for each interface. If you must specify more than one iface stanza; for example, if the configuration for a single interface comes from many places, like a template or a sourced file, make sure the stanzas do not specify the same interface attributes. Otherwise, you see unexpected behavior.

In the following example, swp1 is in two files: /etc/network/interfaces and /etc/network/interfaces.d/speed_settings. ifupdown2 parses this configuration because the same attributes are not in multiple iface stanzas.

cumulus@switch:~$ sudo cat /etc/network/interfaces

source /etc/network/interfaces.d/speed_settings

auto swp1
iface swp1
  address 10.0.14.2/24

cumulus@switch:~$ cat /etc/network/interfaces.d/speed_settings

auto swp1
iface swp1
  link-speed 1000
  link-duplex full

ifupdown2 and sysctl

For sysctl commands in the pre-up, up, post-up, pre-down, down, and post-down lines that use the $IFACE variable, if the interface name contains a dot (.), ifupdown2 does not change the name to work with sysctl. For example, the interface name bridge.1 does not convert to bridge/1.

ifupdown2 and the gateway Parameter

The default route that the gateway parameter creates in ifupdown2 does not install in FRR, therefore does not redistribute into other routing protocols. Define a static default route instead, which installs in FRR and redistributes, if needed.

The following shows an example of the /etc/network/interfaces file when you use a static route instead of a gateway parameter:

auto swp2
iface swp2
address 172.16.3.3/24
up ip route add default via 172.16.3.2

Interface Name Limitations

Interface names can be a maximum of 15 characters. You cannot use a number for the first character and you cannot include a dash (-) in the name. In addition, you cannot use any name that matches with the regular expression .{0,13}\-v.*.

If you encounter issues, remove the interface name from the /etc/network/interfaces file, then restart the networking.service.

cumulus@switch:~$ sudo nano /etc/network/interfaces
cumulus@switch:~$ sudo systemctl restart networking.service

IP Address Scope

ifupdown2 does not honor the configured IP address scope setting in the /etc/network/interfaces file and treats all addresses as global. It does not report an error. Consider this example configuration:

auto swp2
iface swp2
    address 35.21.30.5/30
    address 3101:21:20::31/80
    scope link

When you run ifreload -a on this configuration, ifupdown2 considers all IP addresses as global.

cumulus@switch:~$ ip addr show swp2
5: swp2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 74:e6:e2:f5:62:82 brd ff:ff:ff:ff:ff:ff
inet 35.21.30.5/30 scope global swp2
valid_lft forever preferred_lft forever
inet6 3101:21:20::31/80 scope global
valid_lft forever preferred_lft forever
inet6 fe80::76e6:e2ff:fef5:6282/64 scope link
valid_lft forever preferred_lft forever

To work around this issue, configure the IP address scope:

auto swp6
iface swp6
    post-up ip address add 71.21.21.20/32 dev swp6 scope site

The following configuration shows the correct scope:

cumulus@switch:~$ ip addr show swp6
9: swp6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 74:e6:e2:f5:62:86 brd ff:ff:ff:ff:ff:ff
inet 71.21.21.20/32 scope site swp6
valid_lft forever preferred_lft forever
inet6 fe80::76e6:e2ff:fef5:6286/64 scope link
valid_lft forever preferred_lft forever

Related Information

• Troubleshoot Layer 1
• Debian - Network Configuration

Cumulus Linux exposes network interfaces for several types of physical and logical devices:

• lo is the network loopback device
• eth is a switch management port (for out of band management only)
• swp is a switch front panel port
• (optional) br is a bridge (IEEE 802.1Q VLAN)
• (optional) bond is a bond (IEEE 802.3ad link aggregation trunk, or port channel)