Install and Configure NetQ#

NetQ NvLink (previously NMX-M) provides a single interface for management and telemetry collection of NVLink switches. NetQ is deployed on Kubernetes, along with the other components that make up Mission Control.

What’s New in NetQ 5.2#

NetQ 5.2.1#

This release provides updated instructions for installing a custom-signed certificate and introduces the following NetQ NvLink features:

  • Liquid-cooling leak detection: A new /v1/redfish endpoint for detecting leak events in liquid-cooling equipment.

  • Custom certificate management: New /v1/certificates endpoints for managing custom certificates.

  • Firmware version validation: A new /v1/validations/fw-versions endpoint for validating firmware versions.

  • Firmware upgrade: The ability to upgrade firmware through the new /v1/upgrade-switch endpoint.

  • KPI filtering: New /v1/kpis endpoints that support filtering KPIs for GPUs, switches, compute nodes, partitions, and domains.

  • Enhanced GPU endpoint: The /v1/gpus endpoint now supports filtering by UUID, chassis serial, slot ID, and host ID.

  • UUID-based partition management: New /v1/partitions endpoints for managing partitions by UUID.

  • NMX-T polling frequency: The ability to adjust the NMX-T polling frequency.

  • NVIDIA Vera Rubin platform support (beta).

  • NVLink expansion (beta): Support for adding nodes to existing NVLink and Ethernet combined-mode deployments.

  • Switch credential management: New recommendations for managing switch credentials.

Release Considerations#

  • Combined mode data preservation: When your NetQ deployment operates in combined Ethernet and NVLink mode, the backup and restore process does not preserve certain NVLink data. Network entity data — switches, GPUs, and partitions — is not saved. Service, switch profile, and domain data is saved.

  • Compatible agents: The NetQ 5.2 server supports 5.2 and 5.1 agents running on Cumulus Linux 5.17, 5.16, 5.15, or 5.11, and on Ubuntu 24.04 or 22.04.

NetQ Kubernetes Setup#

NetQ Permanent License Generation and Application Guide#

When installing NetQ, the system receives an evaluation license valid for 60 days. When the evaluation license expires, REST API access is blocked until a new license is applied.

Generate a License File#

Before you generate the license file, you need:

  • The MAC address of the NetQ control plane.

  • The entitlement certificate that you received by email.

  • Access to the NVIDIA Licensing Portal (NLP).

To generate and download the license file, follow these steps:

  1. Log in to the NVIDIA Licensing Portal and select NVIDIA Licensing Portal on the main screen.

  2. Select License Management > Entitlements.

  3. Open the vertical three-dot menu at the end of the NetQ License row and select View Entitlement Features.

  4. Find the Product Key that matches the PAC ID in the entitlement certificate you received by email.

  5. Open the vertical three-dot menu for that Product Key and select Add to License Allocation.

  6. In Configure Allocation, complete the required fields.

  7. Set Feature Instance and Total Licenses, then select Add.

  8. Select Create Allocation.

  9. Refresh the page and select License Allocations.

  10. Open the vertical three-dot menu for the row marked Undeployed and select Create NetQ License.

    The portal generates and downloads the license file.

To apply the downloaded license file, follow Apply a Permanent NetQ NvLink License.

Important Notes about License Regeneration#

When you regenerate a license, you need to keep the following in mind:

  • If you replace your NIC or server, repeat the process of generating the license to set new MAC addresses.

  • You can only regenerate a license two times.

  • To regenerate the license after that, contact NVIDIA Sales Administration at enterprisesupport@nvidia.com.

Download the NetQ install packages#

NetQ installation requires three separate packages from two different sources. Download all three before you start the installation wizard:

Package

Source

Notes

NetQ Appliance

NVIDIA Licensing Portal (NLP)

Must have a .tgz file extension.

NetQ Apps (Debian)

URM Artifactory

Match the OS and CPU architecture.

NetQ Agents (Debian)

URM Artifactory

Match the OS and CPU architecture.

Download the Appliance package from NLP#

To download the Appliance package, use the following steps:

  1. Go to the NVIDIA Licensing Portal (NLP) and log in using your credentials.

  2. Click on Software Downloads, filter the product family to NetQ, find the relevant version, and download the Appliance platform package.

  3. Click on Download.

  4. Save the file on your local drive.

  5. Click Close.

  6. Copy the .tgz file to the BCM head node:

rsync -azP <path-to-appliance-package> root@bcm11-head-01:/root

Important

The NetQ appliance package must have a .tgz file extension. If the file is named with any other extension, the installation fails.

Download the Apps and Agents packages from URM Artifactory#

Download both the relevant apps and agents packages from the following link: https://edge.urm.nvidia.com/artifactory/sw-nbu-netq-debian/pool/netq-5.2/

Installing NetQ#

This section describes how to install NetQ on a BCM-managed Kubernetes cluster.

Note

Plan for at least 2 hours to complete the installation.

Prerequisites#

  • Kubernetes 1.35 is installed

  • Three nodes with the minimum hardware requirements (4TB disk space, 512GB Free RAM, Core count 48)

  • An IP address that is reserved (to be used as Virtual IP)

Note

The prerequisite checks verify free memory and disk space. If your environment meets the requirements but the check fails (for example, when disk space is available on a non-default mount point such as /local), you can pass the --skip-netq-prerequisites-checks flag to cm-mission-control-setup to bypass these checks. Verify that the system meets the required CPU core count, memory, and disk space before doing so.

Installation#

To install NetQ, use the following steps:

  1. Start the installation wizard using the cm-mission-control-setup command.

  2. Select the NetQ installation:

    NetQ installation selection screen

  3. Select the NetQ deployment mode, and choose the NVL Mode:

    NetQ deployment mode selection screen showing NVL Mode option

  4. Set the Kong (NMX API) username and password:

    Note

    Use a password that contains at least 8 characters.

    Kong username configuration screen Kong password configuration screen

  5. Choose the related Kubernetes cluster:

    Kubernetes cluster selection screen

  6. Choose the node category for the nodes where NetQ will be installed (usually the category for k8s-admin control plane nodes):

    Node category selection screen

  7. If left empty, you can also choose 3 nodes:

    Node selection screen showing 3 nodes option

  8. Provide the NetQ overlay name and priority (in most cases ‘default’ can be used):

    NetQ overlay name and priority configuration screen

  9. Provide a Virtual IP for the Cluster - An unused IP address allocated from the same subnet assigned to the default interface for your master and worker nodes.

    Virtual IP configuration screen for the cluster

  10. Provide the paths to the NetQ appliance tarball (the .tgz file) and the Debian packages (.deb files) for the apps and agents. The tarball must have a .tgz extension; any other extension causes installation to fail. The screenshot shows example paths.

    NetQ tarball and debian package paths configuration screen

  11. Choose the storage path for Longhorn (replicated storage system). This must be a path with at least 4 TB of available space. On the k8s-admin node, run df -H to check storage availability; /local/longhorn may be a suitable option.

    Longhorn storage path configuration screen

    Important

    The Longhorn storage path must have at least 4 TB of available space. If the chosen path does not meet this requirement, the installation fails.

  12. Save and deploy:

    Save and deploy confirmation screen

  13. Allow the installation to run to completion.

Configure Longhorn to not be a default storageclass#

After installation of NetQ, the default behavior is to set Longhorn as a default storageclass. We only want to use Longhorn for the NetQ components and not for storage.

To configure Longhorn to not be a default storageclass, use the following steps:

  1. Get the current storageclasses:

kubectl get storageclass
NAME                      PROVISIONER                                      RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
local-path (default)      cluster.local/local-path-provisioner             Delete          WaitForFirstConsumer   true                   25h
longhorn (default)        driver.longhorn.io                               Delete          Immediate              true                   24h
longhorn-no-replication   driver.longhorn.io                               Delete          Immediate              true                   24h
longhorn-static           driver.longhorn.io                               Delete          Immediate              true                   24h
shoreline-local-path-sc   cluster.local/shoreline-local-path-provisioner   Delete          WaitForFirstConsumer   true                   18h

  1. Run the following code to patch the Longhorn storageclass:

kubectl patch storageclass longhorn -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'


kubectl get storageclass
NAME                      PROVISIONER                                      RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
local-path (default)      cluster.local/local-path-provisioner             Delete          WaitForFirstConsumer   true                   26h
longhorn                  driver.longhorn.io                               Delete          Immediate              true                   25h
longhorn-no-replication   driver.longhorn.io                               Delete          Immediate              true                   25h
longhorn-static           driver.longhorn.io                               Delete          Immediate              true                   25h
shoreline-local-path-sc   cluster.local/shoreline-local-path-provisioner   Delete          WaitForFirstConsumer   true                   19h

Post-Installation Validation#

CLI Validation#

  1. Verify that all pods are up and running.

    NetQ pods run in a namespace that depends on the deployment mode: netq-nvl for NVLink mode, netq-eth for Ethernet mode, and both namespaces for Combined mode. Shared infrastructure pods always run in netq-infra. Run the commands that apply to your deployment mode:

    kubectl get pods -n netq-nvl    # NVLink and Combined modes
kubectl get pods -n netq-eth    # Ethernet and Combined modes
kubectl get pods -n netq-infra  # All modes

    Ensure that every pod is in a Running or Completed state and that the READY column shows all containers ready (for example, 1/1).

  2. Check the overall NetQ status:

    Note

    Run this command on a node where NetQ is installed. Do not run it on the BCM head node because the NetQ CLI is not installed there.

    netq show status

    Verify that the Installation Status is FINISHED and the NetQ Live State is Active.

UI Validation#

Connect to:

https://<Virtual IP>:30443/nmx/swag/index.html

Use the rw-user or ro-user credentials and the password set during the installation.

Uninstall NetQ#

Run cm-mission-control-setup and select the “NVIDIA Mission Control NetQ uninstallation” option (this option appears if NetQ is installed).

The wizard prompts for confirmation.

Note

Manually delete files in the Longhorn directory after the uninstallation on all nodes where NetQ ran.