Install NetQ for Ethernet and NVLink (Beta)

Follow these steps to set up and configure your VMs in a cluster of servers. First configure the VM on the master node, and then configure the VM on each additional node. NVIDIA recommends installing the virtual machines on different servers to increase redundancy in the event of a hardware failure.

System Requirements

This deployment model requires a cluster comprising a minimum of three nodes. Verify that each node in your cluster meets the VM requirements:

Resource	Minimum Requirements
Processor	48 virtual CPUs
Memory	512GB RAM
Local disk storage	3.2TB NVMe; P99 disk I/O latency < 10ms
Network interface speed	10 Gbps NIC
Hypervisor	KVM/QCOW (QEMU Copy on Write) image for servers running Ubuntu 24.04; VMware ESXi™ 6.5 or later (OVA image) for servers running Cumulus Linux or Ubuntu 24.04

Port Requirements

Confirm that the required ports are open for communications.

Port or Protocol Number	Protocol	Component Access
4	IP Protocol	Calico networking (IP-in-IP Protocol)
22	TCP	SSH
80	TCP	nginx
179	TCP	Calico networking (BGP)
443	TCP	NetQ UI
2379	TCP	etcd datastore
4789	UDP	Calico networking (VxLAN)
5000	TCP	Docker registry
6443	TCP	kube-apiserver
30001	TCP	DPU communication
30008	TCP	gRPC OTLP receiver
30009	TCP	HTTPS OTLP receiver
31980	TCP	NetQ Agent communication
31982	TCP	NetQ Agent SSL communication
32710	TCP	API Gateway

▼ Internal communication ports

Port or Protocol Number	Protocol	Component Access
2181	TCP	Zookeeper client
2379	TCP	etcd
2380	TCP	etcd
2888	TCP	Zookeeper cluster communication
3888	TCP	Zookeeper cluster communication
5000	TCP	Docker registry
6443	TCP	Kubernetes API server
7000	TCP	Cassandra cluster communication
7071	TCP	Cassandra JMX monitoring
7072	TCP	Kafka JMX monitoring
7073	TCP	Zookeeper JMX monitoring
8080	TCP	Admin API
9042	TCP	Cassandra client
9092	TCP	Kafka client
10250	TCP	kubelet health probe
36443	TCP	Kubernetes control plane

Installation and Configuration

Download the NetQ image.
a. Log in to your NVIDIA Application Hub account.
b. Select NVIDIA Licensing Portal.
c. Select Software Downloads from the menu.
d. In the search field above the table, enter NetQ.<br1 KVM Scale** image. For deployments using VMware, download the **NetQ SW 5.2.1 VMware Scale** image
f. If prompted, read the license agreement and proceed with the download.

NVIDIA employees can download NetQ directly from the NVIDIA Licensing Portal.

Open your hypervisor and configure your VM. You can use the following examples for reference or use your own hypervisor instructions.

KVM Example Configuration

This example shows the VM setup process for a system with Libvirt and KVM/QEMU installed.

Confirm that the SHA256 checksum matches the one posted on the NVIDIA Application Hub to ensure the image download has not been corrupted.

$ sha256sum ./Downloads/netq-5.2.1-ubuntu-24.04-ts-qemu-scale.qcow2
$ a0d9a4f9ce8925b7dfb90a5a44616cadbf3fc667013abae07cd774555c08ff6f  ./Downloads/netq-5.2.1-ubuntu-24.04-ts-qemu-scale.qcow2

Copy the QCOW2 image to a directory where you want to run it.
Tip: Copy, instead of moving, the original QCOW2 image to avoid re-downloading it should you need to perform this process again.
```
$ sudo mkdir /vms
$ sudo cp ./Downloads/netq-5.2.1-ubuntu-24.04-ts-qemu-scale.qcow2 /vms/ts.qcow2
```
Create the VM.
For a Direct VM, where the VM uses a MACVLAN interface on the physical host interface for its connectivity:
```
$ virt-install --name=netq_ts --vcpus=48 --memory=524288 --os-type=linux --os-variant=generic --disk path=/vms/ts.qcow2,format=qcow2,bus=virtio,cache=none --network=type=direct,source=eth0,model=virtio --import --noautoconsole
```
For a Direct VM, where the VM uses a MACVLAN interface on a bonded interface for its connectivity:
```
$ virt-install --name=netq_ts --vcpus=48 --memory=524288 --os-type=linux --os-variant=generic --disk path=/vms/ts.qcow2,format=qcow2,bus=virtio,cache=none --network=type=direct,source=bond1,source_mode=vepa,model=virtio --import --noautoconsole
```
- Replace the disk path value with the QCOW2 image path location.
- Replace the network source value with the name of the interface where the VM is connected to the external network. Note that the first example uses eth0 and the second example uses bond1.
- For HA cluster deployments, allmulticast must be enabled on all host macvtap interfaces. Without it, macvtap drops multicast traffic required by keepalived, which can cause the virtual IP address to become unreachable. See step 4 for configuration instructions.
For a Bridged VM, where the VM attaches to a pre-configured bridge interface:
```
$ virt-install --name=netq_ts --vcpus=48 --memory=524288 --os-type=linux --os-variant=generic \ --disk path=/vms/ts.qcow2,format=qcow2,bus=virtio,cache=none --network=bridge=br0,model=virtio --import --noautoconsole
```
Replace the bridge value (br0) with the name of the bridge interface that connects the VM to the external network. The Bridged VM method is recommended for HA cluster deployments because it handles multicast traffic natively without additional configuration.
Make note of the name used during install as this is needed in a later step.

Direct VM/macvtap deployments only: Enable allmulticast on the host macvtap interfaces and within each NetQ VM, and configure both settings to persist across reboots.

Enable allmulticast on each macvtap interface on the host. Replace macvtapX with the macvtap interface created for the VM:

$ ip link show | grep macvtap
$ ip link set dev macvtapX allmulticast on

To make this setting persist across host reboots, use a libvirt hook:

$ mkdir -p /etc/libvirt/hooks

$ cat > /etc/libvirt/hooks/qemu << 'EOF'
#!/bin/bash
VMNAME=$1
ACTION=$2

if [ "$ACTION" == "started" ]; then
    sleep 2
    for iface in $(ip link show | grep macvtap | awk -F: '{print $2}' | tr -d ' '); do
        ip link set dev "$iface" allmulticast on
        logger "allmulticast enabled on $iface for VM $VMNAME"
    done
fi
EOF

$ chmod +x /etc/libvirt/hooks/qemu

Optionally, enable allmulticast on eth0 within each NetQ VM to support components that use standard multicast sockets:

$ ip link set dev eth0 allmulticast on

To make this setting persist across VM reboots:

$ cat > /etc/systemd/system/allmulticast-eth0.service << 'EOF'
[Unit]
Description=Enable allmulticast on eth0
After=network.target

[Service]
Type=oneshot
ExecStart=/sbin/ip link set dev eth0 allmulticast on
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target
EOF

$ systemctl enable --now allmulticast-eth0.service

Watch the boot process in another terminal window.
```
$ virsh console netq_ts
```

VMware Example Configuration

This example shows the VM setup process using an OVA file with VMware ESXi.

Enter the address of the hardware in your browser.
Log in to VMware using credentials with root access.
Click Storage in the Navigator to verify you have an SSD installed.
Click Create/Register VM at the top of the right pane.
Select Deploy a virtual machine from an OVF or OVA file, and click Next.
Provide a name for the VM, for example NetQ.
Tip: Make note of the name used during install as this is needed in a later step.
Drag the NetQ image file you downloaded from the NVIDIA Application Hub to the installation wizard, then click Next.
Select the storage type and data store for the image, then click Next.
Accept the default deployment options or modify them according to your network needs. Click Next when you are finished.
Review the configuration summary. Click Back to change any of the settings, or click Finish to continue with the creation of the VM.
The progress of the request is shown in the Recent Tasks window at the bottom of the application. This may take some time. After the VM is deployed, the wizard displays the full hardware and configuration details.

Use the default credentials to log in the first time:

Username: nvidia
Password: nvidia

$ ssh nvidia@<ipaddr>
Warning: Permanently added '<ipaddr>' (ECDSA) to the list of known hosts.
Ubuntu 24.04 LTS
nvidia@<ipaddr>'s password:
You are required to change your password immediately (root enforced)
System information as of Thu Dec  3 21:35:42 UTC 2024
System load:  0.09              Processes:           120
Usage of /:   8.1% of 61.86GB   Users logged in:     0
Memory usage: 5%                IP address for eth0: <ipaddr>
Swap usage:   0%
WARNING: Your password has expired.
You must change your password now and login again!
Changing password for nvidia.
(current) UNIX password: nvidia
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully
Connection to <ipaddr> closed.

$ ssh nvidia@<ipaddr>
Warning: Permanently added '<ipaddr>' (ECDSA) to the list of known hosts.
Ubuntu 24.04 LTS
nvidia@<ipaddr>'s password:
  System information as of Thu Dec  3 21:35:59 UTC 2024
  System load:  0.07              Processes:           121
  Usage of /:   8.1% of 61.86GB   Users logged in:     0
  Memory usage: 5%                IP address for eth0: <ipaddr>
  Swap usage:   0%
Last login: Thu Dec  3 21:35:43 2024 from <local-ipaddr>
nvidia@ubuntu:~$

Verify that the master node is ready for installation. Fix any errors before installing the NetQ software.

nvidia@hostname:~$ sudo opta-check-scale

Change the hostname for the VM from the default value.

The default hostname for the NetQ virtual machines is ubuntu. Change the hostname to fit your naming conventions while meeting Internet and Kubernetes naming standards.

Kubernetes requires hostnames to be composed of a sequence of labels concatenated with dots. For example, “en.wikipedia.org” is a hostname. Each label must be from 1 to 63 characters long. The entire hostname, including the delimiting dots, has a maximum of 253 ASCII characters.

The Internet standards (RFCs) for protocols specify that labels may contain only the ASCII letters a through z (in lower case), the digits 0 through 9, and the hyphen-minus character ('-').

Set the new hostname using the following command. Replace NEW_HOSTNAME with the name you chose.

nvidia@hostname:~$ sudo hostnamectl set-hostname NEW_HOSTNAME

Open your hypervisor and set up the VM for the additional nodes in the same manner as for the master node.
Run the following command on each node to verify that the node is ready for a NetQ software installation. Fix any errors indicated before installing the software.

nvidia@hostname:~$ sudo opta-check-scale

Install and activate the NetQ software using the CLI.

Run the following command on your master node to initialize the cluster. Copy the output of the command which includes the SSH key. You will use it in the next step.

nvidia@<hostname>:~$ netq install cluster master-init
    Please run the following command on all worker nodes:
    netq install cluster worker-init c3NoLXJzYSBBQUFBQjNOemFDMXljMkVBQUFBREFRQUJBQUFCQVFDM2NjTTZPdVM3dQN9MWTU1a

Run the netq install cluster worker-init <ssh-key> command on each non-master node.
Create a JSON template using the installation command for your deployment model. Run netq install cluster config generate on your master node to generate a template for a 3-node cluster configuration JSON file. To create a template with placeholder values for more than three nodes, run netq install cluster config generate workers and specify the number of additional nodes within the command. For example, netq install cluster config generate workers 2 creates a JSON template with fields for five nodes (three HA nodes plus two worker nodes).

nvidia@netq-server:~$ netq install cluster config generate
2025-10-28 17:29:53.260462: master-node-installer: Writing cluster installation configuration template file @ /tmp/combined-cluster-config.json

Edit the cluster configuration JSON file with the values for each attribute.

Default JSON Template

The netq install cluster config generate command creates a JSON template for a three-node cluster.

nvidia@netq-server:~$ vim /tmp/combined-cluster-config.json
{
        "version": "v3.0",
        "interface": "<INPUT>",
        "cluster-vip": "<INPUT>",
        "master-ip": "<INPUT>",
        "is-ipv6": "<INPUT>",
        "ha-nodes": [
                {
                        "ip": "<INPUT>
                        "description": "Control Plane Node 1"
                },
                {
                        "ip": "<INPUT>"
                        "description": "Control Plane Node 2"
                }
                ],
        "shared-cluster-install": "<INPUT>"
        "storage-path": "/var/lib/longhorn"
        "alertmanager_webhook_url": "<INPUT>"
        "cert-mode": "<INPUT>",
        "worker-nodes": []

}

Attribute	Description
`version`	The version of the JSON template. For NetQ 5.2, specify “v3.0”.
`interface`	The local network interface on your master node used for NetQ connectivity. Use a static IP address.
`cluster-vip`	The cluster virtual IP address must be an unused IP address allocated from the same subnet assigned to the default interface for your server nodes.
`master-ip`	The IP address of the primary master node in your cluster.
`is-ipv6`	Set the value to `true` if your network connectivity and node address assignments are IPv6. Set the value to `false` for IPv4.
`ha-nodes`, `ip`	The IP addresses of the two high-availability control plane nodes in your cluster.
`shared-cluster-install`	Set the value to `true` if Kubernetes was already installed (for example, as part of a Base Command Manager deployment) or `false` to install Kubernetes.
`alertmanager_webhook_url`	Enter the URL of the Alertmanager webhook. You can add multiple URLs as a comma-separated list. Note that you must manually add this line to the JSON template to receive NVLink alerts.
`cert-mode`	The TLS certificate mode for southbound (switch) communication. Set to `self-signed` (default) to use auto-generated certificates, or `user-cert` to provide your own CA and TLS certificates.
`worker-nodes`, `ip`	The IP addresses of the worker nodes in your cluster.

Completed JSON Example

The following example uses the netq install cluster config generate workers 3 command to create a JSON template for a six-node cluster.

nvidia@netq-server:~$ vim /tmp/combined-cluster-config.json 
{
        "version": "v3.0",
        "interface": "eth0",
        "cluster-vip": "10.176.235.101",
        "master-ip": "10.176.235.51",
        "is-ipv6": false,
        "ha-nodes": [
                {
                        "ip": "10.176.235.52"
                        "description": "Control Plane Node 1"
                },
                {
                        "ip": "10.176.235.53"
                        "description": "Control Plane Node 2"
                },
                ],
        "shared-cluster-install": false,
        "storage-path": "/var/lib/longhorn",
        "alertmanager_webhook_url": "",
        "cert-mode": "self-signed",
        "worker-nodes": [
                {
                        "ip": "10.176.235.54",
                        "description": "Worker Node 1"
                },
                {
                        "ip": "10.176.235.55",
                        "description": "Worker Node 2"
                },
                {
                        "ip": "10.176.235.56",
                        "description": "Worker Node 3"
                }
        ]
}

Attribute	Description
`version`	The version of the JSON template. For NetQ 5.2, specify “v3.0”.
`interface`	The local network interface on your master node used for NetQ connectivity.
`cluster-vip`	The cluster virtual IP address must be an unused IP address allocated from the same subnet assigned to the default interface for your server nodes.
`master-ip`	The IP address of the primary master node in your cluster.
`is-ipv6`	Set the value to `true` if your network connectivity and node address assignments are IPv6. Set the value to `false` for IPv4.
`ha-nodes`, `ip`	The IP addresses of the two high-availability control plane nodes in your cluster.
`shared-cluster-install`	Set the value to `true` if Kubernetes was already installed (for example, as part of a Base Command Manager deployment) or `false` to install Kubernetes.
`alertmanager_webhook_url`	Enter the URL of the Alertmanager webhook. You can add multiple URLs as a comma-separated list. Note that you must manually add this line to the JSON template to receive NVLink alerts.
`cert-mode`	The TLS certificate mode for southbound (switch) communication. Set to `self-signed` (default) to use auto-generated certificates, or `user-cert` to provide your own CA and TLS certificates.
`worker-nodes`, `ip`	The IP addresses of the worker nodes in your cluster.

Run the installation command on your master node using the JSON configuration file that you created in the previous step.

New Install

nvidia@<hostname>:~$ netq install cluster bundle /mnt/installables/NetQ-5.2.1.tgz /tmp/combined-cluster-config.json

If this step fails for any reason, run netq bootstrap reset and then try again.

Verify Installation Status

To view the status of the installation, use the netq show status [verbose] command. The following example shows a successful 3-node installation:

State: Active
    NetQ Live State: Active
    Installation Status: FINISHED
    Version: 5.2.1
    Installer Version: 5.2.1
    Installation Type: Cluster
    Installation Mode: Combined
    Activation Key: EhVuZXRxLWVuZHBvaW50LWdhdGV3YXkYsagDIixPSUJCOHBPWUFnWXI2dGlGY2hTRzExR2E5aSt6ZnpjOUvpVVTaDdpZEhFPQ==
    Master SSH Public Key: c3NoLXJzYSBBQUFBQjNOemFDMXljMkVBQUFBREFRQUJBQUFCZ1FDNW9iVXB6RkczNkRC
    Is Cloud: False
    
    Kubernetes Cluster Nodes Status:
    IP Address      Hostname       Role    NodeStatus    Virtual IP
    ------------    -----------    ------  ------------  ------------
    10.176.235.53   10.176.235.53  Worker  Ready         10.176.235.56
    10.176.235.52   10.176.235.52  Worker  Ready         10.176.235.55
    10.176.235.51   10.176.235.51  Master  Ready         10.176.235.54
    
    In Summary, Live state of the NetQ is... Active

Run the netq show opta-health command to verify that all applications are operating properly. Allow at least 15 minutes for all applications to come up and report their status.

If any of the applications or services display a DOWN status after 30 minutes, open a support ticket and attach the output of the opta-support command.

Next Steps

After NetQ is installed, you can log in to NetQ from your browser using the virtual cluster IP address.
If you installed NetQ NVLink with cert-mode set to user-cert, upload your certificates. After that, perform a system bringup.

Install NetQ for Ethernet and NVLink (Beta)

System Requirements

Port Requirements

Installation and Configuration

Verify Installation Status

Next Steps

Products

Solutions

Learn