HBN Service Release Notes

1.0

The following subsections provide information on HBN service new features, interoperability, known issues, and bug fixes.

HBN 2.2.0 offers the following new features and updates:

  • Added support for single-port BlueField-3 SuperNIC SKUs

  • Added GA-level support for Local VRF Route Leaking

  • Added support for EVPN Downstream VNI (DVNI) for Symmetric EVPN Route Leaking

  • Added support for VRF-Lite configuration with Layer-3 VLAN subinterfaces

  • Added support for Network-to-Network Hairpin routing across BlueField uplink ports

  • Added GA-level support for Stateful ACLs over L2 VXLAN

  • Added initial support for VLAN trunk configuration on host-facing interfaces

Supported BlueField Networking Platforms

HBN 2.2.0 has been validated on the following NVIDIA BlueField Networking Platforms:

  • BlueField-2 DPUs:

    • BlueField-2 P-Series DPU 25GbE Dual-Port SFP56; PCIe Gen4 x8; Crypto Enabled; 16GB on-board DDR; 1GbE OOB management; HHHL

    • BlueField-2 P-Series DPU 25GbE Dual-Port SFP56; integrated BMC; PCIe Gen4 x8; Secure Boot Enabled; Crypto Enabled; 16GB on-board DDR; 1GbE OOB management; FHHL

    • BlueField-2 P-Series DPU 25GbE Dual-Port SFP56; integrated BMC; PCIe Gen4 x8; Secure Boot Enabled; Crypto Enabled; 32GB on-board DDR; 1GbE OOB management; FHHL

    • BlueField-2 P-Series DPU 100GbE Dual-Port QSFP56; integrated BMC; PCIe Gen4 x16; Secure Boot Enabled; Crypto Enabled; 32GB on-board DDR; 1GbE OOB management; FHHL

  • BlueField-3 DPUs:

    • BlueField-3 B3210 P-Series FHHL DPU; 100GbE (default mode)/HDR100 IB; Dual-port QSFP112; PCIe Gen5.0 x16 with x16 PCIe extension option; 16 Arm cores; 32GB on-board DDR; integrated BMC; Crypto Enabled

    • BlueField-3 B3220 P-Series FHHL DPU; 200GbE (default mode)/NDR200 IB; Dual-port QSFP112; PCIe Gen5.0 x16 with x16 PCIe extension option; 16 Arm cores; 32GB on-board DDR; integrated BMC; Crypto Enabled

    • BlueField-3 B3240 P-Series Dual-slot FHHL DPU; 400GbE/NDR IB (default mode); Dual-port QSFP112; PCIe Gen5.0 x16 with x16 PCIe extension option; 16 Arm cores; 32GB on-board DDR; integrated BMC; Crypto Enabled

  • BlueField-3 SuperNICs:

    • BlueField-3 B3210L E-series FHHL SuperNIC, 100GbE (default mode)/HDR100 IB, Dual port QSFP112, PCIe Gen4.0 x16, 8 Arm cores, 16GB on-board DDR, integrated BMC, Crypto Enabled

    • BlueField-3 B3220L E-Series FHHL SuperNIC, 200GbE (default mode)/NDR200 IB, Dual-port QSFP112, PCIe Gen5.0 x16, 8 Arm cores, 16GB on-board DDR, integrated BMC, Crypto Enabled

    • BlueField-3 B3140L E-Series FHHL SuperNIC, 400GbE/ NDR IB (default mode), Single-port QSFP112, PCIe Gen5.0 x16, 8 Arm cores, 16GB on-board DDR, integrated BMC, Crypto Enabled

    • BlueField-3 B3140H E-series HHHL SuperNIC, 400GbE (default mode)/NDR IB, Single-port QSFP112, PCIe Gen5.0 x16, 8 Arm cores, 16GB on board DDR, integrated BMC, Crypto Enabled

Note

BlueField platforms with 8GB on-board DDR memory are currently not supported with HBN.


Supported BlueField OS

HBN 2.2.0 supports DOCA 2.7.0 (BSP 4.7.0) on Ubuntu 22.04 OS.

Verified Scalability Limits

HBN 2.2.0 has been tested to sustain the following maximum scalability limits:

Limit

BlueField-2

BlueField-3

Comments

VTEP peers (BlueFields per control plane) in the fabric

2k

2k

Number of BlueFields (VTEPs) within a single overlay fabric (reachable in the underlay)

L2 VNIs/Overlay networks per BlueField

20

20

Total number of L2 VNIs in the fabric for L2 VXLAN use-case assuming every interface is associated with its own VLAN + L2 VNI

L3 VNIs/Overlay networks per BlueField

20

20

Total number of L3 VNIs in the fabric for L3 VXLAN use-case assuming every interface is associated with its own VLAN + L2 VNI + L3 VNI + VRF

BlueFields per a single L2 VNI network

2k

2k

Total number of DPUs, configured with the same L2 VNI (3 real DPUs, 2000 emulated VTEPs)

BlueFields per a single L3 VNI network

2k

2k

Total number of DPUs, configured with the same L3 VNI (3 real DPUs, 2000 emulated VTEPs)

Maximum number of local MAC/ARP entries per BlueField

20

20

Max total number of MAC/ARP entries learned from the host on the DPU

Maximum number of local BGP routes per BlueField

200

200

Max total number of BGP routes advertised by the host to the BlueField (BGP peering with the host): 100 IPv4 + 100 IPv6

Maximum number of remote L3 LPM routes (underlay)

2K

2K

IPv4 or IPv6 underlay LPM routes per BlueField (default + host routes + LPM)

Maximum number of EVPN type-2 entries

16K

16k

Remote overlay MAC/IP entries for compute peers stored on a single BlueField (L2 EVPN use case)

Maximum number of EVPN type-5 entries

16K

16K

Remote overlay L3 LPM entries for compute peers stored on a single BlueField (L3 EVPN use case)

Maximum number of PFs on the Host side

2

2

Total number of PFs visible to the host

Maximum number of VFs on the Host side

16

16

Total number of VFs created on the host

Maximum number of SFs on BlueField side

2

2

Total number of SF devices created on BlueField Arm


The following table lists the known issues and limitations for this release of HBN.

Reference

Description

3769309

Description: A ping or other IP connectivity from a locally connected host in vrf-X to an interface IP address on the DPU/HBN itself in vrf-Y will not work, even if VRF route-leaking is enabled between these two VRFs.

Workaround: N/A

Keyword:

Reported in HBN version: 2.2.0

3886379

Description: Deleting and re-adding SR-IOV ports might result in some ports in br-hbn bridge going in error state.

Workaround: If possible, an appropriate number of SR-IOV ports should be chosen at BFB install time. But if a change is made and if the system has this error, the host must undergo a power cycle to resolve the issue.

Keyword: Bridge; SR-IOV

Reported in HBN version: 2.2.0

3864080

Description: When an interface is toggled off and on, its sub-interfaces lose their IPv6 addresses and do not get them back.

Workaround: Perform "ifreload -a" to re-apply the IPv6 addresses to the sub-interfaces.

Keyword: Subinterface; IPv6 address

Reported in HBN version: 2.2.0

3835295

Description: Traffic entering HBN service on a host PF/VF main-interface and exiting on a sub-interface of the same PF/VF (and vice versa) is not hardware offloaded. Similarly, traffic entering HBN service on one sub-interface and exiting on another sub-interface of the same host PF/VF is also not hardware offloaded.

Workaround: N/A

Keyword: Hardware offload; interfaces

Reported in HBN version: 2.2.0

3360699

Description: If it is required to decrease the default MTU on interfaces on which HBN operates, after the change is made on the BlueField as well as within HBN, the BlueField must be rebooted for the change to take effect properly.

Workaround: N/A

Keyword: MTU; reboot

Reported in HBN version: 2.2.0

3538167

Description: An explicit restart of FRR service may be required if the BGP AS number is changed via NVUE.

Workaround: N/A

Keyword: FRR; BGP

Reported in HBN version: 2.2.0

3772552

Description: The DHCP relay gateway-interface IP address does not automatically pick up the IP address assigned to the associated VRF.

Workaround: The gateway-interface IP address must be explicitly configured.

Keyword: DHCP relay gateway; IP

Reported in HBN version: 2.2.0

3873506

Description: Rarely, after deletion then creation of an interface, BGP peering over that interface may announce IPv6 routes with an IPv4-mapped IPv6 address as the next hop, which the BGP peer device at the other end can reject.

Workaround: A flap of the BGP peering session resolves this problem.

The NVUE command line to perform this is:

Copy
Copied!
            

nv set vrf <vrf-name> router bgp neighbor <neighbor> shutdown on nv config apply

Followed by:

Copy
Copied!
            

nv set vrf <vrf-name> router bgp neighbor <neighbor> shutdown off nv config apply

Keyword: BGP; toggle

Reported in HBN version: 2.2.0

3891542

Description: If NVUE-based routing policy (route map) configuration is used to associated route target extended communities with a EVPN route, only one route target can be specified.

Workaround: N/A

Keyword: NVUE; route target

Reported in HBN version: 2.2.0

3757686

Description: When the HBN container is coming up and applying a large configuration through the NVUE-startup service which includes entities used by DHCP relay (e.g., interfaces, SVIs and VRFs), the DHCP relay service may go into FATAL state. It can be observed using the following command:

Copy
Copied!
            

supervisorctl status | grep isc-dhcp-relay isc-dhcp-relay-vrf11 RUNNING pid 2069, uptime 0:11:31 isc-dhcp-relay-vrf12 RUNNING pid 2071, uptime 0:11:31 isc-dhcp-relay-vrf13 FATAL Exited too quickly (process log may have details) isc-dhcp-relay-vrf14 FATAL Exited too quickly (process log may have details)

Workaround: Restart the DHCP relay service which is in FATAL state using the command:

Copy
Copied!
            

supervisorctl restart <relay-service-name>

Keyword: DHCP relay; fatal; container; restart

Reported in HBN version: 2.1.0

3605486

Description: When the DPU boots up after issuing a "reboot" command from the DPU itself, some host-side interfaces may remain down.

Workaround:

  1. Restart openibd:

    Copy
    Copied!
                

    systemctl restart openibd

  2. Recreate SR-IOV interfaces if they are needed.

  3. Replay interface config. For example:

    • If using ifupdown2:

      Copy
      Copied!
                  

      ifreload -a 

    • If using Netplan:

      Copy
      Copied!
                  

      netplan apply

Keyword: Reboot

Reported in HBN version: 1.5.0

3547103

Description: IPv6 stateless ACLs are not supported.

Workaround: N/A

Keyword: IPv6 ACL

Reported in HBN version: 1.5.0

3339304

Description: Statistics for hardware-offloaded traffic are not reflected on SFs inside an HBN container.

Workaround: Look up the stats using ip -s link show on PFs outside of the HBN container. PFs would show Tx/Rx stats for traffic that is hardware-accelerated in the HBN container.

Keyword: Statistics; container

Reported in HBN version: 1.4.0

3352003

Description: NVUE show, config, and apply commands malfunction if the nvued and nvued-startup services are not in the RUNNING and EXITED states respectively.

Workaround: N/A

Keyword: NVUE commands

Reported in HBN version: 1.3.0

3184745

Description: The command nv show interface <intf> acl does not show correct information if there are multiple ACLs bound to the interface.

Workaround: Use the command nv show interface <intf> to view the ACLs bound to an interface.

Keyword: ACLs

Reported in HBN version: 1.2.0

3158934

Description: Deleting an NVUE user by removing their password file and restarting the decrypt-user-add service on the HBN container does not work.

Workaround: Either respawn the container after deleting the file, or delete the password file corresponding to the user by running userdel -r username.

Keyword: User deletion

Reported in HBN version: 1.2.0

3185003

Description: When a packet is encapsulated with a VXLAN header, it adds extra bytes which may cause the packet to exceed the MTU of link. Typically, the packet would be fragmented but its silently dropped and no fragmentation happens.

Workaround: Make sure that the MTU on the uplink port is always 50 bytes more than host ports so that even after adding VXLAN headers, ingress packets do not exceed the MTU.

Keyword: MTU; VXLAN

Reported in HBN version: 1.2.0

3184905

Description: On VXLAN encapsulation, the DF flag is not propagated to the outer header. Such a packet may be truncated when forwarded in the kernel, and it may be dropped when hardware offloaded.

Workaround: Make sure that the MTU on the uplink port is always 50 bytes more than host ports so that even after adding VXLAN headers, ingress packets do not exceed the MTU.

Keyword: VXLAN

Reported in HBN version: 1.2.0

3188688

Description: When stopping the container using the command crictl stop an error may be reported because the command uses a timeout of 0 which is not enough to stop all the processes in the HBN container.

Workaround: Pass a timeout value when stopping the HBN container by running:

Copy
Copied!
            

crictl stop --timeout 60 <hbn-container>

Keyword: Timeout

Reported in HBN version: 1.2.0

3129749

Description: The same ACL rule cannot be applied in both the inbound and outbound direction on a port.

Workaround: N/A

Keyword: ACLs

Reported in HBN version: 1.2.0

3126560

Description: The system's time zone cannot be modified using NVUE in the HBN container.

Workaround: The timezone can be manually changed by symlinking the /etc/localtime file to a binary time zone's identifier in the /usr/share/zoneinfo directory. For example:

Copy
Copied!
            

sudo ln -sf /usr/share/zoneinfo/GMT /etc/localtime

Keyword: Time zone; NVUE

Reported in HBN version: 1.2.0

3118204

Description: Auto-BGP functionality (where the ASN does not need to be configured but is dynamically inferred by the system based on the system's role as a leaf or spine device) is not supported on HBN.

Workaround: If BGP is configured and used on HBN, the BGP ASN must be manually configured.

Keyword: BGP

Reported in HBN version: 1.2.0

3233088

Description: Since checksum calculation is offloaded to the hardware (not done by the kernel), it is expected to see an incorrect checksum in the tcpdump for locally generated, outgoing packets. BGP keepalives and updates are some of the packets that show such incorrect checksum in tcpdump.

Workaround: N/A

Keyword: BGP

Reported in HBN version: 1.2.0

2821785

Description: MAC addresses are not learned in the hardware but only in software. This may affect performance in pure L2 unicast traffic.

Workaround: N/A

Keyword: MAC; L2

Reported in HBN version: 1.3.0

3017202

Description: Due to disabled backend foundation units, some NVUE commands return 500 INTERNAL SERVER ERROR/404 NOT FOUND. These commands are related to features or subsystems which are not supported on HBN.

Workaround: N/A

Keyword: Unsupported NVUE commands

Reported in HBN version: 1.3.0

2828838

Description: NetworkManager and other services not directly related to HBN may display the following message in syslog:

Copy
Copied!
            

"netlink: read: too many netlink events. Need to resynchronize platform cache"

The message has no functional impact and may be ignored.

Workaround: N/A

Keyword: Error

Reported in HBN version: 1.3.0

The following table lists the known issues which have been fixed for this release of HBN.

Reference

Description

3632344

Description: HBN interfaces on the BlueField side (outside the HBN container) may not get their proper MTU set from systemd-network.

Fixed in HBN version: 2.2.0

3760869

Description: Datapath flow with very low PPS may be deleted before aging time (60 sec) in large scale of number of routes (16K+).

Fixed in HBN version: 2.2.0

3770992

Description: It is not possible to configure an IPv6 default (::/0) static route using NVUE.

Fixed in HBN version: 2.2.0

3824881

Description: When the number of unique ECMP groups used is more than 6, it results in failure of programming prefixes using ECMP-groups greater than 6. Uniqueness is based on ECMP content, so if multiple routes have same nexthop paths, they just use 1 ECMP group.

Fixed in HBN version: 2.2.0

3705894

Description: In an EVPN Symmetric Routing scenario, IPv6 traffic is not hardware offloaded.

Fixed in HBN version: 2.2.0

3168683

Description: If many interfaces are participating in EVPN/routing, it is possible for the routing process to run out of memory.

Fixed in HBN version: 2.1.0

3519324

Description: The DOCA HBN container takes about 1 minute longer to spawn, as compared to previous HBN release (1.4.0)

Fixed in HBN version: 2.1.0

3219539

Description: TC rules are programmed by OVS to map uplink and host representor ports to HBN service. These rules are ageable and can result in packets needing to get software forwarded periodically to refresh the rules.

Fixed in HBN version: 2.1.0

3610971

Description: The output of the command nv show interface does not display information about VRFs, VXLAN, and bridge.

Fixed in HBN version: 2.0.0

3452914

Description: IPv6 OOB connectivity from the HBN container stops working if the br-mgmt interface on the DPU goes down. When going down, the br-mgmt interface loses its IPv6 address, which is used as the gateway address for the HBN container. If the br-mgmt interface comes back up, its IPv6 address is not added back and IPv6 OOB connectivity from the HBN container will not work

Fixed in HBN version: 1.5.0

3191433

Description: ECMP selection for the underlay path uses the ingress port and identifies uplink ports via round robin. This may not result in uniform spread of the traffic.

Fixed in HBN version: 1.4.0

3049879

Description: When reloading (ifreload) an empty /etc/network/interfaces file, the previously created interfaces are not deleted.

Fixed in HBN version: 1.4.0

3284607

Description: When an ACL is configured for IPv4 and L4 parameters (protocol tcp/udp, source, and destination ports) match, the ACL also matches IPv6 traffic with the specified L4 parameters.

Fixed in HBN version: 1.4.0

3282113

Description: Some DPUs experience an issue with the clock settings after installing a BlueField OS in an HBN setting in which the date reverts back to "Thu Sep 8, 2022".

Fixed in HBN version: 1.4.0

3354029

Description: If interfaces on which BGP unnumbered peering is configured are not defined in the /etc/network/interfaces configuration file, BGP peering does not get established on them.

Fixed in HBN version: 1.4.0

© Copyright 2024, NVIDIA. Last updated on Feb 9, 2024.