HBN Service Release Notes
The following subsections provide information on HBN service new features, interoperability, known issues, and bug fixes.
HBN 2.5.0 offers the following new features and updates:
Added support for Destination NAT (DNAT) for Overlay Gateways
Added Configurable Control Plane Policing (CoPP) support
Added ACL support for L3 sub-interfaces
Added initial Bidirectional Forwarding Detection (BFD) support for BGP (alpha level)
Bug fixes
HBN 2.5.0 does not include any user affecting changes if upgrading from the previous HBN version.
Supported BlueField Networking Platforms
HBN 2.5.0 has been validated on the following NVIDIA® BlueField® networking platforms:
BlueField-2 DPUs:
BlueField-2 P-Series DPU 25GbE Dual-Port SFP56; PCIe Gen4 x8; Crypto Enabled; 16GB on-board DDR; 1GbE OOB management; HHHL
BlueField-2 P-Series DPU 25GbE Dual-Port SFP56; integrated BMC; PCIe Gen4 x8; Secure Boot Enabled; Crypto Enabled; 16GB on-board DDR; 1GbE OOB management; FHHL
BlueField-2 P-Series DPU 25GbE Dual-Port SFP56; integrated BMC; PCIe Gen4 x8; Secure Boot Enabled; Crypto Enabled; 32GB on-board DDR; 1GbE OOB management; FHHL
BlueField-2 P-Series DPU 100GbE Dual-Port QSFP56; integrated BMC; PCIe Gen4 x16; Secure Boot Enabled; Crypto Enabled; 32GB on-board DDR; 1GbE OOB management; FHHL
BlueField-3 DPUs:
BlueField-3 B3210E E-Series FHHL DPU; 100GbE (default mode) / HDR100 IB; Dual-port QSFP112; PCIe Gen5.0 x16 with x16 PCIe extension option; 16 Arm cores; 32GB on-board DDR; integrated BMC; Crypto Enabled
BlueField-3 B3220 P-Series FHHL DPU; 200GbE (default mode)/NDR200 IB; Dual-port QSFP112; PCIe Gen5.0 x16 with x16 PCIe extension option; 16 Arm cores; 32GB on-board DDR; integrated BMC; Crypto Enabled
BlueField-3 B3240 P-Series Dual-slot FHHL DPU; 400GbE/NDR IB (default mode); Dual-port QSFP112; PCIe Gen5.0 x16 with x16 PCIe extension option; 16 Arm cores; 32GB on-board DDR; integrated BMC; Crypto Enabled
BlueField-3 SuperNICs:
BlueField-3 B3210L E-series FHHL SuperNIC, 100GbE (default mode)/HDR100 IB, Dual port QSFP112, PCIe Gen4.0 x16, 8 Arm cores, 16GB on-board DDR, integrated BMC, Crypto Enabled
BlueField-3 B3220L E-Series FHHL SuperNIC, 200GbE (default mode)/NDR200 IB, Dual-port QSFP112, PCIe Gen5.0 x16, 8 Arm cores, 16GB on-board DDR, integrated BMC, Crypto Enabled
BlueField-3 B3140L E-Series FHHL SuperNIC, 400GbE/NDR IB (default mode), Single-port QSFP112, PCIe Gen5.0 x16, 8 Arm cores, 16GB on-board DDR, integrated BMC, Crypto Enabled
BlueField-3 B3140H E-series HHHL SuperNIC, 400GbE (default mode)/NDR IB, Single-port QSFP112, PCIe Gen5.0 x16, 8 Arm cores, 16GB on board DDR, integrated BMC, Crypto Enabled
BlueField platforms with 8GB on-board DDR memory are currently not supported with HBN.
Supported BlueField OS
HBN 2.5.0 supports DOCA 2.10.0 (BSP 4.10.0) on Ubuntu 22.04 OS.
Verified Scalability Limits
HBN 2.5.0 has been tested to sustain the following maximum scalability limits:
Limit | BlueField-2 | BlueField-3 | Comments |
VTEP peers (BlueFields per control plane) in the fabric | 8k 1 | 8k 1 | Number of BlueFields (VTEPs) within a single overlay fabric (reachable in the underlay) |
L2 VNIs/Overlay networks per BlueField | 20 | 20 | Total number of L2 VNIs in the fabric for L2 VXLAN use-case assuming every interface is associated with its own VLAN + L2 VNI |
L3 VNIs/Overlay networks per BlueField | 20 - for up to 4K VTEPs 10 - for up to 8K VTEPs | 20 - for up to 4K VTEPs 10 - for up to 8K VTEPs | Total number of L3 VNIs in the fabric for L3 VXLAN use-case assuming every interface is associated with its own VLAN + L2 VNI + L3 VNI + VRF |
BlueFields per a single L2 VNI network | 8k | 8k | Total number of DPUs, configured with the same L2 VNI (3 real DPUs, 2000 emulated VTEPs) |
BlueFields per a single L3 VNI network | 8k | 8k | Total number of DPUs, configured with the same L3 VNI (3 real DPUs, 2000 emulated VTEPs) |
Maximum number of local MAC/ARP entries per BlueField | 20 | 20 | Max total number of MAC/ARP entries learned from the host on the DPU |
Maximum number of local BGP routes per BlueField | 200 | 200 | Max total number of BGP routes advertised by the host to the BlueField (BGP peering with the host): 100 IPv4 + 100 IPv6 |
Maximum number of remote L3 LPM routes (underlay) | 8k | 8k | IPv4 or IPv6 underlay LPM routes per BlueField (default + host routes + LPM) |
Maximum number of EVPN type-2 entries | 16K | 16k | Remote overlay MAC/IP entries for compute peers stored on a single BlueField (L2 EVPN use case) |
Maximum number of EVPN type-5 entries | 32K | 80K | Remote overlay L3 LPM entries for compute peers stored on a single BlueField (L3 EVPN use case) |
Maximum number of Next-hops in ECMP Next-hop group | 16 | 16 | Max number of next-hops in ECMP Next-hop group (for overlay ECMP) |
Maximum number of PFs on the Host side | 2 | 2 | Total number of PFs visible to the host |
Maximum number of VFs on the Host side | 16 | 16 | Total number of VFs created on the host |
Maximum number of SFs on BlueField side | 2 | 2 | Total number of SF devices created on BlueField Arm |
The following table lists the known issues and limitations for this release of HBN.
Reference | Description |
4200335 | Description: DHCP issues may lead to incomplete resolve.conf on the DPU. The consequences can be DNS resolution failures and/or the hostname being set to 'localhost'. |
Workaround: Reboot. | |
Keywords: DPU, DHCP, resolve.conf, hostname, localhost, DNS | |
Reported in HBN Version: 2.5.0 | |
4196880 | Description: DHCP issues may lead to incomplete resolve.conf on the HBN container. The consequences can be DNS resolution failures and/or the hostname being set to 'localhost'. |
Workaround: Following is a list of possible workarounds.
| |
Keywords: DHCP, resolve.conf, hostname, localhost, DNS | |
Reported in HBN Version: 2.5.0 | |
4257285 | Description: ARP packets between DPU and Outside World that are forwarded using HBN are not HW offloaded. |
Workaround: N/A | |
Keywords: ARP, Outside World, HW offload | |
Reported in HBN Version: 2.5.0 | |
4279243 | Description: OVS does not punt the IPv6 Neighbour Advertisements with unicast Destination MAC address to CPU, hence endpoint MAC may not be learnt on the VTEP as long as the endpoint is silent (resulting in traffic towards endpoint to be software forwarded). This is applicable only for absolutely silent end hosts which do not initiate any IPv6 Neighbour Solicitation messages. Once the silent end host initiates the traffic, traffic will be hardware forwarded. This issue will persist only if the end points never initiate any traffic but only send IPv6 Neighbour Advertisements as a response to IPv6 Neighbour Solicitation (rare scenario) |
Workaround: User need to add following ovs rule on DPU: sudo ovs-ofctl add-flow br-hbn 'table=3,priority=100,icmp6,icmp_type=136 actions=resubmit(,98)' This rule will punt IPv6 NA packets to HBN To check if the rule is present: sudo ovs-ofctl dump-flows br-hbn --color --names table=3 | |
Keywords: IPV6 ND (Neighbour Discovery), NA (Neighbour Advertisement), silent host | |
Reported in HBN Version: 2.5.0 | |
4255708 | Description: When several ports are configured to be part of a bridge and later reconfigured to be L3 interfaces, only one port (the 1st port that was enslaved previously to bridge) gets correctly reprogrammed as an L3 interface in nl2doca. The remaining ports continue to appear as bridged ports in nl2doca. |
Workaround: Restart HBN container after unconfiguring bridge ports and before reconfiguring those ports as L3 interfaces | |
Keywords: Bridge port, L3 Port | |
Reported in HBN Version: 2.5.0 | |
4257285 | Description: ARP packets between DPU and Outside World that are forwarded using HBN are not HW offloaded. |
Workaround: N/A | |
Keywords: ARP | |
Reported in HBN Version: 2.5.0 | |
4214631 | Description: Packets with destination port 4789 coming from host side will be dropped if HBN is configured as L3 evpn, leading to Customer / Tenant encapsulated VxLan traffic drop in L3EVPN scenario. This prevents running vxlan underlay over hbn vxlan overlay in L3 evpn scenarios. |
Workaround: N/A | |
Keywords: 4789, vxlan overlay, vxlan underlay | |
Reported in HBN Version: 2.5.0 | |
4193046 | Description: When LLDP is enabled on BlueField, it may not work on uplink ports when HBN service is running. This might happen if LLDP is running without any interface filter configuration. |
Workaround: Configure LLDP to run only on interfaces where LLDP is required to be run, using a configuration file,
If this configuration file is changed while the LLDP service is running, it must be restarted using | |
Keywords: LLDP | |
Reported in HBN version: 2.4.1 | |
4200335 | Description: Sometimes the DNS resolution may fail if |
Workaround: N/A | |
Keywords: DNS; OOB connectivity | |
Reported in HBN version: 2.4.1 | |
4011688 | Description: The following critical error message is generated during HBN POD reboot. It can be safely ignored.
|
Workaround: N/A | |
Keywords: Log | |
Reported in HBN version: 2.4.0 | |
4098158 | Description: When using default BGP timers, an OVS restart may lead to extended traffic loss due to BGP peering reset. |
Workaround: N/A | |
Keywords: BGP; OVS | |
Reported in HBN version: 2.4.0 | |
3743942 | Description: HBN container may hang in init-sfs during container restart when the HBN YAML file (i.e., |
Workaround: If the container hangs in init-sfs for more than 1 minute, reload the DPU. | |
Keywords: Hang; container | |
Reported in HBN version: 2.3.0 | |
3961387 | Description: The changing of the port number for NVUE REST API using nv CLI/API is not supported. The following command should not be used to change the port number:
|
Workaround: On HBN, NVUE is accessible through 8765 (i.e., default port number). | |
Keywords: NVUE API; port number | |
Reported in HBN version: 2.3.0 | |
3967748 | Description: The command |
Workaround: N/A | |
Keywords: REST API; nginx | |
Reported in HBN version: 2.3.0 | |
3865633 | Description: Packets with destination port 4789/8472 coming from host side will be dropped if HBN is configured as L3 evpn. Functional effect is that Customer / Tenant encapsulated VxLan traffic will be dropped in L3EVPN scenario. |
Workaround: N/A | |
Keywords: 4789, 8472 | |
Reported in HBN version: 2.2.0 | |
3769309 | Description: A ping or other IP connectivity from a locally connected host in vrf-X to an interface IP address on the DPU/HBN itself in vrf-Y will not work, even if VRF route-leaking is enabled between these two VRFs. |
Workaround: N/A | |
Keyword: IP | |
Reported in HBN version: 2.2.0 | |
3835295 | Description: Traffic entering HBN service on a host PF/VF main-interface and exiting on a sub-interface of the same PF/VF (and vice versa) is not hardware offloaded. Similarly, traffic entering HBN service on one sub-interface and exiting on another sub-interface of the same host PF/VF is also not hardware offloaded. |
Workaround: N/A | |
Keyword: Hardware offload; interfaces | |
Reported in HBN version: 2.2.0 | |
3772552 | Description: The DHCP relay gateway-interface IP address does not automatically pick up the IP address assigned to the associated VRF. |
Workaround: The gateway-interface IP address must be explicitly configured. | |
Keyword: DHCP relay gateway; IP | |
Reported in HBN version: 2.2.0 | |
3891542 | Description: If NVUE-based routing policy (route map) configuration is used to associated route target extended communities with a EVPN route, only one route target can be specified. |
Workaround: N/A | |
Keyword: NVUE; route target | |
Reported in HBN version: 2.2.0 | |
3757686 | Description: When the HBN container is coming up and applying a large configuration through the NVUE-startup service which includes entities used by DHCP relay (e.g., interfaces, SVIs and VRFs), the DHCP relay service may go into FATAL state. It can be observed using the following command:
|
Workaround: Restart the DHCP relay service which is in FATAL state using the command:
| |
Keyword: DHCP relay; fatal; container; restart | |
Reported in HBN version: 2.1.0 | |
3605486 | Description: When the DPU boots up after issuing a "reboot" command from the DPU itself, some host-side interfaces may remain down. |
Workaround:
| |
Keyword: Reboot | |
Reported in HBN version: 1.5.0 | |
3547103 | Description: IPv6 stateless ACLs are not supported. |
Workaround: N/A | |
Keyword: IPv6 ACL | |
Reported in HBN version: 1.5.0 | |
3339304 | Description: Statistics for hardware-offloaded traffic are not reflected on SFs inside an HBN container. |
Workaround: Look up the stats using | |
Keyword: Statistics; container | |
Reported in HBN version: 1.4.0 | |
3352003 | Description: NVUE show, config, and apply commands malfunction if the |
Workaround: N/A | |
Keyword: NVUE commands | |
Reported in HBN version: 1.3.0 | |
3184745 | Description: The command |
Workaround: Use the command | |
Keyword: ACLs | |
Reported in HBN version: 1.2.0 | |
3158934 | Description: Deleting an NVUE user by removing their password file and restarting the |
Workaround: Either respawn the container after deleting the file or delete the password file corresponding to the user by running | |
Keyword: User deletion | |
Reported in HBN version: 1.2.0 | |
3185003 | Description: When a packet is encapsulated with a VXLAN header, it adds extra bytes which may cause the packet to exceed the MTU of link. Typically, the packet would be fragmented but its silently dropped and no fragmentation happens. |
Workaround: Make sure that the MTU on the uplink port is always 50 bytes more than host ports so that even after adding VXLAN headers, ingress packets do not exceed the MTU. | |
Keyword: MTU; VXLAN | |
Reported in HBN version: 1.2.0 | |
3184905 | Description: On VXLAN encapsulation, the DF flag is not propagated to the outer header. Such a packet may be truncated when forwarded in the kernel, and it may be dropped when hardware offloaded. |
Workaround: Make sure that the MTU on the uplink port is always 50 bytes more than host ports so that even after adding VXLAN headers, ingress packets do not exceed the MTU. | |
Keyword: VXLAN | |
Reported in HBN version: 1.2.0 | |
3188688 | Description: When stopping the container using the command |
Workaround: Pass a timeout value when stopping the HBN container by running:
| |
Keyword: Timeout | |
Reported in HBN version: 1.2.0 | |
3129749 | Description: The same ACL rule cannot be applied in both the inbound and outbound direction on a port. |
Workaround: N/A | |
Keyword: ACLs | |
Reported in HBN version: 1.2.0 | |
3126560 | Description: The system's time zone cannot be modified using NVUE in the HBN container. |
Workaround: The time zone can be manually changed by symlinking the
| |
Keyword: Time zone; NVUE | |
Reported in HBN version: 1.2.0 | |
3118204 | Description: Auto-BGP functionality (where the ASN does not need to be configured but is dynamically inferred by the system based on the system's role as a leaf or spine device) is not supported on HBN. |
Workaround: If BGP is configured and used on HBN, the BGP ASN must be manually configured. | |
Keyword: BGP | |
Reported in HBN version: 1.2.0 | |
3233088 | Description: Since checksum calculation is offloaded to the hardware (not done by the kernel), it is expected to see an incorrect checksum in the tcpdump for locally generated, outgoing packets. BGP keepalives and updates are some of the packets that show such incorrect checksum in tcpdump. |
Workaround: N/A | |
Keyword: BGP | |
Reported in HBN version: 1.2.0 | |
2821785 | Description: MAC addresses are not learned in the hardware but only in software. This may affect performance in pure L2 unicast traffic. |
Workaround: N/A | |
Keyword: MAC; L2 | |
Reported in HBN version: 1.3.0 | |
3017202 | Description: Due to disabled backend foundation units, some NVUE commands return |
Workaround: N/A | |
Keyword: Unsupported NVUE commands | |
Reported in HBN version: 1.3.0 | |
2828838 | Description: NetworkManager and other services not directly related to HBN may display the following message in syslog:
The message has no functional impact and may be ignored. |
Workaround: N/A | |
Keyword: Error | |
Reported in HBN version: 1.3.0 |
The following table lists the known issues which have been fixed for this release of HBN.
Reference | Description |
4155959 | Description: With uplinks in the br-sfc bridge, IPv6 traffic in uplink-to-uplink direction results in OVS crash resulting in complete traffic drop. |
Fixed in HBN Version: 2.5.0 | |
4197067 | Description: The management VRF does not have an IPv6 address configured, resulting in the absence of a default IPv6 route in the management VRF. Consequently, IPv6 connectivity on the management port is unavailable, and only IPv4 connectivity is supported. |
Fixed in HBN Version: 2.5.0 | |
4093502 | Description: VRF interfaces have a loopback address, but these loopback addresses have scope global, not scope host which can break source IP address lookup for packets originating from the VRF. |
Fixed in HBN version: 2.4.0 | |
4029473 | Description: Rarely, after deletion then creation of an interface, BGP peering over that interface may announce IPv6 routes with an IPv4-mapped IPv6 address as the next hop, which the BGP peer device at the other end can reject. |
Fixed in HBN version: 2.4.0 | |
4125363 | Description: On newer BlueField-2 and BlueField-3 devices, |
Fixed in HBN version: 2.4.0 | |
3965589 | Description: When SR-IOV VFs are created or deleted and recreated, some ports may stay in ethX naming format and not be properly renamed to pfXvfY format. This results in the port remaining in error state as when running the command |
Fixed in HBN version: 2.4.0 | |
4004191 | Description: Due to security fixes on BlueField-2, the number of context switches increased by 20% which may result in user applications (e.g., nl2doca) running slower. |
Fixed in HBN version: 2.4.0 | |
3880352 | Description: Deleting and re-adding SR-IOV ports might result in some ports in br-hbn bridge going in error state. |
Fixed in HBN version: 2.4.0 | |
3960825 | Description: When either |
Fixed in HBN version: 2.3.0 | |
3538167 | Description: An explicit restart of FRR service may be required if the BGP AS number is changed via NVUE. |
Fixed in HBN version: 2.3.0 | |
3360699 | Description: If it is required to decrease the default MTU on interfaces on which HBN operates, after the change is made on the BlueField as well as within HBN, the BlueField must be rebooted for the change to take effect properly. |
Fixed in HBN version: 2.3.0 | |
3864080 | Description: When an interface is toggled off and on, its sub-interfaces lose their IPv6 addresses and do not get them back. |
Fixed in HBN version: 2.3.0 | |
3632344 | Description: HBN interfaces on the BlueField side (outside the HBN container) may not get their proper MTU set from systemd-network. |
Fixed in HBN version: 2.2.0 | |
3760869 | Description: Datapath flow with very low PPS may be deleted before aging time (60 sec) in large scale of number of routes (16K+). |
Fixed in HBN version: 2.2.0 | |
3770992 | Description: It is not possible to configure an IPv6 default ( |
Fixed in HBN version: 2.2.0 | |
3824881 | Description: When the number of unique ECMP groups used is more than 6, it results in failure of programming prefixes using ECMP-groups greater than 6. Uniqueness is based on ECMP content, so if multiple routes have same nexthop paths, they just use 1 ECMP group. |
Fixed in HBN version: 2.2.0 | |
3705894 | Description: In an EVPN Symmetric Routing scenario, IPv6 traffic is not hardware offloaded. |
Fixed in HBN version: 2.2.0 | |
3519324 | Description: The DOCA HBN container takes about 1 minute longer to spawn, as compared to previous HBN release (1.4.0) |
Fixed in HBN version: 2.1.0 | |
3219539 | Description: TC rules are programmed by OVS to map uplink and host representor ports to HBN service. These rules are ageable and can result in packets needing to get software forwarded periodically to refresh the rules. |
Fixed in HBN version: 2.1.0 | |
3610971 | Description: The output of the command |
Fixed in HBN version: 2.0.0 | |
3452914 | Description: IPv6 OOB connectivity from the HBN container stops working if the br-mgmt interface on the DPU goes down. When going down, the br-mgmt interface loses its IPv6 address, which is used as the gateway address for the HBN container. If the br-mgmt interface comes back up, its IPv6 address is not added back and IPv6 OOB connectivity from the HBN container will not work |
Fixed in HBN version: 1.5.0 | |
3191433 | Description: ECMP selection for the underlay path uses the ingress port and identifies uplink ports via round robin. This may not result in uniform spread of the traffic. |
Fixed in HBN version: 1.4.0 | |
3049879 | Description: When reloading ( |
Fixed in HBN version: 1.4.0 | |
3284607 | Description: When an ACL is configured for IPv4 and L4 parameters (protocol tcp/udp, source, and destination ports) match, the ACL also matches IPv6 traffic with the specified L4 parameters. |
Fixed in HBN version: 1.4.0 | |
3282113 | Description: Some DPUs experience an issue with the clock settings after installing a BlueField OS in an HBN setting in which the date reverts back to "Thu Sep 8, 2022". |
Fixed in HBN version: 1.4.0 | |
3354029 | Description: If interfaces on which BGP unnumbered peering is configured are not defined in the |
Fixed in HBN version: 1.4.0 |