Known Issues

3654779

Description: In the current state of the library, the TCP keep-alive feature is partially supported.

The known limitations are:

1. The keep-alive internal parameters are as follows:

a. TCP_KEEPIDLE - set to value in /proc/sys/net/ipv4/tcp_keepalive_time

b. TCP_KEEPINTVL - 75,000ms or 75 seconds(1:15 minutes)

c. TCP_KEEPCNT - 9 (probes to send before timeout)

2. The following setsockopt options are:

a. setsockopt(sock, SOL_SOCKET, SO_KEEPALIVE, &enable, sizeof(enable)) - Disabling/Enabling the feature not operational

b. setsockopt(sock, IPPROTO_TCP, TCP_KEEPIDLE, &idle, sizeof(idle)); - Change the idle time before the first probe is allowed

c. setsockopt(sock, IPPROTO_TCP, TCP_KEEPINTVL, &interval, sizeof(interval)); - Change the intervals between the keep-alive probes not operational

d. setsockopt(sock, IPPROTO_TCP, TCP_KEEPCNT, &retries, sizeof(retries)); - Change the number of probes not operational

3. The following system configurations are disregarded:

`sysctl -w net.ipv4.tcp_keepalive_intvl=60`

`sysctl -w net.ipv4.tcp_keepalive_probes=3`

4. The contents of the following files are disregarded:

/proc/sys/net/ipv4/tcp_keepalive_intvl

/proc/sys/net/ipv4/tcp_keepalive_probes

Workaround: N/A

Keywords: keep-alive, TCP_KEEPALIVE

Discovered in Version: 3.10

3440429

Description: XLIO fails to create RFS rule and prints the following error:

Create RFS flow failed, Tag: Y, Flow: dst: X.X.X.X:X, src: X.X.X.X:X, proto: TCP, family: INET, Priority: 1, errno: 17 - File exists

This happens when a new socket allocates the same fd as a socket in cleanup process.

Workaround: Set XLIO_TCP_ABORT_ON_CLOSE=1 to lower the chances of such a scenario to take place.

Keywords: RFS; steering failure error

Discovered in Version: 3.0.2

2970828

Description: XLIO ignores all signals except SIGINT signal, occasionally resulting in blocked IO calls.

Workaround: N/A

Keywords: Signal; SIGINT; IO

Discovered in Version: 2.1.4

3204491

Description: Note the following limitations when working with setsockopt for UDP multicast socket.

1. Setting either IP_DROP_MEMBERSHIP or IP_DROP_SOURCE_MEMBERSHIP reverts IP_MULTICAST_LOOP=0 to 1.

2. Only the first multicast group will be offloaded by XLIO using IP_ADD_SOURCE_MEMBERSHIP or MCAST_JOIN_SOURCE_GROUP. The rest of the groups will be handled by the Kernel.

3. Binding to a specific IP (not any_address) after setting multicast setsockopt options will result in binding to any_address instead.

Workaround: N/A

Keywords: multicast; setsockopt; bind

Discovered in Version: 2.1.4

3217617

Description: When XLIO binds to any IP, resource allocation is required for each IP per port. This action may take up to several seconds with a limit of 64K steerings per family.

Workaround: To avoid this scenario, bind a socket to a specific IP instead.

Keywords: Bind to any IP; resource allocation

Discovered in Version: 2.0.6

3190147

Description: TCP User Timeout options are not supported with select(); pselect(); poll(); ppoll(); __poll_chk(); and __ppoll_chk.

Workaround: Application may use epoll instead.

Keywords: TCP; user timeout; select; poll; ppoll

Discovered in Version: 2.0.6

3217629

Description: iperf3 usage of longjmp operation in the signal handler results in abandoning previous thread context, leaving the state of internal objects in an undefined state.

Workaround: N/A

Keywords: iperf3; signal handler termination

Discovered in Version: 2.0.6

3231754/3208316

Description: Traffic over IPv6 link-local addresses is not supported.

Workaround: N/A

Keywords: IPv6; link-local

Discovered in Version: 2.0.6

3231768

Description: IPv6 Flow Labels are not supported.

Workaround: N/A

Keywords: IPv6 flow labels

Discovered in Version: 2.0.6

3231775

Description: XLIO configuration file does not support IPv6 addresses.

Workaround: N/A

Keywords: xlio.conf; IPv6

Discovered in Version: 2.0.6

3231781

Description: The following IPv6 multicast socket options are not supported:

• IPV6_MULTICAST_LOOP

• MCAST_BLOCK_SOURCE

• MCAST_UNBLOCK_SOURCE

Workaround: N/A

Keywords: IPV6_MULTICAST_LOOP; MCAST_BLOCK_SOURCE; MCAST_UNBLOCK_SOURCE; socket; IPv6

Discovered in Version: 2.0.6

3232106

Description: SO_MAX_PACING_RATE socket option is not supported.

Workaround: N/A

Keywords: SO_MAX_PACING_RATE; socket

Discovered in Version: 2.0.6

3117726

Description: TLS RX is not supported with XLIO_STRQ=off.

Workaround: To enable TLS RX support, make sure either XLIO_STRQ=on or XLIO_STRQ=regular_rq.

Keywords: TLS RX

Discovered in Version: 1.3.5

3136901

Description: If a multi-threaded application calls exit() syscall while other threads call socket API concurrently, this can lead to a race between XLIO library destructor and intercepted socket API or in rare cases to a segmentation fault.

Workaround: It is recommended to finish running threads before process termination. Alternatively, avoid calling socket API concurrently with exit() syscall.

Keywords: multithreaded applications

Discovered in Version: 1.3.5

2667588

Description: XLIO conf file does not support IPv6 (see XLIO Configuration Parameters).

Workaround: N/A

Keywords: xlio.conf; IPv6

Discovered in Version: 1.2.9

21371706

Description: Socket API breaks for closed UDP sockets when using XLIO_NGINX_UDP_POOL_SIZE != 0, as the socket is not entirely closed. Socket operations will not fail after the socket is closed.

Workaround: N/A

Keywords: UDP; pool

Discovered in Version: 1.2.9

1542628

Description: Device memory programming is not supported on VMs that lack Blue Flame support.

Workaround: N/A

Keywords: MEMIC, Device Memory, Virtual Machine, Blue flame

Discovered in Version: 1.0.6

—

Description: If XLIO runs when XLIO_HANDLE_SIGINTR is enabled, an error message might be written upon exiting.

Workaround: Ignore the error message, or run XLIO with XLIO_HANDLE_SIGINTR disabled.

Keywords: XLIO_HANDLE_SIGINTR, error message

Discovered in Version: 1.0.6

2783472

Description: The following XLIO_ERRORs will be displayed when running ping with root permissions:XLIO ERROR: ring_simple[0x7f257d18d720]:256:create_resources() ibv_create_comp_channel for tx failed. m_p_tx_comp_event_channel = (nil) (errno=13 Permission denied)

XLIO ERROR: ib_ctx_handler213:mem_dereg() failed de-registering a memory region (errno=13 Permission denied)

Workaround: N/A

Keywords: XLIO_ERROR while running ping with root permissions

Discovered in Version: 1.0.6

965237

Description: The following sockets APIs are directed to the OS and are not offloaded by XLIO:

· int socketpair(int domain, int type, int protocol, int sv[2]);

· int dup(int oldfd);

· int dup2(int oldfd, int newfd);

Workaround: N/A

Keywords: sockets, socketpair, dup, dup2

965227

Description: Multicast (MC) loopback within a process is not supported by XLIO:

• If an application process opens 2 (or more) sockets on the same MC group they will not get each other's traffic.

  Note: MC loopback between different XLIO processes always work.

• Both sockets will receive all ingress traffic coming from the wire

Workaround: N/A

Keywords: Multicast, Loopback

965227

Description: MC loopback between XLIO and the OS limitation.

• The OS will reject loopback traffic coming from the NIC

• MC traffic from the OS to XLIO is functional

Workaround: N/A

Keywords: Multicast, Loopback

965227

Description: MC loopback Tx is currently disabled and setsockopt (IP_MULTICAST_LOOP) is not supported.

Workaround: N/A

Keywords: Multicast, Loopback

1011005

Description: XLIO SELECT option supports up to 200 sockets in TCP.

Workaround: Use ePoll that supports up to 6000 sockets.

Keywords: SockPerf

977899

Description: An unsuccessful trial to connect to a local interface, is reported by XLIO as Connection timeout rather than Connection refused.

Workaround: N/A

Keywords: Verification

1019085

Description: Poll is limited with the number of sockets.

Workaround: Use ePoll for large number of sockets (tested up to 6000)

Keywords: Poll, ePoll

—

Description: The following XLIO_PANIC will be displayed when there are not enough open files defined on the server:XLIO PANIC : si[fd=1023]:51:sockinfo() failed to create internal epoll (ret=-1 Too many open files

Workaround: Verify that the number of max open FDs (File Descriptors) in the system (ulimit -n) is twice as number of needed sockets. XLIO internal logic requires one additional FD per offloaded socket.

Keywords: XLIO_PANIC while opening large number of sockets

—

Description: When a XLIO -enabled application is running, there are several cases when it does not exit as expected pressing CTRL-C.

Workaround: Enable SIGINT handling in XLIO, by using:#export XLIO_HANDLE_SIGINTR=1

Keywords: The XLIO application does not exit when you press CTRL-C.

—

Description: XLIO does not support network interface or route changes during runtime

Workaround: N/A

Keywords: Dynamic route or IP changes

—

Description: XLIO behavior of epoll EPOLLET (Edge Triggered) and EPOLLOUT flags with TCP sockets differs between OS and XLIO.

• XLIO – triggers EPOLLOUT event every received ACK (only data, not syn/fin)

• OS – triggers EPOLLOUT event only after buffer was full

Workaround: N/A

Keywords: XLIO behavior of epoll EPOLLET (Edge Triggered) and EPOLLOUT flags with TCP sockets

—

Description: XLIO does not close connections located on the same node (sends FIN to peers) when its own process is terminated.

Workaround: N/A

Keywords: XLIO connection

—

Description: XLIO is not closed (sends FIN to peers) when its own process is terminated when the /etc/init.d/XLIO is stopped.

Workaround: Launch /etc/init.d/XLIO start

Keywords: XLIO connection

—

Description: When a non-offloaded process joins the same MC address as another XLIO process on the same machine, the non-offloaded process does not get any traffic.

Workaround: Run both processes with XLIO

Keywords: MC traffic with XLIO process and non XLIO process on the same machine

—

Description: Occasionally, epoll with EPOLLONESHOT does not function properly.

Workaround: N/A

Keywords: epoll with EPOLLONESHOT

—

Description: Occasionally, when running UDP SFNT-STREAM client with poll muxer flag, the client-side ends with an expected error:ERROR: Sync messages at end of test lost

ERROR: Test failed.

This only occurs with poll flag

Workaround: Set a higher acknowledgment waiting time value in the sfnt-stream.

Keywords: SFNT-STREAM UDP with poll muxer flag ends with an error on client-side

—

Description: Occasionally, SFNT-STREAM UDP client hangs when running multiple times.

Workaround: Set a higher acknowledgment waiting time value in the sfnt-stream.

Keywords: SFNT-STREAM UDP client hanging issue

—

Description: Ethernet loopback functions only if both sides are either off-loaded or not-offloaded.

Workaround: N/A

Keywords: Ethernet loopback is not functional between the XLIO and the OS

—

Description: The following error may occur when running netperf TCP tests with XLIO:remote error 107 'Transport endpoint is not connected

Workaround: Use netperf 2.6.0

Keywords: Error when running netperf 2.4.4 with XLIO

—

Description: Occasionally, a packet is not sent if the socket is closed immediately after send() (also for blocking socket)

Workaround: Wait several seconds after send() before closing the socket

Keywords: A packet is not sent if the socket is closed immediately after send()

—

Description: It can take for XLIO more time than the OS to return from an iomux call if all sockets in this iomux are empty sockets

Workaround: N/A

Keywords: Iomux call with empty sockets

—

Description: Sharing of HW resources between the different working threads might cause lock contentions which can affect performance.

Workaround: Use different ring allocation logic.

Keywords: Issues with performance with some multi-threaded applications

—

Description: XLIO does not support broadcast traffic.

Workaround: Use libxlio.conf to pass broadcast through OS

Keywords: No support for direct broadcast

—

Description: Directing XLIO to access non-valid memory area will cause a segmentation fault.

Workaround: N/A

Keywords: There is no non-valid pointer handling in XLIO

—

Description: XLIO allocates resources on the first connect/send operation, which might take up to several tens of milliseconds.

Workaround: N/A

Keywords: First connect/send operation might take more time than expected

—

Description: Calling select upon shutdown of socket will return “ready to write” instead of timeout.

Workaround: N/A

Keywords: Calling select() after shutdown (write) returns socket ready to write, while select() is expected to return timeout

—

Description: XLIO does not raise sigpipe in connection shutdown.

Workaround: N/A

Keywords: XLIO does not raise sigpipe

—

Description: XLIO polls the CQ for packets; if no packets are available in the socket layer, it takes longer.

Workaround: N/A

Keywords: When there are no packets in the socket, it takes longer to return from the read call

—

Description: Select with more than 1024 sockets is not supported

Workaround: Compile XLIO with SELECT_BIG_SETSIZE defined.

Keywords: 1024 sockets

—

Description: Possible performance degradation running NGINX QUIC.

Workaround: Use XLIO_CQ_MODERATION_ENABLE=0.

Keywords: QUIC, NGINX, XLIO_CQ_MODERATION_ENABLE