NVIDIA Accelerated IO (XLIO) Documentation Rev 3.31.2
NVIDIA Accelerated IO (XLIO) Documentation Rev 3.31.2

XLIO Configuration

You can control the behavior of XLIO by configuring:

  • The libxlio.conf file

  • XLIO configuration parameters, which are Linux OS environment variables

  • XLIO extra API

The installation process creates a default configuration file, /etc/libxlio.conf, in which you can define and change the following settings:

  • The target applications or processes to which the configured control settings apply. By default, XLIO control settings are applied to all applications.

  • The transport to be used for the created sockets.

  • The IP addresses and ports in which you want offload.

By default, the configuration file allows XLIO to offload everything except for the DNS server-side protocol (UDP, port 53) which will be handled by the OS.

In the libxlio.conf file:

  • You can define different XLIO control statements for different processes in a single configuration file. Control statements are always applied to the preceding target process statement in the configuration file.

  • Comments start with # and cause the entire line after it to be ignored.

  • Any beginning whitespace is skipped.

  • Any line that is empty is skipped.

  • It is recommended to add comments when making configuration changes.

The following sections describe configuration options in libxlio.conf. For a sample libxlio.conf file, see Example of XLIO Configuration below.

Configuring Target Application or Process

The target process statement specifies the process to which all control statements that appear between this statement and the next target process statement apply.

Each statement specifies a matching rule that all its sub-expressions must evaluate as true (logical and) to apply.

If not provided (default), the statement matches all programs.

The format of the target process statement is:

Copy
Copied!
            

application-id <program-name|*> <user-defined-id|*>

Option

Description

<program-name|*>

Define the program name (not including the path) to which the control statements appearing below this statement apply.

Wildcards with the same semantics as "ls" are supported (* and ?).

For example:

  • db2* matches any program with a name starting with db2.

  • t?cp matches ttcp, etc.

<user-defined-id|*>

Specify the process ID to which the control statements appearing below this statement apply.

Note

You must also set the XLIO_APPLICATION_ID environment variable to the same value as user-defined-id.


Configuring Socket Transport Control

Use socket control statements to specify when libxlio will offload AF_INET/SOCK_STREAM or AF_INET/SOCK_DATAGRAM sockets (currently SOCK_RAW is not supported).

Each control statement specifies a matching rule that all its sub-expressions must evaluate as true (logical and) to apply. Statements are evaluated in order of definition according to "first-match".

Socket control statements use the following format:

Copy
Copied!
            

use <transport> <role> <address|*>:<port range|*>

Where:

Option

Description

transport

Define the mode of transport:

  • xlio – XLIO should be used.

  • os – the socket should be handled by the OS network stack. In this mode, the sockets are not offloaded.

The default is xlio.

role

Specify one of the following roles:

  • tcp_server – for listen sockets. Accepted sockets follow listen sockets. Defined by local_ip:local_port.

  • tcp_client – for connected sockets. Defined by remote_ip:remote_port:local_ip:local_port

  • udp_sender – for TX flows. Defined by remote_ip:remote_port

  • udp_receiver – for RX flows. Defined by local_ip:local_port

  • udp_connect – for UDP connected sockets. Defined by remote_ip:remote_port:local_ip:local_port

address

You can specify the local address the server is bind to or the remote server address the client connects to.

The syntax for address matching is:

<IPv4 address>[/<prefix_length>]|*

  • IPv4 address – [0-9]+\.[0-9]+\.[0-9]+\.[0-9]+ each sub number < 255

  • prefix_length – [0-9]+ and with value <= 32. A prefix_length of 24 # matches the subnet mask 255.255.255.0 . A prefix_length of 32 requires matching of the exact IP.

port range

Define the port range as:

Copy
Copied!
            

start-port[-end-port]

Port range: 0-65536


Example of XLIO Configuration

To set the following:

  • Apply the rules to program tcp_lat with ID B1

  • Use XLIO by TCP clients connecting to machines that belong to subnet 192.168.1.*

  • Use OS when TCP server listens to port 5001 of any machine

In libxlio.conf, configure:

Copy
Copied!
            

application-id tcp-lat B1 use xlio tcp_client 192.168.1.0/24:*:*:* use os tcp_server *:5001 use os udp_connect *:53

Note

You must also set the XLIO parameter:

Copy
Copied!
            

XLIO_APPLICATION_ID=B1


XLIO configuration is performed using environment variables. For the full list of XLIO parameters, please see libxlio README file.

Note

XLIO parameters must be set prior to loading the application with XLIO. You can set the parameters in a system file, which can be run manually or automatically.

All the parameters have defaults that can be modified.

On default startup, the XLIO library prints the XLIO version information, as well as the configuration parameters being used and their values to stderr.

XLIO always logs the values of the following parameters, even when they are equal to the default value:

  • XLIO_TRACELEVEL

  • XLIO_LOG_FILE

For all other parameters, XLIO logs the parameter values only when they are not equal to the default value.

Beta Level Features Configuration Parameters

The following table lists configuration parameters and their possible values for new XLIO Beta level features. The parameters below are disabled by default.

These XLIO features are still experimental and subject to changes. They can help improve performance of multithread applications.

We recommend altering these parameters in a controlled environment until reaching the best performance tuning.

XLIO Configuration Parameter

Description and Examples

XLIO_RING_MIGRATION_RATIO_TX

XLIO_RING_MIGRATION_RATIO_RX

Ring migration ratio is used with the "ring per thread" logic in order to decide when it is beneficial to replace the socket's ring with the ring allocated for the current thread.

Each XLIO_RING_MIGRATION_RATIO iteration (of accessing the ring), the current thread ID is checked to see whether the ring matches the current thread.

If not, ring migration is considered. If the ring continues to be accessed from the same thread for a certain iteration, the socket is migrated to this thread ring.

Use a value of -1 in order to disable migration.

Default: -1

XLIO_RING_LIMIT_PER_INTERFACE

Limits the number of rings that can be allocated per interface.

For example, in ring allocation per socket logic, if the number of sockets using the same interface is larger than the limit, several sockets will share the same ring.

Note

XLIO_RX_BUFS might need to be adjusted in order to have enough buffers for all rings in the system. Each ring consumes XLIO_RX_WRE buffers.

Use a value of 0 for an unlimited number of rings.

Default: 0 (no limit)

XLIO_RING_DEV_MEM_TX

XLIO can use the on-device-memory to store the egress packet if it does not fit into the BF inline buffer. This improves application egress latency by reducing the PCI transactions.

Using XLIO_RING_DEV_MEM_TX, enables the user to set the amount of the on-device-memory buffer allocated for each TX ring.

The total size of the on-device-memory is limited to 256k for a single port HCA and to 128k for dual port HCA.

Default value is 0

XLIO_TCP_CC_ALGO

TCP congestion control algorithm.

The default algorithm coming with LWIP is a variation of Reno/New-Reno.

The new Cubic algorithm was adapted from FreeBsd implementation.

Use value of 0 for LWIP algorithm.

Use value of 1 for the Cubic algorithm.

Use value of 2 in order to disable the congestion algorithm.

Default: 0 (LWIP).


XLIO can be loaded using Dynamically Loaded (DL) libraries. These libraries are not automatically loaded at program link time or start-up as with LD_PRELOAD. Instead, there is an API for opening a library, looking up symbols, handling errors, and closing the library.

The example below demonstrates how to load socket() function. Similarly, users should load all other network-related functions as declared in sock-redirect.h:

Copy
Copied!
            

#include <stdlib.h> #include <stdio.h> #include <dlfcn.h> #include <arpa/inet.h> #include <sys/socket.h>   typedef int (*socket_fptr_t) (int __domain, int __type, int __protocol);   int main(int argc, const char** argv) { void* lib_handle; socket_fptr_t xlio_socket; int fd;   lib_handle = dlopen("libxlio.so", RTLD_LAZY); if (!lib_handle) { printf("FAILED to load libxlio.so\n"); exit(1); }   xlio_socket = (socket_fptr_t)dlsym(lib_handle, "socket"); if (xlio_socket == NULL) { printf("FAILED to load socket()\n"); exit(1); }   fd = xlio_socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP); if (fd < 0) { printf("FAILED open socket()\n"); exit(1); }   printf("socket creation succeeded fd = %d\n", fd); close(fd); dlclose(lib_handle); return 0; }

For more information, please refer to dlopen man page.

For a complete example that includes all the necessary functions, see sockperf’s xlio-redirect.h and xlio_socket-redirect.cpp files.

© Copyright 2024, NVIDIA. Last updated on Aug 14, 2024.